Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

Lyft Engineering

OCTOBER 3, 2023

We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. In future posts we will discuss how Verity addresses data quality elsewhere in our data platform. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Airbnb Tech

NOVEMBER 28, 2023

However, for all of our uncertified data, which remained the majority of our offline data, we lacked visibility into its quality and didn’t have clear mechanisms for up-leveling it. How could we scale the hard-fought wins and best practices of Midas across our entire data warehouse?

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Data Engineering Podcast

SEPTEMBER 11, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Interview Introduction How did you get involved in the area of data management?

Systems 100

Systems Metadata Building MongoDB 100

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Towards Data Science

NOVEMBER 4, 2023

Often it is a data warehouse solution (DWH) in the central part of our infrastructure. Data warehouse exmaple. It’s worth mentioning that its data frame transformations have been included in one of the basic methods of data loading for many modern data warehouses. Consider this example below.

Data Engineering 86

Data Engineering Data Engineer Engineering BI 86

Knowledge Hut

MARCH 7, 2024

This implies two things, one, the data coming from one source is out of date when compared to another source. Two, it creates a commonality of data definitions, concepts, metadata and the like.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Monte Carlo

AUGUST 30, 2023

Your test passes when there are no rows returned, which indicates your data meets your defined conditions. Often, teams run custom data tests as part of a deployment pipeline, or scheduled on production systems via job schedulers like Apache Airflow, dbt Cloud, or via in-built schedulers in your data warehouse solution.

SQL 52

SQL Datasets Database High Quality Data 52

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. You can observe your pipelines with built in metadata search and column level lineage. What is your working definition of that term?

BI 100

BI Data Warehouse Metadata SQL 100

Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. What is your working definition of "data governance" and how does that influence your product focus and priorities?

Data Governance 130

Data Governance Government Cloud Building 130

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

MAY 18, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Interview Introduction How did you get involved in the area of data management?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Cloudera

SEPTEMBER 29, 2022

This team or domain expert will be responsible for the data produced by the team. The data itself is then treated as a product. The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. What is a data mesh contract?

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Christophe Blefari

FEBRUARY 3, 2023

What's often stays is that AE is between DE and DA, so the definition is often done complementarily to other positions. Microsoft Azure announced managed Airflow — Starting this week you'll be able to launch Apache Airflow within Azure Data Factory. In the end actually, the answer depends. Nothing new under the sun.

BI 130

BI Google Cloud SQL Machine Learning 130

RandomTrees

FEBRUARY 6, 2024

Transformation: Shaping Data for the Future: LLMs facilitate standardizing date formats with precision and translation of complex organizational structures into logical database designs, streamline the definition of business rules, automate data cleansing, and propose the inclusion of external data for a more complete analytical view.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Monte Carlo

MARCH 28, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Hightouch interrupted SaaS eating the world in an attempt to shift the center of gravity to the data warehouse. Other common light transformations done within the ingestion phase are data formatting and deduplication.

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. Lack of governance Our platform lacked governance policies for metric definitions.

SQL 82

SQL Metadata Raw Data Government 82

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. What is a Data Vault model? Pie Insurance , a leading small business insurtech, leverages a data vault 2.0

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Monte Carlo

JANUARY 23, 2024

Every team has their own analytic domain they own, forcing decentralized team structures around broadly standardized definitions of what “good” data looks like. The data warehouse is the mirror of the organization in many ways. Data debt or data pipeline chaos. So, pick your poison?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Podcast

SEPTEMBER 19, 2021

He also discusses the technical underpinnings of what he is building and how using the data warehouse as a shared resource drastically shortens the time required to see value. That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data.

Machine Learning 100

Machine Learning Data Warehouse Banking Metadata 100

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

NOVEMBER 20, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Monte Carlo

MARCH 12, 2024

Google Cloud’s Dataplex is a data fabric tool that enables organizations to discover, manage, monitor, and govern their data across all of their data systems, including their data lakes, data warehouses, data lakehouses, and data marts. Dataplex works with your metadata.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

ProjectPro

FEBRUARY 8, 2023

It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., Glue uses ETL jobs for extracting data from various AWS cloud services and integrating it into data warehouses and lakes.

AWS 98

AWS Scala Metadata Data Lake 98

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Monte Carlo

FEBRUARY 26, 2024

A data catalog is a constantly updated inventory of the universe of data assets within an organization. It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems.

Metadata 52

Metadata Government Data Data Governance 52

Ascend.io

FEBRUARY 28, 2024

This process reduces noise in the data, which is crucial for the effectiveness of AI algorithms, especially in complex predictive models and deep learning applications. Such comprehensive metadata management is crucial in adhering to privacy and compliance standards, safeguarding AI operations against potential legal and ethical pitfalls.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Cloudera

MARCH 17, 2023

When implementing a data lakehouse, the table format is a critical piece because it acts as an abstraction layer, making it easy to access all the structured, unstructured data in the lakehouse by any engine or tool, concurrently. Also, the data lake layer is not limited to cloud object stores.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Towards Data Science

APRIL 3, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Hightouch interrupted SaaS eating the world in an attempt to shift the center of gravity to the data warehouse. Other common light transformations done within the ingestion phase are data formatting and deduplication.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98