Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. How is the governance of DataHub being managed?

Metadata 100

Metadata BI Data Warehouse Government 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ? Data As Code is a very strong choice : we do not want any UI because it is an heritage of the ETL period.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

APRIL 5, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

SEPTEMBER 20, 2022

Here’s how Prefect , Series B startup and creator of the popular data orchestration tool, harnessed the power of data observability to preserve headcount, improve data quality and reduce time to detection and resolution for data incidents. But a growing company means growing data needs. Scaling data governance.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Databand.ai

AUGUST 30, 2023

DataOps tools should provide a comprehensive data cataloging solution that allows organizations to create a centralized repository of their data assets, complete with metadata, data lineage information, and data samples.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Monte Carlo

JUNE 14, 2023

The modern data stack era , roughly 2017 to present data, saw the widespread adoption of cloud computing and modern data repositories that decoupled storage from compute such as data warehouses, data lakes, and data lakehouses. They also recently acquired Apache Flink , another streaming solution.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52