Data Engineering Podcast

AUGUST 13, 2022

The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Can you share your definition of "data discovery" and the technical/social/process components that are required to make it viable? Sifflet also offers a 2-week free trial.

Metadata 100

Metadata MongoDB Scala MySQL 100

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. For example, a single table named ‘Customers’ is actually an aggregation of metadata that manages and references several data files, ensuring that the table behaves as a cohesive unit.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Monte Carlo

JULY 11, 2023

Look for potential biases, flaws, or limitations in the data collection process. Examples of inaccurate data Inaccurate data can arise in a few different ways, all primarily due to manual errors, system hiccups, code changes, or data pipeline breakages. is the gas station actually where the map says it is?).

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40

Monte Carlo

AUGUST 30, 2023

But there’s a lot to understand in order to both create the most value from your dbt tests and avoid leaning too heavily on a time-intensive process. Custom data tests are generally run every time a dbt `run` command is executed to process your transformations. And one of the most popular approaches is dbt testing.

SQL 52

SQL Datasets Database High Quality Data 52

Christophe Blefari

JANUARY 20, 2024

Every company out there has his own definition for the data engineer role. What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Reddit r/dataengineering wiki a place where some data eng definitions are written. Who are the data engineers? This is not.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. Workflow Definitions Below you can see a typical file structure of a sample workflow package written in SparkSQL. ??? backfill.sch.yaml ??? daily.sch.yaml ???

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

AltexSoft

NOVEMBER 23, 2021

But this data is all over the place: It lives in the cloud, on social media platforms, in operational systems, and on websites, to name a few. The traditional way of data integration involves consolidating disparate data within a single repository — commonly a data warehouse — via the extract, transform, load (ETL) process.

Process 69

Process Data Lake Metadata Data Warehouse 69

Lyft Engineering

OCTOBER 3, 2023

Data quality is an amorphous term, with various definitions depending on the context. In Verity, we defined data quality as follows: Verity’s Definition of Data Quality The measure of how well data can be used as intended. Five aspects of data quality with the definition in italics and an example in quotes.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Data Engineering Podcast

FEBRUARY 26, 2023

In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. What are the technical systems that you are relying on to power the different data domains? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Metadata Machine Learning Data Integration 147

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. As data pipelines sprawl across disparate systems, it becomes harder to trace data lineage, track data quality, and quickly pinpoint issues when they arise.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Monte Carlo

JULY 10, 2023

That’s where the definition of data completeness comes in. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. This can lead to data that’s incomplete in one system, even if it’s present in another. The same is true with data.

Data Collection 40

Data Collection Data Governance Government Data 40

Knowledge Hut

MARCH 7, 2024

Foresighted enterprises are the ones who will be able to leverage this data for maximum profitability through data processing and handling techniques. The greatest data processing challenge of 2024 is the lack of qualified data scientists with the skill set and expertise to handle this gigantic volume of data.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Data Engineering Podcast

JULY 24, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What was your process for determining what subject areas to include in the book? Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. Here are some of the primary differences compared to the widely used architecture in the industry: The GraphQL type system used at LinkedIn is completely autogenerated.

Architecture 109

Architecture Metadata Java Transportation 109

Marc Lamberti

SEPTEMBER 22, 2023

That cake doesn’t get magicked into existence; it involves a process – a step-by-step recipe you carefully need to follow; otherwise, you will get something different. Digging into the official definition, you will see that Airflow is noted as an open-source platform that programmatically authors, schedules, and monitors workflows.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

LinkedIn Engineering

JUNE 26, 2023

However, we've found that this vertical self-service model doesn't work particularly well for data pipelines, which involve wiring together many different systems into end-to-end data flows. For streaming data pipelines, this involves writing stream processing jobs. This sort of mismatch can occur between any pair of systems.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Airbnb Tech

NOVEMBER 28, 2023

To meet this challenge, we introduced the “Midas” process to certify our data. Starting in 2020, the Midas process, along with the work to re-architect our most critical data models, has brought a dramatic increase in data quality and timeliness to Airbnb’s most critical data. Align on the input components for assessing data quality.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Tweag

APRIL 19, 2023

Most training pipelines and systems are designed to handle fairly small, sub-megapixel images. To allow innovation in medical imaging with AI, we need efficient and affordable ways to store and process these WSIs at scale. A single hospital makes many captures a day, producing terabytes of such data to store and process.

Medical 60

Medical Process Cloud Bytes 60

Christophe Blefari

FEBRUARY 3, 2023

Lyft, powering millions of real-time decisions with LyftLearn Serving — Architecture of the decentralized system Lyft use to deploy and serve ml models. What's often stays is that AE is between DE and DA, so the definition is often done complementarily to other positions. In the end actually, the answer depends.

BI 130

BI Google Cloud SQL Machine Learning 130

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Cloudera

MARCH 9, 2021

While Cloudera Data Platform (CDP) already supports the entire data lifecycle from ‘Edge to AI’, we at Cloudera are fully aware that enterprises have more systems outside of CDP. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets.

Data Governance 95

Data Governance Government Metadata Datasets 95

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. The type system directives are available for the locations listed below. This process is cumbersome, but it is on purpose. The directive @sensitive is used for this purpose. directive @extensibleEnum(values: [String!]!)

Metadata 86

Metadata Coding Banking Designing 86

Ascend.io

FEBRUARY 28, 2024

The foundation of any AI system is high-quality data. Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Databand.ai

AUGUST 30, 2023

It utilizes various techniques to identify and resolve data quality issues, ensuring that high-quality data is used for business processes and decision-making. Consistency: The uniformity of data across different sources or systems. Completeness: The extent that all required data is present and available.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Monte Carlo

JANUARY 23, 2024

Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. The ability of LLMs to process unstructured data is going to change a lot of the foundational table stakes that make up the core of engineering ,” he says. Those who don’t embrace it will be left behind.

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Data Engineering Podcast

MAY 22, 2022

Get started with Kyligence today at dataengineeringpodcast.com/kyligence Your host is Tobias Macey and today I’m interviewing Ketan Umare and Haytham Abuelfutuh about Flyte, the open source and kubernetes-native orchestration engine for your data systems Interview Introduction How did you get involved in the area of data management?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Cloudera

OCTOBER 7, 2021

Application Logic: Application logic refers to the type of data processing, and can be anything from analytical or operational systems to data pipelines that ingest data inputs, apply transformations based on some business logic and produce data outputs. CDF Capabilities Aligned with Key Design Principles for Data Mesh Implementations.

Architecture 124

Architecture Metadata Government Kafka 124

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. As data structure changes in connected systems, the changes are automatically captured and imported to the data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Cloudera

MARCH 14, 2023

For example, a developer may be asked to tap into the data of a newly acquired application, parsing and transforming it before delivering it to the business’s favorite analytical system where it can be joined with existing data sets. Allows them to iteratively develop processing logic and test with as little overhead as possible.

Data Pipeline 83

Data Pipeline Designing Kafka Metadata 83

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. We will also dive deep into our design and implementation processes and the lessons we learnt.

SQL 82

SQL Metadata Raw Data Government 82

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Engineering at Meta

OCTOBER 31, 2023

SCARF scans production data systems to identify tables or assets that are unused and safely removes them. This process can be unexpectedly difficult, because large software systems are inevitably interconnected. If any of the assets are not in use, SCARF initiates a process to safely remove them.

Data 93

Data Metadata Coding Relational Database 93

ProjectPro

FEBRUARY 8, 2023

For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. It offers a simple and efficient solution for data processing in organizations. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog.

AWS 98

AWS Scala Metadata Data Lake 98

Data Engineering Podcast

JANUARY 7, 2022

Summary Data observability is a set of technical and organizational capabilities related to understanding how your data is being processed and used so that you can proactively identify and fix errors in your workflows. You can observe your pipelines with built in metadata search and column level lineage. No more scripts, just SQL.

BI 100

BI Data Warehouse Metadata SQL 100

ProjectPro

NOVEMBER 30, 2021

This guide provides definitions, a step-by-step tutorial, and a few best practices to help you understand ETL pipelines and how they differ from data pipelines. It was a manual process and very challenging as well. The invention of the pipeline system automated the water supply delivery to homes via pipes from various water sources.

Process 52

Process Data Pipeline Data Warehouse AWS 52

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52