Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Ascend.io

MAY 23, 2023

The data engineering world is full of tips and tricks on how to handle specific patterns that recur with every data pipeline. In general, the root of the orphaned data problem lies in the interim storage of data between the individual processing steps that make up a data pipeline.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Precisely

JULY 21, 2023

Confluent data streams help accelerate innovation for data and analytics initiatives – but only when sourcing from data you can trust. Precisely – the global leader in data integrity and a proud Connect with Confluent program member – helps you build trust in your data to derive the insights your business users need.

Data Integration 52

Data Integration Kafka Bytes Banking 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Towards Data Science

JUNE 19, 2023

Photo by CHUTTERSNAP on Unsplash This article is about Meerschaum Compose , a tool for defining ETL pipelines in YAML and a plugin for the data engineering framework Meerschaum. Docker was a game-changer, revolutionizing the way we design, build, and run our cloud applications.

PostgreSQL 74

PostgreSQL SQL Python Project 74

Rockset

SEPTEMBER 26, 2023

Traditional approaches rely upon cascading batch-oriented data pipelines, meaning data takes hours or even days to flow through the enterprise. As a result, data made available is stale and of low fidelity. Decouple data from your data science tools and production AI apps to test and build faster.

Metadata 64

Metadata Kafka Cloud Database 64

Rockset

APRIL 18, 2023

In real-time ML, the freshness of the features, the serving latency, and the uptime and availability of the data pipeline and model matter. Generating Features for Real-Time Anomaly Detection Now download the anomaly detection dataset to the data / directory. 0 0 Ok, we have the imported data.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Atlan is the metadata hub for your data ecosystem. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

AltexSoft

NOVEMBER 7, 2022

The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. How data engineering works. Apache Airflow is an open-source Python -based workflow orchestrator that enables you to design, schedule, and monitor data pipelines.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. Those who don’t embrace it will be left behind.

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Rockset

AUGUST 26, 2022

For instance, joining two data tables showing hourly temperatures and humidity and how much feed the cows have consumed could allow farmers to improve feeding efficiencies and optimize milk production. When a change is detected, the data is ingested into a data lake hosted on Amazon S3.

IT 52

IT Food Data Lake Aggregated Data 52

The Modern Data Company

JANUARY 17, 2023

These include internal data lakes, apps (1st or 3rd party), web SDK, internal or 3rd-party databases, and even spreadsheets generated within the marketing department. Without lineage or governance metadata, the quality of this data is anyone’s guess. New data is added to the catalog automatically upon ingestion.

Metadata 59

Metadata Government Data Lake Data 59

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

Data Engineering Podcast

JANUARY 28, 2018

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. What have been the most challenging aspects of building and promoting Dat? What is the Dat project and how did it get started?

Data 100

Data Project Electronics Data Management 100

AltexSoft

OCTOBER 8, 2021

So, why is data integration such a big deal? For this purpose, you will have to build a 360-degree customer picture. But here’s the thing, even though you receive all customer data you need, it exists in a bunch of disjointed sources such as CRMs, customer-facing apps, websites, and others. Data profiling and cleansing.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Ascend.io

APRIL 7, 2023

With the monolithic architectures most organizations have today, business users are stuck, constantly waiting for new data pipelines to be built or amended based on their requests. Data engineers aren’t huge fans of this paradigm either. In other words, the product one person builds might have inputs from another product.

Metadata 52

Metadata Datasets Data Governance Government 52

The Modern Data Company

JANUARY 22, 2024

Transforming Data Management and Gaining Recognition Today, enterprises are investing heavily in varied data products, often necessitating intricate integration and customization efforts, Modern has identified a crucial need for a humanized and flexible data platform.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

The Modern Data Company

JANUARY 22, 2024

Transforming Data Management and Gaining Recognition Today, enterprises are investing heavily in varied data products, often necessitating intricate integration and customization efforts, Modern has identified a crucial need for a humanized and flexible data platform.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

AltexSoft

MARCH 30, 2023

Source: The Data Team’s Guide to the Databricks Lakehouse Platform Integrating with Apache Spark and other analytics engines, Delta Lake supports both batch and stream data processing. Besides that, it’s fully compatible with various data ingestion and ETL tools.

Scala 64

Scala Data Lake BI Google Cloud 64

Monte Carlo

APRIL 20, 2023

I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen. This way no decisions get made on bad data and our team becomes a proactive part of the solution,” said then Senior Director of Data at Freshly, Vitaly Lilich.

Data Warehouse 52

Data Warehouse BI Data Government 52