Data Management, High Quality Data and SQL

Building ETL Pipelines With Generative AI

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Building

Building BI SQL Machine Learning

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL

SQL Data Lake High Quality Data Data Pipeline

5 Layers of Data Lakehouse Architecture Explained

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture is an increasingly popular choice for many businesses because it supports interoperability between data lake formats. It supports ACID transactions and can run fast queries, typically through SQL commands, directly on object storage in the cloud or on-prem on structured and unstructured data.

Architecture

Architecture Data Lake Metadata Unstructured Data

Webinars

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

Data Lakehouse Architecture Explained: 5 Layers

Monte Carlo

JANUARY 5, 2024

Data lakehouse architecture is an increasingly popular choice for many businesses because it supports interoperability between data lake formats. It supports ACID transactions and can run fast queries, typically through SQL commands, directly on object storage in the cloud or on-prem on structured and unstructured data.

Architecture

Architecture Data Lake Metadata Unstructured Data

Data Integrity vs. Data Quality: 4 Key Differences You Can’t Confuse

Monte Carlo

JUNE 16, 2023

Data integrity and quality may seem similar at first glance, and they are sometimes used interchangeably in everyday life, but they play unique roles in successful data management. Impact Now that you understand the purpose of data integrity and data quality, what is their impact on data management and decision-making?

Data Integration

Data Integration Hospitality Banking Healthcare

5 ETL Best Practices You Shouldn’t Ignore

Monte Carlo

OCTOBER 5, 2023

Ensure data quality Even if there are no errors during the ETL process, you still have to make sure the data meets quality standards. High-quality data is crucial for accurate analysis and informed decision-making. Ready to leap to the next level of data management prowess?

Data Cleanse

Data Cleanse ETL Tools Datasets High Quality Data

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka

Kafka Data Lake High Quality Data SQL

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. To start, can you share your definition of what constitutes a "Data Lakehouse"?

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

How to become Azure Data Engineer I Edureka

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering

Data Engineering Data Engineer Engineering Programming Language

Low Code And High Quality Data Engineering For The Whole Organization With Prophecy

Data Engineering Podcast

JULY 16, 2021

In this episode he shares his motivations for creating Prophecy, how he is leveraging the magic of compilers to translate between UI and code oriented representations of logic, and the organizational benefits of having a cohesive experience designed to bring business users and domain experts into the same platform as data engineers and analysts.

High Quality Data

High Quality Data Data Engineering Data Engineer Coding

The Symbiotic Relationship Between AI and Data Engineering

Ascend.io

FEBRUARY 28, 2024

While data engineering and Artificial Intelligence (AI) may seem like distinct fields at first glance, their symbiosis is undeniable. The foundation of any AI system is high-quality data. Here lies the critical role of data engineering: preparing and managing data to feed AI models.

Data Engineering

Data Engineering Data Engineer Engineering Metadata

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

In this episode Pete Hunt, CEO of Dagster labs, outlines these new capabilities, how they reduce the burden on data teams, and the increased collaboration that they enable across teams and business units. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database

Non-relational Database Relational Database Database Designing

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Validating that the information is stored and processed correctly can be complex and time-consuming, especially when the source and destination speak different dialects of SQL. Data lakes are notoriously complex. Can you start by outlining some of the situations where reconciling data between databases is needed?

Database

Database Data Lake High Quality Data Data Workflow

Designing Data Platforms For Fintech Companies

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Designing

Designing Data Lake High Quality Data SQL

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building

Building Data Lake High Quality Data Machine Learning

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Data lakes are notoriously complex. Your first 30 days are free!

Building

Building Data Lake High Quality Data Machine Learning

Shining Some Light In The Black Box Of PostgreSQL Performance

Data Engineering Podcast

NOVEMBER 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

PostgreSQL

PostgreSQL Data Lake High Quality Data SQL

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. What do you have planned for the future of your work at VAST Data? Your first 30 days are free!

Programming

Programming Data Lake High Quality Data Data Pipeline

Adding An Easy Mode For The Modern Data Stack With 5X

Data Engineering Podcast

DECEMBER 17, 2023

In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Data Lake

Data Lake High Quality Data SQL Architecture

Designing Data Transfer Systems That Scale

Data Engineering Podcast

DECEMBER 3, 2023

Andrei Tserakhau has dedicated his careeer to this problem, and in this episode he shares the lessons that he has learned and the work he is doing on his most recent data transfer system at DoubleCloud. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Systems

Systems Designing Data Lake SQL

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Database

Database Technology Data Lake High Quality Data

Data Migration Strategies For Large Scale Systems

Data Engineering Podcast

MAY 26, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you start by sharing some of your experiences with data migration projects? Closing Announcements Thank you for listening! Don't forget to check out our other shows.

Systems

Systems Data Lake High Quality Data Google Cloud

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you start by giving some context and scope of what we mean by "data sharing" for the purposes of this conversation? Closing Announcements Thank you for listening!

Data Lake

Data Lake High Quality Data Government Data Pipeline

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. What are the open questions today in technical scalability of data engines? What are the open questions today in technical scalability of data engines?

Data Process

Data Process Process Data Lake High Quality Data

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Lake

Data Lake Building High Quality Data AWS

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

In this episode Alex Merced explains how the branching and merging functionality in Nessie allows you to use the same versioning semantics for your data lakehouse that you are used to from Git. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Project

Project Data Lake High Quality Data Data Workflow

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

Monte Carlo

AUGUST 31, 2023

Not long after data warehouses moved to the cloud, so too did data lakes (a place to transform and store unstructured data), giving data teams even greater flexibility when it comes to managing their data assets. That is the question – at least if you ask a data engineer.

Data Lake

Data Lake Data Pipeline Unstructured Data Data Warehouse

A Day in the Life of a Data Scientist

Knowledge Hut

JANUARY 24, 2024

They excel in handling unstructured data, a crucial asset for businesses in today's data-driven landscape. The skill set of a data scientist encompasses proficiency in statistics, data science, Big Data management , and expertise in programming languages such as R, Python, and SAS.

Database-centric

Database-centric Data Science Machine Learning Datasets

Forge Your Career Path with Best Data Engineering Certifications

ProjectPro

FEBRUARY 21, 2023

Due to the enormous amount of data being generated and used in recent years, there is a high demand for data professionals, such as data engineers, who can perform tasks such as data management, data analysis, data preparation, etc. Basic understanding of the developments in the IT industry.

Certification

Certification Data Engineering Data Engineer Engineering

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Data lakes are notoriously complex. With Materialize, you can!

Architecture

Architecture Data Lake High Quality Data SQL

Unlocking Your dbt Projects With Practical Advice For Practitioners

Data Engineering Podcast

NOVEMBER 19, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data projects are notoriously complex. With multiple stakeholders to manage across varying backgrounds and toolchains even simple reports can become unwieldy to maintain. Data lakes are notoriously complex.

Project

Project Data Lake High Quality Data SQL

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you describe what the role of the CDP is in the context of a businesses data ecosystem? What do you have planned for the future of ActionIQ?

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Data Engineering Podcast

NOVEMBER 12, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Software Engineer

Software Engineer Software Engineering Engineering Data Lake

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake

Data Lake High Quality Data SQL Architecture

What is dbt Testing? Definition, Best Practices, and More

Monte Carlo

AUGUST 30, 2023

Data testing is the first step in many data engineers’ journey toward reliable data. dbt (data build tool) is a SQL-based command-line tool that offers native testing features. dbt also happens to be an incredibly handy tool for running many data quality tests. This model will be saved as `.sql`

SQL

SQL Datasets Database High Quality Data

Build Your Python Data Processing Your Way And Run It Anywhere With Fugue

Data Engineering Podcast

FEBRUARY 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Python

Python Data Process IT Building

Data Engineering Weekly #107

Data Engineering Weekly

NOVEMBER 13, 2022

It moved from the speculation to the data engineers understanding the benefit of it and asking when we can get the implementation soon. I met many data leaders about Data Contracts, my project Schemata, and how the extended version we are building can help them create high-quality data. Not Endless DAGs!

Data Engineering

Data Engineering Data Engineer Engineering Kafka

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure

DataKitchen

JULY 27, 2023

Azure Databricks Delta Live Table s: These provide a more straightforward way to build and manage Data Pipelines for the latest, high-quality data in Delta Lake. SQL Server Integration Services (SSIS): You know it; your father used it. It does the job.

Data Pipeline

Data Pipeline BI Machine Learning Data Preparation

What is Data Observability? 5 Key Pillars To Know

Monte Carlo

AUGUST 10, 2023

Data observability tools employ automated monitoring, root cause analysis, data lineage, and data health insights to proactively detect, resolve, and prevent data anomalies. Time and again, we’d deliver a report, only to be notified minutes later about issues with our data. It is still relevant today.

Data Pipeline

Data Pipeline Software Engineer Software Engineering Machine Learning

Best Data Observability Tools (with RFP Template and Analyst Reports)

Monte Carlo

FEBRUARY 8, 2024

Gartner While Gartner hasn’t produced a data observability magic quadrant or report ranking data observability vendors, they have named it one of the hottest emerging technologies and placed it on the 2023 Data Management Hype Cycle. That means data engineers and high quality data are integral to the solution.

BI

BI Data Pipeline Data Warehouse Data

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

In this episode Artyom Keydunov, creator of Cube, discusses the evolution and applications of the semantic layer as a component of your data platform, and how Cube provides speed and cost optimization for your data consumers. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data BI Data Workflow

Building ETL Pipelines With Generative AI

Tackling Real Time Streaming Data With SQL Using RisingWave

Webinars

Trending Sources

5 Layers of Data Lakehouse Architecture Explained

Webinars

Data Lakehouse Architecture Explained: 5 Layers

Data Integrity vs. Data Quality: 4 Key Differences You Can’t Confuse

5 ETL Best Practices You Shouldn’t Ignore

Troubleshooting Kafka In Production

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

How to become Azure Data Engineer I Edureka

Low Code And High Quality Data Engineering For The Whole Organization With Prophecy

The Symbiotic Relationship Between AI and Data Engineering

Making Email Better With AI At Shortwave

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Designing A Non-Relational Database Engine

Reconciling The Data In Your Databases With Datafold

Designing Data Platforms For Fintech Companies

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Build Your Second Brain One Piece At A Time

Shining Some Light In The Black Box Of PostgreSQL Performance

When And How To Conduct An AI Program

Adding An Easy Mode For The Modern Data Stack With 5X

Designing Data Transfer Systems That Scale

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Migration Strategies For Large Scale Systems

Data Sharing Across Business And Platform Boundaries

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Build A Data Lake For Your Security Logs With Scanner

Version Your Data Lakehouse Like Your Software With Nessie

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

A Day in the Life of a Data Scientist

Forge Your Career Path with Best Data Engineering Certifications

Addressing The Challenges Of Component Integration In Data Platform Architectures

Unlocking Your dbt Projects With Practical Advice For Practitioners

Modern Customer Data Platform Principles

Enhancing The Abilities Of Software Engineers With Generative AI At Tabnine

Run Your Own Anomaly Detection For Your Critical Business Metrics With Anomstack

What is dbt Testing? Definition, Best Practices, and More

Build Your Python Data Processing Your Way And Run It Anywhere With Fugue

Data Engineering Weekly #107

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure

What is Data Observability? 5 Key Pillars To Know

Best Data Observability Tools (with RFP Template and Analyst Reports)

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Stay Connected