Data Engineering Podcast

NOVEMBER 26, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Data Engineering Podcast

SEPTEMBER 3, 2018

Summary With the proliferation of data sources to give a more comprehensive view of the information critical to your business it is even more important to have a canonical view of the entities that you care about. Can you start by establishing a definition of data mastering that we can work from?

Data Management 100

Data Management Management Relational Database Business Intelligence 100

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. How have the design and goals of the project changed since you first started working on it?

Data Integration 162

Data Integration BI SQL Python 162

Knowledge Hut

MARCH 28, 2024

Data Engineer Career: Overview Currently, with the enormous growth in the volume, variety, and veracity of data generated and the will of large firms to store and analyze their data, data management is a critical aspect of data science. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

AUGUST 3, 2021

Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge. By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Can you describe how Hudi is architected?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

OCTOBER 30, 2022

Summary One of the most impactful technologies for data analytics in recent years has been dbt. It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

AUGUST 10, 2021

In this episode Mark Grover explains what he is building at Stemma, how it expands on the success of the Amundsen project, and why trust is the most important asset for data teams. RudderStack’s smart customer data pipeline is warehouse-first. Go to dataengineeringpodcast.com/census today to get a free 14-day trial.

IT 130

IT Building Data Warehouse Python 130

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. or any other destination you choose.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares the design of the project and how it fits into your development practices. Your host is Tobias Macey and today I’m interviewing Ananth Packkildurai about Schemata, a modelling framework for decentralised domain-driven ownership of data. Can you describe what Schemata is and the story behind it?

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

MAY 29, 2022

Summary The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. In order to ensure that you can explore and analyze your data without spending money on inefficient queries Mingsheng Hong and Zheng Shao created Bluesky Data.

Cloud 100

Cloud Data Warehouse Data Lake Data 100

Data Engineering Podcast

MARCH 27, 2022

Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. What do you have planned for the future of Privacera?

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

JANUARY 7, 2022

He discusses the factors that influenced his decision to start with the data warehouse, the potential shortcomings of that approach, and where he plans to go from there. This is a great exploration of what it means to treat your data platform as a living system and apply state of the art engineering to it.

BI 100

BI Data Warehouse Metadata SQL 100

Knowledge Hut

APRIL 23, 2024

Why are Database Management Skills Important? In the current digital era, data has become a crucial asset for businesses in all industries. Effective data management is essential for businesses to make informed decisions, enhance customer experiences, and acquire a competitive edge.

Database 52

Database Management Relational Database SQL 52

Data Engineering Podcast

DECEMBER 19, 2021

In this episode Pardhu Gunnam and Mars Lan explain how they have designed the architecture and user experience to allow everyone to collaborate on the data lifecycle and provide opportunities for automation and extensible workflows. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

NOVEMBER 20, 2021

Summary The technology for scaling storage and processing of data has gone through massive evolution over the past decade, leaving us with the ability to work with massive datasets at the cost of massive complexity. With their managed Kubernetes platform and blazing fast NVMe storage there’s nothing slowing you down.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

MAY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. This is an ambitious and exciting project, so listen and try it out today. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

DECEMBER 11, 2021

In this episode he explains how it can be used to build rich and useful profiles of your users, the system architecture that powers it, and some of the ways that it is being integrated into an organization’s broader data ecosystem. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

JANUARY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. No more scripts, just SQL.

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Data Engineering Podcast

AUGUST 20, 2021

In this episode Vikrant Dubey discusses his work on the Cuelake project which allows data analysts to build a lakehouse with SQL queries. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

MARCH 8, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Data Warehouse MongoDB Kafka 100

Data Engineering Podcast

FEBRUARY 15, 2021

This is a great conversation about the initial steps that every group should be thinking of as they start down the road to making data informed decisions. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

Building 100

Building Data Warehouse BI Data Pipeline 100

Data Engineering Podcast

JULY 27, 2021

Summary Everyone expects data to be transmitted, processed, and updated instantly as more and more products integrate streaming data. This is a fascinating conversation with a lot of useful lessons for anyone who wants to understand the operational aspects of Pulsar and the benefits that it can provide to data workloads.

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. What are your plans for Data Observability?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

MARCH 29, 2021

There are many aspects of data quality management and it’s always a treat to learn from people who are dedicating their time and energy to solving it for everyone. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Management 100

Management Data Warehouse BI Data Pipeline 100

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

APRIL 26, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Data Engineering Podcast

FEBRUARY 9, 2020

Summary Designing the structure for your data warehouse is a complex and challenging process. As businesses deal with a growing number of sources and types of information that they need to integrate, they need a data modeling strategy that provides them with flexibility and speed.

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Data Engineering Podcast

JULY 5, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

FEBRUARY 22, 2021

In this episode co-founders John Lafleur and Michel Tricot share the story of how and why they created Airbyte, discuss the project’s design and architecture, and explain their vision of what an open soure data integration platform should offer. RudderStack’s smart customer data pipeline is warehouse-first.

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Data Engineering Podcast

MAY 3, 2021

If you are wondering how to get control of your data platforms and bring all of your stakeholders onto the same page then this conversation is for you. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100