Data Engineering Podcast

MAY 26, 2024

When that system is responsible for the data layer the process becomes more challenging. Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. Can you start by sharing some of your experiences with data migration projects?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Data Lake 182

Data Lake High Quality Data Data Pipeline Machine Learning 182

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

MAY 12, 2024

Summary Building a data platform is a substrantial engineering endeavor. Once it is running, the next challenge is figuring out how to address release management for all of the different component parts. Data lakes are notoriously complex. Our current challenge is one that plagues all data teams.

Management 147

Management Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

MARCH 10, 2024

Summary Data lakehouse architectures are gaining popularity due to the flexibility and cost effectiveness that they offer. The link that bridges the gap between data lake and warehouse capabilities is the catalog. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Data Engineering Podcast

FEBRUARY 11, 2024

Summary Sharing data is a simple concept, but complicated to implement well. There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

APRIL 23, 2023

Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. Data stacks are becoming more and more complex. Sifflet also offers a 2-week free trial.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Building 147

Building Data Lake High Quality Data Machine Learning 147

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

JANUARY 7, 2024

Summary Data processing technologies have dramatically improved in their sophistication and raw throughput. Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. What are the open questions today in technical scalability of data engines?

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Struggling with broken pipelines? Missing data? Stale dashboards?

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

OCTOBER 28, 2021

Summary Streaming data systems have been growing more capable and flexible over the past few years. Despite this, it is still challenging to build reliable pipelines for stream processing. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today! Stale dashboards?

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. Join us at the top event for the global data community, Data Council Austin.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Want to see Starburst in action?

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Your first 30 days are free! Want to see Starburst in action?

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

MARCH 31, 2024

Summary Working with data is a complicated process, with numerous chances for something to go wrong. Identifying and accounting for those errors is a critical piece of building trust in the organization that your data is accurate and up to date. Dagster offers a new approach to building and running data platforms and data pipelines.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. Data lakes are notoriously complex.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Ascend.io

AUGUST 22, 2023

Businesses are in a continuous race to harness data as a powerful propellant for innovation. Yet, the task of efficient and prompt data delivery remains a challenge. The demands of building, scaling, and maintaining data pipelines have grown increasingly complex and error-prone. What Are Automated Data Pipelines?

Data Pipeline 52

Data Pipeline Raw Data Bytes Transportation 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. Structuring data refers to converting unstructured data into tables and defining data types and relationships based on a schema. What is Data Warehouse? . What is Data Lake? .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

MAY 28, 2023

Summary Building a data team is hard in any circumstance, but at a startup it can be even more challenging. The requirements are fluid, you probably don't have a lot of existing data talent to manage the hiring and onboarding, and there is a need to move fast. What are the concepts that the new hire needs to know?

Data Lake 162

Data Lake Machine Learning Data Warehouse Education 162

Data Engineering Podcast

MAY 14, 2023

In this episode the host Tobias Macey shares his reflections on recent experiences where the abstractions leaked and some observances on how to deal with that situation in a data platform architecture. What do you have planned for the future of your data platform? When is ELT the wrong choice?

Data Lake 147

Data Lake Machine Learning Data Warehouse AWS 147

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

JUNE 17, 2023

In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design.

Data Lake 130

Data Lake Metadata Machine Learning Data Architecture 130

Data Engineering Podcast

JUNE 11, 2023

Summary Data engineering is all about building workflows, pipelines, systems, and interfaces to provide stable and reliable data. Your data can be stable and wrong, but then it isn't reliable. Confidence in your data is achieved through constant validation and testing.

Project 130

Project Building Data Lake Machine Learning 130