Data Engineering Podcast

OCTOBER 29, 2023

In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection.

Database 189

Database BI SQL Machine Learning 189

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. With Materialize, you can! Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!

Building 189

Building BI SQL Python 189

Data Engineering Podcast

APRIL 23, 2023

Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. Can you describe what Meroxa is and the story behind it?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Data Engineering Podcast

OCTOBER 22, 2023

Summary The primary application of data has moved beyond analytics. With the broader audience comes the need to present data in a more approachable format. This has led to the broad adoption of data products being the delivery mechanism for information. With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162

Jesse Anderson

OCTOBER 5, 2023

Everything should be streaming – No, it shouldn’t. There should be a clear and compelling need for streaming to deal with its various downsides. The talks went from ksqlDB and Kafka Streams to Flink. I think the writing on the wall is clear for ksqlDB and remains unclear for Kafka Streams. Tiered storage?

Kafka 195

Kafka Finance Cloud Designing 195

Data Engineering Podcast

SEPTEMBER 24, 2023

In this episode Louis Brandy discusses the applications for vector search capabilities both in and outside of AI, as well as the challenges of maintaining real-time indexes of vector data. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

BI 147

BI SQL Python Machine Learning 147

Data Engineering Podcast

OCTOBER 8, 2023

In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Insurance 162

Insurance BI SQL Machine Learning 162

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JUNE 25, 2023

Summary Data transformation is a key activity for all of the organizational roles that interact with data. Because of its importance and outsized impact on what is possible for downstream data consumers it is critical that everyone is able to collaborate seamlessly. Can you describe what SQLMesh is and the story behind it?

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Want to see Starburst in action?

Designing 130

Designing Data Lake High Quality Data SQL 130

Snowflake

MARCH 2, 2023

Snowflake enables organizations to be data-driven by offering an expansive set of features for creating performant, scalable, and reliable data pipelines that feed dashboards, machine learning models, and applications. But before data can be transformed and served or shared, it must be ingested from source systems.

Kafka 119

Kafka Data Ingestion Data Pipeline Cloud Storage 119

Data Engineering Podcast

NOVEMBER 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The team at 5X understand the pain involved and the barriers to productivity and set out to solve it by pre-integrating the best tools from each layer of the stack.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 12, 2023

In this episode Eran Yahav shares the journey that he has taken in building this product and the ways that it enhances the ability of humans to get their work done, and when the humans have to adapt to the tool. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data projects are notoriously complex. Data lakes are notoriously complex. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

AUGUST 27, 2023

Summary Data persistence is one of the most challenging aspects of computer systems. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. This is an interesting and insightful look at pragmatic engineering for reliability and scale.

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. Data lakes are notoriously complex.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

JANUARY 22, 2023

Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time consuming and error-prone.

PostgreSQL 130

PostgreSQL MongoDB MySQL Machine Learning 130

Data Engineering Podcast

DECEMBER 10, 2023

Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. With Materialize, you can!

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

JANUARY 29, 2023

Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems.

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? The Machine Learning Podcast helps you go from idea to production with machine learning.

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

FEBRUARY 11, 2023

Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the business members who play a crucial role in the success of any data project.

PostgreSQL 130

PostgreSQL SQL Machine Learning Data Warehouse 130

Data Engineering Podcast

AUGUST 20, 2023

In this episode Greg Werner explains how they are using generative AI as an assistive tool for creating educational material, as well as building a data driven experience for learners. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Education 130

Education BI SQL Database 130

Knowledge Hut

APRIL 24, 2024

In today's market, the demand for commerce career options and commerce stream jobs is at an all-time high. Top 15 Highest-paying Jobs for Commerce Students in India The following is an in-depth commerce stream jobs list with salaries for students thinking of pursuing commerce but don't know what to expect.

Retail 80

Retail Finance Certification Banking 80

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations.

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

JANUARY 15, 2023

Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery.

Coding 130

Coding Building PostgreSQL Data Lake 130

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? The Platform Integration Data Store Transformation Orchestration Presentation Transportation Observability Closing What’s changed? Over the last three years, my life has changed as well.

Building 81

Building Transportation Data Lake Metadata 81

Data Engineering Podcast

MARCH 10, 2023

Summary With the rise of the web and digital business came the need to understand how customers are interacting with the products and services that are being sold. Product analytics has grown into its own category and brought with it several services with generational differences in how they approach the problem.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Data Engineering Podcast

JANUARY 8, 2023

Summary Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration of data living in disparate systems. Missing data? Start trusting your data with Monte Carlo today!

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Engineering at Meta

MARCH 18, 2024

Logarithm’s data model Logarithm represents logs as a named log stream of (host-local) time-ordered sequences of immutable unstructured text, corresponding to a single log file. A process can emit multiple log streams ( stdout , stderr , and custom log files). Figure 1: Logarithm data model. in PyTorch).

Engineering 89

Engineering Metadata Architecture Designing 89

Lyft Engineering

JUNE 28, 2023

However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others. To meet the needs of our customers, we kicked off the Real-time Machine Learning with Streaming initiative.

Machine Learning 115

Machine Learning Building Metadata Software Engineer 115

Confluent

JUNE 8, 2023

ChatGPT and data streaming can work together for any company. Learn a basic framework for using GPT-4 and streaming to build a real-world production application.

Data 145

Data Building 145

Data Engineering Podcast

MARCH 25, 2023

Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be.

MySQL 130

MySQL Python Architecture Machine Learning 130

Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. At the time of writing, there are hundreds of these Postgres-sourced event streams out in the wild at Zalando.

PostgreSQL 139

PostgreSQL Java Database Bytes 139