Data Engineering Podcast

DECEMBER 31, 2023

In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Designing 130

Designing Data Lake High Quality Data SQL 130

Engineering at Meta

MARCH 18, 2024

In this post, we present the design behind Logarithm, and show how it powers AI training debugging use cases. We designed the system to support service-level guarantees on log freshness, completeness, durability, query latency, and query result completeness. Each log line can have zero or more metadata key-value pairs attached to it.

Engineering 89

Engineering Metadata Architecture Designing 89

Data Engineering Podcast

OCTOBER 22, 2023

This has led to the broad adoption of data products being the delivery mechanism for information. In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the development, delivery, and evolution of data products. Can you describe what is encompassed by the idea of a data product strategy?

BI 162

BI SQL Machine Learning Programming Language 162

Data Engineering Podcast

JANUARY 22, 2023

Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time consuming and error-prone. Don't forget to check out our other shows.

PostgreSQL 130

PostgreSQL MongoDB MySQL Machine Learning 130

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 10, 2023

It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack Data projects are notoriously complex. What other tools/products might teams be evaluating while they consider Anomstack?

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

OCTOBER 8, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. With Materialize, you can!

Insurance 162

Insurance BI SQL Machine Learning 162

Data Engineering Podcast

MAY 21, 2023

Summary Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. Stream processing technologies have been around for around a decade.

Data Lake 162

Data Lake Machine Learning Kafka Data Warehouse 162

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations.

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

JANUARY 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. product is a platform that you're calling DataOS. Your flagship (only?)

Coding 130

Coding Building PostgreSQL Data Lake 130

Lyft Engineering

JUNE 28, 2023

However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others. While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort.

Machine Learning 115

Machine Learning Building Metadata Software Engineer 115

Data Engineering Podcast

JANUARY 29, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. How have the design/scope/goals of the product changed since you first started working on it?

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Data Engineering Podcast

JUNE 4, 2023

Agile Data Engine is a platform designed to handle the infrastructure side of the DataOps equation, as well as providing the insights that you need to manage the human side of the workflow. How have the design and goals of the system changed since you first started working on it? Can you describe how Agile Data Engine is architected?

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Data Engineering Podcast

NOVEMBER 5, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

AUGUST 27, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. With Materialize, you can! Why Postgres?

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

JANUARY 8, 2023

Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems. What is the core problem that you are trying to solve?

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Data Engineering Podcast

NOVEMBER 12, 2023

In this episode Eran Yahav shares the journey that he has taken in building this product and the ways that it enhances the ability of humans to get their work done, and when the humans have to adapt to the tool. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

MARCH 25, 2023

Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be.

MySQL 130

MySQL Python Architecture Machine Learning 130

Data Engineering Podcast

NOVEMBER 19, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. The Netflix blog emphasizes the importance of finding the zombie data and the system design around deleting unused data.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Data Engineering Podcast

DECEMBER 3, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

AUGUST 20, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. With Materialize, you can!

Education 130

Education BI SQL Database 130

Zalando Engineering

APRIL 22, 2024

The Stream Consumer is the microservice that acts as the entry point for the events into the entire platform. It is responsible for receiving the events from stream consumer and making them available for the downstream services. Stream Consumer will observe those metrics and adjust the consumption rate.

Algorithm 79

Algorithm Systems Kafka Designing 79

Knowledge Hut

APRIL 25, 2024

Hadoop provides a file system (HDFS) that is designed for scalability and reliability, as well as a resource manager (YARN) that enables efficient scheduling of job execution. NoSQL databases are designed for scalability and flexibility, making them well-suited for storing big data. HDFS, Cassandra, Hive). HDFS, Cassandra, Hive).

Big Data 69

Big Data Technology NoSQL Hadoop 69

Data Engineering Podcast

MARCH 10, 2023

Summary With the rise of the web and digital business came the need to understand how customers are interacting with the products and services that are being sold. Product analytics has grown into its own category and brought with it several services with generational differences in how they approach the problem.

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Knowledge Hut

DECEMBER 22, 2023

Value Stream Management Has Come to the Fore Value Stream Mapping helps change the way teams think about the Definition of Done (DoD) from an ‘I-did-my-job' to ‘the-value-is-realized' result. Design thinking focuses on uncovering and solving problems and completing tasks by tweaking through regular feedback loops.

Machine Learning 104

Machine Learning Portfolio Cloud Computing Healthcare 104

Knowledge Hut

APRIL 16, 2024

As enterprises approach some degree of maturity in managing projects , it becomes necessary for streamlining and standardizing the way these projects are executed, be it product development or providing services. System Implementation: Productionizing the software Maintenance: Operation, maintenance of the production software.

Project 81

Project Management Programming Designing 81

Knowledge Hut

APRIL 26, 2024

The product development process is just as vital as product management; both seem similar but have subtle variances. Product development focuses on the creation of a product, whereas The entire process is overseen by product management. What Is the Product Development Process?

Process 71

Process Manufacturing Retail Electronics 71

LinkedIn Engineering

OCTOBER 5, 2023

Right now, you can find EBR at work in a number of features we offer, such as Jobs You Might Be Interested In (JYMBII), the content you see in LinkedIn Feed and Notifications, and a number of our search products, such as Job Search.

IT 133

IT Metadata Designing Algorithm 133

Zalando Engineering

NOVEMBER 8, 2023

Postgres Logical Replication at Zalando Builders at Zalando have access to a low-code solution that allows them to declare event streams that source from Postgres databases. At the time of writing, there are hundreds of these Postgres-sourced event streams out in the wild at Zalando.

PostgreSQL 139

PostgreSQL Java Database Bytes 139

Data Engineering Podcast

JULY 23, 2023

To bring streaming data in reach of application engineers Matteo Pelati helped to create Dozer. In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together. Can you describe how Dozer is implemented?

Building 130

Building SQL Machine Learning Data Ingestion 130

Data Engineering Weekly

FEBRUARY 25, 2024

The article emphasizes that this trend towards compound systems opens new avenues for optimizing AI application design, promising significant improvements in AI's effectiveness and efficiency. The article explores various design choices and how they impacted the Apache Samza adoption in the industry.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Knowledge Hut

OCTOBER 24, 2023

” They employ iterations to “develop the product through a series of repeated cycles and increments to successively add to the functionality of the product. Regular interactions with stakeholders, small batches of work, regular reviews and retrospectives improve the process and therefore, the product.

Portfolio 86

Portfolio Certification Programming Architecture 86

Christophe Blefari

FEBRUARY 23, 2024

Understand the design principles of Snowflake — Someone took a few hours to understand Snowflake internals and this is a great wrap-up. A newsletter about the streaming data space — Robin collected a lot of cool articles about the streaming ecosystem. Thank you Microsoft. Data will not tell you what to do.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 4) Building Data Products and why should you?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

JUNE 17, 2023

By incorporating column-level lineage in the data modeling process it encourages a more robust and well-informed design. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 130

Data Lake Metadata Data Architecture Machine Learning 130

Towards Data Science

FEBRUARY 9, 2024

Streaming Although I briefly mentioned streaming in my 2021 article, you’ll see a renewed focus in the 2024 version. I’m a strong believer that data has to move at the speed of business , and the only way to truly accomplish this in modern times is through data streaming.

Building 81

Building Transportation Data Lake Metadata 81