Data Engineering Podcast

APRIL 7, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

APRIL 14, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

FEBRUARY 25, 2024

When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow ecosystem ready and waiting with useful building blocks to accelerate the process. In this episode he explains how he used the combination of Apache Arrow, Flight, Datafusion, and Parquet to lay the foundation of the newest version of his time-series database.

Database 162

Database Technology Data Lake High Quality Data 162

Uber Engineering

MARCH 12, 2017

With the evolution of storage formats like Apache Parquet and Apache ORC and query engines like Presto and Apache Impala , the Hadoop ecosystem has the potential to become a general-purpose, unified serving layer for workloads that can tolerate latencies … The post Hudi: Uber Engineering’s Incremental Processing Framework on Apache Hadoop appeared (..)

Hadoop 105

Hadoop Process Engineering Data Architecture 105

Data Engineering Podcast

MARCH 10, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Why would someone choose Nessie over native table-level branching in the Apache Iceberg spec? Want to see Starburst in action? Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Data Engineering Podcast

DECEMBER 24, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

JANUARY 28, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

FEBRUARY 11, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake High Quality Data Government Data Pipeline 147

Data Engineering Podcast

JANUARY 7, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

MARCH 31, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

JANUARY 21, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

MARCH 3, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

MARCH 17, 2024

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Weekly

APRIL 14, 2024

[link] Alibaba: Building a Streaming Lakehouse: Performance Comparison Between Paimon and Hudi I’m not a big fan of these comparison studies since it all depends on the nature of the data and the business use cases. It's a good read that familiarizes you with the LakeHouse formats.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Data Engineering Podcast

NOVEMBER 26, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 31, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

AUGUST 3, 2021

Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge. By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Can you describe how Hudi is architected?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

DECEMBER 17, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 19, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Weekly

APRIL 21, 2024

[link] Grab: Enabling near real-time data analytics on the data lake Apache Hudi’s Merge On Read (MoR) is a game changer in developing low-latency analytics on top of the data lake. Grab narrates how it integrated Debeizium, Kafka, and Apache Hudi to enable near real-time data analytics on the data lake.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Data Engineering Podcast

NOVEMBER 5, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 10, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

DECEMBER 3, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

NOVEMBER 12, 2023

And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Christophe Blefari

FEBRUARY 23, 2024

— Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Spark future — I'm convinced that Apache Spark will have to transform itself if it is not to disappear (disappear in the sense of Hadoop, still present but niche). But for sure I'll add Arrow in the v2.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Data Engineering Podcast

FEBRUARY 19, 2023

Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses.

IT 147

IT Data Lake Metadata Data Warehouse 147

Christophe Blefari

NOVEMBER 18, 2023

This isn't a new format but a way to create interoperability between Delta, Iceberg and Hudi. If you are curious about Iceberg and Hudi ACID guarantees read the article. LinkedIn, how to use 4 trillion events daily — Leveraging Apache Beam and Samza. Code faster with Ruff , a Python formatter written in Rust.

Food 147

Food Data Data Lake Retail 147

Data Engineering Weekly

DECEMBER 2, 2023

The LakeHouse tools, such as Apache Hudi, support incremental querying to reduce the latency. It will amplify the innovations on LakeHouse formats like Apache Hudi, Iceberg & Delta. The caching layer from the BI tools with in-memory databases is on the BI tools side.

IT 85

IT BI AWS Kafka 85

Christophe Blefari

AUGUST 4, 2023

this list can become infinite) Conclusion After this design exercice I have mix feeling.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

Data Engineering Weekly

MARCH 3, 2024

The article explains the challenges with frequent updates in the vanilla data lake setup and how Hudi allows for faster writes and guarantees atomicity, consistency, isolation, and durability (ACID). The article concludes with a look at data contracts as a concrete example of these principles in practice.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Weekly

NOVEMBER 26, 2023

We saw back-to-back articles comparing the features and the performance of Delta Lake, Apache Hudi, and Iceberg. The companies backing Apache Hudi and Iceberg write articles about comparative ACID support in both the platforms here and here. One of the engines I’m excited to watch is an alternative to Apache Spark.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Engineering Weekly

AUGUST 13, 2023

Joe Reis, author of "The Fundamentals of Data Engineering," and Vinoth Chandar, creator of Apache Hudi and founder of OneHouse.ai. The author narrates different types of queries with Apache Hudi is a good foundational understanding of thinking about LakeHouse access patterns. 🚀 Stay tuned for all the details!

Data Engineering 52

Data Engineering Data Engineer Engineering Food 52

Data Engineering Podcast

SEPTEMBER 1, 2021

By connecting to your pipeline orchestrator like Apache Airflow and centralizing your end-to-end metadata, Databand.ai Hudi, Delta Lake, Iceberg, Nessie, LakeFS, etc.). By connecting to your pipeline orchestrator like Apache Airflow and centralizing your end-to-end metadata, Databand.ai

Data Lake 100

Data Lake Cloud AWS SQL 100

Data Engineering Weekly

DECEMBER 10, 2023

[link] Dipankar Mazumdar: OneTable- Interoperability for Apache Hudi, Iceberg & Delta Lake One of the 2024 predictions is that “The Great Battle of LakeHouses” will amplify greatly in 2024. The OneTable, as promised in this article, enables seamless interoperability among Hudi, Iceberg, and Delta Lake.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Data Engineering Weekly

MARCH 24, 2024

As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service. The author writes excellent tips on optimizing Apache Hudi LakeHouse infrastructure by adopting indexes, partition pruning, handling incomplete versions, etc.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Christophe Blefari

MARCH 17, 2023

We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. Other few articles but with no comment: Introducing multi-modal index for the Lakehouse in Apache Hudi. I'll try to continue doing deep-dives on article from time to time. dbt Reimagined.

Data 130

Data SQL Deep Learning Kafka 130

Data Engineering Weekly

MARCH 19, 2023

link] Uber: Setting Uber’s Transactional Data Lake in Motion with Incremental ETL Using Apache Hudi Uber writes a comprehensive guide on running incremental ETL using Apache Hudi. Hadoop put forward the schema-on-read strategy that leads to the disruption of data modeling techniques as we know until then.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57