Data Engineering Weekly

MARCH 19, 2023

link] Uber: Setting Uber’s Transactional Data Lake in Motion with Incremental ETL Using Apache Hudi Uber writes a comprehensive guide on running incremental ETL using Apache Hudi. The blog discusses implementing Type-2 SCD modeling and strategies to generate surrogate keys and bridge tables to handle many-to-many relationships.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

DareData

JULY 5, 2023

In this blog post, we aim to share practical insights and techniques based on our real-world experience in developing data lake infrastructures for our clients - let's start! Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. Data Sources: How different are your data sources?

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Robinhood

FEBRUARY 7, 2024

In this blog, we explore the evolution of our in-house batch processing infrastructure and how it helps Robinhood work smarter. Our V1 batch processing architecture was robust, anchored by Apache Spark on multiple Hadoop clusters (Spark is known for effectively handling large-scale data processing). Authored by: Grace L.,

Process 75

Process Hadoop Architecture Accessible 75

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Written by Ritesh Varyani and Jeana Choi at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

ProjectPro

JANUARY 25, 2022

Did you know that, according to Linkedin, over 24,000 Big Data jobs in the US list Apache Spark as a required skill? One of the most in-demand technical skills these days is analyzing large data sets, and Apache Spark and Python are two of the most widely used technologies to do this. This is where Apache Spark PySpark comes in.

Big Data 52

Big Data Data Process Process Kafka 52

ProjectPro

NOVEMBER 15, 2021

This blog will walk through the most popular and fascinating open source big data projects. This blog will walk through the most popular and fascinating open source big data projects. Apache Beam Source: Google Cloud Platform Apache Beam is an advanced unified programming open-source model launched in 2016.

Big Data 52

Big Data Project Metadata Programming Language 52

Airbnb Tech

APRIL 13, 2021

Apache Airflow , Apache Hive, Apache Spark ) and extensive analytics infrastructure (i.e., Minerva , Apache Druid , DataPortal , Apache Superset , SLA monitoring ) to make data-informed decisions. A foundation of robust and actionable data is essential for a successful efficiency program.

AWS 52

AWS Raw Data Amazon Web Services Cloud 52