LinkedIn Engineering

OCTOBER 19, 2023

In 2010, they introduced Apache Kafka , a pivotal Big Data ingestion backbone for LinkedIn’s real-time infrastructure. To transition from batch-oriented processing and respond to Kafka events within minutes or seconds, they built an in-house distributed event streaming framework, Apache Samza.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Podcast

JUNE 1, 2020

What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert. What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

ProjectPro

AUGUST 24, 2021

This architecture shows that simulated sensor data is ingested from MQTT to Kafka. The data in Kafka is analyzed with Spark Streaming API, and the data is stored in a column store called HBase. Machine Learning web service to host forecasting code. Blob Storage for intermediate storage of generated predictions.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52