Data Engineering Podcast

NOVEMBER 20, 2021

Summary One of the perennial challenges posed by data lakes is how to keep them up to date as new data is collected. In this episode Ori Rafael shares his experiences from Upsolver and building scalable stream processing for integrating and analyzing data, and what the tradeoffs are when coming from a batch oriented mindset.

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JUNE 16, 2019

Summary Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Rockset

FEBRUARY 6, 2019

Traditional Data Processing: Batch and Streaming MapReduce, most commonly associated with Apache Hadoop, is a pure batch system that often introduces significant time lag in massaging new data into processed results. Most processing in the Lambda architecture happens in the pipeline and not at query time.

Lambda Architecture 40

Lambda Architecture Architecture MongoDB Kafka 40

ProjectPro

AUGUST 24, 2021

Learn how to process Wikipedia archives using Hadoop and identify the lived pages in a day. Utilize Amazon S3 for storing data, Hive for data preprocessing, and Zeppelin notebooks for displaying trends and analysis. Understand the importance of Qubole in powering up Hadoop and Notebooks. The final step is Publish.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52