Data Engineering Weekly

MARCH 26, 2023

Come and hear talks from companies like StarTree, Confluent, LinkedIn, DoorDash, Imply, and Uber on how they are advancing the state-of-the-art in user-facing analytics delivered instantly. Here are the top 5 key learnings from the report. Go to rtasummit.com and register with DEW30 for 30% off.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. We’ve been able to go from raw event streams to a fast SQL query in 5 minutes ?.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Rockset

AUGUST 28, 2023

Here we track every click to every product in our webshop, and then ingest this data into Rockset via Confluent Cloud. In this blog, we’ll focus on the API gateway part - be sure to check the AWS documentation on how to create an S3 bucket and the IAM role and policy. Our cron schedule will be 0 8 * * 1-5.

AWS 52

AWS SQL Datasets Metadata 52

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. For new platform engineers, it presented a steeper learning curve. Overall, these systems allow us to make timely decisions for the business with accurate predictions.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Monte Carlo

JANUARY 16, 2024

In this blog post, we’ll look at six innovations that are shaping the future of the data warehousing, as well as challenges and considerations that organizations should keep in mind. At the center of it all is the data warehouse, the lynchpin of any modern data stack. Table of Contents 1. Data lake and data warehouse convergence 2.

Data Lake 52

Data Lake Data Warehouse Unstructured Data AWS 52

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. We selected Gradle as our build tool, and wrote a custom Gradle plugin called gradle-confluent to structure and enable this methodology. gradlew composeUp. KSQL primer.

Kafka 95

Kafka Management Bytes SQL 95

Picnic Engineering

SEPTEMBER 8, 2022

We continue our story on the Analytics Platform setup in Picnic. In the “Picnic Analytics Platform: Migration from AWS Kinesis to Confluent Cloud” we described why and how we migrated from AWS Kinesis to Confluent Cloud. Our pipeline captures these events and sends them to Confluent Cloud.

Kafka 52

Kafka Metadata AWS Java 52