Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Netflix Tech

MARCH 10, 2023

Hence we built the data pipeline that can be used to extract the existing assets metadata and process it specifically to each new use case. This feature support required a significant update in the data table design (which includes new tables and updating existing table columns). N, first N rows are fetched from the table.

Management 94

Management Kafka Metadata Media 94

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. The bias toward correctness will increase the processing time, which may not be feasible when speed is a priority. Let’s talk about the data processing types.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Kafka 100

Kafka Finance Media Architecture 100

Data Engineering Podcast

NOVEMBER 18, 2018

Summary Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. Can state be shared across processes or tasks within a Flink cluster?

Process 100

Process Scala Google Cloud Kafka 100

Lyft Engineering

NOVEMBER 29, 2023

Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. It also allowed us to optimize for handling time-series data and event data at scale. ioConfig: Kafka server info, topic names, etc. This was our main form of ingestion.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Cloudera

MAY 13, 2021

Are you using the right stream processing engine for the job at hand? but have you really examined the stream processing engines out there in a side-by-side comparison to make sure? Specifically, the whitepaper analyzes the technical and operational differences between modern processing engines from the Apache open source community.

Process 101

Process Engineering Kafka Architecture 101

Start Data Engineering

SEPTEMBER 26, 2020

Table of Contents Table of Contents Introduction Project description and requirements Infrastructure overview Apache Flink Apache Kafka Design Detect fraudulent accounts Log account actions Prerequisites Code Defining dependencies Inheritance Server logs generator Defining data flow in Apache Flink Create a streaming environment Creating a consumer (..)

Data Engineering 130

Data Engineering Data Engineer Project Engineering 130

Rockset

APRIL 26, 2022

As Kafka Summit is in full swing in London this week and the topic of event streaming is all over my Linkedin feed, I saw a post asking " Is streaming dead? That streaming is rocking and with Kafka Summit this week, I thought it a good time to emphasize the importance of streaming data in today’s modern real-time data stack.

Kafka 52

Kafka Data Warehouse Database Data 52

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. Each local transaction is designed to be reversible so that if something goes wrong during the process, the system can roll back any changes that have been made. A true heroic story indeed! How does it work?

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Data Engineering Podcast

DECEMBER 31, 2018

Summary As more companies and organizations are working to gain a real-time view of their business, they are increasingly turning to stream processing technologies to fullfill that need. How does it compare with systems such as Kafka and Pulsar for ingesting and persisting unbounded data? How do you represent a stream on-disk?

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. Useful to know in case of a disaster event). How far behind is data replication?

Kafka 83

Kafka Metadata Java Cloud 83

Zalando Engineering

JULY 12, 2017

While developing Zalando’s real-time business process monitoring solution, we encountered the need to generate complex events upon the detection of specific patterns of input events. In this blog post we describe the generation of such events using Apache Flink, and share our experiences and lessons learned in the process.

Process 40

Process Kafka AWS Architecture 40

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? For this example, we will use Kafka. The service then pushes the geohash along with the coordinates to a Kafka topic. Rockset has a built-in connector to Apache Kafka.

Kafka 40

Kafka Systems Designing Food 40

Netflix Tech

AUGUST 30, 2021

In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing. The RAE is configured to be effectively a router that devices under test (DUTs) are connected to.

Management 85

Management Kafka Transportation Cloud 85

Knowledge Hut

APRIL 25, 2024

In the past, this data was too large and complex for traditional data processing tools to handle. However, advances in technology have now made it possible to store, process, and analyze big data quickly and effectively. NoSQL databases are designed for scalability and flexibility, making them well-suited for storing big data.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Weekly

FEBRUARY 4, 2024

The user journey, sales process, marketing campaign, everything falls under a state machine. Data modeling is a collaborative process across business units to capture state changes in business activity. The state-machine analogy is vital; your design approach and usage will significantly improve once you get that perspective.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

APRIL 21, 2024

link] RevenueCat: How we solved RevenueCat’s biggest challenges on data ingestion into Snowflake A common design feature of modern data lakes and warehouses is that Inserts and deletes are fast, but the cost of scattered updates grows linearly with the table size.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Towards Data Science

FEBRUARY 18, 2024

How to Stream and Apply Real-Time Prediction Models on High-Throughput Time-Series Data Photo by JJ Ying on Unsplash Most of the stream processing libraries are not python friendly while the majority of machine learning and data mining libraries are python based. An event is generated by a producer (e.g. online dashboard).

Kafka 80

Kafka Python Process Google Cloud 80

Cloudera

JANUARY 31, 2024

It is also important to have multiple options (like normal and prefixless replication) to do the replication process, since every solution has its own advantages. Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication.

Management 68

Management Kafka Big Data Cloud 68

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Can you describe your experiences with Kafka? What are the operational challenges that you have had to overcome while working with Kafka?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Knowledge Hut

JUNE 26, 2023

From exploratory data analysis (EDA) and data cleansing to data modeling and visualization, the greatest data engineering projects demonstrate the whole data process from start to finish. Source Code: Stock and Twitter Data Extraction Using Python, Kafka, and Spark 2. Data pipeline best practices should be shown in these initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka BI Big Data Data Engineering 100

Cloudera

NOVEMBER 16, 2021

This is particularly powerful whenever the use case is event driven and there is no need for NiFi instances to always be up and running. By serverless we mean that resources are provisioned only when and while data is being processed by the application. Event driven use cases. How does this translate in NiFi?

Google Cloud 105

Google Cloud AWS Cloud Computing Kafka 105

Rockset

AUGUST 21, 2019

Rockset continuously ingests data streams from Kafka, without the need for a fixed schema, and serves fast SQL queries on that data. We created the Kafka Connect Plugin for Rockset to export data from Kafka and send it to a collection of documents in Rockset. Implementing a working plugin What is Kafka Connect and Confluent Hub?

Kafka 40

Kafka IT Data Storage Relational Database 40

Data Engineering Podcast

OCTOBER 5, 2020

Summary Analytical workloads require a well engineered and well maintained data integration process to ensure that your information is reliable and up to date. Learn how we streamline and accelerate manual processes to help you derive real results from your data at dataengineeringpodcast.com/immuta.

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100

Data Engineering Podcast

JULY 6, 2020

For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to dataengineeringpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!

Systems 100

Systems Kafka SQL Government 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. What are some of the differences in usage for real-time CDC data vs. event streams that have been the driving force for a large amount of real-time data?

Food 100

Food MongoDB Scala MySQL 100

Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Efficient batch data processing. Exactly once support.

BI 83

BI Digital Media Data Warehouse Kafka 83

Data Engineering Podcast

MARCH 8, 2021

This is a fascinating conversation about what the future looks like when you revisit your assumptions about how systems are designed. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

IT 100

IT Data Warehouse MongoDB Kafka 100

Cloudera

MAY 1, 2023

By breaking down monolithic apps into microservices architectures, for example, or making modularized data products, organizations do their best to enable more rapid iterative cycles of design, build, test, and deployment of innovative solutions. For example, one of the tables in your project might be backed by a Kafka topic.

SQL 76

SQL Project Kafka Accessible 76

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. What is your evaluation process for determining what components to use in your platform?

Data Lake 100

Data Lake Building BI Architecture 100