Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Rockset

AUGUST 16, 2022

Streams for Everyone If you have come this far it means you have already considered or are considering using event streaming in your data architecture for the wide variety of benefits it can offer. In either case, both Amazon Kinesis and Apache Kafka can help but which one is the right fit for you and your goals.

Kafka 52

Kafka AWS Cloud Java 52

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. What did the previous system look like?

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Kafka 100

Kafka Finance Media Architecture 100

Netflix Tech

MARCH 10, 2023

This feature support required a significant update in the data table design (which includes new tables and updating existing table columns). Data Sharding strategy in elasticsearch is updated to provide low search latency (as described in blog post) Design of new Cassandra reverse indices to support different sets of queries.

Management 94

Management Kafka Metadata Media 94

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. Why is Data Quality Expensive? How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Confluent

APRIL 23, 2019

When it was first created, Apache Kafka ® had a client API for just Scala and Java. Since then, the Kafka client API has been developed for many other programming languages which enables you to pick the language you want. At Confluent, we have an engineering team dedicated to the development of these Kafka clients.

Programming Language 101

Programming Language Kafka Programming Scala 101

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. FROM SYSMON_JOIN.

Process 80

Process Kafka Datasets SQL 80

Lyft Engineering

NOVEMBER 29, 2023

Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. It also allowed us to optimize for handling time-series data and event data at scale. ioConfig: Kafka server info, topic names, etc. (ex. This was our main form of ingestion.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Start Data Engineering

SEPTEMBER 26, 2020

Table of Contents Table of Contents Introduction Project description and requirements Infrastructure overview Apache Flink Apache Kafka Design Detect fraudulent accounts Log account actions Prerequisites Code Defining dependencies Inheritance Server logs generator Defining data flow in Apache Flink Create a streaming environment Creating a consumer (..)

Data Engineering 130

Data Engineering Data Engineer Project Engineering 130

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. Useful to know in case of a disaster event). How far behind is data replication?

Kafka 84

Kafka Metadata Java Cloud 84

Rockset

APRIL 26, 2022

As Kafka Summit is in full swing in London this week and the topic of event streaming is all over my Linkedin feed, I saw a post asking " Is streaming dead? That streaming is rocking and with Kafka Summit this week, I thought it a good time to emphasize the importance of streaming data in today’s modern real-time data stack.

Kafka 52

Kafka Data Warehouse Database Data 52

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? For this example, we will use Kafka. The service then pushes the geohash along with the coordinates to a Kafka topic. Rockset has a built-in connector to Apache Kafka.

Kafka 40

Kafka Designing Systems Food 40

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. Each local transaction is designed to be reversible so that if something goes wrong during the process, the system can roll back any changes that have been made. A true heroic story indeed!

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Netflix Tech

AUGUST 30, 2021

In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing. The RAE is configured to be effectively a router that devices under test (DUTs) are connected to.

Management 85

Management Kafka Transportation Cloud 85

Data Engineering Weekly

APRIL 21, 2024

link] RevenueCat: How we solved RevenueCat’s biggest challenges on data ingestion into Snowflake A common design feature of modern data lakes and warehouses is that Inserts and deletes are fast, but the cost of scattered updates grows linearly with the table size.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Can you describe your experiences with Kafka? What are the operational challenges that you have had to overcome while working with Kafka?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Lyft Engineering

JUNE 28, 2023

The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. Designing a Common Interface: RealtimeMLPipeline Recall that our aim was to simplify integrating streaming into ML models for developers by abstracting away technical complexity.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Introduction Kafka as an event streaming component can be applied to a wide variety of use cases. and its use is no longer recommended.

Management 69

Management Kafka Big Data Cloud 69

Knowledge Hut

APRIL 25, 2024

Hadoop provides a file system (HDFS) that is designed for scalability and reliability, as well as a resource manager (YARN) that enables efficient scheduling of job execution. NoSQL databases are designed for scalability and flexibility, making them well-suited for storing big data. HDFS, Cassandra, Hive). log files, clickstreams).

Big Data 98

Big Data Technology NoSQL Hadoop 98

Data Engineering Weekly

FEBRUARY 4, 2024

The state-machine analogy is vital; your design approach and usage will significantly improve once you get that perspective. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. To help you drive data quality at the source, RudderStack just launched a Data Quality Toolkit.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka BI Big Data Data Engineering 100

Rockset

AUGUST 21, 2019

Rockset continuously ingests data streams from Kafka, without the need for a fixed schema, and serves fast SQL queries on that data. We created the Kafka Connect Plugin for Rockset to export data from Kafka and send it to a collection of documents in Rockset. Implementing a working plugin What is Kafka Connect and Confluent Hub?

Kafka 40

Kafka IT Data Storage Relational Database 40

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Using fixed lookback windows to always reprocess data, assuming that most late-arriving events will occur within that window. a cancel event for a missed signup would have had no effect).

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Native streaming ingestion support from Kafka and Kinesis.

BI 84

BI Digital Media Data Warehouse Kafka 84

Data Engineering Podcast

JULY 6, 2020

For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to dataengineeringpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!

Systems 100

Systems Kafka SQL Government 100

Zalando Engineering

JULY 12, 2017

While developing Zalando’s real-time business process monitoring solution, we encountered the need to generate complex events upon the detection of specific patterns of input events. In this blog post we describe the generation of such events using Apache Flink, and share our experiences and lessons learned in the process.

Process 40

Process Kafka AWS Architecture 40

Data Engineering Podcast

JANUARY 5, 2020

Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. Pulsar, Bookkeeper, Pravega)?

Database 100

Database Kafka PostgreSQL MySQL 100

Data Engineering Podcast

MARCH 8, 2021

This is a fascinating conversation about what the future looks like when you revisit your assumptions about how systems are designed. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming.

IT 100

IT Data Warehouse MongoDB Kafka 100

Cloudera

NOVEMBER 16, 2021

This is particularly powerful whenever the use case is event driven and there is no need for NiFi instances to always be up and running. This is the most cost effective way of running applications that only need to process data following specific events. Event driven use cases. How does this translate in NiFi?

Google Cloud 106

Google Cloud AWS Cloud Computing Kafka 106

Knowledge Hut

JUNE 26, 2023

Stock and Twitter Data Extraction Using Python, Kafka, and Spark Project Overview: The rising and falling of GameStop's stock price and the proliferation of cryptocurrency exchanges have made stocks a topic of widespread attention. Source Code: Stock and Twitter Data Extraction Using Python, Kafka, and Spark 2.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. When is a data lake architecture the wrong choice?

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

MAY 10, 2021

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. What are the areas of complexity that come up when building analytical APIs that are often overlooked when first designing a system to operate on and expose real-time data?

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Data Engineering Podcast

OCTOBER 5, 2020

In this episode founders DeVaris Brown and Ali Hamidi explain how their tenure at Heroku informed their approach to making data integration self service, how the platform is architected, and how they have designed their system to adapt to the continued evolution of the data ecosystem.

Data Integration 100

Data Integration Kafka Data Lake Machine Learning 100