ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! By processing a smaller set of data, more often , you effectively divide and conquer a data problem that may otherwise be cost and time prohibitive.

Process 83

Process Data Lake Systems Bytes 83

Cloudera

JULY 18, 2022

In this blog we will conclude the implementation of our fraud detection use case and understand how Cloudera Stream Processing makes it simple to create real-time stream processing pipelines that can achieve neck-breaking performance at scale. Flink is a “streaming first” modern distributed system for data processing.

Process 94

Process Kafka Scala SQL 94

Rockset

MARCH 5, 2020

Apache Kafka has seen broad adoption as the streaming platform of choice for building applications that react to streams of data in real time. In many organizations, Kafka is the foundational platform for real-time event analytics, acting as a central location for collecting event data and making it available in real time.

Kafka 40

Kafka Data Warehouse Data Lake Relational Database 40

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Cloudera

JUNE 28, 2022

We discussed how Cloudera Stream Processing (CSP) with Apache Kafka and Apache Flink could be used to process this data in real time and at scale. Building real-time streaming analytics data pipelines requires the ability to process data in the stream. This is what we call the first-mile problem. The use case.

Process 88

Process Kafka SQL Machine Learning 88

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 85

Systems Metadata Media Kafka 85

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. And if you weren’t there? Well, dig in and start learning! 3 keynotes.

Kafka 109

Kafka Cloud Database Engineering 109

Afterpay Tech

SEPTEMBER 6, 2022

Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. The data is spread across other systems within Afterpay. Instead a message queue, with opt-in subscription models, would be ideal.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Confluent

MAY 13, 2019

In the last year, we’ve experienced enormous growth on Confluent Cloud, our fully managed Apache Kafka ® service. As Confluent Cloud has grown, we’ve noticed two gaps that very clearly remain to be filled in managed Apache Kafka services. Five seconds to Kafka (or, never make another cluster again!).

Kafka 100

Kafka Cloud Building Management 100

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. We used Elasticsearch dashboards to analyze results.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Knowledge Hut

MAY 20, 2024

Scala is now the next wave of computation engines and more importance has been given to the speed processing rather than the size of the batch, and the ability to process event streaming in real-time. Spark is developed by AMPLab and it is a fast and general purpose engine for large-scale data processing.

Scala 52

Scala Technology Kafka Hadoop 52

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. The bias toward correctness will increase the processing time, which may not be feasible when speed is a priority. Let’s talk about the data processing types.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

JULY 1, 2019

Summary Anomaly detection is a capability that is useful in a variety of problem domains, including finance, internet of things, and systems monitoring. In this interview he explains the system design that he tested, his findings for how these tools were able to work together, and how they behaved at different orders of scale.

Kafka 100

Kafka Finance Media Architecture 100

Cloudera

MAY 13, 2021

Are you using the right stream processing engine for the job at hand? but have you really examined the stream processing engines out there in a side-by-side comparison to make sure? Specifically, the whitepaper analyzes the technical and operational differences between modern processing engines from the Apache open source community.

Process 103

Process Engineering Kafka Architecture 103

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Then it is really up to you what you want to with the event data.

Kafka 85

Kafka Java Programming Language Data Pipeline 85

Monte Carlo

MAY 28, 2024

This involves connecting to multiple data sources, using extract, transform, load ( ETL ) processes to standardize the data, and using orchestration tools to manage the flow of data so that it’s continuously and reliably imported – and readily available for analysis and decision-making.

Data Ingestion 52

Data Ingestion Architecture Designing Hadoop 52

Confluent

JUNE 27, 2019

Confluent’s clients for Apache Kafka ® recently passed a major milestone—the release of version 1.0. Magnus Edenhill first started developing librdkafka about seven years ago, later joining Confluent in the very early days to help foster the community of Kafka users outside the Java ecosystem. Builder classes are also new in v1.0

Kafka 105

Kafka Designing Java Coding 105

Rockset

JULY 28, 2022

And you have now introduced another process that has to run, be monitored, scale etc. CDC enables true real-time analytics on your application data, assuming the platform you send the data to can consume the events in real time. Debezium It is also possible to capture MongoDB change data capture events using Debezium.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Data Engineering Podcast

DECEMBER 30, 2019

In order to support their customers, they need to capture, process, and analyze massive amounts of timeseries data with a high degree of uptime and reliability. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh.

Process 100

Process Building Hadoop Java 100

Rockset

SEPTEMBER 7, 2021

Event stream processing has lately become the most-requested feature among data practitioners, who are ever being pushed by their business counterparts for more fresh, real-time insights to improve their operational decisions and boost the digital customer experience.

Kafka 52

Kafka SQL Education Data 52

Precisely

DECEMBER 28, 2023

Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. It also allows for applications, analytics, and reporting to process information as it happens. That enables you to collect, analyze, and store large amounts of information.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Data Engineering Podcast

DECEMBER 31, 2018

Summary As more companies and organizations are working to gain a real-time view of their business, they are increasingly turning to stream processing technologies to fullfill that need. How does it compare with systems such as Kafka and Pulsar for ingesting and persisting unbounded data? How do you represent a stream on-disk?

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Netflix Tech

AUGUST 30, 2021

In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing. In particular, the Kafka integration is the most relevant for this blog post.

Management 85

Management Kafka Transportation Cloud 85

Rockset

APRIL 26, 2022

As Kafka Summit is in full swing in London this week and the topic of event streaming is all over my Linkedin feed, I saw a post asking " Is streaming dead? That streaming is rocking and with Kafka Summit this week, I thought it a good time to emphasize the importance of streaming data in today’s modern real-time data stack.

Kafka 52

Kafka Data Warehouse Database Data 52

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. Each local transaction is designed to be reversible so that if something goes wrong during the process, the system can roll back any changes that have been made. A true heroic story indeed! How does it work?

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Confluent

FEBRUARY 20, 2019

How many Kafka Summits should there be in a year? Some say there should be one giant event where everybody gathers at once. Others say you should live every day like it’s Kafka Summit. Anyway, let me remind you of the dates and places: Kafka Summit New York – April 2, 2019. Kafka Summit London – May 13-14, 2019.

Kafka 66

Kafka Programming Architecture Technology 66

Confluent

APRIL 10, 2019

Enterprises run modern data systems and services across multiple cloud providers, private clouds and on-prem multi-datacenter deployments. Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites.

Kafka 83

Kafka Metadata Java Cloud 83

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? We’ll break our system into pieces such that each component is responsible for one primary job. Let’s take a look at components that constitute the system. For this example, we will use Kafka.

Kafka 40

Kafka Designing Systems Food 40

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Knowledge Hut

APRIL 25, 2024

If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. In the past, this data was too large and complex for traditional data processing tools to handle.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Data Engineering Weekly

FEBRUARY 4, 2024

The user journey, sales process, marketing campaign, everything falls under a state machine. Data modeling is a collaborative process across business units to capture state changes in business activity. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

SEPTEMBER 24, 2023

Airflow isn’t meant to process the data. It instead, say, triggers a Spark job that processes the data. The customer data that is being processed by AI and for what purpose? link] Sponsored: Re: Govern Data governance has often been misunderstood as a controlling bureaucratic process.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 116

Machine Learning Building Metadata Kafka 116