Confluent

JULY 29, 2024

Apache Kafka 3.8 adds 17 new KIPs (13 for Core, 3 for Streams & 1 for Connect). Highlights include 2 new Docker images, the ability to set task assignors, and more!

Kafka 136

Kafka 136

Confluent

MAY 17, 2022

I’m proud to announce the release of Apache Kafka 3.2.0 on behalf of the Apache Kafka® community. release contains many new features and improvements. This blog will highlight […].

Kafka 139

Kafka 139

Confluent

MARCH 18, 2025

Major milestone release Apache Kafka 4.0 removes ZooKeeper entirely, provides early access to Queues for Kafka, and enables faster rebalances, in addition to many other new KIPs.

Kafka 119

Kafka Accessible Accessibility 119

ProjectPro

JUNE 6, 2025

Today, Kafka is used by thousands of companies, including over 80% of the Fortune 100. Kafka's popularity is skyrocketing, and for good reason—it helps organizations manage real-time data streams and build scalable data architectures. As a result, there's a growing demand for highly skilled professionals in Kafka.

Kafka 40

Kafka Certification AWS Retail 40

Waitingforcode

MAY 4, 2023

Today I'll focus on these differences for Amazon Kinesis service and Apache Kafka ecosystem. The managed cloud services often share the same fundamentals as their Open alternatives. However, there is always something different.

Kafka 147

Kafka Cloud Management 147

ProjectPro

JUNE 6, 2025

Kafka Topics are your trusty companions. Learn how Kafka Topics simplify the complex world of big data processing in this comprehensive blog. More than 80% of all Fortune 100 companies trust, and use Kafka. Apache Kafka The meteoric rise of Apache Kafka's popularity is no accident, as it plays a crucial role in data engineering.

Kafka 40

Kafka Big Data Python Java 40

Confluent

SEPTEMBER 5, 2024

Dive into Kafka internals with a four-part series examining client requests and brokers. Part 1 covers what a producer does to prepare raw event data for the broker.

Kafka 139

Kafka Data 139

ProjectPro

JUNE 6, 2025

Looking for the ultimate guide on mastering Apache Kafka in 2024? The ultimate hands-on learning guide with secrets on how you can learn Kafka by doing. Discover the key resources to help you master the art of real-time data streaming and building robust data pipelines with Apache Kafka. How Difficult Is It To Learn Kafka?

Kafka 40

Kafka Java Big Data Data Pipeline 40

Confluent

DECEMBER 19, 2024

Queue support in Apache Kafka 4.0, enabled by share groups, lets you accommodate traditional queue-type workloads through cooperative consumption.

Kafka 136

Kafka Process 136

ProjectPro

JUNE 6, 2025

Explore the full potential of AWS Kafka with this ultimate guide. Elevate your data processing skills with Amazon Managed Streaming for Apache Kafka, making real-time data streaming a breeze. According to IDC , the worldwide streaming market for event-streaming software, such as Kafka, is likely to reach $5.3

Kafka 40

Kafka AWS Amazon Web Services Data Pipeline 40

Confluent

MAY 15, 2023

Take a tour of the internals of Confluent’s Apache Kafka® service, powered by Kora: the next-generation, cloud-native streaming engine.Kora.

Kafka 145

Kafka Cloud Engineering 145

ProjectPro

JUNE 6, 2025

If you’re looking for everything a beginner needs to know about using Apache Kafka for real-time data streaming, you’ve come to the right place. This blog post explores the basics about Apache Kafka and its uses, the benefits of utilizing real-time data streaming, and how to set up your data pipeline. Let's dive in.

Kafka 40

Kafka Hadoop Big Data Data Warehouse 40

Waitingforcode

FEBRUARY 20, 2024

You can limit the max throughput for Apache Spark Structured Streaming jobs for popular data sources such as Apache Kafka, Delta Lake, or raw files. Have you known that you can also control the lower limit, at least for Apache Kafka? I bet you know it already.

Kafka 130

Kafka IT Data 130

Snowflake

MARCH 2, 2023

As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. Now we are able to ingest our data in near real time directly from Kafka topics to a Snowflake table, drastically reducing the cost of ingestion and improving our SLA from 15 minutes to within 60 seconds.

Kafka 137

Kafka Data Ingestion Data Pipeline Cloud Storage 137

Confluent

JANUARY 6, 2023

Introducing fully managed Apache Kafka® + Flink for the most robust, cloud-native data streaming platform with stream processing, integration, and streaming analytics in one.

Kafka 145

Kafka Cloud Management Process 145

Confluent

JULY 26, 2022

Explore GitHub Actions for your Kafka CI/CD pipeline, automate Schema Registry, and transform the development and testing of Kafka client applications.

Kafka 141

Kafka 141

Start Data Engineering

MAY 9, 2020

Change data capture is a software design pattern used to capture changes to data and take corresponding action based on that change. The change to data is usually one of read, update or delete. The corresponding action usually is supposed to occur in another system in response to the change that was made in the source system.

Kafka 246

Kafka Data Designing Systems 246

Confluent

DECEMBER 2, 2022

ksqlDB use case: see how apps can use ksqlDB to ingest, filter, enrich, aggregate, and query data directly with Kafka—no complex architectures or data stores needed.

Kafka 144

Kafka Building Architecture Data 144

Confluent

MAY 11, 2023

Running Kafka is costly, but Confluent has created a far more efficient product to lower your costs. Join the Cost Savings challenge to see for yourself.

Kafka 142

Kafka 142

RudderStack

JUNE 6, 2025

A breakdown of event streaming approaches comparing RudderStack to Apache Kafka, detailing when to use each tool and when to use both.

Kafka 40

Kafka 40

Confluent

NOVEMBER 25, 2024

In the final article of this four-part series on Kafka producer and consumer internals, observe the inner workings of brokers as they attempt to serve data up to consumers.

Kafka 133

Kafka Data 133

Confluent

DECEMBER 2, 2022

Major improvements to the Kafka consumer, Streams, and ksqlDB for incremental cooperative rebalancing while maintaining at-least-once and exactly-once guarantees.

Kafka 138

Kafka Process 138

Confluent

SEPTEMBER 7, 2023

Kafka-on-Windows tutorials are everywhere, but most run Kafka directly on Windows. Here's how to use Kafka on Windows in a Linux environment backed by WSL 2, maximizing performance and stability

Kafka 116

Kafka 116

Confluent

MAY 29, 2025

Learn how to easily extract the data you need from Apache Kafka by generating Apache Flink SQL commands with natural language prompts or questions in this step-by-step demo.

Kafka 49

Kafka SQL Data 49

DoorDash Engineering

MARCH 27, 2024

At DoorDash, we rely on message queue systems based on Kafka to handle billions of real-time events. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka. In Kafka, a test tenant processing production event can cause data inconsistencies, including outages and other incidents.

Kafka 103

Kafka Architecture Systems Process 103

Confluent

FEBRUARY 15, 2023

There are two main consumer group memberships in Apache Kafka®. Here’s how static and dynamic consumer groups work, how they affect rebalancing, and which to choose for your application.

Kafka 122

Kafka 122

Monte Carlo

NOVEMBER 12, 2024

Before diving into what makes each company unique, let’s look at the three tools that kept showing up everywhere: Apache Kafka : A distributed event streaming platform that is the standard for moving large amounts of data in real-time. When you request a ride, Uber grabs your location and streams it through Kafka to Flink.

Architecture 96

Architecture Data Engineer Data Engineering Engineering 96

Confluent

NOVEMBER 27, 2023

The top 7 free online courses, tutorials, get started guides, and examples for the easiest way to learn Apache Kafka.

Kafka 132

Kafka 132

Waitingforcode

MAY 11, 2023

I have used to say "Kinesis Data Streams is like Apache Kafka, an append-only streaming broker with partitions and offsets". Although often it's true, it's not that simple unfortunately.

Kafka 130

Kafka IT Data 130

KDnuggets

SEPTEMBER 5, 2023

Build a streaming data pipeline using Formula 1 data, Python, Kafka, RisingWave as the streaming database, and visualize all the real-time data in Grafana.

Data Pipeline 134

Data Pipeline Kafka Building Python 134

Data Engineering Podcast

APRIL 23, 2023

Email hosts@dataengineeringpodcast.com ) with your story. Email hosts@dataengineeringpodcast.com ) with your story.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

RudderStack

JUNE 6, 2025

With RudderStack’s Kafka Source, you can combine the power of Kafka with RudderStack’s CDP to easily extract business value from your Kafka data.

Kafka 40

Kafka Data 40

Confluent

JUNE 28, 2023

Get an introduction into the world of events and event-driven architecture in Apache Kafka. Learn what events are and the role they play in event design, event streaming, and event-driven design.

Kafka 111

Kafka Architecture Designing 111

Knowledge Hut

MAY 3, 2024

Spark Streaming Vs Kafka Stream Now that we have understood high level what these tools mean, it’s obvious to have curiosity around differences between both the tools. Spark Streaming Kafka Streams 1 Data received from live input data streams is Divided into Micro-batched for processing. 6 Spark streaming is a standalone framework.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. On the other hand, log-based message brokers such as Apache Kafka combine the durability of database storage with the low-latency notification capabilities of messaging systems.

Kafka 98

Kafka Python Process Google Cloud 98

Pinterest Engineering

NOVEMBER 18, 2024

Calls the data plane Apache Kafka® Connect API to obtain information about the current state of the system, such as the status of currently running connectors and their configurations. All hosts in this cluster join the same group and run Kafka Connect in distributed mode. Solution : Upgrading to Kafka 2.8.2

Kafka 59

Kafka MySQL Database Software Engineer 59

DoorDash Engineering

AUGUST 13, 2024

This journey began with Kafka, one of our most critical and widely used infrastructure components. Kafka is a distributed event streaming platform that DoorDash uses to handle billions of real-time events. To address this, we developed Kafka Self-Serve, our flagship self-serve storage infrastructure platform.

Kafka 82

Kafka Engineering AWS Designing 82