Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. FROM SYSMON_JOIN.

Process 80

Process Kafka Datasets SQL 80

Confluent

SEPTEMBER 24, 2020

Operating critical Apache Kafka® event streaming applications in production requires sound automation and engineering practices. Streaming applications are often at the center of your transaction processing and data systems, requiring […].

Kafka 143

Kafka Engineering Systems Process 143

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 91

Kafka Technology IT Government 91

Confluent

NOVEMBER 30, 2022

This is the second part of our guide on streaming data and Apache Kafka. This guide will contain specific advice on how to go about building an event streaming platform in your organization.

Kafka 52

Kafka Building Data Process 52

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Confluent

SEPTEMBER 29, 2020

ksqlDB, the event streaming database, is becoming one of the most popular ways to work with Apache Kafka®. Every day, we answer many questions about the project, but here’s a […].

Process 145

Process Kafka Database Project 145

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 78

Kafka Data Cloud Big Data 78

Rockset

MARCH 5, 2020

Apache Kafka has seen broad adoption as the streaming platform of choice for building applications that react to streams of data in real time. In many organizations, Kafka is the foundational platform for real-time event analytics, acting as a central location for collecting event data and making it available in real time.

Kafka 40

Kafka Data Warehouse Data Lake Relational Database 40

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. You can use this as an example of how to add graph abilities to any event streaming application.

Kafka 55

Kafka Process Algorithm Cloud 55

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! By processing a smaller set of data, more often , you effectively divide and conquer a data problem that may otherwise be cost and time prohibitive.

Process 87

Process Data Lake Systems Bytes 87

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. And if you weren’t there? Well, dig in and start learning! 3 keynotes.

Kafka 109

Kafka Cloud Database Engineering 109

Cloudera

JULY 18, 2022

In this blog we will conclude the implementation of our fraud detection use case and understand how Cloudera Stream Processing makes it simple to create real-time stream processing pipelines that can achieve neck-breaking performance at scale. Flink is a “streaming first” modern distributed system for data processing.

Process 87

Process Kafka Scala SQL 87

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. In this episode James Cunningham and Ted Kaemming describe the process of rearchitecting a production system, what they learned in the process, and some useful tips for anyone else evaluating Clickhouse.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Afterpay Tech

SEPTEMBER 6, 2022

Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. Context The asynchronous processing capability that Kafka offers opens up numerous innovation opportunities to interact with other services.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Cloudera

JUNE 28, 2022

We discussed how Cloudera Stream Processing (CSP) with Apache Kafka and Apache Flink could be used to process this data in real time and at scale. Building real-time streaming analytics data pipelines requires the ability to process data in the stream. This is what we call the first-mile problem.

Process 82

Process Kafka SQL Machine Learning 82

Confluent

MAY 13, 2019

In the last year, we’ve experienced enormous growth on Confluent Cloud, our fully managed Apache Kafka ® service. As Confluent Cloud has grown, we’ve noticed two gaps that very clearly remain to be filled in managed Apache Kafka services. Five seconds to Kafka (or, never make another cluster again!).

Kafka 100

Kafka Cloud Management Building 100

Netflix Tech

MARCH 10, 2023

Hence we built the data pipeline that can be used to extract the existing assets metadata and process it specifically to each new use case. After reading the asset ids using one of the ways, an event is created per asset id to be processed synchronously or asynchronously based on the use case.

Management 94

Management Kafka Metadata Media 94

Data Engineering Weekly

MAY 16, 2023

The bias toward correctness will increase the processing time, which may not be feasible when speed is a priority. Let’s talk about the data processing types. The industry follows two categories of data processing techniques to handle the trade-off between speed and correctness.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

NOVEMBER 18, 2018

Summary Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. Can state be shared across processes or tasks within a Flink cluster?

Process 100

Process Scala Google Cloud Kafka 100

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Then it is really up to you what you want to with the event data.

Kafka 86

Kafka Java Programming Language Data Pipeline 86

Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Kafka 100

Kafka Finance Media Architecture 100

Lyft Engineering

NOVEMBER 29, 2023

It also allowed us to optimize for handling time-series data and event data at scale. Druid Architecture at Lyft Druid Architecture at Lyft With Lyft as a Kubernetes shop, we ran all of the above processes into our stateful and stateless Kubernetes pods within the service namespaces. ioConfig: Kafka server info, topic names, etc.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Confluent

JUNE 27, 2019

Confluent’s clients for Apache Kafka ® recently passed a major milestone—the release of version 1.0. Magnus Edenhill first started developing librdkafka about seven years ago, later joining Confluent in the very early days to help foster the community of Kafka users outside the Java ecosystem. Builder classes are also new in v1.0

Kafka 106

Kafka Designing Java Coding 106

Cloudera

MAY 13, 2021

Are you using the right stream processing engine for the job at hand? but have you really examined the stream processing engines out there in a side-by-side comparison to make sure? Specifically, the whitepaper analyzes the technical and operational differences between modern processing engines from the Apache open source community.

Process 101

Process Engineering Kafka Architecture 101

Rockset

DECEMBER 14, 2022

Rockset’s native connector for Amazon Managed Streaming for Apache Kafka (MSK) makes it simpler and faster to ingest streaming data for real-time analytics. Amazon MSK is a fully managed AWS service that gives users the ability to build and run applications using Apache Kafka.

Kafka 52

Kafka MongoDB SQL AWS 52

Rockset

JULY 28, 2022

And you have now introduced another process that has to run, be monitored, scale etc. CDC enables true real-time analytics on your application data, assuming the platform you send the data to can consume the events in real time. Debezium It is also possible to capture MongoDB change data capture events using Debezium.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Data Engineering Podcast

DECEMBER 30, 2019

In order to support their customers, they need to capture, process, and analyze massive amounts of timeseries data with a high degree of uptime and reliability. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh.

Process 100

Process Building Hadoop Java 100

Rockset

SEPTEMBER 7, 2021

Event stream processing has lately become the most-requested feature among data practitioners, who are ever being pushed by their business counterparts for more fresh, real-time insights to improve their operational decisions and boost the digital customer experience.

Kafka 52

Kafka SQL Education Data 52

Confluent

DECEMBER 7, 2022

Learn how to build a real-time streaming application using Apache Kafka® and.NET producer and consumer clients.

Building 57

Building Kafka Process 57

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Precisely

DECEMBER 28, 2023

Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. It also allows for applications, analytics, and reporting to process information as it happens. That enables you to collect, analyze, and store large amounts of information.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Confluent

APRIL 3, 2024

Learn why Confluent was named a Leader in the analytic stream processing and event brokering software markets. We believe we innovate every industry with real-time stream processing and analytics, cloud-native Apache Kafka®, and robust developer tooling.

Kafka 61

Kafka Cloud Process 61

Zalando Engineering

JULY 12, 2017

While developing Zalando’s real-time business process monitoring solution, we encountered the need to generate complex events upon the detection of specific patterns of input events. In this blog post we describe the generation of such events using Apache Flink, and share our experiences and lessons learned in the process.

Process 40

Process Kafka AWS Architecture 40

Confluent

FEBRUARY 20, 2019

How many Kafka Summits should there be in a year? Some say there should be one giant event where everybody gathers at once. Others say you should live every day like it’s Kafka Summit. Anyway, let me remind you of the dates and places: Kafka Summit New York – April 2, 2019. Kafka Summit London – May 13-14, 2019.

Kafka 66

Kafka Programming Architecture Technology 66

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Cloudera

FEBRUARY 21, 2023

This transformation can be performed on incoming records of a Kafka topic before SSB sees the data. If the Kafka topic has CSV data that we want to add keys and types to it. If the schema you want does not match the incoming Kafka topic. data transformations can be defined using the Kafka Table Wizard.

SQL 109

SQL Kafka Raw Data Data 109