AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka?

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way. To meet the needs of our customers, we kicked off the Real-time Machine Learning with Streaming initiative.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection. Kafka REST Proxy for streaming data.

Data Pipeline 92

Data Pipeline Building Kafka Java 92

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period. “To Keep an eye on the Monte Carlo blog for updates!

Kafka 64

Kafka Database High Quality Data Datasets 64

Cloudera

SEPTEMBER 13, 2021

There are two big gaps in the Apache Kafka project when we think of operating a cluster. There are no solutions for these inside the Kafka project but there are many good 3rd party tools for both problems. Cruise Control is integrated with Kafka through metrics reporting. About Cruise Control. Architecture. Metrics Reporting.

Kafka 71

Kafka Bytes Utilities Management 71

Confluent

MARCH 12, 2019

On the heels of part one in this blog series, Spring for Apache Kafka – Part 1: Error Handling, Message Conversion and Transaction Support , here in part two we’ll focus on another project that enhances the developer experience when building streaming applications on Kafka: Spring Cloud Stream.

Kafka 111

Kafka Cloud Programming Coding 111

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. What technology should you use to build microservice architectures?

Kafka 109

Kafka Designing Architecture ETL Tools 109

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®. The pipe symbol | (i.e.,

Kafka 92

Kafka Cloud Data Pipeline PostgreSQL 92

Cloudera

OCTOBER 19, 2022

In this post I will demonstrate how Kafka Connect is integrated in the Cloudera Data Platform (CDP), allowing users to manage and monitor their connectors in Streams Messaging Manager while also touching on security features such as role-based access control and sensitive information handling. Kafka Connect. Streams Messaging Manager.

Kafka 72

Kafka MySQL Bytes Data 72

Confluent

JUNE 18, 2019

Software projects of all sizes and complexities have a common challenge: building a scalable solution for search. Building a resilient and scalable solution is not always easy. It involves many moving parts, from data preparation to building indexing and query pipelines. Building an indexing pipeline at scale with Kafka Connect.

Architecture 94

Architecture Building Kafka Database-centric 94

Confluent

APRIL 30, 2019

With the release of Apache Kafka ® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. In what follows, we provide some context around how a processor topology was generated inside Kafka Streams before 2.1, Kafka Streams topology generation 101.

Kafka 89

Kafka Coding Process Bytes 89

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This article describes how to instrument Kafka-based applications with distributed tracing capabilities in order to make dataflows between event-based components more visible.

Kafka 111

Kafka Transportation Metadata Consulting 111

Confluent

JULY 16, 2019

How cool would it be to build your own burglar alarm system that can alert you before the actual event takes place simply by using a few network-connected cameras and analyzing the camera images with Apache Kafka ® , Kafka Streams, and TensorFlow? I will show how to implement this use case in this blog post.

Machine Learning 100

Machine Learning Kafka Java Datasets 100

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. Let’s take a look at our build.gradle file to get an understanding of how Gradle builds and tests our UDFs. gradlew composeUp.

Kafka 85

Kafka Java Bytes SQL 85

Rockset

OCTOBER 10, 2019

In this blog, we walk through how to build a real-time dashboard for operational monitoring and analytics on streaming event data from Kafka, which often requires complex SQL, including filtering, aggregations, and joins with other data sets. As opposed to posts, let’s look at comments - perhaps a better proxy for engagement.

Kafka 52

Kafka SQL Building Data 52

Confluent

JANUARY 29, 2021

What happens when you let engineers go wild building an application to track the score of a table football game? This blog post shares SPOUD’s story of engineering a simple […].

Kafka 113

Kafka Engineering Building 113

Data Engineering Podcast

APRIL 19, 2020

Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. This was an interesting inside look at building a business on top of open source stream processing frameworks and how to reduce the burden on end users.

Building 100

Building PostgreSQL MongoDB SQL 100

Data Engineering Podcast

NOVEMBER 26, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What did the previous system look like?

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Rockset

JULY 28, 2022

Options For Change Data Capture on MongoDB Apache Kafka The native CDC architecture for capturing change events in MongoDB uses Apache Kafka. MongoDB provides Kafka source and sink connectors that can be used to write the change events to a Kafka topic and then output those changes to another system such as a database or data lake.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Rockset

OCTOBER 1, 2020

In this post, I’ll walk through what a customer 360 profile is and how to build one that updates in real-time. marketing_emails (Kafka): Marketing data including items sent and any interactions. weblogs (Kafka): Website analytics such as logins, browsing, purchases and any errors. What Is a Customer 360 Profile?

MongoDB 40

MongoDB Kafka Building Data Warehouse 40

Confluent

APRIL 10, 2019

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 83

Kafka Metadata Java Cloud 83

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Data Engineering Weekly

MAY 19, 2024

The blog compares the two modern-day alternatives for Parquet, Nimble & Lance. link] Sponsored: Build next-generation data experiences faster with Cube Cloud Join us for a live webinar on May 22nd at 9 am PT | 12 pm ET, where we'll showcase how to accelerate the development of AI and embedded analytics solutions using Cube Cloud.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Confluent

JUNE 11, 2019

It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline. Here in part 4 of the Spring for Apache Kafka Deep Dive blog series, we will cover: Common event streaming topology patterns supported in Spring Cloud Data Flow.

Kafka 83

Kafka Cloud Java MongoDB 83

Data Engineering Weekly

FEBRUARY 4, 2024

[link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. The goal is to deliver, not detour.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

MARCH 31, 2024

The blog further emphasizes its increased investment in Data Mesh and clean data. The blog is an excellent overview of all the improvements made to PySpark in 2023. The blog narrates how Uber utilizes the existing infrastructure to support various AI/ML use cases. The problem statement is certainly pretty interesting.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. Anna shares a good checklist to build data team foundations.

Kafka 130

Kafka BI Finance Data 130

RandomTrees

AUGUST 8, 2020

Apache Kafka is used for building real-time data pipelines and streaming apps. Apache NiFi can work as a Producer and a Consumer for Kafka. Why NiFi and Kafka together? Integration of Kafka and NiFi helps us to avoid writing lines of code to make it work. Easy to scale.

Kafka 52

Kafka BI Data Pipeline Coding 52

Cloudera

OCTOBER 16, 2023

In this multi-part blog post, we’re going to show you how to use the latest Cloudera Iceberg innovation to build an Open Data Lakehouse on a private cloud. We use NiFi to ingest an airport route data set (JSON) and send that data to Kafka and Iceberg. and follow our Getting Started blog series.

Cloud 78

Cloud Kafka SQL Data 78

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. The blog specifically focuses on the following areas of LLM implementations.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Jesse Anderson

JANUARY 9, 2023

I’ve always been vocal about ksqlDB’s and Kafka Stream’s limitations. The Future of ksqlDB and Kafka Streams With this announcement, the future of primarily ksqlDB and, to a lesser extent, Kafka Streams comes into view. Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly.

Kafka 147

Kafka Coding Technology SQL 147

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Confluent

MAY 29, 2019

In part 1 , we discussed an event streaming architecture that we implemented for a customer using Apache Kafka ® , KSQL from Confluent, and Kafka Streams. Now in part 2, we’ll discuss the challenges we faced developing, building, and deploying the KSQL portion of our application and how we used Gradle to address them.

Kafka 93

Kafka Management Bytes SQL 93

Pinterest Engineering

JANUARY 10, 2023

A single and shared Kafka topic will be used for all logs going through this pipeline. To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site. Logs will be persisted in hive, thus supporting any SQL-based queries. To explore life at Pinterest, visit our Careers page.

Systems 54

Systems Building Software Engineer Software Engineering 54

Data Engineering Weekly

DECEMBER 3, 2023

Github writes an excellent blog to capture the current state of the LLM integration architecture. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications. link] Sophie Blee-Goldman: Kafka Streams and Rebalancing through the Ages Consumers come and go.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Cloudera

JANUARY 31, 2024

It forms a foundational element for building robust and reliable distributed architectures. Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Replication can be dynamically enabled for topics and consumer groups.

Management 69

Management Kafka Big Data Cloud 69