Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 78

Kafka Data Cloud Big Data 78

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. Email hosts@dataengineeringpodcast.com ) with your story.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Confluent

JULY 1, 2020

This is the third month of Project Metamorphosis, where we discuss new features in Confluent’s offerings that bring together event streams and the best characteristics of modern cloud data systems. […].

Project 119

Project Cloud Kafka Systems 119

Confluent

AUGUST 4, 2020

We launched Project Metamorphosis in April with the goal of making it dramatically easier for organizations to leverage an event streaming platform for their business needs. By reimagining event streaming […].

Project 95

Project Kafka Cloud Google Cloud 95

Rockset

AUGUST 16, 2022

Streams for Everyone If you have come this far it means you have already considered or are considering using event streaming in your data architecture for the wide variety of benefits it can offer. In either case, both Amazon Kinesis and Apache Kafka can help but which one is the right fit for you and your goals.

Kafka 52

Kafka AWS Cloud Java 52

Afterpay Tech

SEPTEMBER 6, 2022

Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. Context The asynchronous processing capability that Kafka offers opens up numerous innovation opportunities to interact with other services.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Confluent

FEBRUARY 28, 2019

Only a little more than one month after the first release, we are happy to announce another milestone for our Kafka integration. Today, you can grab the Kafka Connect Neo4j Sink from Confluent Hub. . Neo4j extension – Kafka sink refresher. Then it is really up to you what you want to with the event data.

Kafka 86

Kafka Java Programming Language Data Pipeline 86

Lyft Engineering

NOVEMBER 29, 2023

It also allowed us to optimize for handling time-series data and event data at scale. Real-time Ingestion Events from our real-time analytics pipeline were configured to be sent into our internal Flink application, streamed to Kafka, and written into Druid. ioConfig: Kafka server info, topic names, etc. (ex.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Confluent

MAY 13, 2019

In the last year, we’ve experienced enormous growth on Confluent Cloud, our fully managed Apache Kafka ® service. As Confluent Cloud has grown, we’ve noticed two gaps that very clearly remain to be filled in managed Apache Kafka services. Five seconds to Kafka (or, never make another cluster again!).

Kafka 100

Kafka Cloud Management Building 100

Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. Coming up this fall is the combined events of Graphorum and the Data Architecture Summit.

Kafka 100

Kafka Finance Media Architecture 100

Confluent

JUNE 12, 2020

Apache Kafka® is the defacto standard for event streaming today. The semantics of the partitioned consumer model that Kafka pioneered have enabled scale at a level and at a cost […].

Kafka 98

Kafka Cloud Project 98

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. FROM SYSMON_JOIN.

Process 80

Process Kafka Datasets SQL 80

Precisely

DECEMBER 28, 2023

Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. One very popular platform is Apache Kafka , a powerful open-source tool used by thousands of companies. How do streaming data pipelines work? You need a separate tool to do that.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Confluent

JUNE 17, 2020

In April, we kicked off Project Metamorphosis. Project Metamorphosis is an effort to bring the simplicity of best of breed cloud systems to the world of event streaming. It is […].

Project 90

Project Kafka Cloud Systems 90

Rockset

SEPTEMBER 7, 2021

Event stream processing has lately become the most-requested feature among data practitioners, who are ever being pushed by their business counterparts for more fresh, real-time insights to improve their operational decisions and boost the digital customer experience. We’d love to discuss your initiatives.

Kafka 52

Kafka SQL Education Data 52

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing. We used Elasticsearch dashboards to analyze results.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. The Azure Data Engineer certification aspirants frequently seek out real-world projects in order to obtain hands-on experience and demonstrate their skills.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Confluent

DECEMBER 2, 2022

Confluent announces revolutionary event streaming features so any businesses can leverage modernized, scalable, multi-cloud data systems.

Cloud 57

Cloud Kafka Project Systems 57

Christophe Blefari

MARCH 17, 2023

Airflow alternatives meetup Click on the image to go the the LinkedIn event. We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. We will host another event soon after with others. You can either register on LinkedIn either join the meetup event.

Data 130

Data SQL Deep Learning Kafka 130

Confluent

SEPTEMBER 29, 2020

ksqlDB, the event streaming database, is becoming one of the most popular ways to work with Apache Kafka®. Every day, we answer many questions about the project, but here’s a […].

Process 145

Process Kafka Database Project 145

Jesse Anderson

DECEMBER 12, 2022

Apache Kafka came in 2011 and gave the industry a much better way to move real-time data. Apache Kafka has its architectural limitations, and Apache Pulsar was released in 2016. There was (and still is) an overall problem in the industry because most projects failed to get into production. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

OCTOBER 15, 2023

As more people start using AI for projects, two things are clear: It’s a rapidly advancing field, but it’s tough to navigate. If you've learned something or tried out a project from the show then tell us about it! If you've learned something or tried out a project from the show then tell us about it!

Process 182

Process Building SQL BI 182

Netflix Tech

AUGUST 30, 2021

In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing. In particular, the Kafka integration is the most relevant for this blog post.

Management 85

Management Kafka Transportation Cloud 85

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. Can you describe your experiences with Kafka? What are the operational challenges that you have had to overcome while working with Kafka?

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

MAY 11, 2020

In this episode Sijie Guo discusses his motivations for spending so much of his time and energy on contributing to the project and growing the community. This was a great conversation about the strengths of the Pulsar project, how it has evolved in recent years, and some of the innovative ways that it is being used.

Lambda Architecture 100

Lambda Architecture Cloud Kafka Hadoop 100

Data Engineering Podcast

SEPTEMBER 28, 2020

Summary Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine.

Kafka 100

Kafka BI Big Data Data Engineering 100

Lyft Engineering

JUNE 28, 2023

The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. One key component is the Analytics Event Abstraction layer. Note that in the code block we defined a RealtimeMLPipline as pipe. Over time, we settled on an engagement model.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Data Engineering Podcast

JULY 27, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. How do you handle migrating existing projects, particularly if they are using Kafka currently?

Building 100

Building Management Kafka Data Warehouse 100

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Introduction Kafka as an event streaming component can be applied to a wide variety of use cases. and its use is no longer recommended.

Management 69

Management Kafka Big Data Cloud 69

Data Engineering Podcast

AUGUST 3, 2021

Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge. By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Can you describe how Hudi is architected?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

JULY 6, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Kafka SQL Government 100

Cloudera

SEPTEMBER 13, 2023

Soon after, I became a huge fan of Apache Kafka. An opportunity to pursue an exciting new project at a major fortune 500 company came up and I decided to give it a try. An opportunity to pursue an exciting new project at a major fortune 500 company came up and I decided to give it a try.

Technology 83

Technology Hadoop Kafka Project 83

Rockset

FEBRUARY 7, 2020

In this IoT example, we examine how to enable complex analytic queries on real-time Kafka streams from connected car sensors. Building Real-Time Analytics on Connected Car IoT Data For our example, we have a fleet of connected vehicles that send the sensor data they generate to a Kafka cluster.

Kafka 40

Kafka Transportation Data SQL 40

Data Engineering Weekly

FEBRUARY 4, 2024

The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. It opens some old memory; try to solve this problem first with Presto-Kafka connector and then using OLAP engines like Druid & Apache Pinot. If you have a data quality problem, success like this can seem out of reach.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

SEPTEMBER 24, 2023

Many teams struggle to effectively use their customer data from Kafka to drive value downstream because of the custom integration and pipeline work required. Many teams struggle to effectively use their customer data from Kafka to drive value downstream because of the custom integration and pipeline work required.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Podcast

MARCH 8, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Data Warehouse MongoDB Kafka 100