dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. In this blog post we’ll dive into how we tackled one source of quality issues: directly relying on upstream database schemas. So our solution was to start using an intentional contract: Events.

Kafka 52

Kafka ETL Tools BI Database 52

Cloudera

SEPTEMBER 26, 2023

The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog. Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

Kafka 91

Kafka Technology IT Government 91

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Kafka 78

Kafka Data Cloud Big Data 78

Confluent

MAY 21, 2021

Apache Kafka® applications run in a distributed manner across multiple containers or machines. This blog post […]. And in the world of distributed systems, what can go wrong often goes wrong.

Kafka 126

Kafka Systems 126

Data Engineering Podcast

NOVEMBER 26, 2019

Summary The team at Sentry has built a platform for anyone in the world to send software errors and events. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period. “To Keep an eye on the Monte Carlo blog for updates!

Kafka 64

Kafka Database High Quality Data Datasets 64

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously.

Kafka 111

Kafka Transportation Metadata Consulting 111

Rockset

NOVEMBER 6, 2019

Events are messages that are sent by a system to notify operators or other systems about a change in its domain. With event-driven architectures powered by systems like Apache Kafka becoming more prominent, there are now many applications in the modern software stack that make use of events and messages to operate effectively.

Kafka 40

Kafka Data Lake SQL Hadoop 40

Rockset

JULY 28, 2022

CDC enables true real-time analytics on your application data, assuming the platform you send the data to can consume the events in real time. Options For Change Data Capture on MongoDB Apache Kafka The native CDC architecture for capturing change events in MongoDB uses Apache Kafka.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Rockset

FEBRUARY 24, 2022

Event-based architectures have been gaining popularity for some time. With increased adoption has come a flood of options for aggregating and analyzing events. Which databases are optimized for ingesting streaming events and analyzing them in real time? Supporting separation of data access (read/write) mechanisms like CQRS.

AWS 52

AWS Amazon Web Services Kafka SQL 52

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. Event Routers typically don’t alter the payload.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Weekly

FEBRUARY 25, 2024

Regarding which city to host the DEWCon event, Berlin and Amsterdam got an almost equal vote!! Our India event was a huge success because we kept the focus on practitioners, not vendors. link] Though the online events are amazing, nothing can match the in-person networking and learning from each other.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Data Engineering Weekly

APRIL 21, 2024

The blog narrates how Chronon fits into Stripe’s online and offline requirements. Grab narrates how it integrated Debeizium, Kafka, and Apache Hudi to enable near real-time data analytics on the data lake. The blog narrates using Apache Arrow Flight RPC to build data querying, post-processing, and caching layers.

Data Engineering 52

Data Engineering Data Engineer Engineering Medical 52

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. Useful to know in case of a disaster event). How far behind is data replication?

Kafka 84

Kafka Metadata Java Cloud 84

Christophe Blefari

MARCH 8, 2024

This is mainly due to the fact that I've done my annual University lectures and talked at 7 events since the beginning of the year. At every event I talk, I decide to do a new presentation. I mean, there is a big difference between podcast for instance and news blogging like I'm doing.

Data 130

Data Algorithm Database Education 130

Cloudera

FEBRUARY 21, 2023

This transformation can be performed on incoming records of a Kafka topic before SSB sees the data. If the Kafka topic has CSV data that we want to add keys and types to it. If the schema you want does not match the incoming Kafka topic. data transformations can be defined using the Kafka Table Wizard.

SQL 109

SQL Kafka Raw Data Data 109

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. It also allowed us to optimize for handling time-series data and event data at scale. ioConfig: Kafka server info, topic names, etc. (ex. This was our main form of ingestion.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Rockset

MARCH 18, 2024

While it's well-known that Flink excels at filtering, joining and enriching streaming data from Apache Kafka® or Confluent Cloud , what is less known is that it is increasingly becoming ingrained in the end-to-end stack for AI-powered applications. We setup a Kafka producer to publish events to a Kafka cluster.

Cloud 64

Cloud Building Metadata Kafka 64

Netflix Tech

MARCH 10, 2023

Data Sharding strategy in elasticsearch is updated to provide low search latency (as described in blog post) Design of new Cassandra reverse indices to support different sets of queries. After reading the asset ids using one of the ways, an event is created per asset id to be processed synchronously or asynchronously based on the use case.

Management 94

Management Kafka Metadata Media 94

Data Engineering Weekly

MARCH 31, 2024

The blog further emphasizes its increased investment in Data Mesh and clean data. The blog is an excellent overview of all the improvements made to PySpark in 2023. The blog narrates how Uber utilizes the existing infrastructure to support various AI/ML use cases. The problem statement is certainly pretty interesting.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Christophe Blefari

MARCH 17, 2023

Airflow alternatives meetup Click on the image to go the the LinkedIn event. We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. We will host another event soon after with others. You can either register on LinkedIn either join the meetup event.

Data 130

Data SQL Deep Learning Kafka 130

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Scribd Technology

MAY 18, 2021

Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd’s data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. To serve this need, we created kafka-delta-ingest.

Kafka 52

Kafka Data Warehouse Bytes Metadata 52

Netflix Tech

AUGUST 30, 2021

In this blog post, we will focus on the latter feature set. In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing.

Management 85

Management Kafka Transportation Cloud 85

Picnic Engineering

MARCH 28, 2023

Pipeline overview Since the TS is an externally developed hardware and software system, Picnic cannot change it to expose analytics events directly. So how can we get our hands on events for analytical purposes if we cannot change the TS to expose them? It detects any changes made to a database and exposes them as events.

Kafka 52

Kafka Transportation Data Warehouse Database 52

Data Engineering Weekly

FEBRUARY 4, 2024

[link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on. Here we go!

Kafka 55

Kafka Process Algorithm Cloud 55

Data Engineering Weekly

SEPTEMBER 24, 2023

The blog talks about the limitations of rule engines and how LLM can enrich additional context to make the rule engine more effective. The blog specifically focuses on the following areas of LLM implementations. The blog is an excellent read about the “Scale First, Learn Last” problem in the data processing world.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Weekly

AUGUST 6, 2023

😃🎊 Click here to Register for DEWCon - 2023 → 🤺Hot Take on The Great Orchestration Battle for Data Land ⚔️ Last week a sequence of blog posts revealed the underlying ambition of the data orchestration companies. the best way to run dbt Core in Airflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 56

Data Ingestion Kafka Java Data Pipeline 56

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Introduction Kafka as an event streaming component can be applied to a wide variety of use cases. and its use is no longer recommended.

Management 69

Management Kafka Big Data Cloud 69

Rockset

OCTOBER 10, 2019

In this blog, we walk through how to build a real-time dashboard for operational monitoring and analytics on streaming event data from Kafka, which often requires complex SQL, including filtering, aggregations, and joins with other data sets. As opposed to posts, let’s look at comments - perhaps a better proxy for engagement.

Kafka 52

Kafka SQL Building Data 52

Uber Engineering

DECEMBER 8, 2016

As Uber continues to scale, our systems generate continually more events, interservice messages, and logs. Those data needs go through Kafka to get processed. To monitor our Kafka pipeline … The post Introducing Chaperone: How Uber Engineering Audits Apache Kafka End-to-End appeared first on Uber Engineering Blog.

Kafka 73

Kafka Engineering Systems Process 73

Scott Logic

NOVEMBER 14, 2023

Rather, I am talking here about asynchronously distributing data to other services that maintain their own copy of the data in order to do their work or where we want to emit and distribute events for other services to process. That said, I’ll try and call out when a technology is better for one or another.

Technology 52

Technology Kafka AWS Data 52