Rockset

JULY 28, 2022

Options For Change Data Capture on MongoDB Apache Kafka The native CDC architecture for capturing change events in MongoDB uses Apache Kafka. The out-of-the-box connectors make it fairly simple to set up the CDC solution, however they do require the use of a Kafka cluster.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

ProjectPro

DECEMBER 7, 2021

This blog will give you an in-depth knowledge of what is a data pipeline and also explore other aspects such as data pipeline architecture, data pipeline tools, use cases, and so much more. Features of a Data Pipeline Data Pipeline Architecture How to Build an End-to-End Data Pipeline from Scratch? What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid Architecture at Lyft Druid Architecture at Lyft With Lyft as a Kubernetes shop, we ran all of the above processes into our stateful and stateless Kubernetes pods within the service namespaces.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Data Engineering Weekly

DECEMBER 3, 2023

Github: The architecture of today’s LLM applications LLM is slowly changing the application architecture landscape as it becomes integral to app development. Github writes an excellent blog to capture the current state of the LLM integration architecture. Visit rudderstack.com to learn more.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Weekly

MAY 16, 2023

In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. I won’t bore you with the importance of data quality in the blog. Instead, Let’s examine the current data pipeline architecture and ask why data quality is expensive. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Jesse Anderson

JANUARY 9, 2023

I’ve always been vocal about ksqlDB’s and Kafka Stream’s limitations. The Future of ksqlDB and Kafka Streams With this announcement, the future of primarily ksqlDB and, to a lesser extent, Kafka Streams comes into view. Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly.

Kafka 147

Kafka Coding Technology SQL 147

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. First and foremost, Confluent Control Center can manage multi-datacenter Apache Kafka ® deployments, whether on prem or in the cloud.

Kafka 86

Kafka Metadata Java Cloud 86

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Data Engineering Weekly

SEPTEMBER 24, 2023

The blog talks about the limitations of rule engines and how LLM can enrich additional context to make the rule engine more effective. The blog specifically focuses on the following areas of LLM implementations. The blog is an excellent read about the “Scale First, Learn Last” problem in the data processing world.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Jesse Anderson

DECEMBER 12, 2023

We talk about his time evolving architectures and creating real-time systems at Auchan (grocery) and Adeo/Leroy Merlin (Home Improvement). We go in-depth into Conduktor as to how it works and more even so how it works with your Kafka deployment. We discuss the issues of British food and how to find good food in London.

Food 100

Food Kafka Architecture Systems 100

Data Engineering Weekly

FEBRUARY 4, 2024

[link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

LinkedIn Engineering

OCTOBER 19, 2023

In 2010, they introduced Apache Kafka , a pivotal Big Data ingestion backbone for LinkedIn’s real-time infrastructure. To transition from batch-oriented processing and respond to Kafka events within minutes or seconds, they built an in-house distributed event streaming framework, Apache Samza. hours to 25 minutes).

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Uber Engineering

FEBRUARY 16, 2018

Given the scope … The post Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka appeared first on Uber Engineering Blog.

Kafka 109

Kafka Building Engineering Systems 109

Cloudera

JANUARY 31, 2024

It forms a foundational element for building robust and reliable distributed architectures. Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. Up until now, this type of replication was not available or fully supported.

Management 68

Management Kafka Big Data Cloud 68

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on.

Kafka 55

Kafka Process Algorithm Cloud 55

Christophe Blefari

JANUARY 20, 2024

Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). workflows (Airflow, Prefect, Dagster, etc.) This is not.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

AUGUST 6, 2023

😃🎊 Click here to Register for DEWCon - 2023 → 🤺Hot Take on The Great Orchestration Battle for Data Land ⚔️ Last week a sequence of blog posts revealed the underlying ambition of the data orchestration companies. the best way to run dbt Core in Airflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Uber Engineering

OCTOBER 21, 2021

Apache Kafka ® is widely used across Uber’s multiple business lines. Take the example of an Uber ride: When a user opens up the Uber app, demand and supply data are aggregated in Kafka queues to serve fare calculations. Background.

Management 131

Management Kafka Engineering Data 131

Netflix Tech

AUGUST 30, 2021

In this blog post, we will focus on the latter feature set. The challenge, then, is to be able to ingest and process these events in a scalable manner, i.e., scaling with the number of devices, which will be the focus of this blog post. In particular, the Kafka integration is the most relevant for this blog post.

Management 85

Management Kafka Transportation Cloud 85

Cloudera

SEPTEMBER 28, 2020

Cloudera has found that customers have spent many years investing in their big data assets and want to continue to build on that investment by moving towards a more modern architecture that helps leverage the multiple form factors. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams. Background: .

Cloud 131

Cloud Kafka Professional Services Metadata 131

Cloudera

MAY 1, 2023

By breaking down monolithic apps into microservices architectures, for example, or making modularized data products, organizations do their best to enable more rapid iterative cycles of design, build, test, and deployment of innovative solutions. For example, one of the tables in your project might be backed by a Kafka topic.

SQL 76

SQL Project Kafka Accessible 76

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Cloudera

MARCH 5, 2024

The latest generation of our platform includes Ozone features like improved replication, improved quotas for volumes, buckets to facilitate cloud-native architectures, and snapshots, which are also now able to support data storage at the bucket and volume levels. ZDU gives organizations a more convenient means of upgrading.

Data Lake 86

Data Lake Data Storage Government Kafka 86

Uber Engineering

DECEMBER 8, 2016

Those data needs go through Kafka to get processed. To monitor our Kafka pipeline … The post Introducing Chaperone: How Uber Engineering Audits Apache Kafka End-to-End appeared first on Uber Engineering Blog. As Uber continues to scale, our systems generate continually more events, interservice messages, and logs.

Kafka 71

Kafka Engineering Systems Process 71

Monte Carlo

SEPTEMBER 8, 2023

Tenable One architecture. A microservice architecture brings all the benefits of minimizing blast radii and faster CI/CD pipelines to production, but it can be hard to monitor. Luckily the pipeline is well instrumented with start and end times of each stage saved to a central Kafka topic.

Kafka 75

Kafka SQL Data Pipeline Database 75

Knowledge Hut

APRIL 25, 2024

In this blog post, we will discuss such technologies. Big data storage technologies is a compute-and-storage architecture that collects and manages large data sets while also allowing real-time data analytics. Due to its user-friendly API and scalable architecture, it has a growing community of users and developers.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Data Engineering Weekly

NOVEMBER 26, 2023

The Battle for Supremacy: Inside the Fierce Lakehouse Architecture War One of the hot topics in the data industry is which LakeHouse format to choose. The data industry clearly understands the power of blog storage, and using S3 as a database is not a new concept either. Let me know your thoughts in the comments. But I had my doubts.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Rockset

AUGUST 21, 2019

Rockset continuously ingests data streams from Kafka, without the need for a fixed schema, and serves fast SQL queries on that data. We created the Kafka Connect Plugin for Rockset to export data from Kafka and send it to a collection of documents in Rockset. This blog covers how we implemented the plugin.

Kafka 40

Kafka IT Data Storage Relational Database 40

Netflix Tech

JANUARY 13, 2022

This blog will explore these two systems and how they perform auto-diagnosis and remediation across our Big Data Platform and Real-time infrastructure. Streaming Pensive Streaming Pensive Architecture Apache Flink powers real-time stream processing jobs in the Netflix data platform.

Kafka 95

Kafka Big Data Data Machine Learning 95

Data Engineering Weekly

FEBRUARY 19, 2023

The blog discusses fairness in ML and demonstrates why high accuracy doesn’t mean the algorithm is fair. link] Sponsored: [New Guide] The Ultimate Guide to Data Mesh Architecture If implementing data mesh is high on your list of priorities, you’re not alone.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Jesse Anderson

DECEMBER 12, 2022

Apache Kafka came in 2011 and gave the industry a much better way to move real-time data. Apache Kafka has its architectural limitations, and Apache Pulsar was released in 2016. Apache Flink came in 2011 and gave us our first real streaming engine. It handled the stateful problems of real-time elegantly.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

JUNE 25, 2023

A16Z: Emerging Architectures for LLM Applications LLM is slowly hitting the enterprise architecture, and every company is looking to see how they can adopt LLM with their private data. A16Z published a reference architecture for the emerging LLM stack. The blog narrates the essence of the asset model below.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Rockset

JULY 8, 2020

For this example, we will use Kafka. The service then pushes the geohash along with the coordinates to a Kafka topic. Rockset ingests data from this Kafka topic and updates it into a collection called locations. Rockset has a built-in connector to Apache Kafka. For this example, we will use DynamoDB.

Kafka 40

Kafka Designing Systems Food 40

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Cloudera

JANUARY 19, 2022

Run on your desktop or cloud node, connecting to Kafka or other sources/sinks via API calls to their respective clusters. We hope this helps clear up some questions around what CSA CE can be used for and suggested configurations and architectures. We plan future blog posts on this workflow.

Java 85

Java Kafka SQL Cloud 85

Cloudera

AUGUST 10, 2021

With Apache Ozone on the Cloudera Data Platform (CDP) , they can implement a scale-out model and build out their next generation storage architecture without sacrificing security, governance and lineage. Then you can import Kafka lineage using the Atlas Kafka import tool provided with CDP. Writing to Ozone in Spark.

Hadoop 104

Hadoop Kafka Datasets Government 104

Cloudera

APRIL 20, 2021

What is modern streaming architecture? A modern streaming architecture consists of critical components that provide data ingestion, security and governance, and real-time analytics. Streamings Messaging , powered by Apache Kafka, buffers and scales massive volumes of data streams for streaming analytics. Watch a video.

Hospitality 92

Hospitality Kafka Retail Data Ingestion 92