DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Towards Data Science

FEBRUARY 18, 2024

Although the Faust library aims to bring Kafka Streaming ideas into the Python ecosystem, it may pose challenges in terms of ease of use. Traditional databases are ill-suited for storing events in high throughput event streams. This document serves as a tutorial and offers best practices for effectively utilizing Faust.

Kafka 81

Kafka Python Process Google Cloud 81

Pinterest Engineering

NOVEMBER 22, 2023

Goku is our in-house time series database providing cost efficient and low latency storage for metrics data. From these kafka topics, an ingestion service would consume the data points and push them into the GokuS cluster(s) with a retry mechanism (via a separate kafka + small ingestion service) to handle failure.

Database 94

Database Bytes Kafka Architecture 94

Towards Data Science

FEBRUARY 9, 2024

This involves getting data from an API and storing it in a PostgreSQL database. In the second phase, we’ll develop an application that uses a language model to interact with this database. The second article, which will come later, will delve into creating agents using tools like LangChain to communicate with external databases.

Kafka 76

Kafka Data Engineering Data Engineer PostgreSQL 76

Confluent

FEBRUARY 23, 2023

How to use Kafka Streams to aggregate change data capture (CDC) messages from a relational database into transactional messages, powering a scalable microservices architecture.

Kafka 108

Kafka Relational Database Architecture Database 108

KDnuggets

SEPTEMBER 5, 2023

Build a streaming data pipeline using Formula 1 data, Python, Kafka, RisingWave as the streaming database, and visualize all the real-time data in Grafana.

Data Pipeline 104

Data Pipeline Kafka Building Python 104

Snowflake

MARCH 2, 2023

The volume of data generated in real time from application databases, sensors, and mobile devices continues to grow exponentially. As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka. How does Snowpipe Streaming work?

Kafka 123

Kafka Data Ingestion Data Pipeline Cloud Storage 123

Confluent

AUGUST 11, 2020

However, not all databases can be in the […]. Building a Cloud ETL Pipeline on Confluent Cloud shows you how to build and deploy a data pipeline entirely in the cloud.

Database 100

Database Kafka Data Pipeline Cloud 100

Cloudera

SEPTEMBER 26, 2023

The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog. Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

Kafka 95

Kafka Technology IT Government 95

Workfall

JUNE 27, 2023

It means that there is a high risk of data loss but Apache Kafka solves this because it is distributed and can easily scale horizontally and other servers can take over the workload seamlessly. So if a user requests a piece of information, the request goes to the Database server. This is where Apache Kafka comes in.

Kafka 75

Kafka Architecture AWS Transportation 75

Data Engineering Podcast

MARCH 2, 2020

The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing. Developers can work with the SQL constructs that they are familiar with while automatically getting the durability and reliability that Kafka offers. How is ksqlDB architected?

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Towards Data Science

DECEMBER 1, 2023

Why static workload is insufficient and what I learned by comparing HNSWLIB and DiskANN using streaming workload Image by DALLE-3 Vector databases are built for high-dimensional vector retrieval. Many vector databases are now measuring their performance using this approach in their tech blogs. O(log n) ) with respect to index size.

Database 52

Database Algorithm Datasets Kafka 52

Data Engineering Podcast

MAY 8, 2022

TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. How has the ecosystem of graph databases changed in usage and design in recent years? Start trusting your data with Monte Carlo today! Visit [link] to learn more.

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

JANUARY 5, 2020

Summary Databases are useful for inspecting the current state of your application, but inspecting the history of that data can get messy without a way to track changes as they happen. How has the tight coupling with Kafka impacted the direction and capabilities of Debezium? What, if any, other substrates does Debezium support (e.g.

Database 100

Database Kafka PostgreSQL MySQL 100

Rockset

MARCH 27, 2023

In this post, we’ll explore the differences between real-time analytics databases and stream processing frameworks. Differing Paradigms Stream processing systems and real-time analytics (RTA) databases are both exploding in popularity. Let’s start with a quick summary of both stream processing and RTA databases.

Database 52

Database Process Scala SQL 52

Confluent

JULY 29, 2020

Change Data Capture (CDC) is an excellent way to introduce streaming analytics into your existing database, and using Debezium enables you to send your change data through Apache Kafka®. Although […].

Kafka 104

Kafka Database MongoDB Data 104

Knowledge Hut

JULY 24, 2023

Recently, the advent of stream processing has unlocked the door for a new era in database technology. In today’s data-driven world, the future of SQL is entwined with the future of databases and becoming highly significant. According to recent studies, the global database market will grow from USD 63.4

SQL 52

SQL Database Process NoSQL 52

Data Engineering Podcast

SEPTEMBER 12, 2021

Summary The Cassandra database is one of the first open source options for globally scalable storage systems. The community recently released a new major version that marks a milestone in its maturity and stability as a project and database. Since its introduction in 2008 it has been powering systems at every scale.

Database 100

Database Metadata Kafka Data Storage 100

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! This year’s London event was almost double the size of last year, so I’m especially happy to see these numbers on the page: 2 days of Apache Kafka ®. World-class Kafka keynotes. Privacy Engineering for the World of Kafka by Alexander Cook of Privitar.

Kafka 109

Kafka Cloud Database Engineering 109

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). COD is an operational database-as-a-service that brings ease of use and flexibility to Apache HBase. Read about Building a Scalable Process Using NiFi, Kafka, and HBase on CDP. Cloudera DataFlow .

Database 86

Database Machine Learning Data Lake Kafka 86

Cloudera

SEPTEMBER 23, 2020

Cloudera’s platform can support piping of audit data to HDFS, Kafka, Syslog or to SIEM systems for long-term retention and archival. Database object security. Database object-level security is available through the centralized authorization framework of Apache Ranger. . Security levels.

Database 82

Database Data Lake Metadata Java 82

Rockset

MARCH 22, 2022

Real-time analytics can be achieved in a number of ways, but approaches can generally be split into two camps: streaming analytics and analytics databases. Analytics databases ingest data in as near real time as possible, and allow fast analytical queries to be done on this data.

Database 52

Database Kafka Transportation Unstructured Data 52

Confluent

NOVEMBER 21, 2019

ksqlDB is a new kind of database purpose-built for stream processing apps, allowing users to build stream processing applications against data in Apache Kafka® and enhancing developer productivity. ksqlDB simplifies […].

Kafka 108

Kafka Database Process Building 108

Rockset

OCTOBER 21, 2022

When building a real-time customer 360 app, you’ll definitely need event data from a streaming data source, like Kafka. You’ll also need a transactional database to store customers’ transactions and personal information. We’ll be building a basic version of this using Kafka, S3, Rockset, and Retool.

Kafka 52

Kafka Building SQL Database 52

Cloudera

JANUARY 19, 2022

Gartner® recognized Cloudera in three recent reports – Magic Quadrant for Cloud Database Management Systems (DBMS), Critical Capabilities for Cloud Database Management Systems for Analytical Use Cases and Critical Capabilities for Cloud Database Management Systems for Operational Use Cases. Get started with CDP.

Database 92

Database Cloud Data Warehouse Data Lake 92

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This article describes how to instrument Kafka-based applications with distributed tracing capabilities in order to make dataflows between event-based components more visible.

Kafka 111

Kafka Transportation Metadata Consulting 111

Afterpay Tech

SEPTEMBER 6, 2022

Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. Context The asynchronous processing capability that Kafka offers opens up numerous innovation opportunities to interact with other services.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Data Engineering Podcast

JUNE 1, 2020

What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert. What used to be entirely managed by the database engine is now a composition of multiple systems that need to be properly configured to work in concert.

Data Lake 100

Data Lake Database Building Lambda Architecture 100

Rockset

JULY 28, 2022

MongoDB has grown from a basic JSON key-value store to one of the most popular NoSQL database solutions in use today. So, this means we generally want to allow our application to talk directly to a database. These services would still need to talk directly to the database. It turns out this is hard to do in practice.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Rockset

APRIL 26, 2022

As Kafka Summit is in full swing in London this week and the topic of event streaming is all over my Linkedin feed, I saw a post asking " Is streaming dead? That streaming is rocking and with Kafka Summit this week, I thought it a good time to emphasize the importance of streaming data in today’s modern real-time data stack.

Kafka 52

Kafka Data Warehouse Database Data 52

Cloudera

FEBRUARY 9, 2021

Cloudera Operational Database is now available in three different form-factors in Cloudera Data Platform (CDP). . If you are new to Cloudera Operational Database, see this blog post. In this blog post, we’ll look at both Apache HBase and Apache Phoenix concepts relevant to developing applications for Cloudera Operational Database.

Database 103

Database Java Data Ingestion SQL 103

Confluent

MAY 18, 2020

What is stopping you from using Kafka Streams as your data layer for building applications? After all, it comes with fast, embedded RocksDB storage, takes care of redundancy for you, […].

Kafka 111

Kafka Building IT Database 111

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

Data Engineering Podcast

JULY 1, 2019

Scaling the volume of events that can be processed in real-time can be challenging, so Paul Brebner from Instaclustr set out to see how far he could push Kafka and Cassandra for this use case. And don’t forget to thank them for supporting the show! And don’t forget to thank them for supporting the show!

Kafka 100

Kafka Finance Media Architecture 100

Data Engineering Podcast

FEBRUARY 4, 2024

RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. There are numerous stream processing engines, near-real-time database engines, streaming SQL systems, etc. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Jesse Anderson

JANUARY 9, 2023

I’ve always been vocal about ksqlDB’s and Kafka Stream’s limitations. The Future of ksqlDB and Kafka Streams With this announcement, the future of primarily ksqlDB and, to a lesser extent, Kafka Streams comes into view. Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly.

Kafka 147

Kafka Coding Technology SQL 147