Confluent

APRIL 10, 2019

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 86

Kafka Metadata Java Cloud 86

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. Given that around 50% of Java clients at Pinterest are on Flink, PSC integration with Flink was key to achieving our platform goals of fully migrating Java clients to PSC.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Cloudera

AUGUST 10, 2021

Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Deep Dive 2: Atlas / Kafka integration. This will expose newly created Kafka topics to Atlas.

Cloud 96

Cloud Kafka Metadata SQL 96

Netflix Tech

JUNE 1, 2023

It also included metadata about ads, such as ad placement and impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Netflix Tech

NOVEMBER 14, 2023

At Netflix, our backend microservices continuously generate real-time event data that gets streamed into Kafka. It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Christophe Blefari

DECEMBER 3, 2022

How LinkedIn cleans up unused metadata for its Kafka clusters — Kafka garbage collection, if you like to understand Kafka internals this post is for you. This time Apple engineers shared at Trino—good old Presto for lost people—Summit how they use it.

Kafka 130

Kafka BI Finance Data 130

Confluent

AUGUST 31, 2022

Kafka Raft lets you use Apache Kafka without ZooKeeper by consolidating metadata management. Here’s how you can learn and do more with KRaft.

Kafka 76

Kafka Metadata Management 76

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Weekly

MAY 16, 2023

Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. Like the staging environment, Fronting Kafka receives all the events without validation. Now, Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Precisely

JULY 21, 2023

Used by more than 75% of the Fortune 500, Apache Kafka has emerged as a powerful open source data streaming platform to meet these challenges. But harnessing and integrating Kafka’s full potential into enterprise environments can be complex. This is where Confluent steps in.

Data Integration 52

Data Integration Kafka Bytes Banking 52

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Weekly

MARCH 24, 2024

The author goes beyond comparing the tools to various offerings from streaming vendors in stream processing and Kafka protocol-supported systems. The APIs support emitting unstructured log lines and typed metadata key-value pairs (per line). The extracted key-value pairs are written to the line’s metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on.

Kafka 55

Kafka Process Algorithm Cloud 55

Rockset

SEPTEMBER 26, 2023

Rockset has built an integration that offers native support for Confluent Cloud and Apache Kafka®, making it simple and fast to ingest real-time streaming data for AI applications. The integration frees users from having to build, deploy or operate any infrastructure component on the Kafka side.

Metadata 64

Metadata Kafka Cloud Database 64

Cloudera

SEPTEMBER 28, 2020

The customer also wanted to utilize the new features in CDP PvC Base like Apache Ranger for dynamic policies, Apache Atlas for lineage, comprehensive Kafka streaming services and Hive 3 features that are not available in legacy CDH versions. Support Kafka connectivity to HDFS, AWS S3 and Kafka Streams. Kafka, SRM, SMM.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Big Data Tools

JUNE 8, 2022

DataHub 0.8.36 – Metadata management is a big and complicated topic. DataHub is a completely independent product by LinkedIn, and the folks there definitely know what metadata is and how important it is. If you haven’t found your perfect metadata management system just yet, maybe it’s time to try DataHub!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

DataHub 0.8.36 – Metadata management is a big and complicated topic. DataHub is a completely independent product by LinkedIn, and the folks there definitely know what metadata is and how important it is. If you haven’t found your perfect metadata management system just yet, maybe it’s time to try DataHub!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Cloudera

MARCH 2, 2023

Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

Process 112

Process SQL Kafka Database 112

Netflix Tech

SEPTEMBER 29, 2022

Timestone clients can create queues, enqueue messages with user-defined deadlines and metadata, then dequeue these messages in an earliest-deadline-first (EDF) fashion. messages that belong to queue X and have metadata Y”) is also supported. Allow for dequeueing and queue depth querying using filters (metadata key-value pairs) 4.

Systems 85

Systems Metadata Media Kafka 85

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Zalando Engineering

NOVEMBER 29, 2017

See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series. Running Kafka Streams applications in AWS At Zalando, Europe’s leading online fashion platform, we use Apache Kafka for a wide variety of use cases. Our team at Zalando was an early adopter of the Kafka Streams API.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Christophe Blefari

JUNE 16, 2023

Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). Masthead does not run SQL on your data—which generate costs uplift—but reading logs and metadata to identify anomalies. At the same time Masthead raises raises $1.3m

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Christophe Blefari

JANUARY 20, 2024

Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. new concepts — in today's data engineering a lot of new concepts enter the field every year like quality, lineage, metadata management, governance, privacy, sharing, etc. This is not.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.);

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

FEBRUARY 9, 2021

A new solution integrating cloud object storage, with Cloudera’s NiFi dataflows, a Kafka datahub, and a Hive virtual warehouse in the CDW service allows businesses to take the best advantage of this public cloud trend. In the real-time layer, Kafka automatically cleans the historical events with configurable retention. Cost-Effective.

Data Warehouse 69

Data Warehouse Cloud Kafka Cloud Storage 69

Rockset

JULY 8, 2020

For this example, we will use Kafka. The service then pushes the geohash along with the coordinates to a Kafka topic. Rockset ingests data from this Kafka topic and updates it into a collection called locations. Rockset has a built-in connector to Apache Kafka. For this example, we will use DynamoDB.

Kafka 40

Kafka Designing Systems Food 40

Christophe Blefari

DECEMBER 9, 2022

Zero trust with Kafka — Sorry I've read too many articles these days, my brain can't process this one but I like diagrams. 👀 See the slides PS: shout-out to people I met there reading the newsletter, your kind words are important and it gives me a lot of motivation.

SQL 130

SQL Data Data Science Metadata 130

Lyft Engineering

OCTOBER 3, 2023

These flow through Kafka , our event streaming platform, before being processed by Flink , our streaming compute framework. Check Result— The numeric measurement of data quality at a point in time, a boolean pass/fail value, and metadata about this run. We log these events asynchronously at the order of millions per second.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Rockset

APRIL 18, 2023

Under the hood, Rockset utilizes its Converged Index technology, which is optimized for metadata filtering, vector search and keyword search, supporting sub-second search, aggregations and joins at scale. Fast Search: Combine vector search and selective metadata filtering to deliver fast, efficient results.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Lyft Engineering

JUNE 28, 2023

The interface was designed such that a minimal amount of metadata was needed to construct a pipeline object which performs a given capability. To facilitate this, we created a common interface called RealtimeMLPipeline for defining all real-time ML applications. Note that in the code block we defined a RealtimeMLPipline as pipe.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Cloudera

MARCH 14, 2023

You want to see how to get data from Kafka and write it to Iceberg? Just create a new draft from the Kafka to Iceberg ReadyFlow and explore it in the Designer. Attributes contain key metadata like the source directory of a file or the source topic of a Kafka message.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Data Engineering Weekly

JANUARY 6, 2023

Recap makes it easy for engineers to build infrastructure and tools that need metadata. Recap is a data catalog for machines–a metadata service. Recap focuses on metadata that software needs–schema, access controls, data profiles, indexes, and queries. Humans use traditional data catalogs.

Metadata 40

Metadata ETL Tools MySQL Data Lake 40

Precisely

MAY 31, 2023

Shared Business and Technical Metadata As IT landscapes grow larger and more complex, organizations struggle to gain a holistic understanding of where data comes from, how it’s used, and which data is most critical to the business. The Data Integrity Suite’s Data Catalog of business and technical metadata connects all Suite services.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Cloudera

JULY 15, 2021

All three will be quorums of Zookeepers and HDFS Journal nodes to track changes to HDFS Metadata stored on the Namenodes. The key requirements for ingest are a number of dedicated disks for both the Kafka broker roles and the Nifi roles. Data flow and streaming (NiFi, Kafka, etc.) Networking . policies can also be defined.

Architecture 97

Architecture Cloud Kafka Hadoop 97

Netflix Tech

JUNE 7, 2021

Flow Collector consumes two data streams, the IP address change events from Sonar via Kafka and eBPF flow log data from the Flow Exporter sidecars. It performs real time attribution of flow data with application metadata from Sonar. Sonar is an IPv6 and IPv4 address identity tracking service.

Transportation 130

Transportation AWS Cloud Metadata 130