Data Engineering Weekly

MAY 16, 2023

Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. Like the staging environment, Fronting Kafka receives all the events without validation. Now, Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Confluent

APRIL 10, 2019

Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites. Useful to know in case of a disaster event). How far behind is data replication?

Kafka 84

Kafka Metadata Java Cloud 84

Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. With the Kafka Integration, users no longer need to build, deploy or operate any infrastructure component on the Kafka side.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Confluent

FEBRUARY 21, 2019

During a recent talk titled Hunters ATT&CKing with the Right Data , which I presented with my brother Jose Luis Rodriguez at ATT&CKcon, we talked about the importance of documenting and modeling security event logs before developing any data analytics while preparing for a threat hunting engagement. FROM SYSMON_JOIN.

Process 80

Process Kafka Datasets SQL 80

Cloudera

AUGUST 10, 2021

Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Deep Dive 2: Atlas / Kafka integration. This will expose newly created Kafka topics to Atlas.

Cloud 96

Cloud Kafka Metadata SQL 96

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Lyft Engineering

OCTOBER 3, 2023

For example, we can almost instantly validate that each record is well-formed and complete during event generation. Our Analytic Event Lifecycle below demonstrates the workflow of how much of our data gets to Hive. We log these events asynchronously at the order of millions per second.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Rockset

SEPTEMBER 26, 2023

As smart as ChatGPT appears to be, it can’t summarize current events accurately if it was last trained a year ago and not told what’s happening now. Models may need to know about events, computed metrics, and embeddings based on locality.

Metadata 64

Metadata Kafka Cloud Database 64

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases. 8, and so on. Here we go!

Kafka 55

Kafka Process Algorithm Cloud 55

Lyft Engineering

JUNE 28, 2023

The Event Driven Decisions capability in particular turned out to be general enough as to be applicable to a wide range of use cases. The interface was designed such that a minimal amount of metadata was needed to construct a pipeline object which performs a given capability. One key component is the Analytics Event Abstraction layer.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Big Data Tools

JUNE 8, 2022

DataHub 0.8.36 – Metadata management is a big and complicated topic. DataHub is a completely independent product by LinkedIn, and the folks there definitely know what metadata is and how important it is. If you haven’t found your perfect metadata management system just yet, maybe it’s time to try DataHub!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

DataHub 0.8.36 – Metadata management is a big and complicated topic. DataHub is a completely independent product by LinkedIn, and the folks there definitely know what metadata is and how important it is. If you haven’t found your perfect metadata management system just yet, maybe it’s time to try DataHub!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Netflix Tech

SEPTEMBER 29, 2022

Timestone clients can create queues, enqueue messages with user-defined deadlines and metadata, then dequeue these messages in an earliest-deadline-first (EDF) fashion. messages that belong to queue X and have metadata Y”) is also supported. Allow for dequeueing and queue depth querying using filters (metadata key-value pairs) 4.

Systems 85

Systems Metadata Media Kafka 85

Christophe Blefari

DECEMBER 9, 2022

It happens for instance when you have 2 producers of the same event but they are not using the same type for a column. Using server sent events to simplify real-time streaming at scale — Interesting discussion about concepts around real-time communication for apps. in Terraform) but for data.

SQL 130

SQL Data Data Science Metadata 130

Cloudera

MARCH 14, 2023

When a draft is ready to be deployed in production, it is published to the Catalog, and can be productionalized with serverless DataFlow Functions for event-driven, micro-bursty use cases or auto-scaling DataFlow Deployments for low latency, high throughput use cases. You want to see how to get data from Kafka and write it to Iceberg?

Data Pipeline 80

Data Pipeline Designing Kafka Metadata 80

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

LinkedIn Engineering

DECEMBER 19, 2023

The framework incorporates proactive measures to monitor critical events, provides deployment insights, and issues proper notifications to enable seamless deployments. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes.

Big Data 92

Big Data Hadoop Metadata Data 92

Cloudera

FEBRUARY 9, 2021

A new solution integrating cloud object storage, with Cloudera’s NiFi dataflows, a Kafka datahub, and a Hive virtual warehouse in the CDW service allows businesses to take the best advantage of this public cloud trend. In the real-time layer, Kafka automatically cleans the historical events with configurable retention.

Data Warehouse 67

Data Warehouse Cloud Kafka Cloud Storage 67

Netflix Tech

JUNE 7, 2021

Flow Collector consumes two data streams, the IP address change events from Sonar via Kafka and eBPF flow log data from the Flow Exporter sidecars. It performs real time attribution of flow data with application metadata from Sonar. Sonar is an IPv6 and IPv4 address identity tracking service.

Transportation 130

Transportation AWS Cloud Metadata 130

Data Engineering Podcast

AUGUST 6, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. manage versions of vectors, metadata management, etc.) manage versions of vectors, metadata management, etc.)

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Cloudera

SEPTEMBER 23, 2020

Access audits are mastered centrally in Apache Ranger which provides comprehensive non-repudiable audit log for every access event to every resource with rich access event metadata such as: IP. Cloudera’s platform can support piping of audit data to HDFS, Kafka, Syslog or to SIEM systems for long-term retention and archival.

Database 80

Database Data Lake Metadata Java 80

Zalando Engineering

NOVEMBER 29, 2017

See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series. Running Kafka Streams applications in AWS At Zalando, Europe’s leading online fashion platform, we use Apache Kafka for a wide variety of use cases. Our team at Zalando was an early adopter of the Kafka Streams API.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Cloudera

DECEMBER 9, 2022

For example, if your dataflow is reading events from a Kafka topic, which you want to filter and process but you’re not sure about the exact schema the events are in, you might want to peek at the events before writing your filter condition.

Designing 95

Designing Coding Google Cloud AWS 95

Data Engineering Podcast

OCTOBER 28, 2019

Upcoming events include the combined events of the Data Architecture Summit and Graphorum, the Data Orchestration Summit, and Data Council in NYC. Go to dataengineeringpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today.

Building 100

Building Data Pipeline Programming Language Metadata 100

Rockset

JULY 8, 2020

For this example, we will use Kafka. The service then pushes the geohash along with the coordinates to a Kafka topic. Rockset ingests data from this Kafka topic and updates it into a collection called locations. Rockset has a built-in connector to Apache Kafka. For this example, we will use DynamoDB.

Kafka 40

Kafka Designing Systems Food 40

Data Engineering Podcast

OCTOBER 14, 2019

Dataform is a platform that helps you apply engineering principles to your data transformations and table definitions, including unit testing SQL scripts, defining repeatable pipelines, and adding metadata to your warehouse to improve your team’s communication.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Cloudera

AUGUST 10, 2021

This integration mechanism does not provide a direct Atlas Hook or Atlas Bridge option to listen to the entity events in Ozone. If you want to continue experimenting with Ozone and Atlas, you can try writing to Ozone via Kafka using our documented configuration examples. Before we begin. ozone sh bucket list /data.

Hadoop 103

Hadoop Kafka Datasets Government 103

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Cloudera

AUGUST 21, 2020

For example, if a credit card was used in the United States and shortly afterward the same card was used in Spain to withdraw the same amount, these two events in isolation could appear legitimate. However, in the context of time and geography, these two events point to a pattern of fraud.

Banking 59

Banking Kafka Cloud Storage Government 59

AltexSoft

AUGUST 29, 2023

Such an object storage model allows metadata tagging and incorporating unique identifiers, streamlining data retrieval and enhancing performance. Apache Kafka and AWS Kinesis are popular tools for handling real-time data ingestion. Advanced metadata management. Watch our video explaining how data engineering works.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 90

Kafka Programming Language Metadata Architecture 90

Striim

JANUARY 4, 2023

On a technical level, data contracts handle schemas of entities and events. This means that your contract should include metadata about your schema, which you can use to describe your data and add value constraints for certain fields (e.g., Maybe, you first load data into a data warehouse and later go on to load data into a data lake.

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Cloudera

OCTOBER 28, 2020

These processors will be dedicated to specific services that you would want to target, e.g. HDFS, YARN, Apache Kafka, Hive, Impala, etc. We want to evaluate based on an event, not a group of events, so a processor to split incoming records is needed. If the first level stops unwanted services, this stage stops unwanted events.

Kafka 63

Kafka SQL Professional Services Consulting 63

Rock the JVM

JANUARY 21, 2023

Giannis is a proud alumnus of Rock the JVM, working as a Solutions Architect with a focus on Event Streaming and Stream Processing Systems. Introduction Apache Kafka is a well-known event streaming platform used in many organizations worldwide. Environment Setup First, we want to have a Kafka Cluster up and running.

Kafka 68

Kafka Java Scala Coding 68

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. So our solution was to start using an intentional contract: Events. What are Events? Events are facts about what happened within your service.

Kafka 52

Kafka ETL Tools BI Database 52

Confluent

MARCH 20, 2019

Back in May 2017, we laid out why we believe that Kafka Streams is better off without a concept of watermarks or triggers , and instead opts for a continuous refinement model. By continuous refinement , I mean that Kafka Streams emits new results whenever records are updated. Use case: Alerting. But how can you build alerts?

Kafka 106

Kafka Programming Process Designing 106

Scott Logic

APRIL 22, 2024

In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. I’ve coloured the data entities according to their types and we see there’s a few different patterns like events and state which we’ll discuss in a moment.

AWS 52

AWS Metadata Systems Kafka 52