Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 95

Kafka Technology IT Government 95

DoorDash Engineering

NOVEMBER 21, 2022

Instead, since the inventory levels are stored in specific CockroachDB tables, we decided to leverage CockroachDB’s change feed to send data changes to Kafka, which then starts Cadence workflows to accomplish whatever task needs to be done. We have set up one Kafka consumer per table (more details on how the consumer is set up below).

Data Process 57

Data Process Process Kafka Database 57

ProjectPro

FEBRUARY 8, 2023

Did you know thousands of businesses, including over 80% of the Fortune 100, use Apache Kafka to modernize their data strategies? Apache Kafka is the most widely used open-source stream-processing solution for gathering, processing, storing, and analyzing large amounts of data. What is Apache Kafka Used For?

Kafka 52

Kafka Banking Medical Healthcare 52

Afterpay Tech

SEPTEMBER 6, 2022

Photo by Leon S on Unsplash By: Jing Li Summary This article articulates the challenges, innovation and success of the Kafka implementation in Afterpay’s Global Payments Platform in the PCI zone. Context The asynchronous processing capability that Kafka offers opens up numerous innovation opportunities to interact with other services.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

ProjectPro

JANUARY 25, 2022

Features of PySpark Features that contribute to PySpark's immense popularity in the industry- Real-Time Computations PySpark emphasizes in-memory processing, which allows it to perform real-time computations on huge volumes of data. PySpark is used to process real-time data with Kafka and Streaming, and this exhibits low latency.

Big Data 52

Big Data Data Process Process Kafka 52

Data Engineering Weekly

MAY 16, 2023

Architectural Patterns for Data Quality Now we understand the trade-off between speed & correctness and the difference between data testing and observability. Let’s talk about the data processing types. Two-Phase WAP The Two-Phase WAP, as the name suggests, follows two copy processes.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Podcast

DECEMBER 31, 2018

What are the use cases for Pravega and how does it fit into the data ecosystem? How does it compare with systems such as Kafka and Pulsar for ingesting and persisting unbounded data? One of the compelling aspects of Pravega is the automatic sharding and resource allocation for variations in data patterns.

Lambda Architecture 100

Lambda Architecture Process Data Process Kafka 100

Netflix Tech

MARCH 10, 2023

After reading the asset ids using one of the ways, an event is created per asset id to be processed synchronously or asynchronously based on the use case. For asynchronous processing, events are sent to Apache Kafka topics to be processed. Apache Kafka topic is configured as a message broker.

Management 94

Management Kafka Metadata Media 94

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Data Engineering Weekly

APRIL 28, 2023

LinkedIn writes about its experience adopting Apache Beam’s approach, where Apache Beam follows unified pipeline abstraction that can run in any target data processing runtime such as Samza, Spark & Flink.

Consulting 52

Consulting Kafka Lambda Architecture Engineering 52

Precisely

DECEMBER 28, 2023

How do streaming data pipelines work? The first step in a streaming data pipeline is where information enters the pipeline. One very popular platform is Apache Kafka , a powerful open-source tool used by thousands of companies. But in all likelihood, Kafka doesn’t natively connect with the applications that contain your data.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

ProjectPro

JUNE 29, 2021

Your search for Apache Kafka interview questions ends right here! Let us now dive directly into the Apache Kafka interview questions and answers and help you get started with your Big Data interview preparation! How to study for Kafka interview? What is Kafka used for? What are main APIs of Kafka?

Kafka 40

Kafka Bytes Big Data Java 40

Monte Carlo

FEBRUARY 20, 2024

Let’s explore what to consider when thinking about data ingestion tools and explore the leading tools in the field. Consideration What to Look For Integration Capabilities Support for a diverse array of data sources and destinations, ensuring compatibility with your data ecosystem. Cost : It’s open-source software, so free.

Data Ingestion 52

Data Ingestion Google Cloud Kafka AWS 52

Data Engineering Weekly

DECEMBER 2, 2023

Many companies restrict themselves to batch processing with high latency to access the data to keep the design simple enough to manage. Previously, we even tried to query Kafka directly using Presto-Kafka Connector. Presto tried with RaptorX. It doesn’t fly. What is Next?

IT 85

IT BI AWS Kafka 85

Data Engineering Weekly

SEPTEMBER 24, 2023

Check out the agenda and save your spot to join Re: Govern on October 05 Pinterest: Last Mile Data Processing with Ray Pinterest writes about its assessment of the ML developer velocity bottlenecks and delves deeper into the adoption of Ray , the open-source framework to scale AI and machine learning workloads.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Data Engineering Weekly

FEBRUARY 25, 2024

My challenge with Samza during my time at Slack is the decision to co-locate Samza's state in Kafka. At that time, operating Kafka comes with its challenges. Samza’s stream-stream join relies on Kafka’s key partition to shift the streaming operation burden to Kafka.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Rockset

MARCH 18, 2024

That’s why we’re thrilled that Confluent Cloud is making it easier to use Flink, providing efficient and performant stream processing while saving engineers from complex infrastructure management. LLMs like ChatGPT are trained on vast amounts of text data available up to a cutoff date. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Rockset

AUGUST 21, 2019

Rockset continuously ingests data streams from Kafka, without the need for a fixed schema, and serves fast SQL queries on that data. We created the Kafka Connect Plugin for Rockset to export data from Kafka and send it to a collection of documents in Rockset. This blog covers how we implemented the plugin.

Kafka 40

Kafka IT Data Storage Relational Database 40

Zalando Engineering

NOVEMBER 29, 2017

See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series. Running Kafka Streams applications in AWS At Zalando, Europe’s leading online fashion platform, we use Apache Kafka for a wide variety of use cases. Our team at Zalando was an early adopter of the Kafka Streams API.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Christophe Blefari

DECEMBER 16, 2022

Query your data in Kafka using SQL — This is a post that compares Flink, ksqlDB, Trino, Materialize, RisingWave and timeplus (the authors) in order to query Kafka. Even if it's vendor oriented this is a good starting point to have an overview of what you can expect from these tools.

Kafka 130

Kafka Data SQL Cloud 130

Cloudera

SEPTEMBER 11, 2018

The open data processing pipeline. IoT is expected to generate a volume and variety of data greatly exceeding what is being experienced today, requiring modernization of information infrastructure to realize value. The Enterprise Data Hub. Telemetry data routed to the Cloudera Enterprise Data Hub flows into Apache Kafka.

Data Process 40

Data Process Process Building Machine Learning 40

Cloudera

OCTOBER 16, 2023

We use NiFi to ingest an airport route data set (JSON) and send that data to Kafka and Iceberg. Please reference user documentation for installation and configuration of Cloudera Data Platform Private Cloud Base 7.1.9 Stay tuned for part two, Data Processing with Apache Spark.

Cloud 80

Cloud Kafka SQL Data 80

Pinterest Engineering

NOVEMBER 28, 2023

Pinterest’s real-time metrics asynchronous data processing pipeline, powering Pinterest’s time series database Goku, stood at the crossroads of opportunity. The mission was clear: identify bottlenecks, innovate relentlessly, and propel our real-time analytics processing capabilities into an era of unparalleled efficiency.

Kafka 81

Kafka Bytes Architecture Utilities 81

Knowledge Hut

JUNE 26, 2023

If you want to break into the field of data engineering but don't yet have any expertise in the field, compiling a portfolio of data engineering projects may help. Data pipeline best practices should be shown in these initiatives. Source Code: Stock and Twitter Data Extraction Using Python, Kafka, and Spark 2.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Data Engineering Weekly

DECEMBER 3, 2023

The solution is as simple but highly effective as adopting incremental data processing and applying ownership and lining style conventions. link] Sophie Blee-Goldman: Kafka Streams and Rebalancing through the Ages Consumers come and go. The workspace evolution is an exciting read for incremental architectural improvements.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Weekly

JULY 30, 2023

😄🎢🚀 High Scalability: Lessons Learned Running Presto At Meta Scale Presto, potentially ranking as one of the most influential open-source initiatives of the past ten years, stands shoulder to shoulder with the likes of Apache Kafka. DuckDB brings an exciting data architecture challenge to the industry.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Christophe Blefari

JANUARY 14, 2023

On the data processing side there is Polars, a DataFrame library that could replace pandas. How to land a job in progressive data — If you want to use your skills to Do Good you have to look at Brittany's post about progressive data. Let's have a quick look at it. I did not read it yet but it looks great.

Python 130

Python Kafka Data Scala 130

Team Data Science

MAY 10, 2020

Kafka, while not in the top 5 most in demand skills, was still the most requested buffer technology requested which makes it worthwhile to include it. I'll use Python and Spark because they are the top 2 requested skills in Toronto.

Data Pipeline 130

Data Pipeline Data Ingestion AWS Kafka 130

Knowledge Hut

MAY 3, 2024

Cluster Computing: Efficient processing of data on Set of computers (Refer commodity hardware here) or distributed systems. It’s also called a Parallel Data processing Engine in a few definitions. Spark is utilized for Big data analytics and related processing. Happy Learning!!!

Scala 98

Scala Hadoop Healthcare Big Data 98

Cloudera

JULY 18, 2022

The streaming analytics process that we will implement in this blog aims to identify potentially fraudulent transactions by checking for transactions that happen at distant geographical locations within a short period of time. Flink is a “streaming first” modern distributed system for data processing. Registering catalogs.

Process 92

Process Kafka Scala SQL 92

Zalando Engineering

OCTOBER 9, 2017

With this selection of Kafka as the outbound event platform, it was also a natural selection for the inbound data processing. With this selection of Kafka as the outbound event platform, it was also a natural selection for the inbound data processing.

Kafka 40

Kafka PostgreSQL Database Data Ingestion 40

Knowledge Hut

APRIL 25, 2024

Big data is a term that refers to the massive volume of data that organizations generate every day. In the past, this data was too large and complex for traditional data processing tools to handle. There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Christophe Blefari

OCTOBER 9, 2023

Confluent announced Kafka roadmap and Flink as a Cloud service — this is the result of Confluent acquisition of Immerok. Ensuring data quality with Verity — Lyft definition of data quality and a tour of the in-house product to address data quality in the data platform: Verity.

Python 130

Python Data Database Java 130

ProjectPro

JUNE 28, 2016

Apache Kafka is breaking barriers and eliminating the slow batch processing method that is used by Hadoop. This is just one of the reasons why Apache Kafka was developed in LinkedIn. Kafka was mainly developed to make working with Hadoop easier. This data is constantly changing, and is voluminous.

Kafka 40

Kafka Systems Hadoop BI 40

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

Process 116

Process SQL Kafka Database 116

Data Engineering Weekly

SEPTEMBER 11, 2023

But the key point here is if we can shrink the incremental data processing that can fit into a single machine, the greater the cost efficiency of data infrastructure. We need to take these concepts but should rethink to fit the data model to take advantage of both the software and hardware advancements.

Data Engineering 62

Data Engineering Data Engineer Engineering Kafka 62

Data Engineering Weekly

FEBRUARY 12, 2023

But compute needs will likely not change much over time; most analysis is done over recent data. Historical data processing is a rare event, where 99% of the computing happens over the last 24 hours of data. A must-read for data engineering professionals. Pipelines for data in motion can quickly turn into DAG hell.

Data Engineering 52

Data Engineering Data Engineer Engineering Hadoop 52