Waitingforcode

MARCH 5, 2024

I'm writing this unexpected blog post because I got stuck with watermarks and checkpoints and felt that I was missing some basics. Even though this introduction is a bit negative, the exploration for the data reading enabled my other discoveries.

Data 130

Data 130

Confluent

MARCH 21, 2024

See the highlights from Kafka Summit London 2024, and learn about how data streaming platforms, Gen AI, and Apache Flink® are driving innovation in the data streaming community.

Kafka 70

Kafka Data 70

Waitingforcode

JANUARY 30, 2024

Previously you could read about transformation of a user job definition into an executable stream graph. Since this explanation was relatively high-level, I decided to deep dive into the final step executing the code.

Coding 130

Coding 130

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Kafka 145

Kafka Process Systems Engineering 145

Waitingforcode

JUNE 30, 2023

I've always been eager to learn, including 5 years ago when for the first time, I left my Apache Spark comfort zone to explore Apache Beam. You shouldn't. Since then I had a chance to write some Dataflow streaming pipelines to fully appreciate this technology and work on AWS, GCP, and Azure.

AWS 130

AWS Technology 130

Confluent

MARCH 19, 2024

Confluent Cloud's serverless Flink offering is now available on all major clouds, offering a unified, managed platform for real-time data processing.

Cloud 93

Cloud Data Process Process Management 93

Confluent

MARCH 18, 2024

Read the highlights from the Flink 1.19 release, including standard YAML support for configurations, dynamic source parallelism inference, and SQL and Table API improvements.

SQL 70

SQL 70

Jesse Anderson

JANUARY 9, 2023

Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly. I think Confluent will eventually reduce staff working on Kafka Streams to concentrate on Flink. It employed some of the top people in the Apache Flink project. Kafka Streams pales in comparison to Flink’s features.

Kafka 147

Kafka Coding Technology SQL 147

Knowledge Hut

APRIL 25, 2024

There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB. Apache Hadoop: It is one of the most popular big data technologies in 2024. Apache Spark: Spark is an open-source big data processing engine that can perform batch and real-time analytics on large data sets.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Pinterest Engineering

NOVEMBER 7, 2023

A central component of data ingestion infrastructure at Pinterest is our PubSub stack, and the Logging Platform team currently runs deployments of Apache Kafka and MemQ. Given that around 50% of Java clients at Pinterest are on Flink, PSC integration with Flink was key to achieving our platform goals of fully migrating Java clients to PSC.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

Confluent

DECEMBER 12, 2023

Dive into the serverless architecture of Confluent Cloud for Apache Flink and explore its benefits like reduced infrastructure costs, increased reliability, & seamless adoption.

Architecture 125

Architecture Cloud IT 125

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. This enables you to maximize utilization of streaming data at scale.

Process 113

Process SQL Kafka Database 113

Netflix Tech

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Confluent

AUGUST 29, 2023

Apache Flink can be used for multiple stream processing use cases. Learn how developers can use Flink to build real-time applications, run analytical workloads or build real-time pipelines.

Process 70

Process Kafka Building 70

Confluent

JANUARY 6, 2023

Introducing fully managed Apache Kafka® + Flink for the most robust, cloud-native data streaming platform with stream processing, integration, and streaming analytics in one.

Kafka 145

Kafka Cloud Process Management 145

Zalando Engineering

JULY 12, 2017

In this blog post we describe the generation of such events using Apache Flink, and share our experiences and lessons learned in the process. You can read more on why we have chosen Apache Flink over other stream processing frameworks here: Apache Showdown: Flink vs. Spark. This consumes a lot of memory.

Process 40

Process Kafka AWS Architecture 40

Christophe Blefari

FEBRUARY 23, 2024

— Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Spark future — I'm convinced that Apache Spark will have to transform itself if it is not to disappear (disappear in the sense of Hadoop, still present but niche). But for sure I'll add Arrow in the v2.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Start Data Engineering

SEPTEMBER 26, 2020

Table of Contents Table of Contents Introduction Project description and requirements Infrastructure overview Apache Flink Apache Kafka Design Detect fraudulent accounts Log account actions Prerequisites Code Defining dependencies Inheritance Server logs generator Defining data flow in Apache Flink Create a streaming environment Creating a consumer (..)

Data Engineering 130

Data Engineering Data Engineer Project Engineering 130

Data Engineering Podcast

FEBRUARY 19, 2023

Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses.

IT 147

IT Data Lake Metadata Data Warehouse 147

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. This was our main form of ingestion.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Your other engines that may be writing to the table, such as Apache Spark or Apache Flink, can continue to write, and Snowflake can read.

Building 85

Building Metadata Cloud Storage AWS 85

Cloudera

JUNE 7, 2021

We think that this is a good validation of our data-in-motion philosophy that a streaming architecture is made up of needs across data ingestion , messaging and analytics and in our case, this is powered by Apache NiFi, Apache Kafka and Apache Flink.

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Data Engineering Weekly

NOVEMBER 26, 2023

We saw back-to-back articles comparing the features and the performance of Delta Lake, Apache Hudi, and Iceberg. The companies backing Apache Hudi and Iceberg write articles about comparative ACID support in both the platforms here and here. One of the engines I’m excited to watch is an alternative to Apache Spark.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Christophe Blefari

APRIL 21, 2023

We announced this week the May Paris Apache Airflow meetup. Fast News ⚡️ Building a Flink self-serve platform on Kubernetes at scale — Instacart engineering team migrated from Flink on EMR to Flink on Kubernetes. When you look at the first edition we came a long way—lmao.

Raw Data 130

Raw Data Data Datasets SQL 130

Cloudera

JUNE 7, 2021

It enabled users to easily write, run and manage real-time SQL queries on streams from Apache Kafka with an exceptionally smooth user experience. . Flink SQL DDL and Catalog support. Flink SQL DDL Support. At the end of March, we released the first version of Cloudera SQL StreamBuilder as part of CSA 1.3. Feature Highlights.

Kafka 101

Kafka SQL Accessible Accessibility 101

Data Engineering Weekly

MARCH 24, 2024

As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service. The author writes excellent tips on optimizing Apache Hudi LakeHouse infrastructure by adopting indexes, partition pruning, handling incomplete versions, etc.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Jesse Anderson

DECEMBER 12, 2022

Doug Cutting took those papers and created Apache Hadoop in 2005. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. Apache Pig in 2008 came too, but it didn’t ever see as much adoption. Apache HBase came in 2007, and Apache Cassandra came in 2008. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Apache Flink.

Process 86

Process Kafka Scala SQL 86

Data Engineering Weekly

DECEMBER 25, 2023

Streaming Architectures and the Rise of Apache Flink Streaming architectures have gained significant traction, with Apache Flink leading the way. The support and innovation around Apache Flink suggest a continued focus on streaming architectures in the coming year.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Cloudera

MARCH 5, 2024

Cloudera is now the only provider to offer an open data lakehouse with Apache Iceberg for cloud and on-premises. This approach brings new compute engines into the fold, adding Spark, Flink, Impala, and NiFi, enabling concurrent access and processing of datasets within Iceberg.

Data Lake 87

Data Lake Government Data Storage Kafka 87

Confluent

SEPTEMBER 26, 2023

Confluent's Q3 '23 Launch brings intelligent, secure, cost-effective data pipelines with cloud-native Apache Flink, enterprise clusters powered by Kora engine, and more.

Data Pipeline 70

Data Pipeline Data Cloud Engineering 70

Netflix Tech

JANUARY 13, 2022

These steps launch jobs on clusters running Apache Spark and Presto via Genie. Streaming Pensive Streaming Pensive Architecture Apache Flink powers real-time stream processing jobs in the Netflix data platform. If a workflow step fails, Scheduler asks Pensive to diagnose the step’s error.

Kafka 95

Kafka Big Data Data Machine Learning 95

Data Engineering Weekly

APRIL 14, 2024

However, TIL about Apache Paimon, a lake format that enables building a real-time lakehouse architecture with Flink and Spark for both streaming and batch operations. The blog post brings in-depth insights into how Apache Hudi works on the right side, read side & underlying storage formats.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Cloudera

SEPTEMBER 26, 2023

As lakehouse architectures (including offerings from Cloudera and IBM) become the norm for data processing and building AI applications, a robust streaming service becomes a critical building block for modern data architectures. Apache Kafka has evolved into the most widely-used streaming platform, capable of ingesting and processing (..)

Kafka 90

Kafka Technology IT Government 90

Cloudera

OCTOBER 4, 2023

What we need is: An openness to support a wide range in streaming ingest sources, including NiFi, Spark Streaming, Flink, as well as APIs for languages like C++, Java, and Python. However, there are some key capabilities that will make real-time analytics a practical and applied reality. Kudu has this covered.

Telecommunication 85

Telecommunication Data Warehouse Java Manufacturing 85

Lyft Engineering

JUNE 28, 2023

The RealtimeMLPipline object constructed can be executed in a Jupyter notebook, a local test environment, and against staging and production Flink clusters. In a notebook and on a local test environment, the pipeline runs against an ad hoc Flink cluster and the output is written to the local file system for validation.

Machine Learning 115

Machine Learning Building Metadata Software Engineer 115

Cloudera

OCTOBER 16, 2023

Cloudera recently released a fully featured Open Data Lakehouse , powered by Apache Iceberg in the private cloud, in addition to what’s already been available for the Open Data Lakehouse in the public cloud since last year. Please note, you can also leverage Flink and SQL Stream Builder in CSA 1.11 Cloudera Flow Management 2.1.6

Cloud 76

Cloud Kafka SQL Data 76