Waitingforcode

AUGUST 2, 2023

When you wrote your first arbitrary stateful processing pipelines, the state expiration is maybe the first tricky point you had to deal with. Why is that? After all, it's just about setting the timeout, doesn't it? Most of the time, yes, but there is an exception.

Process 130

Process IT 130

Waitingforcode

JULY 21, 2023

Starting from Apache Spark 3.2.0 is now possible to load an initial state of the arbitrary stateful pipelines. Even though the feature is easy to implement, it hides some interesting implementation details!

IT 130

IT 130

Waitingforcode

MAY 24, 2023

Apache Spark is infamous for its correctness issue for chained stateful operations. Fortunately things get improved in each release. The most recent one, the 3.4.0, also got some important changes on that field!

IT 130

IT 130

databricks

JUNE 28, 2023

In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance.

Project 95

Project Engineering 95

Waitingforcode

FEBRUARY 20, 2024

You can limit the max throughput for Apache Spark Structured Streaming jobs for popular data sources such as Apache Kafka, Delta Lake, or raw files. Have you known that you can also control the lower limit, at least for Apache Kafka? I bet you know it already.

Kafka 130

Kafka IT Data 130

Knowledge Hut

APRIL 25, 2024

If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Waitingforcode

FEBRUARY 27, 2024

Apache Spark Structured Streaming relies on the micro-batch pattern which evaluates the same query in each execution. That's only a high level vision, though. Under-the-hood, there are many other interesting things that happen.

130

130

Waitingforcode

SEPTEMBER 14, 2023

I won't hide it, I'm still a fresher in the Apache Flink world and despite my past streaming experiences with Apache Spark Structured Streaming and GCP Dataflow, I need to learn. And to learn a new tool or concept, there is nothing better than watching some conference talks!

IT 130

IT 130

Christophe Blefari

FEBRUARY 23, 2024

— Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Spark future — I'm convinced that Apache Spark will have to transform itself if it is not to disappear (disappear in the sense of Hadoop, still present but niche). Thank you Microsoft.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

databricks

FEBRUARY 28, 2024

This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this.

Data Engineering 88

Data Engineering Data Engineer Engineering Data 88

Waitingforcode

NOVEMBER 1, 2023

Watermark, or rather multiple watermarks management, has been a thorn in the side of Apache Spark Structured Streaming. It has improved in the previous release (3.4.0) but still had some room for improvement. Well, it did have because the 3.5.0 release brought a serious fix for the multiple watermarks scenario.

Management 130

Management IT 130

Waitingforcode

APRIL 15, 2023

journey with Apache Spark I've met the "checkpointing" world in the context of Structured Streaming mostly. But this term also applies to other modules including Apache Spark SQL, so batch processing! In my long - but not long enough!

SQL 130

SQL Process 130

Waitingforcode

MARCH 17, 2023

If you need to go back in time and analyze your past Apache Spark applications, you can use the native Apache Spark History server. However, it can also be an infrastructure problem because of the continuously increasing historical logs for streaming applications.

IT 130

IT 130

Waitingforcode

JUNE 30, 2023

I've always been eager to learn, including 5 years ago when for the first time, I left my Apache Spark comfort zone to explore Apache Beam. Since then I had a chance to write some Dataflow streaming pipelines to fully appreciate this technology and work on AWS, GCP, and Azure. You shouldn't.

AWS 130

AWS Technology 130

LinkedIn Engineering

OCTOBER 19, 2023

Authors: Bingfeng Xia and Xinyu Liu Background At LinkedIn, Apache Beam plays a pivotal role in stream processing infrastructures that process over 4 trillion events daily through more than 3,000 pipelines across multiple production data centers.

Process 119

Process Lambda Architecture Kafka Machine Learning 119

Waitingforcode

FEBRUARY 3, 2023

Apache Spark innovates in this field a lot, including new metrics for Structured Streaming and an important update added in the 3.0.0 Observability is a hot topic nowadays, not only for the data but also the software industry. release that I missed at the time, which are the observable metrics.

Data 130

Data 130

Jesse Anderson

JANUARY 9, 2023

I’ve always been vocal about ksqlDB’s and Kafka Stream’s limitations. The Future of ksqlDB and Kafka Streams With this announcement, the future of primarily ksqlDB and, to a lesser extent, Kafka Streams comes into view. Since Kafka Streams is part of the Apache project, I don’t see it going away as quickly.

Kafka 147

Kafka Coding Technology SQL 147

Netflix Tech

DECEMBER 14, 2023

Learn more about how batch and streaming data pipelines are built at Netflix. Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Marc Lamberti

SEPTEMBER 22, 2023

What is Apache Airflow? Now you know what an orchestrator is, we can move on to the question, “What is apache airflow?” ” What is Apache Airflow? Now you know what is Apache Airflow from a benefit Airflow Concepts Airflow has a few concepts to know. A task is an Operator in Apache Airflow.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

Jesse Anderson

DECEMBER 12, 2022

Doug Cutting took those papers and created Apache Hadoop in 2005. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. Apache Pig in 2008 came too, but it didn’t ever see as much adoption. Apache HBase came in 2007, and Apache Cassandra came in 2008. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Weekly

NOVEMBER 26, 2023

We’ve seen a fleet of tools like TextToSQL ; Slack bots to ask questions to your data warehouse , Chat interface for spreadsheets , and even the English SDK for Spark ! We saw back-to-back articles comparing the features and the performance of Delta Lake, Apache Hudi, and Iceberg. Let me know your thoughts in the comments.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Cloudera

OCTOBER 4, 2023

What we need is: An openness to support a wide range in streaming ingest sources, including NiFi, Spark Streaming, Flink, as well as APIs for languages like C++, Java, and Python. Kudu also has native support for C++, Java, and Python APIs for capturing data streams from applications and components based on those languages.

Telecommunication 85

Telecommunication Data Warehouse Java Manufacturing 85

Data Engineering Weekly

APRIL 14, 2024

[link] Alibaba: Building a Streaming Lakehouse: Performance Comparison Between Paimon and Hudi I’m not a big fan of these comparison studies since it all depends on the nature of the data and the business use cases. It's a good read that familiarizes you with the LakeHouse formats.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Cloudera

OCTOBER 16, 2023

Cloudera recently released a fully featured Open Data Lakehouse , powered by Apache Iceberg in the private cloud, in addition to what’s already been available for the Open Data Lakehouse in the public cloud since last year. to stream ingest data sets to Iceberg. to stream ingest data sets to Iceberg.

Cloud 76

Cloud Kafka SQL Data 76

Data Engineering Weekly

MARCH 24, 2024

link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. This enables you to maximize utilization of streaming data at scale.

Process 113

Process SQL Kafka Database 113

ProjectPro

FEBRUARY 16, 2023

Apache Spark is an open-source, distributed computing system for big data processing and analytics. Today, the Apache Spark project has over 1,000 contributors from over 250 companies worldwide. Thus, there is a tremendous demand for spark professionals in the coming years. Indeed recently posted nearly 2.4k

Big Data 52

Big Data Scala Machine Learning Hadoop 52

ProjectPro

FEBRUARY 21, 2023

With around 35k stars and over 26k forks on Github, Apache Spark is one of the most popular big data frameworks used by 22,760 companies worldwide. Apache Spark is the most efficient, scalable, and widely used in-memory data computation tool capable of performing batch-mode, real-time, and analytics operations.

Scala 52

Scala Programming Language Java Hadoop 52

Data Engineering Weekly

MARCH 3, 2024

These improvements, part of Project Lightspeed, aim to reduce latency and increase throughput for stateful streaming queries, making them more efficient and cost-effective. The article also highlights sink-specific improvements and operator-specific enhancements that contribute to the overall performance boost.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

FEBRUARY 20, 2022

Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. How does it work?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Precisely

DECEMBER 28, 2023

The concept of streaming data was born of necessity. That’s where streaming data pipelines come into play. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. What is a streaming data pipeline? How do streaming data pipelines work?

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Data Engineering Podcast

OCTOBER 16, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP.

Data Lake 100

Data Lake Food MongoDB Scala 100

Knowledge Hut

DECEMBER 26, 2023

Most Popular Programming Certifications C & C++ Certifications Oracle Certified Associate Java Programmer OCAJP Certified Associate in Python Programming (PCAP) MongoDB Certified Developer Associate Exam R Programming Certification Oracle MySQL Database Administration Training and Certification (CMDBA) CCA Spark and Hadoop Developer 1.

Certification 98

Certification Programming MongoDB R (Programming) 98

Data Engineering Weekly

AUGUST 6, 2023

My mental model for the dbt workload is more similar to Spark's RDD [It is one of the reasons why I am excited about dbt-duckdb integration]. The goal of the Spark RDD framework is to minimize the data shuffle, as the dbt can potentially minimize the number of database calls. Look no further than the Gartner latest report.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

MAY 16, 2023

I posted this LinkedIn post that sparked some exciting conversation. A streaming consumer, often implemented in stream processing frameworks like Flink or Spark, consumes the events from the fronting Kafka and runs through data contract validation. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

FEBRUARY 16, 2021

COD is an operational database-as-a-service that brings ease of use and flexibility to Apache HBase. Cloudera Shared Data Experience (SDX) Data Lake helps you configure and manage authorization and access control through the Apache Ranger user interface that ensures consistent policy administration for Apache HBase in COD.

Database 83

Database Machine Learning Data Lake Kafka 83

Towards Data Science

APRIL 3, 2023

Building a Mental Model for Engineers and Anyone in Between Stream Processing can be handled gently and with care, or wildly, and almost out of control! How you transition from a batch mindset to a streaming mindset although can also be tricky, so let’s start small and build. You be the judge of what future you’d rather embrace.

Process 87

Process Data Lake Systems Bytes 87