Waitingforcode

MARCH 5, 2024

I'm writing this unexpected blog post because I got stuck with watermarks and checkpoints and felt that I was missing some basics. Even though this introduction is a bit negative, the exploration for the data reading enabled my other discoveries.

Data 130

Data 130

Waitingforcode

FEBRUARY 27, 2024

Apache Spark Structured Streaming relies on the micro-batch pattern which evaluates the same query in each execution. That's only a high level vision, though. Under-the-hood, there are many other interesting things that happen.

130

130

Waitingforcode

FEBRUARY 20, 2024

I bet you know it already. You can limit the max throughput for Apache Spark Structured Streaming jobs for popular data sources such as Apache Kafka, Delta Lake, or raw files. Have you known that you can also control the lower limit, at least for Apache Kafka?

Kafka 130

Kafka IT Data 130

Waitingforcode

FEBRUARY 13, 2024

It's time for another part of "What's new on the cloud for data engineers" Let's see what happened in the last 5 months.

Data Engineering 130

Data Engineering Data Engineer Cloud Engineering 130

Waitingforcode

FEBRUARY 6, 2024

The previous blog from the series we discovered streaming reader. However, an end-to-end streaming Delta Lake pipeline also requires a writer which will be our focus today.

130

130

Waitingforcode

JANUARY 30, 2024

Previously you could read about transformation of a user job definition into an executable stream graph. Since this explanation was relatively high-level, I decided to deep dive into the final step executing the code.

Coding 130

Coding 130

Waitingforcode

JANUARY 23, 2024

Data enrichment is one of common data engineering tasks. It's relatively easy to implement with static datasets because of the data availability. However, this apparently easy task can become a nightmare if used with inappropriate technologies.

Datasets 130

Datasets Data Engineering Data Engineer Technology 130

Waitingforcode

JANUARY 17, 2024

Even though I'm into streaming these days, I haven't really covered streaming in Delta Lake yet. I only slightly blogged about Change Data Feed but completely missed the fundamentals. Hopefully, this and next blog posts will change this!

Data 130

Data 130

Waitingforcode

JANUARY 9, 2024

It's technically possible to process files in a continuous way from a streaming job. However, if you are expecting some latency sensitive job, this will always be slower than processing data directly from a streaming broker. Why?

Process 130

Process IT Data 130

Waitingforcode

JANUARY 2, 2024

If you're interested in stream processing, I bet your thinking is technology-based. It's not wrong, after all, the ability to use a tool gives you and me a job. However, for a long-term consideration it's better to reason in terms of patterns or models. Being aware of a more general vision helps assimilate new tools.

Process 130

Process Technology IT 130

Waitingforcode

JANUARY 2, 2024

This is one of my favorite blog posts, the yearly retrospective. Every year I summarize what happened in the past 12 months and share with you my future plans. It's time for the 2023 Edition!

IT 130

IT 130

Waitingforcode

DECEMBER 26, 2023

I must admit it, if you want to catch my attention, you can use some keywords. One of them is "stream". Knowing that, the topic of my new blog post shouldn't surprise you.

IT 130

IT 130

Waitingforcode

DECEMBER 19, 2023

Even though nowadays data processing frameworks and data stores have smart query planners, they don't take our responsibility to correctly design the job logic.

Designing 130

Designing Data Process Process Data 130

Waitingforcode

DECEMBER 12, 2023

One week later than initially announced, but here it is, the second part for Data+AI Summit 2023 retrospective. I don't know how, but I managed to include some streaming-related talks here too!

Data 130

Data Management IT 130

Waitingforcode

DECEMBER 5, 2023

The "vertical scaling" has caught my attention a few times already when I have been reading about cloud updates. I've always considered horizontal scaling as the single true scaling policy for elastic data processing pipelines. Have I been wrong?

Data Process 130

Data Process Process Cloud Data 130

Waitingforcode

NOVEMBER 28, 2023

In March I wrote a blog showing how to use accumulators to know the application of each filter statement. Turns out, the solution may not be perfect as mentioned by Aravind in one of the comments. I bet you already have an idea but if not, keep reading. Everything will be clear in the end!

130

130

Waitingforcode

NOVEMBER 21, 2023

Even though you may be thinking now about Data+AI Summit 2024, I still owe you my retrospective for the 2023 edition. Let's start with the first part covering stream processing talks!

Data 130

Data Process 130

Waitingforcode

NOVEMBER 14, 2023

Have you written your first successful Apache Flink job and are still wondering the high-level API translates into the executable details? I did and decided to answer the question in the new blog post.

130

130

Waitingforcode

NOVEMBER 7, 2023

Checkpoints are a well-known fault-tolerance mechanism in stream processing. But what does it have to do with Delta Lake?

Process 130

Process IT 130

Waitingforcode

NOVEMBER 1, 2023

Watermark, or rather multiple watermarks management, has been a thorn in the side of Apache Spark Structured Streaming. It has improved in the previous release (3.4.0) but still had some room for improvement. Well, it did have because the 3.5.0 release brought a serious fix for the multiple watermarks scenario.

Management 130

Management IT 130

Waitingforcode

OCTOBER 24, 2023

It's time to start the series covering Apache Spark 3.5.0 features. As the first topic I'm going to cover Structured Streaming which has got a lot of RocksDB improvements and some major API changes.

IT 130

IT 130

Waitingforcode

OCTOBER 17, 2023

I've already written about watermarks in a few places in the blog but despite that, I still find things to refresh. One of them is the watermark used to filter out the late data, which will be the topic of this blog post.

Data 130

Data 130

Waitingforcode

OCTOBER 10, 2023

If you have some experience with RDBMS, who doesn't btw, you have probably run a VACUUM command to reclaim the storage space occupied by deleted or obsolete rows. If you're now working with Delta Lake, you can do the same!

130

130

Waitingforcode

OCTOBER 4, 2023

Do not get the title wrong! Having applyInPandasWithState in the PySpark API is huge! However, due to Python duck typing, some operations are more difficult and more risky to express in the code than in the strongly typed Scala API.

Scala 147

Scala Python Coding 147

Waitingforcode

SEPTEMBER 27, 2023

It's always a huge pleasure to see the PySpark API covering more and more Scala API features. Starting from Apache Spark 3.4.0 you can even write arbitrary stateful processing jobs! But since the API is a little bit different than the one available on the Scala side, I wanted to take a deeper look.

Process 147

Process Scala IT 147

Waitingforcode

SEPTEMBER 20, 2023

It's time for another part of "What's new on the cloud for data engineers" Let's see what happened in the last 4 months.

Data Engineering 130

Data Engineering Data Engineer Cloud Engineering 130

Waitingforcode

SEPTEMBER 14, 2023

I won't hide it, I'm still a fresher in the Apache Flink world and despite my past streaming experiences with Apache Spark Structured Streaming and GCP Dataflow, I need to learn. And to learn a new tool or concept, there is nothing better than watching some conference talks!

IT 130

IT 130

Waitingforcode

SEPTEMBER 6, 2023

In our social media and marketing-driven era, it's quite hard to get things right. For me there is one common misconception brought by the Modern Data Stack idea that everything should be now ELT. In fact no, it shouldn't but only can.

Media 228

Media IT Data 228

Waitingforcode

AUGUST 29, 2023

If Delta Lake implemented the commits only, I could stop exploring this transactional part after the previous article. But as for RDBMS, Delta Lake implements other ACID-related concepts. One of these are isolation levels.

130

130

Waitingforcode

AUGUST 22, 2023

One of the great features of modern table file formats is the ability to handle write conflicts. It wouldn't be possible without commits that are the topic of this new blog post.

IT 130

IT 130

Waitingforcode

AUGUST 17, 2023

Lessons learned why it's always worth checking the code implementation to avoid surprises later. Even for vendor-supported solutions.

Coding 130

Coding IT 130

Waitingforcode

AUGUST 10, 2023

There are probably not that many people working today on the flat files with Structured Streaming than 5 years ago thanks to the table file formats. However, if you are in this group and are still generating CSVs or JSONs with the streaming sink, brace yourself, the memory problems are coming if you don't take action!

130

130

Waitingforcode

AUGUST 2, 2023

When you wrote your first arbitrary stateful processing pipelines, the state expiration is maybe the first tricky point you had to deal with. Why is that? After all, it's just about setting the timeout, doesn't it? Most of the time, yes, but there is an exception.

Process 130

Process IT 130

Waitingforcode

JULY 25, 2023

You certainly know it, the watermark (aka GC Watermark) is responsible for cleaning state store in Apache Spark Structured Streaming. But you may not know that it's not the single time-based condition. There is a different one involved in the stream-to-stream joins.

IT 130

IT 130

Waitingforcode

JULY 21, 2023

Starting from Apache Spark 3.2.0 is now possible to load an initial state of the arbitrary stateful pipelines. Even though the feature is easy to implement, it hides some interesting implementation details!

IT 130

IT 130

Waitingforcode

JULY 13, 2023

That's the conference I've heard only recently about. What a huge mistake! Despite the lack of "data" word in the name, it covers many interesting data topics and before I share with you my notes from this year's Data+AI Summit, let me do the same for Berlin Buzzwords!

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Waitingforcode

JULY 6, 2023

That's often a dilemma, whether we should put multiple sinks working on the same data source in the same or in different Apache Spark Structured Streaming applications? Both solutions may be valid depending on your use case but let's focus here on the former one including multiple sinks together.

Data 130

Data 130