article thumbnail

Data Engineering Annotated Monthly – September 2021

Big Data Tools

Zingg is a tool that integrates with Spark and tries to answer this question automatically, without the quadratic complexity of the task! Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 com | 2021-07-15T05:33:52+08:00 | + + + Which script is more readable?

article thumbnail

Data Engineering Annotated Monthly – September 2021

Big Data Tools

Zingg is a tool that integrates with Spark and tries to answer this question automatically, without the quadratic complexity of the task! Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 com | 2021-07-15T05:33:52+08:00 | + + + Which script is more readable?

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Data Engineering Annotated Monthly – July 2021

Big Data Tools

Rack-aware Kafka streams – Kafka has already been rack-aware for a while, which gives its users more confidence. When data is replicated between different racks housed in different locations, if anything bad happens to one rack, it won’t happen to another. Most of the topics, from data quality to DWH architecture, are hot!

article thumbnail

Data Engineering Annotated Monthly – July 2021

Big Data Tools

Rack-aware Kafka streams – Kafka has already been rack-aware for a while, which gives its users more confidence. When data is replicated between different racks housed in different locations, if anything bad happens to one rack, it won’t happen to another. Most of the topics, from data quality to DWH architecture, are hot!

article thumbnail

Data Engineering Annotated Monthly – November 2021

Big Data Tools

It’s developed by LinkedIn, which means it has very tight integrations with other LinkedIn tools, like Apache Kafka! This release brings 2 big features: Segment Merge and Rollup, both of which can be used for better (i.e. And, unlike Kafka, it doesn’t need ZooKeeper and it supports message scheduling! Apache Pinot 0.9.0

article thumbnail

Data Engineering Annotated Monthly – November 2021

Big Data Tools

It’s developed by LinkedIn, which means it has very tight integrations with other LinkedIn tools, like Apache Kafka! This release brings 2 big features: Segment Merge and Rollup, both of which can be used for better (i.e. And, unlike Kafka, it doesn’t need ZooKeeper and it supports message scheduling! Apache Pinot 0.9.0

article thumbnail

Data Engineering Annotated Monthly – October 2021

Big Data Tools

Future improvements Data engineering technologies are evolving every day. Kafka: Allow configuring num.network.threads per listener – Sometimes you find yourself in a situation with Kafka brokers where some listeners are less active than others (and are in some sense more equal than others).