Confessions of a Data Guy

FEBRUARY 10, 2023

But we all get sloppy, sometimes we got the string and varchar route because we don’t spend enough time on the […] The post Data Types in Delta Lake + Spark. I mean we know the difference between boolean, string, and integers, those are easy to get right. Join and Storage Performance.

Data 238

Data Big Data Data Engineering Data Engineer 238

Confessions of a Data Guy

JANUARY 12, 2023

Anyone who’s been roaming around the forest of Data Engineering has probably run into many of the newish tools that have been growing rapidly around the concepts of Data Warehouses, Data Lakes, and Lake Houses … the merging of the old relational database functionality with TB and PB level cloud-based file storage systems.

Data Lake 162

Data Lake Relational Database Data Warehouse Data Engineering 162

Waitingforcode

NOVEMBER 7, 2023

But what does it have to do with Delta Lake? Checkpoints are a well-known fault-tolerance mechanism in stream processing.

Process 130

Process IT 130

Waitingforcode

FEBRUARY 3, 2023

In the previous blog post about Delta Lake you discovered the logic for the writing part. Meantime Delta Lake 2 was released and it's for this brand new version that I'm going to share with you some findings related to the data reading.

IT 130

IT Data 130

Confessions of a Data Guy

AUGUST 4, 2023

The post Introduction to Delta Lake appeared first on Confessions of a Data Guy.

Data 130

Data Data Engineering Data Engineer Engineering 130

Waitingforcode

APRIL 28, 2023

Data lakes have made the data-on-read schema popular. Things seem to change with the new open table file formats, like Delta Lake or Apache Iceberg. Let's try to understand that by analyzing their schema evolution parts.

Data Lake 130

Data Lake Data 130

Waitingforcode

MARCH 29, 2023

In my recent exploration of the compaction, aka OPTIMIZE command, in Delta Lake, I found this famous Z-Ordering mode. It was one of the most outstanding features when I first heard about Delta Lake. You can't even imagine how impatient I was to see what it is doing under-the-hood. Fortunately, this time has come!

IT 130

IT 130

Confessions of a Data Guy

JUNE 22, 2023

The post Conceptual Introduction to Delta Lake. appeared first on Confessions of a Data Guy.

Data 130

Data Big Data Data Engineering Data Engineer 130

Waitingforcode

FEBRUARY 3, 2023

As for the 3rd part, I'm going to start with Delta Lake. It's time to start the 4th part of the Table file formats series. This time the topic will be Change Data Capture, so how to stream all changes made on the table.

Data 147

Data IT 147

Waitingforcode

MARCH 27, 2024

Streaming Delta tables is slightly different from streaming native streaming sources, such as Apache Kafka topics. One of the significant differences is schema enforcement. It leads to the job failure in case of schema changes of the streamed table.

Kafka 130

Kafka IT 130

Waitingforcode

JANUARY 23, 2024

Data enrichment is one of common data engineering tasks. It's relatively easy to implement with static datasets because of the data availability. However, this apparently easy task can become a nightmare if used with inappropriate technologies.

Datasets 130

Datasets Data Engineering Data Engineer Technology 130

databricks

JUNE 28, 2023

We are excited to announce Delta Lake 3.0, the next major release of the Linux Foundation open source Delta Lake Project, available in.

Project 92

Project Engineering 92

Waitingforcode

AUGUST 22, 2023

One of the great features of modern table file formats is the ability to handle write conflicts. It wouldn't be possible without commits that are the topic of this new blog post.

IT 130

IT 130

Towards Data Science

FEBRUARY 7, 2023

In this article, we will try to understand how the output from Delta Lake change feed can be used to feed downstream applications Continue reading on Towards Data Science »

Data Management 77

Data Management Management Data Science Data 77

Waitingforcode

FEBRUARY 16, 2023

I like writing code and each time there is a data processing job to write with some business logic I'm very happy. However, with time I've learned to appreciate the Open Source contributions enhancing my daily work. Mack library, the topic of this blog post, is one of those projects discovered recently.

Coding 130

Coding Data Process Project Process 130

Simon Späti

SEPTEMBER 30, 2022

Image by Rachel Claire on Pexels Ever wanted or been asked to build an open-source Data Lake offloading data for analytics? This article explains the data lake power and which technologies can build one to avoid creating a Data Swamp with no structure and orphaned files. Asked yourself what components and features would that include.

Data Lake 130

Data Lake Data Warehouse Government Data 130

Simon Späti

AUGUST 25, 2022

Image by Rachel Claire on Pexels Ever wanted or been asked to build an open-source Data Lake offloading data for analytics? This article explains the data lake power and which technologies can build one to avoid creating a Data Swamp with no structure and orphaned files. Asked yourself what components and features would that include.

Data Lake 130

Data Lake Data Warehouse Government Data 130

Towards Data Science

NOVEMBER 8, 2023

How are different partitioning/clustering methods implemented in Delta? How do they work in practice? Continue reading on Towards Data Science »

Data Science 93

Data Science Data Data Programming Big Data 93

Waitingforcode

FEBRUARY 3, 2023

The small files is a well known problem in data systems. Fortunately, modern table file formats have built-in features to address it. In the next series we'll see how.

Systems 100

Systems IT Data 100

Monte Carlo

JUNE 28, 2022

To help organizations realize the full potential of their data lake and lakehouse investments, Monte Carlo, the data observability leader, is proud to announce integrations with Delta Lake and Databricks’ Unity Catalog for full data observability coverage. billion in 2020 to 17.60 billion in 2020 to 17.60

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

Scribd Technology

JULY 19, 2021

Delta Lake is integral to our data platform which is why we have invested heavily in delta-rs to support our non-JVM Delta Lake needs. This year I had the opportunity to share the progress of delta-rs at Data and AI Summit. Suffice it to say, there’s a lot of really interesting work to be done!

Python 40

Python Kafka Data Engineering Data Engineer 40

Waitingforcode

FEBRUARY 20, 2024

You can limit the max throughput for Apache Spark Structured Streaming jobs for popular data sources such as Apache Kafka, Delta Lake, or raw files. I bet you know it already. Have you known that you can also control the lower limit, at least for Apache Kafka?

Kafka 130

Kafka IT Data 130

RudderStack

JANUARY 27, 2022

Learn more about how to send event data from RudderStack to your Data Lake and Delta Lake destinations.

Data Lake 40

Data Lake Data 40

databricks

DECEMBER 5, 2023

Recent data show that the number of recall campaigns caused by product deficiencies keeps increasing, while each known recorded case is a multi-million.

Management 85

Management Manufacturing Data 85

Confessions of a Data Guy

MARCH 25, 2024

I built out lakescuman open-source package to help with Databricks Unity Catalog Delta Lake tables querying with Polars, DuckDB, or PyArrow. Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI?

Python 100

Python Building Project Data 100

databricks

MAY 9, 2023

As organizations are maturing their data infrastructure and accumulating more data than ever before in their data lakes, Open and Reliable table formats.

Data Lake 91

Data Lake Data Data Engineering Data Engineer 91

databricks

JUNE 6, 2023

Apache Parquet is one of the most popular open source file formats in the big data world today. Being column-oriented, Apache Parquet allows.

Data Lake 74

Data Lake Big Data Data Data Engineering 74

Towards Data Science

JANUARY 17, 2024

Parquet vs ORC vs Avro vs Delta Lake Photo by Viktor Talashuk on Unsplash The big data world is full of various storage systems, heavily influenced by different file formats. You’ll explore four widely used file formats: Parquet , ORC , Avro , and Delta Lake. These will be used for Parquet, Avro, ORC, and Delta Lake.

Big Data 72

Big Data Data Data Storage SQL 72

Waitingforcode

FEBRUARY 9, 2023

However, it works a little bit differently than for Delta Lake presented last time. Compaction is also a feature present in Apache Iceberg. Let's see in this new blog post!

IT 130

IT 130

Waitingforcode

FEBRUARY 3, 2023

After Delta Lake and Apache Iceberg it's time to see the reading part of Apache Hudi. Despite an apparent similarity with the aforementioned table formats, Apache Hudi has an interesting reading specificity related to the different table types.

IT 130

IT 130

Waitingforcode

APRIL 7, 2023

Last time you discovered the Z-Order compaction in Delta Lake. But guess what? Apache Iceberg also has this feature!

130

130

Waitingforcode

FEBRUARY 3, 2023

Last week you could read about data reading in Delta Lake. Today it's time to cover this part in Apache Iceberg!

IT 130

IT Data 130

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Monte Carlo

MARCH 4, 2024

So let’s dive into a quick high-level overview of how data observability works and then dive into some of the nuances of Iceberg and Delta tables as well as Delta Live Tables and Snowflake Dynamic Tables. Previously a data lake would need to be accompanied by a metastore like Hive to enable many of Monte Carlo’s monitors.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. Your host is Tobias Macey and today I’m interviewing Vinoth Chandar about Apache Hudi, a data lake management layer for supporting fast and incremental updates to your tables.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 130

Data Engineering Data Engineer Engineering Project 130

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes. Delta Lake added transaction support to the data in a lake.

Data Lake 89

Data Lake Data Warehouse BI SQL 89