Cloudera

AUGUST 8, 2022

In June 2022, Cloudera announced the general availability of Apache Iceberg in the Cloudera Data Platform (CDP). Iceberg is a 100% open-table format, developed through the Apache Software Foundation , which helps users avoid vendor lock-in and implement an open lakehouse. . Loading data into Iceberg tables with CDE.

Data Warehouse 88

Data Warehouse BI Machine Learning SQL 88

Confluent

MARCH 19, 2024

Seamlessly integrate Apache Kafka data into your lakehouse as Apache Iceberg tables, bridging the operational and analytical divide, with Tableflow. Read more in our blog post.

Kafka 132

Kafka Data 132

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Are you not already using any Iceberg catalog? If so, then an Iceberg Table using SNOWFLAKE as the catalog source is ideal.

Building 85

Building Metadata Cloud Storage AWS 85

Cloudera

MARCH 5, 2024

Cloudera is now the only provider to offer an open data lakehouse with Apache Iceberg for cloud and on-premises. Iceberg delivers the open table format so that enterprises can put AI to work on their data all in an on-premises setting. Let’s dive deeper into three of the most impactful features included in this update.

Data Lake 87

Data Lake Government Data Storage Kafka 87

Christophe Blefari

SEPTEMBER 25, 2023

Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. Tabular is the company providing a cloud platform on top of Apache Iceberg—developed by Iceberg founders. The source code is on Github.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. Tabular is the company providing a cloud platform on top of Apache Iceberg—developed by Iceberg founders. The source code is on Github.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Cloudera

OCTOBER 16, 2023

Cloudera recently released a fully featured Open Data Lakehouse , powered by Apache Iceberg in the private cloud, in addition to what’s already been available for the Open Data Lakehouse in the public cloud since last year. This release signified Cloudera’s vision of Iceberg everywhere. to stream ingest data sets to Iceberg.

Cloud 76

Cloud Kafka SQL Data 76

Waitingforcode

FEBRUARY 3, 2023

After Delta Lake and Apache Iceberg it's time to see the reading part of Apache Hudi. Despite an apparent similarity with the aforementioned table formats, Apache Hudi has an interesting reading specificity related to the different table types.

IT 130

IT 130

Waitingforcode

APRIL 28, 2023

Things seem to change with the new open table file formats, like Delta Lake or Apache Iceberg. Data lakes have made the data-on-read schema popular. Let's try to understand that by analyzing their schema evolution parts.

Data Lake 130

Data Lake Data 130

Christophe Blefari

DECEMBER 31, 2023

At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. They still use technologies around JVM (Spark and Flink), but with no surprise everything resolves around Iceberg—which has been created at Netflix. And the weight are open-source as well.

Data 100

Data Cloud Storage Datasets Python 100

Netflix Tech

DECEMBER 14, 2023

Streaming SQL on Data Mesh using Apache Flink Mark Cho, Guil Pires and Sujay Jain, Engineers from the Netflix Data Platform talk about how a managed Streaming SQL using Apache Flink can help unlock new Stream Processing use cases at Netflix.

Data Engineering 103

Data Engineering Data Engineer Engineering Metadata 103

Cloudera

JULY 14, 2023

Therefore, Apache Iceberg table format is poised to replace the traditional Hive table format in the coming years. However, as there are already 25 million terabytes of data stored in the Hive table format, migrating existing tables in the Hive table format into the Iceberg table format is necessary for performance and cost.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Christophe Blefari

FEBRUARY 23, 2024

— Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. Spark future — I'm convinced that Apache Spark will have to transform itself if it is not to disappear (disappear in the sense of Hadoop, still present but niche). But for sure I'll add Arrow in the v2.

Data Lake 130

Data Lake PostgreSQL MongoDB Scala 130

Monte Carlo

DECEMBER 20, 2023

The Iceberg will rise (Apache Iceberg) 10. The Iceberg will rise (Apache Iceberg) Apache Iceberg is an open source data lakehouse table format developed by the data engineering team at Netflix to provide a faster and easier way to process large datasets at scale. Big data will get small 8.

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Monte Carlo

DECEMBER 20, 2023

The Iceberg will rise (Apache Iceberg) 10. The Iceberg will rise (Apache Iceberg) Apache Iceberg is an open source data lakehouse table format developed by the data engineering team at Netflix to provide a faster and easier way to process large datasets at scale. Big data will get small 8.

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

phData: Data Engineering

APRIL 4, 2023

Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole. Fivetran today announced support for Amazon Simple Storage Service (Amazon S3) with Apache Iceberg data lake format.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Cloudera

JUNE 18, 2022

An open lakehouse, and the birth of Apache Iceberg. Apache Iceberg was built from inception with the goal to be easily interoperable across multiple analytic engines and at a cloud-native scale. The cloud native table format was open sourced into Apache Iceberg by its creators. How are we embracing Iceberg?

Data Lake 89

Data Lake Data Warehouse BI SQL 89

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. However, you need to regularly maintain Iceberg tables to keep them in a healthy state so that read queries can perform faster.

Bytes 56

Bytes Metadata Data Lake SQL 56

Data Engineering Weekly

NOVEMBER 26, 2023

We saw back-to-back articles comparing the features and the performance of Delta Lake, Apache Hudi, and Iceberg. The companies backing Apache Hudi and Iceberg write articles about comparative ACID support in both the platforms here and here. One of the engines I’m excited to watch is an alternative to Apache Spark.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Cloudera

MARCH 17, 2023

Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID. They can be built on premises or as hybrid deployments leveraging private clouds, HDFS stores, or Apache Ozone. Also, the data lake layer is not limited to cloud object stores.

Education 60

Education Unstructured Data Data Lake Data Warehouse 60

Data Engineering Podcast

AUGUST 3, 2021

Your host is Tobias Macey and today I’m interviewing Vinoth Chandar about Apache Hudi, a data lake management layer for supporting fast and incremental updates to your tables. How does Hudi compare to projects like Iceberg, Delta Lake, Hive, etc.? How does Hudi compare to projects like Iceberg, Delta Lake, Hive, etc.?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

AUGUST 20, 2021

By building on top of Zeppelin, Spark, and Iceberg he and his team at Cuebook have built an autoscaled cloud native system that abstracts the underlying complexity. By building on top of Zeppelin, Spark, and Iceberg he and his team at Cuebook have built an autoscaled cloud native system that abstracts the underlying complexity.

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Weekly

MAY 16, 2023

The modern LakeHouse format, such as Apache Iceberg & Apache Hudi, support a Zero-Copy WAP pattern implementation. In the Iceberg case, it is a simple two-step config change.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

DECEMBER 21, 2021

We also introduced Apache Airflow on Kubernetes as the next generation orchestration service. A flexible orchestration tool that enables easier automation, dependency management, and customization — like Apache Airflow — is needed to meet the evolving needs of organizations large and small. Autoscaling speed and scale.

Data Engineering 116

Data Engineering Data Engineer Engineering Data Pipeline 116

Christophe Blefari

MARCH 4, 2023

A small head's up, the Apache Airflow team has announced the Airflow Summit for 2023 which will be held in Toronto in September. Tabular is a cloud offer using Apache Iceberg. AWS lambdas are still on Python 3.9 — Corey rant about AWS lambdas that are still using Python 3.9 You bring your own compute.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Data Engineering Weekly

DECEMBER 3, 2023

link] Netflix: Incremental Processing using Netflix Maestro and Apache Iceberg Netflix writes about its incremental processing design with its orchestration engine Maestro on top of Iceberg. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Podcast

OCTOBER 16, 2022

To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers Links Dremio Podcast Episode Dremio Sonar Dremio Arctic DML == Data Modification Language Spark Data Lake Trino Presto Dremio Data Reflections Tableau Delta Lake Podcast Episode Apache Impala Apache Arrow DuckDB Podcast Episode Google BigLake (..)

Data Lake 100

Data Lake Food MongoDB Scala 100

Cloudera

AUGUST 9, 2022

We recently announced the industry’s only open data lakehouse , based on Apache Iceberg. Apache Iceberg enables organizations to store very large analytical tables in a standards-based, open-source format, while being able to access them with a standard language.

Data Architecture 88

Data Architecture Architecture Cloud Computing Cloud 88

Netflix Tech

MARCH 10, 2023

We build the data pipeline to persist the assets data in the iceberg in parallel with cassandra and elasticsearch DB. But to build the data facts, we need the complete data set in the iceberg and not just the new. Hence the existing assets data was read and copied to the iceberg tables without any production downtime.

Management 94

Management Kafka Metadata Media 94

Cloudera

FEBRUARY 22, 2022

The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg. Apache Iceberg is a new open table format targeted for petabyte-scale analytic datasets. Key Design Goals .

Metadata 105

Metadata Datasets BI SQL 105

Data Engineering Podcast

JANUARY 6, 2019

How does it compare to the work being done on the Iceberg table format? How does it compare to the work being done on the Iceberg table format? Can you start by explaining what Kudu is and the motivation for building it? Can you start by explaining what Kudu is and the motivation for building it?

Data Analytics 100

Data Analytics Hadoop Kafka Media 100

Data Engineering Weekly

DECEMBER 21, 2022

In the article, Max writes about how the functional data engineering concepts drive the design principles of Apache Airflow. Let’s see how we can implement the principles in Lake House systems like Apache Hudi , Apache Iceberg & Delta Lake. How to Implement Functional Data Engineering?

Data Engineering 57

Data Engineering Data Engineer Engineering Hadoop 57

Netflix Tech

MARCH 7, 2024

Data: Fast Data Our main data lake is hosted on S3, organized as Apache Iceberg tables. For ETL and other heavy lifting of data, we mainly rely on Apache Spark. We use Apache Arrow to decode Parquet and to host an in-memory representation of data.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Netflix Tech

NOVEMBER 20, 2023

We will show how we are building a clean and efficient incremental processing solution (IPS) by using Netflix Maestro and Apache Iceberg. Data Freshness: Large datasets from Iceberg tables needed to be processed quickly and accurately to generate insights to enable faster product decisions.

Process 84

Process Data Pipeline Datasets SQL 84

Monte Carlo

NOVEMBER 14, 2023

Does it feel colder in here or is it all this Apache Iceberg talk? Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. Table of Contents What is Apache Iceberg? Image courtesy of Dremio.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Data Engineering Podcast

MAY 1, 2022

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Links Yotpo Data Platform Architecture Blog Post Greenplum Databricks Metorikku Apache Hive CDC == Change Data Capture Debezium Podcast Episode Apache Hudi Podcast Episode Upsolver Podcast Episode Spark PrestoDB Snowflake Podcast Episode Druid Rockset (..)

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

MAY 22, 2022

That’s why Kyligence has built on top of the leading open source OLAP engine for data lakes, Apache Kylin. Kyligence]([link] Kyligence was founded in 2016 by the original creators of Apache Kylin™, the leading open source OLAP for Big Data.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130