Snowflake

MARCH 21, 2023

In February, Snowflake launched new features around streaming data ingestion and data governance and improved SQL experience and performance, with enhancements to Search Optimization Service and more. This can help improve performance of queries on such tables. Read on to learn about everything new announced in February.

Retail 72

Retail Healthcare Data Ingestion Consulting 72

LinkedIn Engineering

JANUARY 25, 2023

These subscription renewal payments used to go through a rule-based routing engine to selected payment gateways, which often resulted in a less-than-optimal experience. The highlight for the Adyen model is the epsilon-greedy implementation and for Dropbox, the model is to optimize for the best time to process a payment.

Banking 97

Banking Machine Learning Datasets Engineering 97

Netflix Tech

MAY 4, 2023

These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 135

Utilities Systems Architecture Coding 135

Data Engineering Weekly

AUGUST 20, 2023

Learn from today's top data leaders and architects at The Data Observability Summit on how to build more trustworthy and reliable data & AI products with the latest technologies, processes, and strategies shaping our industry (yes, LLMs will be on the table). 📣 Exciting news! 🎤 And guess what?

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Monte Carlo

APRIL 24, 2023

By accommodating various data types, reducing preprocessing overhead, and offering scalability, data lakes have become an essential component of modern data platforms , particularly those serving streaming or machine learning use cases. Databricks Data Catalog and AWS Lake Formation are examples in this vein.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Cloudera

FEBRUARY 22, 2022

The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg. Apache Iceberg is a new open table format targeted for petabyte-scale analytic datasets. Key Design Goals .

Metadata 109

Metadata Datasets BI SQL 109

Monte Carlo

NOVEMBER 14, 2023

Does it feel colder in here or is it all this Apache Iceberg talk? Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. Table of Contents What is Apache Iceberg? Image courtesy of Dremio.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Cloudera

JULY 13, 2023

Iceberg is an emerging open-table format designed for large analytic workloads. The Apache Iceberg project continues developing an implementation of Iceberg specification in the form of Java Library. Different query engines such as Impala, Hive, and Spark can immediately benefit from using Apache Iceberg Java Library.

Metadata 72

Metadata Java PostgreSQL Data Warehouse 72

Snowflake

AUGUST 31, 2023

Apache Iceberg continues to grow in popularity as the industry standard for open table formats. With your invaluable feedback, we have made significant progress in improving our Iceberg offering, now available in private preview. You can easily configure your Iceberg catalog to match the capabilities you need.

Metadata 85

Metadata AWS Data Lake Datasets 85

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore).

Big Data 78

Big Data Data Management Management Metadata 78

Snowflake

NOVEMBER 2, 2023

Beyond that, many customers with a company-wide repository of tables highly optimized for SQL, and highly concurrent business intelligence workloads and reporting have built a data warehouse on Snowflake. Each of these architectures has its own unique strengths and tradeoffs.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Netflix Tech

NOVEMBER 20, 2023

We will show how we are building a clean and efficient incremental processing solution (IPS) by using Netflix Maestro and Apache Iceberg. Data Freshness: Large datasets from Iceberg tables needed to be processed quickly and accurately to generate insights to enable faster product decisions.

Process 84

Process Data Pipeline Datasets SQL 84

Monte Carlo

MAY 30, 2023

If you don’t know Apache Iceberg , you might find yourself skating on thin ice. Initially developed by Netflix and later donated to the Apache Software Foundation, Apache Iceberg is an open-source table format for large-scale distributed data sets.

Metadata 52

Metadata Raw Data Data Lake Data 52

Netflix Tech

JULY 26, 2021

With this batch style approach, several issues have surfaced like data movement is tightly coupled with database tables, database schema is not an exact mapping of business data model, and data being stale given it is not real time etc. tactical) in nature. We will deep dive into Data Delivery and Data Consumption in the following sections.

Data 101

Data Data Pipeline MySQL Data Warehouse 101

Netflix Tech

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. On the other hand, these optimizations themselves need to be sufficiently inexpensive to justify their own processing cost over the gains they bring.

Data Warehouse 81

Data Warehouse Metadata Algorithm Data 81

Monte Carlo

JUNE 27, 2023

We always say in order to have an AI strategy you have to have a data strategy….If If you don’t have your data strategy wired in, you can’t just keep going with the past. These were: Data : Expanded Iceberg table support. That didn’t stop conference attendees from locating, descending, and swarming Ceasar’s.

Data Lake 59

Data Lake Government SQL Unstructured Data 59

Data Engineering Podcast

SEPTEMBER 24, 2021

That has brought with it a number of challenges having to do with the speed and scalability of working with large volumes of information.There have been many projects and strategies for overcoming these challenges, each with their own set of tradeoffs. Summary Python has beome the de facto language for working with data. Stale dashboards?

Data Process 100

Data Process Python Process Data Lake 100

Netflix Tech

APRIL 26, 2022

The first version of our logger library optimized for storage by deduplicating facts and optimized for network i/o using different compression methods for each fact. Then we started hitting roadblocks while optimizing the query performance. Then we started hitting roadblocks while optimizing the query performance.

Metadata 75

Metadata Datasets Machine Learning Designing 75

Netflix Tech

MARCH 10, 2023

It consists of close to a hundred tables (column families) , the majority of which are reverse indices to help query the assets in a more optimized way. And finally, we have an Apache Iceberg layer which stores assets in a denormalized fashion to help answer heavy queries for analytics use cases. Net, Ruby, Perl etc.).

Management 103

Management Metadata Digital Media Kafka 103

Towards Data Science

JULY 24, 2023

Well, the connective tissue that powers and integrates these services is frequently found in the metadata of table formats (storage) and query/job logs (compute). While that may be a legitimate strategy for some, it is a rapidly dwindling minority. Read on to see where that might be. Take data governance for example.

Metadata 97

Metadata Data Warehouse Data Lake Data Science 97

AltexSoft

NOVEMBER 10, 2021

So, unlike data warehouses, the lakehouse system can store and process lots of varied data at a lower cost, and unlike data lakes, that data can be managed and optimized for SQL performance. In 1901, a woman by the name of Julia Davis Chandler published the recipe that changed the world for good. Data warehouse.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Airbnb Tech

JUNE 1, 2021

Because of this multi-year investment, when Airbnb’s business was severely disrupted by COVID-19 last year, we were able to quickly turn data into actionable insights and strategies. On the one hand, core_data standardized table consumption and allowed users to quickly identify which tables to build upon.

Datasets 52

Datasets Pipeline-centric Metadata Data Science 52

Confluent

SEPTEMBER 20, 2019

But deployment is just the tip of the iceberg. As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. Although the concept of managed services is fairly common in databases (e.g., BigQuery, Amazon Redshift, and MongoDB Atlas) and caches (e.g.,

Kafka 21

Kafka Management Cloud AWS 21

Data Engineering Weekly

SEPTEMBER 3, 2023

link] Kyle Weller: Delta, Hudi, Iceberg — A Benchmark Compilation The LakeHouse architecture brings the best of the database and the data lake into the data infrastructure. Following a benchmark result between Delta & Iceberg , Apache Hudi added its benchmark in the same repo. 📣 Exciting news! Which one to choose?

Data Engineering 73

Data Engineering Data Engineer Engineering Data 73

Data Engineering Weekly

JULY 9, 2023

The blog narrates the optimization and cluster architectural changes by ByteDance on top of ClickHouse. RazerPay writes about its high-level data platform architecture and optimization strategies to reduce platform expenses. See how it works today. The call for speakers will be open until the 21st of July. I liked the chapter.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Monte Carlo

JULY 18, 2023

Well, the connective tissue that powers and integrates these services is frequently found in the metadata of table formats (storage) and query/job logs (compute). While that may be a legitimate strategy for some, it is a rapidly dwindling minority. Why are these things related, and more importantly, why should data leaders care?

Metadata 52

Metadata Data Warehouse Data Lake Government 52

Monte Carlo

JULY 18, 2023

Well, the connective tissue that powers and integrates these services is frequently found in the metadata of table formats (storage) and query/job logs (compute). While that may be a legitimate strategy for some, it is a rapidly dwindling minority. Why are these things related, and more importantly, why should data leaders care?

Metadata 40

Metadata Data Warehouse Data Lake Government 40

Cloudera

APRIL 1, 2024

This recognition underscores Cloudera’s commitment to continuous customer innovation and validates our ability to foresee future data and AI trends, and our strategy in shaping the future of data management. Cloudera, a leader in big data analytics, provides a unified Data Platform for data management, AI, and analytics.

Cloud 106

Cloud Unstructured Data Metadata Datasets 106

Monte Carlo

JUNE 28, 2023

It turns out that media, vendors, and customers all took the same strategy of splitting up their teams to have a presence at both. We were greeted with conference pop/techno (our generation’s elevator music) and flashing neon lights. We are going to lead the revolution.

Data Warehouse 52

Data Warehouse Scala Unstructured Data Government 52

AltexSoft

MARCH 30, 2023

Unity catalog serves as a centralized metadata management and data governance layer for all Databricks data assets, including tables, files, dashboards, and machine learning models. In this article, we’ll highlight the reasoning behind this choice and the challenges related to it. Let’s see what exactly Databricks has to offer.

Scala 64

Scala Data Lake BI Google Cloud 64