Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Data projects are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Want to see Starburst in action?

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. Data lakes are notoriously complex.

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 3, 2023

Summary The first step of data pipelines is to move the data to a place where you can process and prepare it for its eventual purpose. Data transfer systems are a critical component of data enablement, and building them to support large volumes of information is a complex endeavor. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 12, 2023

Tabnine is one of the main platforms offering an AI powered assistant for software engineers. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Data Engineering Podcast

NOVEMBER 5, 2023

When an application is slow, there is a good probability that the database needs some attention. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex. Data lakes are notoriously complex.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 10, 2023

Summary If your business metrics looked weird tomorrow, would you know about it first? Unfortunately, it can often be complex or expensive to incorporate anomaly detection into your data platform. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Cloudera

APRIL 1, 2024

We’re excited to share that Gartner has recognized Cloudera as a Visionary among all vendors evaluated in the 2023 Gartner® Magic Quadrant for Cloud Database Management Systems. Cloudera, a leader in big data analytics, provides a unified Data Platform for data management, AI, and analytics.

Cloud 103

Cloud Unstructured Data Metadata Datasets 103

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. But while the modern data stack , and how it’s structured, may be evolving, the need for reliable data is not — and that also has some real implications for your data platform.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Monte Carlo

NOVEMBER 14, 2023

Does it feel colder in here or is it all this Apache Iceberg talk? Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. Table of Contents What is Apache Iceberg? So, is Iceberg right for you?

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 99

Architecture Metadata Unstructured Data Machine Learning 99

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Dremio is one of the companies leading the development of products and services that support the open lakehouse.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Monte Carlo

DECEMBER 20, 2023

“The data and AI space moves fast. Wondering what’s next for the future of data engineering and GenAI? Each year, we chat with one of the data industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. Grab your crystal balls and let’s take a peek!

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Monte Carlo

DECEMBER 20, 2023

“The data and AI space moves fast. Wondering what’s next for the future of data engineering and GenAI? Each year, we chat with one of the data industry’s pioneering leaders about their predictions for the modern data stack – and share a few of our own. Grab your crystal balls and let’s take a peek!

Amazon Web Services 52

Amazon Web Services Metadata AWS Data 52

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Atlan is the metadata hub for your data ecosystem.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Cloudera

SEPTEMBER 12, 2022

It sounds straightforward: you just need data and the means to analyze it. The data is there, in spades. Data volumes have been growing for years and are predicted to reach 175 ZB by 2025. First, organizations have a tough time getting their arms around their data. Unified data fabric. Yes and no.

Unstructured Data 73

Unstructured Data Data Lake Data Architecture Data 73

Data Engineering Weekly

DECEMBER 2, 2023

AWS S3 Express One Zone sparks some delight in the data infrastructure. link] Amazon S3 Express One Zone is a high-performance, single-availability Zone storage class purpose-built to deliver consistent single-digit millisecond data access for your most frequently accessed data and latency-sensitive applications.

IT 85

IT BI AWS Kafka 85

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Cloudera

JULY 13, 2023

Iceberg is an emerging open-table format designed for large analytic workloads. The Apache Iceberg project continues developing an implementation of Iceberg specification in the form of Java Library. Different query engines such as Impala, Hive, and Spark can immediately benefit from using Apache Iceberg Java Library.

Metadata 70

Metadata Java PostgreSQL Data Warehouse 70

Cloudera

JULY 14, 2023

Introduction For more than a decade now, the Hive table format has been a ubiquitous presence in the big data ecosystem, managing petabytes of data with remarkable efficiency and scale. Some of the common issues include constrained schema evolution, static partitioning of data, and long planning time because of S3 directory listings.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Monte Carlo

JUNE 28, 2023

Like the omnipresent San Francisco drizzle, Data + AI keynote attendees steadily trickled down the escalators and toward the Moscone Center Hall B + C. There was quite a bit of speculation that attendance would suffer at both events of the major data conferences taking place this week, but that is clearly not the case.

Data Warehouse 52

Data Warehouse Scala Unstructured Data Government 52

Towards Data Science

JULY 24, 2023

While there is no right answer, there is likely a sweet spot for most organizations’ data platforms. Photo by Kelly Sikkema on Unsplash Data engineers discovered the benefits of conscious uncoupling around the same time as Gwyneth Paltrow and Chris Martin in 2014. Read on to see where that might be.

Metadata 97

Metadata Data Warehouse Data Lake Data Science 97

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

AltexSoft

MARCH 30, 2023

The answer is simple: They use the same technology to make the most of data. Along with thousands of other data-driven organizations from different industries, the above-mentioned leaders opted for Databrick to guide strategic business decisions. The relatively new storage architecture powering Databricks is called a data lakehouse.

Scala 64

Scala Data Lake BI Google Cloud 64

AltexSoft

NOVEMBER 10, 2021

Well, there’s a new phenomenon in data management that received the name of a data lakehouse. The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Monte Carlo

JULY 18, 2023

Data engineers discovered the benefits of conscious uncoupling around the same time as Gwyneth Paltrow and Chris Martin in 2014. A data engineering manager at a Fortune 500 company expressed the pain of on-prem limitations to me by saying: “Our analysts were unable to run the queries they wanted to run when they wanted to run them.

Metadata 52

Metadata Data Warehouse Data Lake Government 52

Monte Carlo

JULY 18, 2023

Data engineers discovered the benefits of conscious uncoupling around the same time as Gwyneth Paltrow and Chris Martin in 2014. A data engineering manager at a Fortune 500 company expressed the pain of on-prem limitations to me by saying: “Our analysts were unable to run the queries they wanted to run when they wanted to run them.

Metadata 40

Metadata Data Warehouse Data Lake Government 40