Data Engineering Podcast

SEPTEMBER 1, 2021

Summary The Presto project has become the de facto option for building scalable open source analytics in SQL for the data lake. That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data. lets you identify data quality issues and their root causes from a single dashboard.

Data Lake 100

Data Lake Cloud AWS SQL 100

Cloudera

AUGUST 30, 2022

As such, managers at different agencies need to sort through multiple systems to make sure these documents are delivered correctly—even though they all apply to the same individuals.”. Data quality issues deter trust and hinder accurate analytics. Disparate systems create issues with transparency and compliance. Forrester ).

Data Architecture 78

Data Architecture Architecture Data Lake NoSQL 78

Monte Carlo

MAY 14, 2024

As organizations seek greater value from their data, data architectures are evolving to meet the demand — and table formats are no exception. At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Introduction to the Data Mesh Architecture and its Required Capabilities.

Architecture 124

Architecture Metadata Government Kafka 124

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Data Engineering Podcast

JUNE 17, 2023

Summary Architectural decisions are all based on certain constraints and a desire to optimize for different outcomes. In data systems one of the core architectural exercises is data modeling, which can have significant impacts on what is and is not possible for downstream use cases. ML, reverse ETL, etc.)

Data Lake 130

Data Lake Metadata Machine Learning Data Architecture 130

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

AltexSoft

AUGUST 22, 2022

In this post, we’ll attempt to explain the idea behind a data fabric, its architectural building blocks, the benefits it brings, and ways to approach its implementation. What is data fabric? to provide a unified view of all enterprise data. to provide a unified view of all enterprise data. Unified data access.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. 2-A truly open data lakehouse. Cloudera has long had the capabilities of a data lakehouse, if not the label. 4-Ready for modern data fabric architectures.

Database 90

Database Cloud Systems Management 90

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Rockset

APRIL 26, 2023

In this post, we’ll get to the details: how does one design an RTA system? We’ve noticed many common patterns across streaming data architectures and we’ll be sharing a blueprint for three of the most popular: anomaly detection, IoT, and recommendations. This could be a cloud data warehouse, a data lake, or a database.

Architecture 52

Architecture Transportation Data Lake Insurance 52

AltexSoft

SEPTEMBER 23, 2021

The larger the company, the more data it has to generate actionable insights. Because it is scattered across disparate systems, hardly available for analytical apps. Evidently, common storage solutions fail to provide a unified data view and meet the needs of companies for seamless data flow. What is Data Hub?

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

Data Engineering Podcast

MAY 22, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Stop struggling to speed up your data lake.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Monte Carlo

MAY 31, 2022

Monte Carlo’s data observability platform does an excellent job mapping table lineage (even field level lineage!) for SQL based transformations, but some of the most popular, Spark-based systems remained a blindspot for us and for the industry at large. Easy compared to Spark lineage? Absolutely. It’s a really elegant solution….here’s

Data Lake 52

Data Lake Building Scala Metadata 52

AltexSoft

JULY 19, 2022

In the last few decades, we’ve seen a lot of architectural approaches to building data pipelines , changing one another and promising better and easier ways of deriving insights from information. There have been relational databases, data warehouses, data lakes, and even a combination of the latter two.

Architecture 52

Architecture Data Lake Medical Datasets 52

Christophe Blefari

JANUARY 20, 2024

What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Still, we can have a common ground when mixing software engineering, DevOps principles, Cloud — or on-prem — systems understanding and data literacy. Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

ProjectPro

JANUARY 31, 2022

This blog walks you through what does Snowflake do , the various features it offers, the Snowflake architecture, and so much more. Table of Contents Snowflake Overview and Architecture What is Snowflake Data Warehouse? Snowflake is not based on existing database systems or big data software platforms like Hadoop.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 92

Data Lake Data Warehouse BI SQL 92

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 86

Data Lake Business Intelligence Metadata Data Warehouse 86

Cloudera

NOVEMBER 17, 2022

With Cloudera Data Platform, we aim to unlock value faster and offer consistent data security and governance to meet this goal. Aqeel Ahmed Jatoi, Lead – Architecture, Governance and Control, Habib Bank Limited. Future-proof data capabilities. ” Prior to the upgrade, HBL’s 27 node cluster ran on CDH 6.1

Banking 84

Banking Management Data Lake Professional Services 84

Snowflake

AUGUST 31, 2023

Because of its leading ecosystem of diverse adopters, contributors and commercial offerings, Iceberg helps prevent storage lock-in and eliminates the need to move or copy tables between different systems, which often translates to lower compute and storage costs for your overall data stack.

Metadata 85

Metadata AWS Data Lake Datasets 85

Cloudera

APRIL 22, 2021

This CVD is built using Cloudera Data Platform Private Cloud Base 7.1.5 on Cisco UCS S3260 M5 Rack Server with Apache Ozone as the distributed file system for CDP. This has been a major architectural enhancement on how Apache Ozone manages data at scale in a data lake. . Data Generation at Scale.

Pipeline-centric 100

Pipeline-centric Data Lake Hadoop Metadata 100

Ascend.io

APRIL 7, 2023

With the monolithic architectures most organizations have today, business users are stuck, constantly waiting for new data pipelines to be built or amended based on their requests. Data engineers aren’t huge fans of this paradigm either. What is a Data Mesh?

Metadata 52

Metadata Datasets Data Governance Government 52

Data Engineering Podcast

JUNE 17, 2021

Summary Working with unstructured data has typically been a motivation for a data lake. Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable. No more scripts, just SQL.

Unstructured Data 100

Unstructured Data Data Warehouse Metadata Media 100

Monte Carlo

JUNE 26, 2023

Over the past decade, Databricks and Apache Spark™ not only revolutionized how organizations store and process their data, but they also expanded what’s possible for data teams by operationalizing data lakes at an unprecedented scale across nearly infinite use cases. billion in 2020 to $17.6

Data Lake 52

Data Lake Metadata Bytes Google Cloud 52

ProjectPro

FEBRUARY 8, 2023

Let us dive deeper into this data integration solution by AWS and understand how and why big data professionals leverage it in their data engineering projects. It offers a simple and efficient solution for data processing in organizations. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog.

AWS 98

AWS Scala Metadata Data Lake 98

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Central to this transformation are two shifts. Let’s take a closer look.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. What is Delta Lake? The data became useless. The Lakehouse architecture was one of them.

Data Lake 77

Data Lake Data Warehouse Hadoop Data Architecture 77

Data Engineering Podcast

SEPTEMBER 23, 2018

Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. How do you define data curation? How does the size and maturity of a company affect the ways that they architect and interact with their data systems?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

JUNE 16, 2022

This blog aims to answer two questions as illustrated in the diagram below: How have stream processing requirements and use cases evolved as more organizations shift to “streaming first” architectures and attempt to build streaming analytics pipelines? The batch nature of these tools increased the overall latency of the analytics.

Kafka 86

Kafka Manufacturing Data Lake SQL 86