Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Architecture 100

Architecture Metadata MongoDB MySQL 100

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. Sample of a high-level data architecture blueprint for Azure BI programs.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

JUNE 7, 2022

Companies can now capitalize on the value in all their data, by delivering a hybrid data platform for modern data architectures with data anywhere. Cloudera Data Platform (CDP) is designed to address the critical requirements for modern data architectures today and tomorrow.

IT 109

IT Unstructured Data Data Architecture Government 109

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Cloudera

MARCH 23, 2022

In fact, we recently announced the integration with our cloud ecosystem bringing the benefits of Iceberg to enterprises as they make their journey to the public cloud, and as they adopt more converged architectures like the Lakehouse. 1: Multi-function analytics . 3: Open Performance.

Metadata 89

Metadata Data Architecture BI Machine Learning 89

Databand.ai

JULY 19, 2023

This capability is useful for businesses, as it provides a clear and comprehensive view of their data’s history and transformations. Data lineage tools are not a new concept. In this article: Why Are Data Lineage Tools Important? It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Data Engineering Weekly

DECEMBER 10, 2023

Capital One: Insights on building a data strategy to drive business value One of the hotly debated and many companies struggling with is to build an agile data strategy to drive business value. There is no shortage of data, but data management is still hard, which requires skillful execution and alignment across the org.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Cloudera

OCTOBER 7, 2021

The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., need to integrate multiple “point solutions” used in a data ecosystem) and organization reasons (e.g., Components of a Data Mesh.

Architecture 123

Architecture Metadata Government Kafka 123

Ascend.io

APRIL 19, 2023

In this post, I’ll explore the origins of the modern data stack, discuss why its promised benefits have proven elusive, and advocate for a post-modern approach to data management that prioritizes productivity and value. Where did the modern data stack come from?

Metadata 52

Metadata Business Analyst Hadoop Software Engineer 52

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 85

Data Lake Business Intelligence Metadata Data Warehouse 85

Data Engineering Podcast

SEPTEMBER 23, 2018

He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. How do you define data curation? How does the size and maturity of a company affect the ways that they architect and interact with their data systems?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

The Modern Data Company

MARCH 1, 2024

For today’s Chief Data Officers (CDOs) and data teams, the struggle is real. We’re drowning in data yet thirsting for actionable insights. We need a new approach, a paradigm shift that delivers data with the agility and efficiency of a speedboat – enter Data Products.

Database-centric 40

Database-centric Data Lake Government Retail 40

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

SEPTEMBER 1, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Cloud AWS SQL 100

Ascend.io

FEBRUARY 28, 2024

Metadata Management for Enhanced Machine Interpretation Data engineering extends its importance to the realm of metadata management, providing AI models with essential context about data definitions, classifications, and governance policies.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Cloudera

OCTOBER 11, 2022

Different data types need different types of analytics – real-time, streaming, operational, data warehouses. As Mason said, all the data management, data analytics, and data science tools should easily work together and run against all this shared data. It should run on any cloud or on-prem.

Data Science 91

Data Science Aggregated Data Data Consulting 91

Towards Data Science

OCTOBER 25, 2023

Not too long ago, almost all data architectures and data team structures followed a centralized approach. As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. Your organization may be undergoing the decentralization of data.

Data Pipeline 82

Data Pipeline SQL Data Architecture Data 82

Monte Carlo

MAY 25, 2023

That’s why we’re excited to announce that uniting diverse data team personas to collectively ensure data quality just got easier, thanks to a new integration between Monte Carlo’s data observability platform and Atlan’s active metadata platform.

Metadata 52

Metadata Data Governance Government Business Intelligence 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. 2023 predictions from the panel are; Unified metadata becomes kingmaker.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Podcast

JULY 28, 2019

Rather than manually defining all of the mappings ahead of time, we can rely on the power of graph databases and some strategic metadata to allow connections to occur as the data becomes available. If you are struggling to maintain a tangle of data pipelines then you might find some new ideas for reducing your workload.

Data Integration 100

Data Integration Metadata Media Architecture 100

Data Engineering Podcast

AUGUST 5, 2019

Announcements Welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Cloudera

NOVEMBER 17, 2022

While Cloudera CDH was already a success story at HBL, in 2022, HBL identified the need to move its customer data centre environment from Cloudera’s CDH to Cloudera Data Platform (CDP) Private Cloud to accommodate growing volumes of data. See other customers’ success here .

Banking 83

Banking Management Data Lake Professional Services 83

Data Engineering Podcast

OCTOBER 28, 2019

In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming. If you are building dataflows then Dagster is definitely worth exploring.

Building 100

Building Data Pipeline Programming Language Metadata 100

Monte Carlo

DECEMBER 8, 2023

Enter: data provenance and data lineage. Data lineage is a visual tool that tracks the movement and transformations of data through various systems, processes, and applications. Data provenance is the record of metadata from data’s original sources, providing the historical context and authenticity of data.

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

DECEMBER 8, 2023

Enter: data provenance and data lineage. Data lineage is a visual tool that tracks the movement and transformations of data through various systems, processes, and applications. Data provenance is the record of metadata from data’s original sources, providing the historical context and authenticity of data.

Metadata 52

Metadata Data Data Warehouse Government 52

Data Engineering Podcast

OCTOBER 14, 2019

Dataform is a platform that helps you apply engineering principles to your data transformations and table definitions, including unit testing SQL scripts, defining repeatable pipelines, and adding metadata to your warehouse to improve your team’s communication. What do you have planned for the future of DataForm?

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Watch our video explaining how data engineering works.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

AltexSoft

NOVEMBER 10, 2021

Well, there’s a new phenomenon in data management that received the name of a data lakehouse. The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. Data warehouse.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Cloudera

JANUARY 5, 2023

In the past year, businesses who doubled down on digital transformation during the pandemic saw their efforts coming to fruition in the form of cost savings and more streamlined data management. These features provide businesses with a common metadata, security, and governance model across all their data.

Banking 52

Banking Machine Learning Insurance Data Architecture 52

DataKitchen

AUGUST 8, 2023

Data Observability leverages five critical technologies to create a data awareness AI engine: data profiling, active metadata analysis, machine learning, data monitoring, and data lineage. This is especially true regarding our one-to-many, producer-to-consumer relationships on our data architecture.

Data Pipeline 52

Data Pipeline Data Government Metadata 52

Towards Data Science

FEBRUARY 15, 2023

Data by itself has no value, it needs to be organized, standardized, and clean. In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture.

Data Lake 80

Data Lake Data Warehouse Hadoop Data Architecture 80

Knowledge Hut

APRIL 20, 2023

It involves establishing a framework for data management that ensures data quality, privacy, security, and compliance with regulatory requirements. The mix of people, procedures, technologies, and systems ensures that the data within a company is reliable, safe, and simple for employees to access.

Data Governance 59

Data Governance Government Data Cleanse Data Security 59

AltexSoft

MARCH 2, 2023

However, decentralized models may result in inconsistent and duplicate master data. There’s a centralized structure that provides a framework, which is then used by autonomous departments that own their data and metadata. Data management is the overall process of collecting, storing, organizing, maintaining, and using data.

Data Governance 52

Data Governance Government Healthcare Programming 52

Silectis

JANUARY 3, 2021

Big Query Google’s cloud data warehouse. Data Architecture Data architecture is a composition of models, rules, and standards for all data systems and interactions between them. Data Catalog An organized inventory of data assets relying on metadata to help with data management.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Snowflake

APRIL 5, 2023

You can see how Data Vault overcomes some limitations of the dimensional model below: Why Data Vault can be a better choice for CQR and management data warehousing In the CQR, data quality and accuracy are critical. Metadata in the Data Vault approach helps to track the origin and processing of data.

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Monte Carlo

APRIL 1, 2021

Data Catalogs Can Drown in a Data Lake Although exceptionally flexible and scalable, data lakes lack the organization necessary to facilitate proper metadata management and data governance. Data discovery tools and platforms can help. Image courtesy of Adrian on Unsplash.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52