Data Pipeline, Designing and Metadata - Data Engineering Digest

Cloudera DataFlow Designer: The Key to Agile Data Pipeline Development

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities.

Data Pipeline

Data Pipeline Designing Kafka Metadata

Build your data pipelines like the Toyota Way

François Nguyen

FEBRUARY 28, 2021

Today, we are going to apply these principles to the data pipelines. The idea is to transpose these 7 principles to data pipeline knowing that Data pipelines are 100% flexible : if you have the skills, you build the pipeline you want. How does a bad data pipeline process look like ?

Data Pipeline

Data Pipeline Building Manufacturing BI

Data Pipeline Observability: A Model For Data Engineers

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability.

Data Pipeline

Data Pipeline Data Engineering Data Engineer Engineering

Webinars

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. How is the governance of DataHub being managed?

Metadata

Metadata BI Data Warehouse Government

How to Simplify Data Pipelines with DBT and Airflow?

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline

Data Pipeline Raw Data Data Database

What Is Data Pipeline Orchestration and Why You Need It

Ascend.io

NOVEMBER 28, 2023

The terms ‘data orchestration’ and ‘data pipeline orchestration’ are often used interchangeably, yet they diverge significantly in function and scope. Data orchestration refers to a wide collection of methods and tools that coordinate any and all types of data-related computing tasks.

Data Pipeline

Data Pipeline IT Data Metadata

1. Streamlining Membership Data Engineering at Netflix with Psyberg

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. This is mainly used to identify new changes since the last update.

Data Engineering

Data Engineering Data Engineer Engineering Metadata

Data Engineering Weekly #176

Data Engineering Weekly

JUNE 16, 2024

Experience Enterprise-Grade Apache Airflow Astro augments Airflow with enterprise-grade features to enhance productivity, meet scalability and availability demands across your data pipelines, and more. The official design document for liquid clustering is here. The more metadata there is, the more readability of the model.

Data Engineering

Data Engineering Data Engineer Engineering Kafka

Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Business Intelligence

Business Intelligence Metadata MongoDB MySQL

Build Data Products Without A Data Team Using AgileData

Data Engineering Podcast

NOVEMBER 13, 2022

With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping. In this episode he explains the design of the platform and how it builds on agile development principles to help you focus on delivering value. Atlan is the metadata hub for your data ecosystem.

Building

Building Metadata MongoDB MySQL

How to Ensure Data Integrity at Scale By Harnessing Data Pipelines

Ascend.io

APRIL 12, 2023

From this research, we developed a framework with a sequence of stages to implement data integrity quickly and measurably via data pipelines. Table of Contents Why does data integrity matter? At every level of a business, individuals must trust the data, so they can confidently make timely decisions. Let’s explore!

Data Pipeline

Data Pipeline Data Integration Datasets Data

Introducing Project Inception: The Next Evolution in Data Automation

Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project

Project Metadata Data Pipeline Data Engineering

What is Apache Airflow?

Marc Lamberti

SEPTEMBER 22, 2023

Stemming from this analogy, “you” is the orchestrator in data orchestration, and the recipe is the data pipeline. It was created in 2014 by Airbnb and has since been widely adopted by the data engineering community, primarily as it was the first orchestrator allowing to author data pipelines programmatically.

Data Pipeline

Data Pipeline Python Metadata Database

Making The Total Cost Of Ownership For External Data Manageable With Crux

Data Engineering Podcast

JULY 17, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Management

Data Management Management Metadata MongoDB

The Data Integration Solution Checklist: Top 10 Considerations

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. A notable capability that achieves this is the data catalog. If so, how do you combine that metadata with other data across the enterprise? #4.

Data Integration

Data Integration Metadata Amazon Web Services Data Governance

Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks

Data Engineering Podcast

OCTOBER 16, 2022

Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics. Atlan is the metadata hub for your data ecosystem. What are the types of data that Pathway is designed to work with?

Database

Database Metadata MongoDB MySQL

Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

Data Engineering Podcast

JUNE 19, 2022

In this episode Ryan explains how he and his team have designed their platform to bring everyone onto a level playing field and the benefits that it provides to the organization. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata

Metadata Unstructured Data MongoDB MySQL

Business Intelligence In The Palm Of Your Hand With Zing Data

Data Engineering Podcast

DECEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Business Intelligence

Business Intelligence Metadata BI MongoDB

Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language

Data Engineering Podcast

SEPTEMBER 25, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building

Building Metadata MongoDB MySQL

Data Reprocessing Pipeline in Asset Management Platform @Netflix

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management

Management Kafka Metadata Media

Charting the Path of Riskified's Data Platform Journey

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata

Metadata MongoDB MySQL Data Pipeline

Toward a Data Mesh (part 2) : Architecture & Technologies

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing. We want to have our hands free and be totally devoted to devops principles.

Technology

Technology Architecture Google Cloud Metadata

A Reflection On Data Observability As It Reaches Broader Adoption

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. RudderStack helps you build a customer data platform on your warehouse or data lake.

IT

IT Metadata MongoDB MySQL

The View From The Lakehouse Of Architectural Patterns For Your Data Platform

Data Engineering Podcast

JULY 3, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture

Architecture Metadata MongoDB MySQL

What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta

Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. The only thing worse than having bad data is not knowing that you have it. Atlan is the metadata hub for your data ecosystem.

IT

IT Metadata MongoDB MySQL

Building A System Of Record For Your Organization's Data Ecosystem At Metaphor

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data.

Systems

Systems Building Metadata Data Warehouse

The Symbiotic Relationship Between AI and Data Engineering

Ascend.io

FEBRUARY 28, 2024

Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing.

Data Engineering

Data Engineering Data Engineer Engineering Metadata

An Engineering Guide to Data Quality - A Data Contract Perspective - Part 2

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering

Engineering Kafka Data Pipeline Data Warehouse

Building And Managing Data Teams And Data Platforms In Large Organizations With Ashish Mrig

Data Engineering Podcast

JANUARY 23, 2022

Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup. In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company.

Building

Building Management Data Pipeline Metadata

A Data Mesh Implementation: Expediting Value Extraction from ERP/CRM Systems

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. This generalisation makes their data models complex and cryptic and require domain expertise. This means that we need to expose global identifiers so our data product might be universally used in other domains.

Systems

Systems Raw Data Metadata Data Cleanse

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

Data Engineering Podcast

MAY 22, 2022

What are some of the ways that that manifests in the design and feature set? How have the design and goals of the platform changed/evolved since you first started working on it? What are the changes in the data ecosystem that have had the most substantial impact on the Flyte project? (e.g.

Machine Learning

Machine Learning Data Engineering Data Engineer Cloud

How to learn data engineering

Christophe Blefari

JANUARY 20, 2024

Then here a list of global resources that can help you navigate through the field: The Data Engineer Roadmap — An image with advices and technology names to watch. Reddit r/dataengineering wiki a place where some data eng definitions are written. My advice on this point is to learn from others. Is it really modern?

Data Engineering

Data Engineering Data Engineer Engineering Hadoop

Understanding The Immune System With Data At ImmunAI

Data Engineering Podcast

FEBRUARY 20, 2022

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. You can observe your pipelines with built in metadata search and column level lineage.

Systems

Systems Software Engineer Software Engineering Data Warehouse

Data Lineage Tools: Key Capabilities and 5 Notable Solutions

Databand.ai

JULY 19, 2023

By visually mapping the data flow, businesses can quickly identify issues, assess the impact of changes, and ensure data quality and accuracy. This allows businesses to trace the lineage of data across multiple systems, providing a comprehensive view of their data landscape.

Pipeline-centric

Pipeline-centric Data Governance Metadata Government

Scalable Annotation Service?—?Marken

Netflix Tech

JANUARY 25, 2023

Scalable Annotation Service — Marken by Varun Sekhri , Meenakshi Jindal Introduction At Netflix, we have hundreds of micro services each with its own data models or entities. For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. In this case it is BOUNDING_BOX.

Algorithm

Algorithm Media Metadata Data Ingestion

Build Your Own End To End Customer Data Platform With Rudderstack

Data Engineering Podcast

FEBRUARY 13, 2022

Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? How does it work?

Building

Building Hadoop Data Pipeline Metadata

Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security

Data Security Metadata MongoDB Scala

Effective Pandas Patterns For Data Engineering

Data Engineering Podcast

JANUARY 30, 2022

Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? How does it work?

Data Engineering

Data Engineering Data Engineer Engineering Data Pipeline

Data Quality Monitoring Explained – You’re Doing It Wrong

Monte Carlo

APRIL 20, 2024

As your data environment grows increasingly complex and interconnected, the need to cover more bases, faster, becomes critical to achieving true data reliability. What is data quality monitoring? Data quality monitoring is the process of automatically monitoring the health of data pipelines and the data that runs through them.

IT

IT Metadata Data Warehouse Machine Learning

Scalable Strategies For Protecting Data Privacy In Your Shared Data Sets

Data Engineering Podcast

FEBRUARY 6, 2022

Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? How does it work?

Data Pipeline

Data Pipeline Data Metadata Software Engineer

Stick All Of Your Systems And Data Together With SaaSGlue As Your Workflow Manager

Data Engineering Podcast

JULY 5, 2021

Summary At the core of every data pipeline is an workflow manager (or several). Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need. How is the SaaSGlue platform architected?

Systems

Systems Management Data Warehouse Programming Language

Data governance beyond SDX: Adding third party assets to Apache Atlas

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. The SDX layer of CDP leverages the full spectrum of Atlas to automatically track and control all data assets. Assets: Files.

Data Governance

Data Governance Government Metadata Datasets

Beyond Garbage Collection: Tackling the Challenge of Orphaned Datasets

Ascend.io

MAY 23, 2023

The data engineering world is full of tips and tricks on how to handle specific patterns that recur with every data pipeline. Already in 2016, IBM estimated the cost of bad data to be over three trillion dollars, and that was before the chaos of data lakes emerged and orphaned datasets began to swamp the land.

Datasets

Datasets Data Pipeline Metadata Database

Real World Change Data Capture At Datacoral

Data Engineering Podcast

MARCH 22, 2021

Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.

Data Warehouse

Data Warehouse Metadata Data Lake Hadoop

Cloudera DataFlow Designer: The Key to Agile Data Pipeline Development

Build your data pipelines like the Toyota Way

Webinars

Trending Sources

Data Pipeline Observability: A Model For Data Engineers

Webinars

Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data

How to Simplify Data Pipelines with DBT and Airflow?

What Is Data Pipeline Orchestration and Why You Need It

1. Streamlining Membership Data Engineering at Netflix with Psyberg

Data Engineering Weekly #176

Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense

Build Data Products Without A Data Team Using AgileData

How to Ensure Data Integrity at Scale By Harnessing Data Pipelines

Introducing Project Inception: The Next Evolution in Data Automation

What is Apache Airflow?

Making The Total Cost Of Ownership For External Data Manageable With Crux

The Data Integration Solution Checklist: Top 10 Considerations

Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks

Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

Business Intelligence In The Palm Of Your Hand With Zing Data

Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language

Data Reprocessing Pipeline in Asset Management Platform @Netflix

Charting the Path of Riskified's Data Platform Journey

Toward a Data Mesh (part 2) : Architecture & Technologies

A Reflection On Data Observability As It Reaches Broader Adoption

The View From The Lakehouse Of Architectural Patterns For Your Data Platform

What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta

Building A System Of Record For Your Organization's Data Ecosystem At Metaphor

The Symbiotic Relationship Between AI and Data Engineering

An Engineering Guide to Data Quality - A Data Contract Perspective - Part 2

Building And Managing Data Teams And Data Platforms In Large Organizations With Ashish Mrig

A Data Mesh Implementation: Expediting Value Extraction from ERP/CRM Systems

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

How to learn data engineering

Understanding The Immune System With Data At ImmunAI

Data Lineage Tools: Key Capabilities and 5 Notable Solutions

Scalable Annotation Service?—?Marken

Build Your Own End To End Customer Data Platform With Rudderstack

Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault

Effective Pandas Patterns For Data Engineering

Data Quality Monitoring Explained – You’re Doing It Wrong

Scalable Strategies For Protecting Data Privacy In Your Shared Data Sets

Stick All Of Your Systems And Data Together With SaaSGlue As Your Workflow Manager

Data governance beyond SDX: Adding third party assets to Apache Atlas

Beyond Garbage Collection: Tackling the Challenge of Orphaned Datasets

Real World Change Data Capture At Datacoral

Stay Connected