Data Management, Data Pipeline and Project

Unpacking The Seven Principles Of Modern Data Pipelines

Data Engineering Podcast

AUGUST 13, 2023

Summary Data pipelines are the core of every data product, ML model, and business intelligence dashboard. The folks at Rivery distilled the seven principles of modern data pipelines that will help you stay out of trouble and be productive with your data. Closing Announcements Thank you for listening!

Data Pipeline

Data Pipeline BI SQL Business Intelligence

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Data Engineering Podcast

SEPTEMBER 11, 2022

Building reliable data pipelines is a complex and costly undertaking with many layered requirements. In order to reduce the amount of time and effort required to build pipelines that power critical insights Manish Jethani co-founded Hevo Data. Data stacks are becoming more and more complex.

Data Pipeline

Data Pipeline Building MongoDB Scala

Streaming Data Pipelines Made SQL With Decodable

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Can you describe what Decodable is and the story behind it? What do you have planned for the future of Decodable?

Data Pipeline

Data Pipeline SQL Data Warehouse Data Lake

Webinars

The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication

Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications

From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

MORE WEBINARS

Moving Machine Learning Into The Data Pipeline at Cherre

Data Engineering Podcast

APRIL 19, 2021

Summary Most of the time when you think about a data pipeline or ETL job what comes to mind is a purely mechanistic progression of functions that move data from point A to point B. Sometimes, however, one of those transformations is actually a full-fledged machine learning project in its own right.

Data Pipeline

Data Pipeline Machine Learning Data Warehouse Datasets

Making The Total Cost Of Ownership For External Data Manageable With Crux

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management

Data Management Management Metadata MongoDB

Ship Faster With An Opinionated Data Pipeline Framework

Data Engineering Podcast

SEPTEMBER 30, 2019

Summary Building an end-to-end data pipeline for your machine learning projects is a complex task, made more difficult by the variety of ways that you can structure it. In this episode Tom Goldenberg explains how it works, how it is being used at Quantum Black for customer projects, and how it can help you structure your own.

Data Pipeline

Data Pipeline Machine Learning Media Data

Strategies And Tactics For A Successful Master Data Management Implementation

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management

Data Management Management MongoDB Scala

Pay Down Technical Debt In Your Data Pipeline With Great Expectations

Data Engineering Podcast

JANUARY 26, 2020

Summary Data pipelines are complicated and business critical pieces of technical infrastructure. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. Closing Announcements Thank you for listening!

Data Pipeline

Data Pipeline PostgreSQL Media Data Validation

Audacy Shifts to Data Pipeline Automation to Revitalize Radio Advertising

Ascend.io

MARCH 2, 2023

“But we also had over 75 projects orchestrated with Airflow and a combination of tools, cloud services, and bespoke code. . “But we also had over 75 projects orchestrated with Airflow and a combination of tools, cloud services, and bespoke code. The system suffered frequent outages and regularly lost months of data.

Data Pipeline

Data Pipeline Media Portfolio Data Ingestion

Advice On Scaling Your Data Pipeline Alongside Your Business with Christian Heinzmann - Episode 61

Data Engineering Podcast

DECEMBER 16, 2018

As the organization grows and gains more customers, the requirements for that pipeline will change. In this episode Christian Heinzmann, Head of Data Warehousing at Grubhub, discusses the various requirements for data pipelines and how the overall system architecture evolves as more data is being processed.

Data Pipeline

Data Pipeline Data Lake Data Warehouse Python

Data Pipeline- Definition, Architecture, Examples, and Use Cases

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline

Data Pipeline Architecture Kafka AWS

Rebuilding Yelp's Data Pipeline with Justin Cunningham - Episode 5

Data Engineering Podcast

JUNE 17, 2017

Preamble Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Can you start by giving an overview of your pipeline and the type of workload that you are optimizing for?

Data Pipeline

Data Pipeline Kafka Business Intelligence Architecture

How to Ensure Data Integrity at Scale By Harnessing Data Pipelines

Ascend.io

APRIL 12, 2023

From this research, we developed a framework with a sequence of stages to implement data integrity quickly and measurably via data pipelines. Table of Contents Why does data integrity matter? At every level of a business, individuals must trust the data, so they can confidently make timely decisions. Let’s explore!

Data Pipeline

Data Pipeline Data Integration Datasets Data

Mastering the Art of ETL on AWS for Data Management

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. This is particularly useful for companies that need to process data in near-real-time.

AWS

AWS Data Management ETL Tools Management

How Data Engineering Teams Power Machine Learning With Feature Platforms

Data Engineering Podcast

JULY 2, 2023

In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features. How is this distinct from other forms of data pipeline development and delivery?

Machine Learning

Machine Learning Data Engineering Data Engineer Engineering

Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense

Data Engineering Podcast

OCTOBER 30, 2022

In this episode Amir Orad discusses the Sisense platform and how it facilitates the embedding of analytics and data insights in every aspect of organizational and end-user experiences. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Business Intelligence

Business Intelligence Metadata MongoDB MySQL

Build Data Products Without A Data Team Using AgileData

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building Metadata MongoDB MySQL

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database

Database MySQL Data Lake MongoDB

DataOps Framework: 4 Key Components and How to Implement Them

Databand.ai

AUGUST 30, 2023

The DataOps framework is a set of practices, processes, and technologies that enables organizations to improve the speed, accuracy, and reliability of their data management and analytics operations. This can be achieved through the use of automated data ingestion, transformation, and analysis tools.

Data Governance

Data Governance Data Pipeline Government Data Cleanse

Build Better Data Products By Creating Data, Not Consuming It

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building

Building IT Metadata MongoDB

The Data Discovery Team

Jesse Anderson

NOVEMBER 14, 2023

First of all, in data science, data discovery means finding patterns in data using database query languages to test hypotheses. This kind of data discovery can be subdivided into several steps, as e.g. suggested by Piethein Strengholt in Data Management at Scale.

Metadata

Metadata Data Science Big Data Data

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

MongoDB

MongoDB Scala MySQL Data Lake

Adopting Real-Time Data At Organizations Of Every Size

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake

Data Lake MongoDB MySQL Data Warehouse

Business Intelligence In The Palm Of Your Hand With Zing Data

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence

Business Intelligence Metadata BI MongoDB

15+ AWS Projects Ideas for Beginners to Practice in 2023

ProjectPro

JULY 23, 2021

AWS (Amazon Web Services) is the world’s leading and widely used cloud platform, with over 200 fully featured services available from data centers worldwide. This blog presents some of the most unique and innovative AWS projects from beginner to advanced levels. Applications of AWS Projects Why should you work on AWS Projects?

AWS

AWS Project Amazon Web Services Cloud Computing

Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building

Building Metadata MongoDB MySQL

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Ascend.io

JANUARY 10, 2024

By blending these elements, we lay a solid foundation, ensuring your AI projects don’t just start strong, but also deliver real, lasting value. AI models are only as good as the data they consume, making continuous data readiness crucial. Your data pipeline platform should excel in collecting data from a wide array of sources.

Data Pipeline

Data Pipeline Government Data Governance Raw Data

DataOps vs. MLOps: Similarities, Differences, and How to Choose

Databand.ai

JULY 17, 2023

DataOps , short for Data Operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data management processes. It aims to streamline the entire data lifecycle—from ingestion and preparation to analytics and reporting.

Data Pipeline

Data Pipeline Machine Learning High Quality Data BI

Effective Pandas Patterns For Data Engineering

Data Engineering Podcast

JANUARY 30, 2022

He recently wrote a book on effective patterns for Pandas code, and in this episode he shares advice on how to write efficient data processing routines that will scale with your data volumes, while being understandable and maintainable. The only thing worse than having bad data is not knowing that you have it.

Data Engineering

Data Engineering Data Engineer Engineering Python

Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks

Data Engineering Podcast

OCTOBER 16, 2022

In this episode Adrian Kosowski explains how the Pathway product got started, how its design simplifies the creation of data products that support supply chain operations, and how developers can help to build an ecosystem of applications that allow businesses to accelerate their time to insight. Closing Announcements Thank you for listening!

Database

Database Metadata MongoDB MySQL

Building And Managing Data Teams And Data Platforms In Large Organizations With Ashish Mrig

Data Engineering Podcast

JANUARY 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building

Building Management Data Pipeline Metadata

Gain Visibility And Insight Into Your Supply Chains Through Operational Analytics Powered By Roambee

Data Engineering Podcast

OCTOBER 2, 2022

In this episode Roambee CEO, Sanjay Sharma, shares the types of questions that companies are asking about their logistics, the technical work that they do to provide ways to answer those questions, and how they approach the challenge of data quality in its many forms. Can you describe what Roambee is and the story behind it?

Metadata

Metadata Electronics MongoDB MySQL

Understanding The Immune System With Data At ImmunAI

Data Engineering Podcast

FEBRUARY 20, 2022

He also explains how he has architected the systems that ingest, process, and distribute the data that he is responsible for and the requirements that are introduced when collaborating with researchers, domain experts, and machine learning developers. RudderStack’s smart customer data pipeline is warehouse-first.

Systems

Systems Software Engineer Software Engineering Data Warehouse

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

Data Engineering Podcast

OCTOBER 30, 2022

It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.

Engineering

Engineering MongoDB Scala MySQL

Self Service Open Source Data Integration With AirByte

Data Engineering Podcast

FEBRUARY 22, 2021

Summary Data integration is a critical piece of every data pipeline, yet it is still far from being a solved problem. There are a number of managed platforms available, but the list of options for an open source system that supports a large variety of sources and destinations is still embarrasingly short.

Data Integration

Data Integration Data Warehouse Data Pipeline BI

Charting A Path For Streaming Data To Fill Your Data Lake With Hudi

Data Engineering Podcast

AUGUST 3, 2021

Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge. By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Can you describe how Hudi is architected?

Data Lake

Data Lake Data Warehouse Hadoop Architecture

Stick All Of Your Systems And Data Together With SaaSGlue As Your Workflow Manager

Data Engineering Podcast

JULY 5, 2021

Summary At the core of every data pipeline is an workflow manager (or several). Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need.

Systems

Systems Management Data Warehouse Programming Language

Charting the Path of Riskified's Data Platform Journey

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data? Need to automate data pipelines with less red tape?

Metadata

Metadata MongoDB MySQL Data Pipeline

Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Unstructured Data

Unstructured Data Metadata MongoDB MySQL

Making The Open Data Lakehouse Affordable Without The Overhead At Iomete

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata

Metadata AWS MongoDB MySQL

Data Testing Tools: Key Capabilities and 6 Tools You Should Know

Databand.ai

AUGUST 30, 2023

Besides these categories, specialized solutions tailored specifically for particular domains or use cases also exist, such as ETL (Extract-Transform-Load) tools for managing data pipelines, data integration tools for combining information from disparate sources/systems, and more.

Data Cleanse

Data Cleanse Data Validation Data Pipeline Datasets

Build Your Own End To End Customer Data Platform With Rudderstack

Data Engineering Podcast

FEBRUARY 13, 2022

In this episode CEO and founder Soumyadeb Mitra explains how Rudderstack compares to the various other tools and platforms that share some overlap, how to set it up for your own data needs, and how it is architected to scale to meet demand. The only thing worse than having bad data is not knowing that you have it.

Building

Building Hadoop Data Pipeline Metadata

Scalable Strategies For Protecting Data Privacy In Your Shared Data Sets

Data Engineering Podcast

FEBRUARY 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline

Data Pipeline Data Metadata Software Engineer

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager.

Machine Learning

Machine Learning Data Engineering Data Engineer Cloud

Unpacking The Seven Principles Of Modern Data Pipelines

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Webinars

Trending Sources

Streaming Data Pipelines Made SQL With Decodable

Webinars

Moving Machine Learning Into The Data Pipeline at Cherre

Making The Total Cost Of Ownership For External Data Manageable With Crux

Ship Faster With An Opinionated Data Pipeline Framework

Strategies And Tactics For A Successful Master Data Management Implementation

Pay Down Technical Debt In Your Data Pipeline With Great Expectations

Audacy Shifts to Data Pipeline Automation to Revitalize Radio Advertising

Advice On Scaling Your Data Pipeline Alongside Your Business with Christian Heinzmann - Episode 61

Data Pipeline- Definition, Architecture, Examples, and Use Cases

Rebuilding Yelp's Data Pipeline with Justin Cunningham - Episode 5

How to Ensure Data Integrity at Scale By Harnessing Data Pipelines

Mastering the Art of ETL on AWS for Data Management

How Data Engineering Teams Power Machine Learning With Feature Platforms

Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense

Build Data Products Without A Data Team Using AgileData

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

DataOps Framework: 4 Key Components and How to Implement Them

Build Better Data Products By Creating Data, Not Consuming It

The Data Discovery Team

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Adopting Real-Time Data At Organizations Of Every Size

Business Intelligence In The Palm Of Your Hand With Zing Data

15+ AWS Projects Ideas for Beginners to Practice in 2023

Build A Common Understanding Of Your Data Reliability Rules With Soda Core and Soda Checks Language

AI Implementation: The Roadmap to Leveraging AI in Your Organization

DataOps vs. MLOps: Similarities, Differences, and How to Choose

Effective Pandas Patterns For Data Engineering

Speeding Up The Time To Insight For Supply Chains And Logistics With The Pathway Database That Thinks

Building And Managing Data Teams And Data Platforms In Large Organizations With Ashish Mrig

Gain Visibility And Insight Into Your Supply Chains Through Operational Analytics Powered By Roambee

Understanding The Immune System With Data At ImmunAI

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

Self Service Open Source Data Integration With AirByte

Charting A Path For Streaming Data To Fill Your Data Lake With Hudi

Stick All Of Your Systems And Data Together With SaaSGlue As Your Workflow Manager

Charting the Path of Riskified's Data Platform Journey

Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas

Making The Open Data Lakehouse Affordable Without The Overhead At Iomete

Data Testing Tools: Key Capabilities and 6 Tools You Should Know

Build Your Own End To End Customer Data Platform With Rudderstack

Scalable Strategies For Protecting Data Privacy In Your Shared Data Sets

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

Stay Connected