Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

JULY 3, 2022

In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. Can you describe what the data diff tool is and the story behind it?

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Data lineage and metadata systems are a hot topic right now. What are the core problems that Manta aims to solve?

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. Can you describe what Sust Global is and the story behind it?

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Knowledge Hut

DECEMBER 28, 2023

Today, organizations seek skilled professionals who can harness data’s power to drive informed decisions. As technology evolves, cloud platforms have emerged as the cornerstone of modern data management. Its comprehensive suite of services can handle data at scale. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. Data lakes are notoriously complex.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JANUARY 1, 2022

In this episode Emily Riederer shares her work to create a controlled vocabulary for managing the semantic elements of the data managed by her team and encoding it in the schema definitions in her data warehouse. star/snowflake schema, data vault, etc.) (e.g. star/snowflake schema, data vault, etc.)

Data Warehouse 100

Data Warehouse BI Data Workflow Coding 100

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP). Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. Enhanced Data Quality Data quality is a critical factor in the success of data analytics projects.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Can you describe what your conception of a data contract is?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 27, 2021

If you are having trouble answering questions for your business with the data that you generate and collect internally, then it is definitely worthwhile to explore the information available from external sources. Start trusting your data with Monte Carlo today! No more scripts, just SQL. Who are your target customers?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

NOVEMBER 14, 2021

In this episode Michael Harper advocates for proactive data quality and starting with the source, rather than being reactive and having to work backwards from when a problem is found. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Podcast

JUNE 1, 2021

This is an interesting conversation about how to make data more accessible and more useful by improving the user experience of the tools that we create. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

Data Pipeline 100

Data Pipeline Data Warehouse Data Workflow Data 100

Data Engineering Podcast

OCTOBER 29, 2021

In this episode Ahmed Elsamadisi shares his journey to find a more flexible and universal data model in the form of the "activity schema" that is powering the Narrator platform, and how it has allowed his customers to perform self-service exploration of their business domains without being blocked by schema evolution in the data warehouse.

Data Warehouse 100

Data Warehouse BI Data Workflow Data Pipeline 100

Meltano

OCTOBER 5, 2022

In the same way, a DataOps engineer designs the data assembly line that enables data scientists to derive insights from data analytics faster and with fewer errors. DataOps engineers improve the speed and quality of the data development process by applying DevOps principles to data workflow, known as DataOps.

Engineering 52

Engineering Raw Data Data Pipeline ETL Tools 52

DataKitchen

APRIL 13, 2021

Piperr.io — Pre-built data pipelines across enterprise stakeholders, from IT to analytics, tech, data science and LoBs. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Genie — Distributed big data orchestration service by Netflix.

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. Database-centric In bigger organizations, Data engineers mainly focus on data analytics since the data flow in such organizations is huge.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Databand.ai

DECEMBER 29, 2022

He believes in making data do the work through proper data management based on strategic rationale and business alignment. On LinkedIn, he posts frequently about big data, master data, data science, data management, and data storytelling.

BI 52

BI Consulting Data Science Data Governance 52