Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Your first 30 days are free!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Knowledge Hut

DECEMBER 28, 2023

This certification, often referred to as the Azure Data Engineer Associate certification, validates the competency of individuals in implementing Azure data solutions. It’s a testament to their ability to create scalable, efficient and secure data pipelines. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. In this episode he explains how he used the combination of Apache Arrow, Flight, Datafusion, and Parquet to lay the foundation of the newest version of his time-series database. Data lakes are notoriously complex.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

MARCH 31, 2024

While there are numerous products available to provide that visibility, they all have different technologies and workflows that they focus on. To bring observability to dbt projects the team at Elementary embedded themselves into the workflow. How have the scope and goals of the project changed since you started working on it?

Project 130

Project Data Lake High Quality Data Data Workflow 130

Hepta Analytics

FEBRUARY 14, 2022

We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design. The process is slow since the data has to be transformed first. The BI team wants to do analysis of the data. It is called Idempotency.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

MAY 18, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Data Engineering Podcast

JUNE 2, 2024

In this episode she shares the practical steps to implementing a data governance practice in your organization, and the pitfalls to avoid. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

DataKitchen

JANUARY 3, 2022

For example, teams working under the VP/Directors of Data Analytics may be tasked with accessing data, building databases, integrating data, and producing reports. Data scientists derive insights from data while business analysts work closely with and tend to the data needs of business units.

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

MAY 26, 2024

Sriram Panyam has been involved in several projects that required migration of large volumes of data in high traffic environments. In this episode he shares some of the valuable lessons that he learned about how to make those projects successful. Can you start by sharing some of your experiences with data migration projects?

Systems 130

Systems Data Lake High Quality Data Google Cloud 130

Data Engineering Podcast

JUNE 9, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Process 130

Process Data Lake High Quality Data Machine Learning 130

Data Engineering Podcast

DECEMBER 24, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

DECEMBER 3, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Knowledge Hut

SEPTEMBER 29, 2023

One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role. To truly understand the importance of data engineers, we can explore the hierarchy of needs in data science. You can browse the data lake files with the interactive training material.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

FEBRUARY 4, 2024

RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

MAY 12, 2024

I listened to the recent episode "Transforming Your Database" and appreciated the valuable advice on how to approach the selection and integration of new databases in applications and the impact on team dynamics. Data lakes are notoriously complex. Data lakes are notoriously complex.

Management 147

Management Data Lake High Quality Data Machine Learning 147

ThoughtSpot

SEPTEMBER 6, 2023

The pathway from ETL to actionable analytics can often feel disconnected and cumbersome, leading to frustration for data teams and long wait times for business users. And even when we manage to streamline the data workflow, those insights aren’t always accessible to users unfamiliar with antiquated business intelligence tools.

Data Workflow 91

Data Workflow Raw Data Data Lake Business Intelligence 91

Data Engineering Podcast

NOVEMBER 5, 2023

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

DECEMBER 31, 2023

In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

DECEMBER 17, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Data projects are notoriously complex. Want to see Starburst in action?

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

DECEMBER 10, 2023

Andrew Maguire got tired of solving that problem for each of the different roles he has ended up in, so he created the open source Anomstack project. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. With Materialize, you can! Sponsored By: Starburst : ![Starburst

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Monte Carlo

MAY 14, 2024

The “legacy” table formats The data landscape has evolved so quickly that table formats pioneered within the last 25 years are already achieving “legacy” status. It was designed to support high-volume data exchange and compatibility across different system versions, which is essential for streaming architectures such as Apache Kafka.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. If you've learned something or tried out a project from the show then tell us about it!

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Ascend.io

DECEMBER 11, 2023

When data starts piling up from all corners — including cloud APIs , cloud warehouses , on-premises databases , and data lakes — that’s when you really start feeling the need for efficient data orchestration. So, why is data orchestration a big deal? But let’s step back for a second.

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Knowledge Hut

MAY 23, 2024

This vast stream of interdisciplinary domains deals with data in different ways. It helps companies understand data and obtain meaningful insights from it. According to the GlobeNewswire report , the projected growth of the data science market will hike up to a CAGR of 25 percent by 2030.

Data Science 52

Data Science MongoDB Programming Language Hadoop 52

Monte Carlo

MARCH 24, 2023

Data migration refers to the movement of data from one location (warehouse or database), format, or SaaS application, to another. Data migrations are also highly visible projects with severe consequences to both the organization and project manager for failed attempts.

Data Warehouse 52

Data Warehouse AWS Cloud Data Governance 52

Monte Carlo

APRIL 24, 2023

During data ingestion, raw data is extracted from sources and ferried to either a staging server for transformation or directly into the storage level of your data stack—usually in the form of a data warehouse or data lake. There are two primary types of raw data.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Ripple Engineering

MAY 27, 2021

Using the open source version of DBT, the Data Engineering team aimed to equip Ripple’s analysts with a tool that cleans data and reduces the amount of time spent manually performing checks. DBT Implementation Basics Setting up the DBT Project We used GitLab to set up our DBT project.

Building 52

Building Raw Data SQL Data 52

Data Engineering Podcast

APRIL 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

phData: Data Engineering

JANUARY 3, 2022

This makes you work for your data instead of your data working for you. This commonly introduces: Database or Data Warehouse API/EDI Integrations ETL software Business intelligence tooling By leveraging off-the-shelf tooling, your company separates disciplines by technology.

IT 52

IT AWS Software Engineer Software Engineering 52