Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 19, 2023

Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data projects are notoriously complex.

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

OCTOBER 2, 2021

Summary Aerospike is a database engine that is designed to provide millisecond response times for queries across terabytes or petabytes. He also discusses the technical implementation that allows for such extreme performance and how the data model contributes to the scalability of the system.

Building 100

Building BI Data Architecture Architecture 100

Data Engineering Podcast

MARCH 10, 2024

In this episode Alex Merced explains how the branching and merging functionality in Nessie allows you to use the same versioning semantics for your data lakehouse that you are used to from Git. Data lakes are notoriously complex. What are the core problems/complexities that Nessie is designed to solve?

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Data Engineering Podcast

MAY 13, 2021

While they provide a measure of convenience, they also require you to sacrifice a certain amount of control over your data. If you want to build a warehouse that gives you both control and flexibility then you might consider building on top of the venerable PostgreSQL project.

PostgreSQL 100

PostgreSQL Data Warehouse Building MySQL 100

Data Engineering Podcast

NOVEMBER 16, 2020

In this episode CEO and co-founder of Isima Darshan Rawal explains how the biOS platform is architected to enable ease of use, the challenges that were involved in building an entirely new system from scratch, and how it can integrate with the rest of your data platform to allow for incremental adoption. When is it the wrong choice?

Data Management 100

Data Management Management BI Business Intelligence 100

Monte Carlo

AUGUST 31, 2023

As the data analyst or engineer responsible for managing this data and making it usable, accessible, and trustworthy, rarely a day goes by without having to field some request from your stakeholders. But what happens when the data is wrong? What is Data Quality?

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

JULY 12, 2021

Summary We have been building platforms and workflows to store, process, and analyze data since the earliest days of computing. Over that time there have been countless architectures, patterns, and "best practices" to make that task manageable. RudderStack’s smart customer data pipeline is warehouse-first.

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Data Engineering Podcast

JULY 27, 2020

In this episode he shares his approach to testing complex systems, the common challenges that are faced by engineers who build them, and why it is important to understand their limitations. If you hand a book to a new data engineer, what wisdom would you add to it? How has the design evolved since you first began working on it?

Systems 100

Systems Building Scala Java 100

Data Engineering Podcast

AUGUST 31, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it?

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Data Engineering Podcast

DECEMBER 14, 2020

In this episode Andrew Gross, Bobby Muldoon, and Anup Segu describe the self service data platform that they have built to allow data analysts to own the end-to-end delivery of data projects and how that has allowed them to scale their output. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Analytics 100

Data Analytics Building MySQL Data 100

Data Engineering Podcast

JANUARY 30, 2022

In addition to that, the host curated the essays contained in the book "97 Things Every Data Engineer Should Know", using the knowledge and context gained from running the show to inform the selection process. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Data Engineering Podcast

JANUARY 15, 2022

Now it is being used to power consumer facing services, influence organizational behaviors, and build sophisticated machine learning systems. Given this increased level of importance it has become necessary for everyone in the business to treat data as a product in the same way that software applications have driven the early 2000s.

Engineering 100

Engineering Electronics ETL Tools Data Pipeline 100

Data Engineering Podcast

DECEMBER 26, 2021

In this episode Zhamak re-joins the show to discuss the real world benefits that have been seen, the lessons that she has learned while working with her clients and the community, and her vision for the future of the data mesh. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

BI 130

BI Data Warehouse Data Engineering Data Engineer 130

Data Engineering Podcast

DECEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

OCTOBER 13, 2021

In this episode he discusses his experiences and how he approached the work of distilling them for his book "Fail Fast, Learn Faster" This is an entertaining and enlightening exploration of the business side of data with an industry veteran. Can you start by discussing the focus of the book and what motivated you to write it?

Entertainment 100

Entertainment Big Data Data Lake Business Intelligence 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Data Engineering Podcast

JUNE 19, 2022

In this episode Ryan explains how he and his team have designed their platform to bring everyone onto a level playing field and the benefits that it provides to the organization. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

JANUARY 30, 2022

He recently wrote a book on effective patterns for Pandas code, and in this episode he shares advice on how to write efficient data processing routines that will scale with your data volumes, while being understandable and maintainable. What are the main tasks that you have seen Pandas used for in a data engineering context?

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

Data Engineering Podcast

APRIL 17, 2022

In this episode Prineha Narang, co-founder and CTO of Aliro, explains how these systems work, the capabilities that they can offer, and how you can start preparing for a post-quantum future for your data systems. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse SQL Data Engineering Data Engineer 100

Snowflake

FEBRUARY 12, 2024

Leveraging Big Data for 5G Monetization : Fawad Qureshi, Snowflake’s Global Industry Field CTO, will join leaders from Vodafone Business UK, AT&T, TELUS and United Airlines to discuss how industries can unlock the full value of IoT applications and the increase of data with the rollout of 5G.

Python 83

Python Manufacturing Cloud Government 83

Data Engineering Podcast

APRIL 10, 2022

Rivery is a platform designed to reduce this incidental complexity and provide a single system for working across the different stages of the data lifecycle. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. How is the Rivery platform designed and implemented?

Data Integration 100

Data Integration SQL Data Engineering Data Engineer 100

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. Can you describe how Planetscale is implemented?

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

APRIL 3, 2022

Summary The flexibility of software oriented data workflows is useful for fulfilling complex requirements, but for simple and repetitious use cases it adds significant complexity. Coalesce is a platform designed to reduce repetitive work for common workflows by adopting a visual pipeline builder to support your data warehouse transformations.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. RudderStack helps you build a customer data platform on your warehouse or data lake.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 9, 2018

Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. He also discusses what you need to know to get it deployed and keep it running in a production environment and how it fits into the overall data ecosystem. Who is the target audience?

Scala 100

Scala MySQL Kafka Hadoop 100

François Nguyen

MARCH 7, 2021

In this article, Juan Sequada gives maybe one of the best definition of Data Mesh ” It is paradigm shift towards a distributed architecture that attempts to find an ideal balance between centralization and decentralization of metadata and data management.”

Government 130

Government Data Governance Data Metadata 130

Data Engineering Podcast

DECEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Data Warehouse Data Pipeline MongoDB 100

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Data Engineering Podcast

MAY 1, 2022

He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the maintenance burden of data integration workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Missing data?

Data Warehouse 100

Data Warehouse Data Lake Data Pipeline BI 100

Data Engineering Podcast

NOVEMBER 27, 2022

He also discusses the improvements that have been incorporated into FeatureBase to simplify integration with the rest of your data stack, and the SQL interface that was added to make working with the product easier. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. Can you describe what FeatureBase is?

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100