Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

MARCH 27, 2022

The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

AUGUST 20, 2021

In this episode Vikrant Dubey discusses his work on the Cuelake project which allows data analysts to build a lakehouse with SQL queries. By building on top of Zeppelin, Spark, and Iceberg he and his team at Cuebook have built an autoscaled cloud native system that abstracts the underlying complexity.

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

AUGUST 10, 2021

In this episode Mark Grover explains what he is building at Stemma, how it expands on the success of the Amundsen project, and why trust is the most important asset for data teams. RudderStack’s smart customer data pipeline is warehouse-first. Tell me more about Stemma and how it compares to Amundsen?

IT 130

IT Building Data Warehouse Python 130

Data Engineering Podcast

FEBRUARY 15, 2021

In order to be able to truly derive insights from the data that an organization collects, there are certain foundational capabilities that they need to have capacity for. This is a great conversation about the initial steps that every group should be thinking of as they start down the road to making data informed decisions.

Building 100

Building Data Warehouse BI Data Pipeline 100

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares the design of the project and how it fits into your development practices. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

MARCH 16, 2020

In this episode Adam Kocoloski shares the history of the project, how it works under the hood, and how the new design will improve the project for our new era of computation. This was an interesting conversation about the challenges of maintaining a large and mission critical project and the work being done to evolve it.

Building 100

Building Data Warehouse NoSQL Data Lake 100

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. We feel your pain.

IT 147

IT Data Lake Metadata Data Warehouse 147

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. ETL has typically been carried out utilizing data warehouses and on-premise ETL tools.

AWS 52

AWS Data Management ETL Tools Management 52

Data Engineering Podcast

JULY 27, 2021

This is a fascinating conversation with a lot of useful lessons for anyone who wants to understand the operational aspects of Pulsar and the benefits that it can provide to data workloads. RudderStack’s smart customer data pipeline is warehouse-first. Go to dataengineeringpodcast.com/97things today to get your copy!

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

MAY 15, 2022

Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 3, 2023

Summary Cloud data warehouses and the introduction of the ELT paradigm has led to the creation of multiple options for flexible data integration, with a roughly equal distribution of commercial and open source options. What are the benefits of building it in Python? Can you describe what dlt is and the story behind it?

Data Integration 162

Data Integration BI SQL Python 162

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

MAY 19, 2019

Summary In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines. When is it the wrong choice?

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 3, 2018

Using master data management to build a data catalog helps you answer these questions reliably and simplify the process of building your business intelligence reports. Can you start by establishing a definition of data mastering that we can work from?

Data Management 100

Data Management Management Relational Database Business Intelligence 100

Knowledge Hut

OCTOBER 27, 2023

With Fabric, users can create their own data warehouse and create quick reports out of them. Power BI is a wonderful tool, and there are many reasons to learn Power BI like: Power BI has a wide range of offers; it is one tool that can perform multiple functions, from data extraction and transformation to dashboard and dynamic reports.

BI 52

BI Project Datasets Healthcare 52

Data Engineering Podcast

AUGUST 3, 2021

Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge. By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

DECEMBER 22, 2019

Data warehouses are optimized for batched writes and complex analytical queries. Between those use cases there are varying levels of support for fast reads on quickly changing data. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council.

SQL 100

SQL Building Engineering Java 100

Data Engineering Podcast

OCTOBER 30, 2022

It’s hard to have a conversation about data engineering or analysis without mentioning it. Despite its widespread adoption there are still rough edges in its workflow that cause friction for data analysts. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Data teams are increasingly under pressure to deliver.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. or any other destination you choose.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

MAY 1, 2022

Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. This is an excellent exploration of the decisions and tradeoffs that need to be made while building such a complex system.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

OCTOBER 22, 2021

Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. No more scripts, just SQL.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

NOVEMBER 11, 2018

In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization. There are a lot of opinions on both sides of the data lake argument.

Data Lake 100

Data Lake Building Kafka Cloud 100

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data. Use separate data warehouses for cost-effective data loading.

Data Warehouse 101

Data Warehouse Banking Data Cloud 101

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Data Engineering Podcast

MAY 29, 2022

Summary The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. In order to ensure that you can explore and analyze your data without spending money on inefficient queries Mingsheng Hong and Zheng Shao created Bluesky Data.

Cloud 100

Cloud Data Warehouse Data Lake Data 100

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Start trusting your data with Monte Carlo today! No more scripts, just SQL. What are the goals and target use cases of Unomi?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. No more scripts, just SQL.

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. This is an ambitious and exciting project, so listen and try it out today. Start trusting your data with Monte Carlo today! No more scripts, just SQL.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

JULY 5, 2021

In this episode Bart and Rich Wood explain how SaaSGlue is architected to allow for a high degree of flexibility in usage and deployment, their experience building a business with family, and how you can get started using it today. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

FEBRUARY 22, 2021

In this episode co-founders John Lafleur and Michel Tricot share the story of how and why they created Airbyte, discuss the project’s design and architecture, and explain their vision of what an open soure data integration platform should offer. RudderStack’s smart customer data pipeline is warehouse-first.

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Data Engineering Podcast

APRIL 5, 2021

In this episode Prukalpa Sankar shares her experiences across multiple attempts at building a system that brings everyone onto the same page, ultimately bringing her to found Atlan. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

MAY 10, 2021

Summary Building an API for real-time data is a challenging project. The team at Tinybird wants to make it easy to turn a continuous stream of data into a production ready API or data product. RudderStack’s smart customer data pipeline is warehouse-first.

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100