Data Engineering Podcast

NOVEMBER 13, 2022

Summary Building data products is an undertaking that has historically required substantial investments of time and talent. With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping. With the rise in cloud platforms and self-serve data technologies the barrier of entry is dropping.

Building 130

Building Metadata MongoDB MySQL 130

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 89

Kafka Technology IT Government 89

Knowledge Hut

NOVEMBER 19, 2023

With advances in technology, wearable devices do provide some trace of your health. How do they collect, process, and analyze data for you? Data Engineers/Architects set up this infrastructure to extract, process and store this data. Hence, Data Engineering and Data Science in IoT is gaining wide traction.

Data Science 52

Data Science Technology Machine Learning Big Data 52

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. When is Crux the wrong choice?

Data Management 130

Data Management Management Metadata MongoDB 130

Picnic Engineering

OCTOBER 17, 2023

Building the data foundation for world’s best online supermarket The Importance of MDM In the fast-paced world of technology, it’s easy to be captivated by the allure of artificial intelligence. To create the best online grocery experience and ensure flawless operations, mastering data management became a vital practice.

Data Management 52

Data Management Management Food Data 52

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode Nick King discusses how you can be intentional about data creation in your applications and services to reduce the friction and errors involved in building data products and ML applications. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Any business that wants to understand their operations and customers through data requires some form of pipeline. Building reliable data pipelines is a complex and costly undertaking with many layered requirements. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

MARCH 27, 2022

The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project. What do you have planned for the future of Privacera?

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

JUNE 8, 2020

Summary The landscape of data management and processing is rapidly changing and evolving. This is a useful conversation to gain a macro perspective on where businesses are looking to improve their capabilities to work with data. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Management 100

Data Management Management Machine Learning Portfolio 100

The Modern Data Company

APRIL 15, 2024

With this in mind, let’s explore how to demystify the process of building your data-driven strategy, making it accessible and actionable. We’ll uncover how you can transform data into a strategic asset that propels your organization forward without getting lost in the complexity of its creation. It matters a lot.

Building 52

Building Raw Data Food Data 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

Data Engineering Podcast

FEBRUARY 15, 2021

In order to be able to truly derive insights from the data that an organization collects, there are certain foundational capabilities that they need to have capacity for. This is a great conversation about the initial steps that every group should be thinking of as they start down the road to making data informed decisions.

Building 100

Building Data Warehouse BI Data Pipeline 100

Data Engineering Podcast

AUGUST 20, 2021

In this episode Vikrant Dubey discusses his work on the Cuelake project which allows data analysts to build a lakehouse with SQL queries. By building on top of Zeppelin, Spark, and Iceberg he and his team at Cuebook have built an autoscaled cloud native system that abstracts the underlying complexity.

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

MAY 15, 2022

Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

FEBRUARY 8, 2021

Summary With all of the tools and services available for building a data platform it can be difficult to separate the signal from the noise. One of the best ways to get a true understanding of how a technology works in practice is to hear from people who are running it in production.

Data Warehouse 100

Data Warehouse Building BI SQL 100

Data Engineering Podcast

DECEMBER 7, 2020

Summary Building data products are complicated by the fact that there are so many different stakeholders with competing goals and priorities. Different organizations have tried a multitude of organizational strategies to improve the success rate of these data teams with varying levels of success. centralized, embedded, etc.)

Building 100

Building Big Data Consulting Data Engineering 100

Data Engineering Podcast

AUGUST 10, 2021

In this episode Mark Grover explains what he is building at Stemma, how it expands on the success of the Amundsen project, and why trust is the most important asset for data teams. RudderStack’s smart customer data pipeline is warehouse-first. And don’t forget to thank them for their continued support of this show!

IT 130

IT Building Data Warehouse Python 130

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage. When is Firebolt the wrong choice?

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Data Engineering Podcast

JANUARY 23, 2022

In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

JANUARY 21, 2019

Summary Building internal expertise around big data in a large organization is a major competitive advantage. In this episode Jesper Søgaard and Keld Antonsen share the story of starting and growing the big data group at LEGO. My understanding is that the big data group at LEGO is a fairly recent development.

Big Data 100

Big Data Systems Building Hadoop 100

Data Engineering Podcast

FEBRUARY 13, 2022

To simplify the work of managing the full flow of your customer data and keep you in full control the team at Rudderstack created their eponymous open source platform that allows you to work with first and third party data, as well as build and manage reverse ETL workflows. Who are the target users of Rudderstack?

Building 100

Building Hadoop Data Pipeline Metadata 100

Data Engineering Podcast

APRIL 19, 2020

Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor. Eventador is a managed platform designed to let you focus on using the data that you collect, without worrying about how to make it reliable.

Building 100

Building PostgreSQL MongoDB SQL 100

Data Engineering Podcast

APRIL 6, 2020

By organizing and storing data to emphasize the relationship between entities, we can discover the complex connections between multiple sources of information. If you’re wondering how to extract additional business value from existing data, this episode will provide a way to expand your data resources.

Building 100

Building Big Data Machine Learning Python 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

SEPTEMBER 3, 2018

Using master data management to build a data catalog helps you answer these questions reliably and simplify the process of building your business intelligence reports. Can you start by establishing a definition of data mastering that we can work from?

Data Management 100

Data Management Management Relational Database Business Intelligence 100

Data Engineering Podcast

JULY 27, 2021

Summary Everyone expects data to be transmitted, processed, and updated instantly as more and more products integrate streaming data. This is a fascinating conversation with a lot of useful lessons for anyone who wants to understand the operational aspects of Pulsar and the benefits that it can provide to data workloads.

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 26, 2019

Summary Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. Benn Stancil is the chief analyst at Mode Analytics and in this episode he explains the set of considerations and requirements that data analysts need in their tools and.

Building 100

Building Data Analytics Media Data Engineering 100

Data Engineering Podcast

NOVEMBER 9, 2020

Summary A data catalog is a critical piece of infrastructure for any organization who wants to build analytics products, whether internal or external. While there are a number of platforms available for building that catalog, many of them are either difficult to deploy and integrate, or expensive to use at scale.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects. What are the ways that reliability is measured for data assets? Can you describe what SodaCL is and the story behind it?

Building 100

Building Metadata MongoDB MySQL 100

Data Engineering Podcast

MARCH 16, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building Data Warehouse NoSQL Data Lake 100

Data Engineering Podcast

OCTOBER 28, 2019

In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming.

Building 100

Building Data Pipeline Programming Language Metadata 100

Data Engineering Podcast

FEBRUARY 3, 2020

If you are curious about data sharing strategies or data collaboratives, then listen now to learn more! You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management.

Building 100

Building Media AWS Data 100