Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Master Data Management (MDM) is the process of building consensus around what the information actually means in the context of the business and then shaping the data to match those semantics.

Data Management 100

Data Management Management MongoDB Scala 100

Confluent

MARCH 19, 2019

When should you use an event? I will attempt to articulate in layman’s terms what an event-driven architecture (EDA) is and contrast it with service-oriented architecture (SOA). In essence, this is an attempt to differentiate and/or associate APIs with events. Is not life simply a series of events? If it can, should it?

Architecture 110

Architecture Telecommunication Transportation Database 110

Data Engineering Podcast

MARCH 18, 2023

Summary As with all aspects of technology, security is a critical element of data applications, and the different controls can be at cross purposes with productivity. He also explains why data security is distinct from application security and some methods for reducing the challenge of working across different data systems.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Cloudera

APRIL 18, 2024

With the arrival of the latest AI-powered technologies like large language models (LLMs) and generative AI (GenAI), there’s a vast amount of opportunities for innovation, growth, and improved business outcomes right around the corner. All of that technology, though, depends on data to be successful.

Metadata 106

Metadata Government Data Governance Data 106

Data Engineering Podcast

OCTOBER 15, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Process 182

Process Building SQL BI 182

Data Engineering Podcast

APRIL 2, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Businesses that adapt well to change grow 3 times faster than the industry average. As your business adapts, so should your data. As your business adapts, so should your data.

Hadoop 130

Hadoop Machine Learning Python Architecture 130

Data Engineering Podcast

MARCH 5, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze.

Building 147

Building Database Design Metadata Machine Learning 147

Data Engineering Podcast

OCTOBER 22, 2023

In this episode Ranjith Raghunath shares his thoughts on how to build a strategy for the development, delivery, and evolution of data products. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

BI 162

BI SQL Machine Learning Programming Language 162

Data Engineering Podcast

OCTOBER 8, 2023

In this episode he shares his journey of data collection and analysis and the challenges of automating an intentionally manual industry. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Closing Announcements Thank you for listening!

Insurance 162

Insurance BI SQL Machine Learning 162

Data Engineering Podcast

OCTOBER 1, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Building 162

Building BI SQL Machine Learning 162

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

Snowflake

APRIL 29, 2024

This regulation aims to strengthen the operational resilience of financial entities (FEs), and their third-party information and communication technology (ICT) providers. The second batch of DORA policy products, which aims to strengthen oversight and risk management for third-party ICT providers, is currently under development.

Transportation 79

Transportation Data Governance Government Consulting 79

Data Engineering Weekly

MAY 5, 2024

link] LinkedIn: LakeChime - A Data Trigger Service for Modern Data Lakes LinkedIn points out two critical flaws in a partitioned approach to data management. The granularity of partition creation constrained data consumption. The author expands on the possibility of unified data platforms.

Data Engineering 67

Data Engineering Data Engineer Engineering Google Cloud 67

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. Interview Introduction (see Guy’s bio below) How did you get involved in the area of data management?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

AUGUST 3, 2021

In this episode Vinoth shares the history of the project, how its architecture allows for building more frequently updated analytical queries, and the work being done to add a more polished experience to the data lake paradigm. Interview Introduction How did you get involved in the area of data management?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

AUGUST 10, 2021

In this episode Mark Grover explains what he is building at Stemma, how it expands on the success of the Amundsen project, and why trust is the most important asset for data teams. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more.

IT 130

IT Building Data Warehouse Python 130

Data Engineering Podcast

APRIL 24, 2022

In this episode Andy Dang explains why the project was created, how you can apply it to your existing data systems, and how it functions to provide detailed context for being able to gain insight into all of your data processes. How do you structure the log events and metadata to provide detail and context for data applications?

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Data Engineering Podcast

MAY 20, 2021

This is because it is actually a concept that encompasses the entire lifecycle of data, across all of the people in an organization who interact with it. Stijn Christiaens co-founded Collibra with the goal of addressing the wide variety of technological aspects that are necessary to realize such an important and expansive process.

Data Governance 100

Data Governance Government Data Warehouse Data Pipeline 100

Data Engineering Podcast

FEBRUARY 25, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join us at the top event for the global data community, Data Council Austin.

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

APRIL 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Can you describe what MLOps is?

Data Engineering 100

Data Engineering Data Engineer Engineering IT 100

Data Engineering Podcast

JUNE 22, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 17, 2022

Summary Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

MARCH 27, 2022

Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. What do you have planned for the future of Privacera?

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Can you describe the role of data at Riskified?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

JULY 27, 2021

Summary Everyone expects data to be transmitted, processed, and updated instantly as more and more products integrate streaming data. This is a fascinating conversation with a lot of useful lessons for anyone who wants to understand the operational aspects of Pulsar and the benefits that it can provide to data workloads.

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

JUNE 12, 2022

In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Podcast

NOVEMBER 11, 2019

Summary The practice of data management is one that requires technical acumen, but there are also many policy and regulatory issues that inform and influence the design of our systems. He started Datacoral with the goal to make SQL the universal data programming language.

Designing 100

Designing Data Pipeline Programming Language Data 100

Knowledge Hut

DECEMBER 26, 2023

One of the primary reasons is the rapidly changing technological landscape and the fact that software adoption is steadily rising across numerous industries, including finance, government, military, retail, hospitals, education, and energy, to name a few. Cybersecurity is defending sensitive data and important systems from online threats.

Hospitality 98

Hospitality Finance Electronics Retail 98

Data Engineering Podcast

MARCH 9, 2020

Tim provides useful examples for understanding how to adopt this approach in your own organization, including some technology recommendations for making it maintainable and scalable. We have partnered with organizations such as O’Reilly Media, Corinium Global Intelligence, ODSC, and Data Council.

Data Governance 100

Data Governance Government Architecture Data 100

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage. When is Firebolt the wrong choice?

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Data Engineering Podcast

JANUARY 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

AUGUST 3, 2020

This was an informative and enlightening conversation with two experts on graph data applications that will help you start on the right track in your own projects. If you hand a book to a new data engineer, what wisdom would you add to it? Can you start by explaining what your goals are for the Practitioner’s Guide To Graph Data?

NoSQL 100

NoSQL Database Algorithm Relational Database 100

Data Engineering Podcast

FEBRUARY 17, 2020

Summary One of the biggest challenges in building reliable platforms for processing event pipelines is managing the underlying infrastructure. At Snowplow Analytics the complexity is compounded by the need to manage multiple instances of their platform across customer environments.

AWS 100

AWS Google Cloud Kafka Media 100