Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 130

MongoDB Scala MySQL Machine Learning 130

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 17, 2022

In this episode Crux CTO Mark Etherington discusses the different costs involved in managing external data, how to think about the total return on investment for your data, and how the Crux platform is architected to reduce the toil involved in managing third party data. Tired of deploying bad data?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Tired of deploying bad data? What does Riskified do?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Monte Carlo

MAY 26, 2023

Pro-tip : Don’t confuse data freshness with data latency. Data latency is the time between when the event occurs and when the data is available in the core data system (like a data warehouse) whereas data freshness is how recently the data within the final asset (table, BI report) has been updated.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Meltano

JULY 19, 2022

Extraction and Loading Tasks in ELT pipeline A custom extractor must present the extracted data in a form that the tool can use to load it into the target sink like a data warehouse. Singer is a commonly used tool for extracting data, which serves as the specification for implementing data extractors and loaders.

Building 52

Building Python Database Project 52

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

FEBRUARY 24, 2020

Summary Misaligned priorities across business units can lead to tensions that drive members of the organization to build data and analytics projects without the guidance or support of engineering or IT staff. Are you spending too much time maintaining your data pipeline?

IT 100

IT Data Lake Data Pipeline Media 100

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Workfall

SEPTEMBER 18, 2023

Meet Airbyte, the data magician that turns integration complexities into child’s play. In this digital era, businesses thrive on data, and making this data dance harmoniously with your analytics tools is crucial. With its drag-and-drop interface, creating data pipelines becomes as easy as arranging blocks in a puzzle.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Monte Carlo

NOVEMBER 9, 2023

And in many ways, LLMs are going to make data engineers more valuable – and that’s exciting! Still, it’s one thing to show your boss a cool demo of a data discovery tool or text-to-SQL generator – it’s another thing to use it with your company’s proprietary data, or even more concerning, customer data.

Unstructured Data 86

Unstructured Data Database Data Pipeline Architecture 86

Data Engineering Podcast

DECEMBER 21, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Now every team can build a customer360 in Snowflake with RudderStack Profiles.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

JANUARY 4, 2021

In order to make those efforts sustainable, the core capability they need is for data scientists and analysts to be able to build and deploy features in a self service manner. As a result the feature store is becoming a required piece of the data platform. What types of data can feature stores manage?

Python 100

Python Machine Learning Data Lake Computer Science 100

Data Engineering Podcast

JANUARY 11, 2021

In this episode he explains how the TerminusDB system is architected to provide a versioned graph storage engine that allows for branching and merging of data sets, how that opens up new possibilities for individuals and teams to work together on building new data repositories. When is TerminusDB the wrong choice?

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Cloudera

JANUARY 20, 2021

Most of what is written though has to do with the enabling technology platforms (cloud or edge or point solutions like data warehouses) or use cases that are driving these benefits (predictive analytics applied to preventive maintenance, financial institution’s fraud detection, or predictive health monitoring as examples) not the underlying data.

Manufacturing 110

Manufacturing Data Warehouse Kafka Retail 110

Striim

FEBRUARY 10, 2023

It provides networking solutions to support the world’s largest telecommunications service providers, submarine network operators, data and cloud operators, and large enterprises. Because of Striim, we have so much customer and operational data at our fingertips. Help customers visualize their data using a business intelligence tool.

Data Analytics 52

Data Analytics Telecommunication Insurance Google Cloud 52

Monte Carlo

MAY 26, 2023

This could be the result of a sync issue, the type of information each collects, or how each platform treats different data types. Having the data warehouse as a central source of truth can help present a more consistent view of the campaign. Looker also has a semantic layer called LookML.

Datasets 52

Datasets Data Warehouse Data Validation Data 52

ProjectPro

JANUARY 19, 2022

ProjectPro has precisely that in this section, but before presenting it, we would like to answer a few common questions to strengthen your inclination towards data engineering further. What is Data Engineering? Data Engineering refers to creating practical designs for systems that can extract, keep, and inspect data at a large scale.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Data Engineering Podcast

MAY 1, 2022

He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the maintenance burden of data integration workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

JUNE 1, 2021

Summary Every part of the business relies on data, yet only a small team has the context and expertise to build and maintain workflows and data pipelines to transform, clean, and integrate it. RudderStack’s smart customer data pipeline is warehouse-first. task-based, data assets, etc.)

Data Pipeline 100

Data Pipeline Data Warehouse Data Workflow Data 100

Data Engineering Podcast

DECEMBER 28, 2022

In this episode he shares his thoughts on the strategic and tactical elements of moving your work as a data professional from being task-oriented to being product-oriented and the long term improvements in your productivity that it provides. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Data Engineering Podcast

DECEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Pricing for SQLake is simple.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Data Engineering Podcast

DECEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Pricing for SQLake is simple.

Data Lake 100

Data Lake Data Warehouse Data Pipeline MongoDB 100

Data Engineering Podcast

MAY 25, 2021

To eliminate this impedance mismatch Edo Liberty founded Pinecone to build database that works natively with vectors. In this episode he explains how this technology will allow teams to accelerate the speed of innovation, how vectors make it possible to build more advanced search functionality, and how Pinecone is architected.

Database 100

Database Building Data Warehouse Machine Learning 100

Data Engineering Podcast

OCTOBER 2, 2021

If you need to deal with massive data, at high velocities, in milliseconds, then Aerospike is definitely worth learning about. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Building 100

Building BI Data Architecture Architecture 100

Data Engineering Podcast

NOVEMBER 27, 2022

He also discusses the improvements that have been incorporated into FeatureBase to simplify integration with the rest of your data stack, and the SQL interface that was added to make working with the product easier. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

APRIL 3, 2022

Coalesce is a platform designed to reduce repetitive work for common workflows by adopting a visual pipeline builder to support your data warehouse transformations. In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Missing data?

Data Warehouse 100

Data Warehouse Data Lake Data Pipeline BI 100

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

APRIL 17, 2022

In this episode Prineha Narang, co-founder and CTO of Aliro, explains how these systems work, the capabilities that they can offer, and how you can start preparing for a post-quantum future for your data systems. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks.

Data Warehouse 100

Data Warehouse SQL Data Engineering Data Engineer 100