Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. What is the motivating goal for the company and product?

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

The Modern Data Company

DECEMBER 29, 2022

In addition, business users can leverage drag-and-drop engineering to build pipelines without extensive coding. If the finance industry can leverage an operational layer, fractured data ecosystems will become a challenge of the past.

Banking 52

Banking Pipeline-centric Building Finance 52

The Modern Data Company

JANUARY 22, 2024

The Modern Story: Navigating Complexity and Rethinking Data in The Business Landscape Enterprises face a data landscape marked by the proliferation of IoT-generated data, an influx of unstructured data, and a pervasive need for comprehensive data analytics.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Snowflake

JANUARY 15, 2024

Leading companies around the world rely on Informatica data management solutions to manage and integrate data across various platforms from virtually any data source and on any cloud. It’s a few steps and a no-code experience, and it’s super easy to use.

Data Integration 99

Data Integration BI Cloud Data 99

Data Engineering Podcast

SEPTEMBER 30, 2018

Summary There are countless sources of data that are publicly available for use. The team at Enigma builds a knowledge graph for use in your own data projects. This is a great episode to listen to for ideas on how to organize a data engineering organization. How do you manage the software lifecycle for your ETL code?

Building 100

Building Data Lake AWS Machine Learning 100

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Precisely

MAY 2, 2024

When it comes to customer-related transactions and analytics, your data’s integrity, accuracy, and accessibility directly impact your business’s ability to operate efficiently and deliver value to customers. That’s what makes slow, manual customer data management so damaging.

Coding Skills 67

Coding Skills Food Data Portfolio 67

Christophe Blefari

JUNE 16, 2023

Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI. Data warehouses are mutable, this is one of the many root causes proposed by Lucas.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

The Modern Data Company

JANUARY 22, 2024

The Modern Story: Navigating Complexity and Rethinking Data in The Business Landscape Enterprises face a data landscape marked by the proliferation of IoT-generated data, an influx of unstructured data, and a pervasive need for comprehensive data analytics.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Data Engineering Podcast

DECEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Data Warehouse Data Pipeline MongoDB 100

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Data Engineering Podcast

APRIL 17, 2022

In this episode Prineha Narang, co-founder and CTO of Aliro, explains how these systems work, the capabilities that they can offer, and how you can start preparing for a post-quantum future for your data systems. Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams.

Data Warehouse 100

Data Warehouse SQL Data Engineering Data Engineer 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Tired of deploying bad data?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Monte Carlo

MAY 31, 2023

SELECT COUNT(*) as num_invalid_codes FROM table_name WHERE column_name IS NOT NULL AND column_name NOT IN ( SELECT code FROM iso_3166_1_alpha_2 ) Example of data validation test to compare the country code in your column to the list of valid codes in the lookup table. Ready to learn more?

Building 52

Building Data Validation Healthcare Data Integration 52

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Tired of deploying bad data?

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

APRIL 3, 2022

In this episode Satish Jayanthi explains how he is building a framework to allow enterprises to move quickly while maintaining guardrails for data workflows. This allows everyone in the business to participate in data analysis in a sustainable manner. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Warehouse 100

Data Warehouse Data Workflow Data Architecture SQL 100

Data Engineering Podcast

NOVEMBER 27, 2022

He also discusses the improvements that have been incorporated into FeatureBase to simplify integration with the rest of your data stack, and the SQL interface that was added to make working with the product easier. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 100

Data Lake Data Warehouse MongoDB Data Pipeline 100

Data Engineering Podcast

MAY 1, 2022

He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the maintenance burden of data integration workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

MARCH 27, 2022

As these systems are scaled to larger volumes of data and higher throughputs the RocksDB engine can become a bottleneck for performance. In this episode Adi Gelvan shares the work that he and his team at SpeeDB have put into building a drop-in replacement for RocksDB that eliminates that bottleneck.

Relational Database 100

Relational Database Database SQL Data Warehouse 100

Data Engineering Podcast

APRIL 10, 2022

In this episode CEO and founder Itamar Ben hemo explains how his experiences in the industry led to his vision for the Rivery platform as a single place to build end-to-end analytical workflows, including how it is architected and how you can start using it today for your own work. Can you describe what Rivery is and the story behind it?

Data Integration 100

Data Integration SQL Data Engineering Data Engineer 100

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

DECEMBER 21, 2020

If you are starting down the path of implementing a data governance strategy then this episode will provide a great overview of what is involved. If you hand a book to a new data engineer, what wisdom would you add to it? ConfigCat is a feature flag service that lets you easily add flags to your Python code, and 9 other platforms.

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Missing data?

Data Warehouse 100

Data Warehouse Data Lake Data Pipeline BI 100

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JUNE 1, 2021

Summary Every part of the business relies on data, yet only a small team has the context and expertise to build and maintain workflows and data pipelines to transform, clean, and integrate it. RudderStack’s smart customer data pipeline is warehouse-first. What are the main goals that you have for Shipyard?

Data Pipeline 100

Data Pipeline Data Warehouse Data Workflow Data 100

Data Engineering Podcast

MAY 18, 2021

In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification. RudderStack’s smart customer data pipeline is warehouse-first. code available, maturity of the specification, etc.)

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

JANUARY 4, 2021

Summary As more organizations are gaining experience with data management and incorporating analytics into their decision making, their next move is to adopt machine learning. As a result the feature store is becoming a required piece of the data platform. The first 25 will receive a free, limited edition Monte Carlo hat!

Python 100

Python Data Lake Machine Learning Computer Science 100

Data Engineering Podcast

JANUARY 11, 2021

In this episode he explains how the TerminusDB system is architected to provide a versioned graph storage engine that allows for branching and merging of data sets, how that opens up new possibilities for individuals and teams to work together on building new data repositories. When is TerminusDB the wrong choice?

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Workfall

JULY 25, 2023

Sheety is a web application that allows you to turn Google Sheets into RESTful APIs without the need for writing backend code. It acts as an intermediary between your Google Spreadsheet and the applications you build, enabling you to interact with your Google Sheets data programmatically. Hands-On Conclusion What Is Sheety API?

Python 76

Python Programming Language Coding Accessible 76

Data Engineering Podcast

OCTOBER 9, 2018

This was a deep dive on how to build a successful company around a powerful platform, and how that platform simplifies operations for enterprise grade data management. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

PostgreSQL 100

PostgreSQL BI Data Warehouse Machine Learning 100

Data Engineering Podcast

JULY 1, 2019

It was an interesting conversation about how he stress tested the Instaclustr managed service for benchmarking an application that has real-world utility. Use the code BNLLC to get an additional 10% off any pass when you register. At CluedIn they call it “eventual connectivity”.

Kafka 100

Kafka Finance Media Architecture 100

Data Engineering Podcast

JULY 14, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 100

Machine Learning Media Cloud Data 100

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100