Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

MARCH 16, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building Data Warehouse NoSQL Data Lake 100

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

OCTOBER 30, 2022

In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

MAY 1, 2022

He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the maintenance burden of data integration workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Data Engineering Podcast

NOVEMBER 13, 2022

Summary CreditKarma builds data products that help consumers take advantage of their credit and financial capabilities. To make that possible they need a reliable data platform that empowers all of the organization’s stakeholders. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Missing data?

Data Warehouse 100

Data Warehouse Data Lake Data Pipeline BI 100

Data Engineering Podcast

FEBRUARY 24, 2020

Summary Misaligned priorities across business units can lead to tensions that drive members of the organization to build data and analytics projects without the guidance or support of engineering or IT staff. Are you spending too much time maintaining your data pipeline?

IT 100

IT Data Lake Data Pipeline Media 100

Monte Carlo

JUNE 27, 2023

The Snowflake CEO wasted no time outlining Snowflake’s evolution and expanded ambition, “Most in this room are doing data warehousing. These days we view data warehousing as a particular kind of workload.” He then went on to list Snowflake’s data lake, transactional (Unistore), hybrid, and collaborative (Data Sharing) capabilities.

Data Lake 59

Data Lake Government SQL Unstructured Data 59

Data Engineering Podcast

DECEMBER 21, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

MARCH 2, 2020

Summary Building applications on top of unbounded event streams is a complex endeavor, requiring careful integration of multiple disparate systems that were engineered in isolation. The ksqlDB project was created to address this state of affairs by building a unified layer on top of the Kafka ecosystem for stream processing.

Kafka 100

Kafka Process PostgreSQL MySQL 100

Data Engineering Podcast

JANUARY 4, 2021

In order to make those efforts sustainable, the core capability they need is for data scientists and analysts to be able to build and deploy features in a self service manner. As a result the feature store is becoming a required piece of the data platform. What types of data can feature stores manage?

Python 100

Python Data Lake Machine Learning Computer Science 100

Data Engineering Podcast

JANUARY 11, 2021

In this episode he explains how the TerminusDB system is architected to provide a versioned graph storage engine that allows for branching and merging of data sets, how that opens up new possibilities for individuals and teams to work together on building new data repositories. When is TerminusDB the wrong choice?

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Ascend.io

FEBRUARY 12, 2023

Stakeholders have grown frustrated with how long it takes to build data pipelines. Business users are questioning the accuracy and data reliability of the data pipelines and often have shifted back to operating on hunches rather than facts. Data products. The antidote to this harmful situation?

Data Lake 52

Data Lake Business Analyst Data Pipeline Data 52

Ascend.io

FEBRUARY 12, 2023

Stakeholders have grown frustrated with how long it takes to build data pipelines. Business users are questioning the accuracy and data reliability of the data pipelines and often have shifted back to operating on hunches rather than facts. Data products. The antidote to this harmful situation?

Data Lake 52

Data Lake Business Analyst Data Pipeline Data 52

Ascend.io

FEBRUARY 12, 2023

Stakeholders have grown frustrated with how long it takes to build data pipelines. Business users are questioning the accuracy and data reliability of the data pipelines and often have shifted back to operating on hunches rather than facts. Data products. The antidote to this harmful situation?

Data Lake 52

Data Lake Business Analyst Data Pipeline Data 52

Ascend.io

MAY 23, 2023

The data engineering world is full of tips and tricks on how to handle specific patterns that recur with every data pipeline. Already in 2016, IBM estimated the cost of bad data to be over three trillion dollars, and that was before the chaos of data lakes emerged and orphaned datasets began to swamp the land.

Datasets 52

Datasets Data Pipeline Metadata Database 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64

Ascend.io

APRIL 7, 2023

With the monolithic architectures most organizations have today, business users are stuck, constantly waiting for new data pipelines to be built or amended based on their requests. Data engineers aren’t huge fans of this paradigm either. In other words, the product one person builds might have inputs from another product.

Metadata 52

Metadata Datasets Data Governance Government 52

The Modern Data Company

JANUARY 17, 2023

Because DataOS connects to everything in the data stack, no data goes unnoticed. Leaders Can Easily Bring the CDP Under Federated Governance CDPs ingest data through both governed and ungoverned channels. It increases complexity in the data pipeline and creates yet another learning impediment for both users and SI engineers.

Metadata 59

Metadata Government Data Lake Data 59

Rockset

AUGUST 26, 2022

For instance, joining two data tables showing hourly temperatures and humidity and how much feed the cows have consumed could allow farmers to improve feeding efficiencies and optimize milk production. When a change is detected, the data is ingested into a data lake hosted on Amazon S3.

IT 52

IT Food Data Lake Aggregated Data 52

AltexSoft

NOVEMBER 7, 2022

The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. How data engineering works. Apache Airflow is an open-source Python -based workflow orchestrator that enables you to design, schedule, and monitor data pipelines.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

AltexSoft

OCTOBER 8, 2021

They are applied to retrieve data from the source systems, perform transformations when necessary, and load it into a target system ( data mart , data warehouse, or data lake). So, why is data integration such a big deal? For this purpose, you will have to build a 360-degree customer picture.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

AltexSoft

SEPTEMBER 10, 2021

With the right data integration strategy, companies can consolidate the needed data into a single place and ensure its integrity and quality for better and more reliable insights. Let’s imagine you run an eCommerce business and you want to build a predictive propensity model to calculate customer lifetime value.

Data Integration 52

Data Integration Data Consolidation Data Warehouse BI 52