Data Engineering Podcast

AUGUST 21, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Data Engineering Podcast

MAY 11, 2020

You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Tidy Data is the DataOps monitoring platform that you’ve been missing. You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data?

Lambda Architecture 100

Lambda Architecture Cloud Kafka Hadoop 100

Data Engineering Podcast

NOVEMBER 20, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. The Lambda architecture has largely been abandoned, so what is the answer for today’s data lakes?

Data Lake 100

Data Lake Data Integration Lambda Architecture Process 100

Data Engineering Podcast

JUNE 16, 2019

This conversation was useful for getting a better idea of the challenges that exist in large scale data analytics, and the current state of the tradeoffs between data lakes and data warehouses in the cloud. What are some of the common antipatterns in data lake implementations and how does Delta Lake address them?

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Pinterest Engineering

MAY 2, 2023

This platform is also a key component for PinnerFormer work providing real-time user sequence data. Real-Time Indexing Pipeline The main goal of the real-time indexing pipeline is to enrich, store, and serve the last few relevant user actions as they come in. To explore life at Pinterest, visit our Careers page.

Lambda Architecture 65

Lambda Architecture Datasets Software Engineer Software Engineering 65

Monte Carlo

MARCH 14, 2023

Data ingestion is the process of collecting data from various sources and moving it to your data warehouse or lake for processing and analysis. It is the first step in modern data management workflows. Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley. There are trade-offs.

Data Ingestion 52

Data Ingestion Data Warehouse Lambda Architecture Raw Data 52

Cloudera

NOVEMBER 2, 2020

Users today are asking ever more from their data warehouse. As an example of this, in this post we look at Real Time Data Warehousing (RTDW), which is a category of use cases customers are building on Cloudera and which is becoming more and more common amongst our customers. What is Real Time Data Warehousing?

Data Warehouse 91

Data Warehouse Kafka Lambda Architecture Telecommunication 91