Data Engineering Weekly

AUGUST 6, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. the best way to run dbt Core in Airflow. See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Monte Carlo

MARCH 28, 2024

Similar to how DevOps applies CI/CD to software development and operations, DataOps entails a CI/CD-like, automation-first approach to building and scaling data products. What is the Salary of DataOps vs. DevOps? What is the Salary of DataOps vs. DevOps? Table of Contents What Is The Difference Between DataOps and DevOps?

Data Pipeline 52

Data Pipeline Coding Machine Learning Data Warehouse 52

Data Engineering Weekly

MARCH 19, 2023

link] Uber: Setting Uber’s Transactional Data Lake in Motion with Incremental ETL Using Apache Hudi Uber writes a comprehensive guide on running incremental ETL using Apache Hudi. Map table vs. using complex data structure? How do you handle these models without compromising scale and usability?

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. or any other destination you choose.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

APRIL 5, 2021

In this episode Prukalpa Sankar shares her experiences across multiple attempts at building a system that brings everyone onto the same page, ultimately bringing her to found Atlan. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence. In this episode Sabin Thomas shares his experiences building the platform and the interesting ways that it is being used.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

MAY 15, 2022

Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Data Lake 100

Data Lake Building BI Architecture 100

Data Engineering Podcast

APRIL 3, 2022

Summary Building a data platform for your organization is a challenging undertaking. Building multiple data platforms for other organizations as a service without burning out is another thing entirely. He explains the common patterns that have been useful across multiple use cases, as well as when and how to build customized solutions.

Designing 100

Designing Data Warehouse Data Engineering Data Engineer 100

Data Engineering Weekly

OCTOBER 23, 2022

Oh, a humble request to the Data Catalog vendors, Please abstain from the poll ❤️ Rittman Analytics: The dbt Semantic Layer, Data Orchestration, and the Modern Enterprise Data Stack It has been an eventful last week for dbt with Coalesce conference. Is all-in-one data stacks the future?

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Podcast

AUGUST 3, 2021

By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. What are the most interesting, unexpected, or challenging lessons that you have learned while working on Hudi?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. When is Iomete the wrong choice?

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

FEBRUARY 22, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. field constraints, valid enums, etc.)

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Data Engineering Podcast

MAY 1, 2022

Summary Building a data platform is an iterative and evolutionary process that requires collaboration with internal stakeholders to ensure that their needs are being met. This is an excellent exploration of the decisions and tradeoffs that need to be made while building such a complex system.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. When is CDC the wrong choice?

Food 100

Food MongoDB Scala MySQL 100

dbt Developer Hub

MARCH 9, 2022

Special Thanks: Emilie Schario, Matt Winkler dbt has done a great job of building an elegant, common interface between data engineers, analytics engineers, and any data-y role, by uniting our work on SQL. Should we increase the SQL surface area to build ML models? I like to call this interoperability a “baton pass.”

Machine Learning 52

Machine Learning Raw Data SQL Python 52

Data Engineering Podcast

NOVEMBER 20, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Start trusting your data with Monte Carlo today!

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Weekly

MARCH 26, 2023

Last year around this time, Bundling vs. Unbundling was the talk of the town. I’m excited to listen to all the excellent speakers at the conference, and looking forward to the week of learning. dbt: State of Analytics Engineering dbt publishes the state of analytical [data???🤔] 🤔] engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Monte Carlo

OCTOBER 31, 2023

For those unfamiliar, Beane is the pioneering former general manager of the Oakland As who brought data & analytics to baseball, fielding a team of rookie players based on runs and getting to base vs. building a team around more expensive all-stars. Who better to learn from than the pioneers of data observability themselves?

Data Engineering 72

Data Engineering Data Engineer Data Engineering 72

Data Engineering Podcast

JANUARY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Vertically integrated vs. modular data stack?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Monte Carlo

JULY 5, 2023

We spoke with Mark Stange-Tregear, VP of Data at Babylist, and Michael Weiss, Senior Product Manager at Nasdaq, to learn how each has embarked on his respective journey to build a modern data stack premised on overarching data trust. Here are five lessons we learned from their stories. Databricks!)

High Quality Data 52

High Quality Data Data Software Engineer Software Engineering 52

Monte Carlo

JULY 7, 2022

Data contracts could become a key piece of the data quality puzzle, and much can be learned from Andrew’s experience implementing them at GoCardless, detailed below. 7 Key Learnings From Our Experience Implementing Data Contracts A perspective from Andrew Jones, GoCardless One of the core values at GoCardless is to ask why.

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Data Engineering Podcast

APRIL 12, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Data Warehouse 100

Data Warehouse Data Pipeline BI Data 100

Data Engineering Weekly

NOVEMBER 11, 2022

Little did I know I would be building data pipelines to measure system efficiency instead of measuring business efficiency. I worked with many incredible humans in reliability engineering; It’s fun, and there is always tons to learn every day. We got the dbt superpower to make things work; ignore the Snowflake billing.

Systems 40

Systems High Quality Data Consulting Metadata 40

Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold. What are the factors that convince teams to use a NoSQL vs. SQL database?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Towards Data Science

JULY 13, 2023

Building a maintainable and modular LLM application stack with Hamilton in 13 minutes LLM Applications are dataflows, use a tool specifically designed to express them LLM stacks. Hamilton is great for describing any type of dataflow , which is exactly what you’re doing when building an LLM powered application. Image from pixabay.

Building 74

Building Database-centric Database Coding 74

dbt Developer Hub

APRIL 19, 2023

Data modeling techniques on a normalization vs denormalization scale While the relevancy of dimensional modeling has been debated by data practitioners , it is still one of the most widely adopted data modeling technique for analytics. This tutorial aims to solve this by providing the definitive guide to dimensional modeling with dbt.

Building 145

Building PostgreSQL BI Database 145

Ascend.io

AUGUST 31, 2023

Data Exploration and Discovery: Since raw data is maintained and a variety of computation capabilities can be brought to bear, data lakes are excellent environments for data exploration, machine learning, and advanced analytics where unknown patterns need to be discovered from raw data.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

In this episode Eran Yahav shares the journey that he has taken in building this product and the ways that it enhances the ability of humans to get their work done, and when the humans have to adapt to the tool. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Christophe Blefari

NOVEMBER 11, 2022

I'll speak about "How to build the data dream team" Let's jump onto the news. When it comes to storage it's mainly a row-based vs. a column-based discussion, which in the end will impact how the engine will process data. Machine learning at Riot Games If you play video games like me you'll like this video.

BI 130

BI Data Warehouse Data Database 130

Monte Carlo

APRIL 16, 2022

The other camp is mad at me because their modern data stack is fundamentally not set up this way and it isn’t how they have been building out their data products,” said Chad. I’ll let you decide for yourself if the “immutable data warehouse” (or active vs passive ETL) is the right path for your data team.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Data Engineering Podcast

DECEMBER 25, 2022

Summary Encryption and security are critical elements in data analytics and machine learning applications. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

dbt Developer Hub

OCTOBER 17, 2022

So I couldn’t be more excited about dbt’s polyglot capabilities arriving in dbt Core 1.3. Going polyglot is a major next step in the journey of dbt Core. Polyglot dbt allows informed practitioners to choose the language that best fits your use case. Polyglot dbt: An alloy of Python, dataframes, and SQL ​ dbt Core 1.3

SQL 52

SQL Python Programming Language Data Analysis 52

dbt Developer Hub

AUGUST 17, 2022

When running a job that has over 1,700 models, how do you know what a “good” runtime is? While there are many possible answers depending on dataset size, complexity of modeling, and historical run times, the crux of the matter is normally “did you hit your SLAs”? hours to run. The model fct_dbt_invocations takes, on average, 1.5

Data Warehouse 59

Data Warehouse Datasets Cloud Coding 59

Christophe Blefari

DECEMBER 30, 2022

💡 You can also read the 2021's must-read that I've done one year and half ago or how to learn data engineering that contains key articles to understand the field. My guide about managing and scheduling dbt from dev to production. The best website to understand visually machine learning models. Joe Reis blog.

Data Warehouse 130

Data Warehouse BI Data Machine Learning 130

Christophe Blefari

AUGUST 4, 2023

Summer Edition ( credits ) This is the first article of the Data News Summer Edition: how to build a data platform. So I thought it was the perfect data to build a data platform. I tried to be as short as possible in this first article, details will come in the following ones. It will be a Google Sheets. I don't know yet.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

Data Engineering Weekly

MAY 21, 2023

Ramon Marrero: DBT Model Contracts - Importance and Pitfalls dbt introduces model contract with 1.5 There were a few critics of the dbt model implementation, such as The False Promise of dbt Contracts. I found the argument made in the false promise of the dbt contract surprising, especially the below comments.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52