Christophe Blefari

MARCH 4, 2023

This week I've published an compact article about how to get started with dbt. The idea behind this article is to define every dbt concept and objects from the CLI to the Jinja templating or models and sources. I think the SCD type 2 is the most common and lossless one, but other are worth mentioning.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Christophe Blefari

MARCH 17, 2023

On my side I'm slowly starting to get on top of the things I had in queue. Which means that I get easily disturbed by a notification—or even a thought—and do something that I did not plan to do at first. It, probably, explains why you always get the newsletter late on Fridays—or Saturdays.

Data 130

Data SQL Deep Learning Kafka 130

Christophe Blefari

JANUARY 27, 2023

Every week to be honest I get hit by Friday. Every week you get a taste of my current mood. In my manage and schedule dbt guide in a nutshell I say that in dbt projects you have 2 lifecycles. The first one is the developing experience and the second is the dbt runtime. I don't write in advance. SQLAlchemy 2.0

Data 130

Data Algorithm Data Pipeline SQL 130

Data Engineering Podcast

OCTOBER 8, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. How can you get the best results for your use case?

Insurance 162

Insurance BI SQL Machine Learning 162

Christophe Blefari

JULY 8, 2023

The new dbt Semantic Layer Following the acquisition of Transform by dbt Labs a few months ago, dbt Core integrates MetricsFlow. This week, Nick Handel, co-founder of ex-Transform, wrote about how dbt Core specs will adapt. In a dbt model, you can define primary and foreign entities.

Kafka 130

Kafka PostgreSQL Transportation Data 130

Christophe Blefari

JANUARY 20, 2023

This week I don't want to be late, so let's start the weekly wrap-up. Perfect your modeling techniques ( credits ) Fast News ⚡️ Why I moved my dbt workloads to GitHub and saved over $65,000 — With the dbt Cloud price increase I already shared companies started to look for innovative way to run dbt.

Google Cloud 130

Google Cloud Data Hadoop Business Intelligence 130

Data Engineering Podcast

AUGUST 6, 2023

In this episode Barr Moses and Anna Filippova explore that question and provide useful exercises to start answering that in your company. How do your respective solutions, Monte Carlo and dbt, help teams measure and scale data value? So how do you calculate the return on investment for data?

Machine Learning 130

Machine Learning SQL Data Data Engineering 130

Christophe Blefari

DECEMBER 9, 2022

After an small introduction the evening started with a presentation by Clément and Steff from leboncoin data engineering team. Then Qonto data engineering team with Charles & Charles shared how they integrated dbt within Airflow. They shared with us the good practices they implemented to scale their Airflow development.

SQL 130

SQL Data Data Science Metadata 130

Data Engineering Podcast

OCTOBER 22, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free! How can you get the best results for your use case? With Materialize, you can!

BI 162

BI SQL Machine Learning Programming Language 162

Towards Data Science

OCTOBER 25, 2023

Monolithic data architecture is still the norm — even though there are clear drawbacks.” —  dbt labs In this article, I will introduce one of those techniques: contract testing. I will show how you can use dbt to create simple contract tests for your upstream sources and your dbt models’ public interfaces.

Data Pipeline 82

Data Pipeline SQL Data Architecture Data 82

Christophe Blefari

MARCH 22, 2024

I write in one-go starting from top to bottom. dbt Core unit testing in v1.8 — dbt Core has implemented unit testing and it's coming soon. dbt will run and validate the model for you. The email notification feature is almost ready so opt-in on the reco page to get your recommended link by email once ready.

Electronics 130

Electronics Media Data Python 130

Data Engineering Podcast

OCTOBER 1, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!

Building 162

Building BI SQL Machine Learning 162

Christophe Blefari

FEBRUARY 2, 2024

is coming soon and before end of July we might get the v1.0.0. DuckDB adoption numbers are demonstrating a real trend behind the "hype" DuckDB docs website gets 500k unique visitors per month and DuckDB has a new shiny website. DuckDB announcements The Duck creators announced that v0.10.1

Datasets 130

Datasets Data Lake SQL Algorithm 130

Data Engineering Podcast

OCTOBER 30, 2022

Summary One of the most impactful technologies for data analytics in recent years has been dbt. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus. To help simplify the adoption and management of dbt projects Nandam Karthik helped create Optimus.

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 27, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!

Database 130

Database Building PostgreSQL BI 130

Data Engineering Podcast

AUGUST 13, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production.

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Christophe Blefari

SEPTEMBER 28, 2023

When you are a data engineer you're getting paid to build systems that people can rely on. Well, it all depends on whether you're a newcomer and want to start building your data platform, or whether you already have a stack and are wondering what to do next. Good data layers are a good start. Something boring.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Towards Data Science

JULY 10, 2023

A simple framework for building dbt models that actually get used. When I was researching the Ultimate Guide to dbt , I was shocked by the lack of material around actually building models from scratch. This article is the result of research and experiments into how best to design and implement a dbt model.

Designing 98

Designing Electronics Coding Engineering 98

Christophe Blefari

FEBRUARY 3, 2023

You can also get inspired by Stuart's modern data stack. Microsoft Azure announced managed Airflow — Starting this week you'll be able to launch Apache Airflow within Azure Data Factory. You can also use the dbt --defer option which does something similar. Analytics Engineer- A Glorified BI Engineer?

BI 130

BI Google Cloud SQL Machine Learning 130

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. Photo by Gaining Visuals on Unsplash Here is a simple and fast pet project for Data/Analytics Engineers, who want to kick the tires on Modern Data Stack tools including dbt Core, Snowflake, Fivetran, and GitHub Actions.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Towards Data Science

AUGUST 19, 2023

I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded. This can mean pursuing hobbies outside of your 9–5 that bring you joy, or getting involved with different teams at the company you already work at. They were like data engineers with way more business exposure!

Data Engineering 83

Data Engineering Data Engineer Engineering SQL 83

Marc Lamberti

SEPTEMBER 24, 2023

If you are new to Airflow, check out my course here; you will get it at a special discount. But you get the point. Otherwise, you will get an error. Either way, you get an error if you forget to add the prefix to the task ids in the task group. Amazing, isn’t it? I’m kidding.

Coding 130

Coding Python Process Data Pipeline 130

Christophe Blefari

APRIL 8, 2023

Simon started a multipart guide about data modeling. When it comes to modeling it's hard not to mention dbt. In the recent years dbt simplified and revolutionised the tooling to create data models. dbt, as of today, is the leading framework. I hope he will fill the gaps. But alternatives are coming.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

Simon started a multipart guide about data modeling. When it comes to modeling it's hard not to mention dbt. In the recent years dbt simplified and revolutionised the tooling to create data models. dbt, as of today, is the leading framework. I hope he will fill the gaps. But alternatives are coming.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Data Engineering Podcast

OCTOBER 22, 2021

One of the driving forces for that change has been the rise of analytics engineering powered by dbt. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. Get started for free at dataengineeringpodcast.com/hightouch.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

MARCH 5, 2023

And let's not even get started on all the extra tools you have to buy to get it to do its thing. Head over to dataengineeringpodcast.com/timextender where you can do two things: watch us build a data estate in 15 minutes and start for free today. It ends up being anything but that. It ends up being anything but that.

Building 147

Building Database Design Metadata Machine Learning 147

Monte Carlo

FEBRUARY 9, 2024

Getting their data quality in order. We’re working on our infrastructure and getting the right tooling in place, making sure we have the right data quality under the hood,” said Dustin. A vision of where the dbt Semantic Layer should sit within a modern data stack architecture via dbt. “If

Data 64

Data High Quality Data Architecture Coding 64

Workfall

AUGUST 14, 2023

Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. In this blog, we will cover: Understanding DBT and Airflow What is DBT?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Data Engineering Podcast

FEBRUARY 19, 2023

And let's not even get started on all the extra tools you have to buy to get it to do its thing. Head over to timextender.com/dataengineering where you can do two things: watch us build a data estate in 15 minutes and start for free today. It ends up being anything but that. But don't worry, there is a better way.

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

JULY 17, 2022

Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

FEBRUARY 8, 2021

One of the best ways to get a true understanding of how a technology works in practice is to hear from people who are running it in production. In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify. Can you start by giving an overview of what the Shopify platform is?

Data Warehouse 100

Data Warehouse Building BI SQL 100

Data Engineering Podcast

JUNE 19, 2022

Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Christophe Blefari

JULY 15, 2023

Yesterday I found a way to get sensor data of half of the Tour de France peloton, I was sure it was a good dataset to explore new tools with. They use Fivetran, dbt and Census. In order to get started they first built a cost dashboard using information_schema.jobs tables. And here we are on Saturday.

Datasets 130

Datasets Python Machine Learning Data 130

Christophe Blefari

DECEMBER 3, 2022

Modern Data Modeling: Start with the End? — Brice wrote about dbt projects structure and foundations to have a working modelisation. Introduction to Data Modeling — Data modeling is the trendy skill everyone wants to learn today, dbt and the Analytics Engineering trends put modeling back in the front seat.

Kafka 130

Kafka BI Finance Data 130

Workfall

JULY 18, 2023

But then, a game-changer emerged – DBT (Data Build Tool). With DBT’s materializations, our data transformations underwent a magical transformation themselves. In this blog, we’ll whisk you away on an enchanting journey through DBT materializations. In this blog, we will cover: What is DBT? DBT , the Data Build Tool.

Datasets 77

Datasets Entertainment Data Workflow Data 77

Christophe Blefari

JUNE 3, 2023

A dynamic prompt — (Paragraphica camera) Fast News ⚡️ Meltano announced their Cloud — Meltano is an open-source data integration project that has been started at Gitlab. If you want a quick intro about the tool this is the way to start. dbt at Zendesk ; setting foundations for scalability.

Data Pipeline 130

Data Pipeline Data SQL Algorithm 130

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. With DBT, they weave powerful SQL spells to create data models that capture the essence of their organization’s information. In DBT, transformations are like these artisans.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69