Christophe Blefari

APRIL 21, 2023

Analytics engineering future This week Tristan Handy—dbt Labs CEO—wrote a post about the future of analytics engineering: The next big step forwards for analytics engineering. As introduction Tristan gives the original vision of dbt that became mainstream, today. In dbt Core 1.5 CoreDB raises $6.5m seed round.

Raw Data 130

Raw Data Data Datasets SQL 130

Christophe Blefari

JANUARY 27, 2023

I often try to sync my travels on Fridays, even if internet is terrible in the train, this is still a good way to fill the +8 hours travel time I'm used to. Today I take the following commitment: I will never use any generative algorithm to write something in the newsletter. It might be a good use-case for GPTZero.

Data 130

Data Algorithm Data Pipeline SQL 130

Christophe Blefari

SEPTEMBER 28, 2023

The modern data stack has always been nice words to bundle a philosophy used to build data platform. With a handy warehouse at the center and multiple SaaS tools revolving around to answer useful—sometimes not—use-cases. A lot of companies started with Postgres + dbt + Metabase, don't be ashamed.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Data Engineering Podcast

AUGUST 13, 2023

Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. How do the application of these principles influence the ability of organizations and their data teams to encourage and keep pace with the use of data in the business?

Data Pipeline 130

Data Pipeline BI SQL Business Intelligence 130

Christophe Blefari

SEPTEMBER 15, 2023

Gen) AI News 🤖 Reinforcement Learning: an easy introduction to value iteration — Title says easy, but the article contains maths formula. The research also shown that people using Github Copilot feel more 88% more productive and are more efficient and less frustrated. But don't worry I don't forget you.

Data Warehouse 130

Data Warehouse Data SQL Python 130

Data Engineering Podcast

JULY 9, 2023

The major strategies in use today were created decades ago when the software and hardware for warehouse databases were far more constrained. dbt vs. informatica vs. ETL scripts, etc.) What is the impact on the underlying compute engine on the modeling strategies used? How does it compare to dimensional modeling strategies? (e.g.

Database-centric 130

Database-centric Machine Learning SQL Data Engineering 130

Towards Data Science

OCTOBER 25, 2023

This approach was effective for small organizations and startups with limited data sources and use cases. However, as organizations prioritized being data-driven, there was an increased need for more machine learning, analytics, and business intelligence data use cases. How did they do it?

Data Pipeline 85

Data Pipeline SQL Data Architecture Data 85

Data Engineering Weekly

APRIL 7, 2024

dbt: 2024 State of Analytics Engineering The 2024 dbt’s state of analytical engineering report is out. link] Matt Turck: Full Steam Ahead: The 2024 MAD (Machine Learning, AI & Data) Landscape Coninue the week of insights into the world of data & AI landscape, the 2024 MAD landscape is out.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Workfall

JULY 18, 2023

But then, a game-changer emerged – DBT (Data Build Tool). With DBT’s materializations, our data transformations underwent a magical transformation themselves. In this blog, we’ll whisk you away on an enchanting journey through DBT materializations. In this blog, we will cover: What is DBT? DBT , the Data Build Tool.

Datasets 77

Datasets Entertainment Data Workflow Data 77

Monte Carlo

FEBRUARY 9, 2024

Although, we did create the above image using DALL-E, and I think you could say it’s pretty meaningful.) And now GenAI offers even more opportunities to do just that—with organizations like Atlassian already offering early examples of this use case. And GenAI is no different. Interested? Let’s dive in. Table of Contents 1.

Data 64

Data High Quality Data Architecture Coding 64

Data Engineering Weekly

FEBRUARY 18, 2024

Visit rudderstack.com to learn more. Midway through organizing the conference, fear washed over us. Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Podcast

FEBRUARY 8, 2021

In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify. This is a great conversation about the lessons learned from real world use of a specific technology and how well it lives up to its promises. You ultimately decided to standardize on DBT.

Data Warehouse 100

Data Warehouse Building BI SQL 100

Data Engineering Weekly

MARCH 3, 2024

Visit rudderstack.com to learn more. by Aswin James Christy( Qlik/Talend) Aswin establishes a case for why you should start thinking about building data products. GraphRAG significantly improves question-and-answer performance over traditional vector similarity techniques using LLM-generated knowledge graphs for document analysis.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

JUNE 19, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. What is the workflow for a business user that is using Canvas to iterate on a series of questions?

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Marc Lamberti

SEPTEMBER 24, 2023

As of today, SubDAGs are deprecated, and you should stop using them. Use Case To better illustrate a concept, let’s begin with a use case: In this tutorial, we will use taskgroups to organize the giant DAG above. Therefore, Taskgroups are lightweight and easy to use. When to use an Airflow Taskgroup?

Coding 130

Coding Python Process Data Pipeline 130

Data Engineering Podcast

NOVEMBER 6, 2022

Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models. In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Weekly

DECEMBER 3, 2023

Visit rudderstack.com to learn more. link] Netflix: Incremental Processing using Netflix Maestro and Apache Iceberg Netflix writes about its incremental processing design with its orchestration engine Maestro on top of Iceberg. I like the 3G model with Guardrails, Guidelines & Gadget, which I’m sure I will use more often :-).

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Data Engineering Weekly

AUGUST 6, 2023

the best way to run dbt Core in Airflow. The blog narrates how to schedule dbt jobs in Airflow by parsing dbt's manifest.json file, and auto construct Airflow tasks. At Data Engineering Weekly, I wrote an article 15 months back narrating how dbt is not a separate model but part of the larger orchestration system.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

JANUARY 21, 2024

Visit rudderstack.com to learn more. Sebastian Raschka: Ten Noteworthy AI Research Papers of 2023 It is an eventful 2023 in machine learning and AI research. RudderStack just announced a new ML product called Predictions that can use any data in your warehouse to automatically produce churn and conversion scores without MLOps.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Towards Data Science

JUNE 16, 2023

What data is business-critical Data used for decision-making is important and if data is incorrect it may lead to wrong decisions and over time a loss of trust in data. Identifying your business-critical dashboards Looker exposes metadata about content usage in pre-built Explores that you can enrich with your own data to make it more useful.

BI 82

BI Data ETL Tools Machine Learning 82

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. With DBT, they weave powerful SQL spells to create data models that capture the essence of their organization’s information. In DBT, transformations are like these artisans.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Workfall

AUGUST 14, 2023

Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. By combining DBT and Airflow , you get the best of both worlds.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Towards Data Science

APRIL 5, 2023

A Guide to Maximize Your dbt Productivity in Visual Studio Code (Image from Unsplash ) If you are struggling to get VS Code and dbt to work well together, you are not alone. or higher and a virtual environment with dbt installed. A dbt project. This article does not go into detail on how to set up a dbt project.

Coding 82

Coding SQL Python Data Science 82

Monte Carlo

DECEMBER 4, 2023

Ah, dbt unit testing. Learn from these six common missteps in dbt unit testing so you can sidestep them in your own data quality journey. Table of Contents Wait, can’t I avoid mistakes by just skipping dbt unit testing? Define the test in your dbt model’s YAML file 2. Slightly annoying? Absolutely.

Datasets 52

Datasets SQL Data Pipeline Data Engineering 52

Christophe Blefari

MARCH 24, 2023

Arrow is a multi-langage interface to use in-memory data structures. Datafold's data-diff now integrates dbt — You can now run data diff after a dbt run to compare your models to the production state and get a summarised view of the rows impacted. — A good list of use-cases where data lineage would be useful.

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Data Engineering Weekly

APRIL 30, 2023

DoorDash identifies Five big areas for using Generative AI. Intuit provides insightful information about performing a set of tests called machine learning model sanity checks in a pre-production environment. We could instead use a node-cluster mapping table and have each edge add a mini-cluster to it upon initialization.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. and transformation (Spark, dbt, Pandas, etc.) What is Hadoop?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

dbt Developer Hub

JULY 2, 2023

Data Vault is the dam and reservoir: it is the well-engineered data model to structure an organization’s data from source systems for use by downstream data projects – rather than each team collecting data straight from the source. They allow for more flexibility and extensibility and can be used to model complex processes in an agile way.

Cloud 52

Cloud Data Warehouse Data BI 52

Monte Carlo

DECEMBER 13, 2023

Inspired by the practices of DevOps observability, data observability uses automated monitoring , alerting, and triaging — along with end-to-end lineage — to give organizations the ability to fully understand their data health. Win, win, win, win, win. Don’t get too complicated.

BI 52

BI Data Ingestion SQL Data Warehouse 52

dbt Developer Hub

MAY 16, 2023

At Lunar , most of our dbt models are sourcing from event-driven architecture. I also made the following meme in the dbt Community Slack channel #memes-and-off-topic-chatter to encapsulate this method: Meme of writing documentation What pain is being solved? ​ What will you learn after reading this article? ​

Coding 59

Coding Project SQL Utilities 59

Monte Carlo

NOVEMBER 27, 2023

No one would use a mobile banking or navigation app that crashed every 4 hours. Monte Carlo has accelerated the root cause analysis of this data freshness incident by correlating it to a dbt model error resulting from a GitHub pull request where the model code was incorrectly modified with the insertion of a semi-colon on line 113.

Unstructured Data 59

Unstructured Data Data Pipeline Data Banking 59

Data Engineering Weekly

APRIL 16, 2023

1 On dbt; Yes, I completely agree with the author’s take that it lacks the basic functionality expected of a best-in-class tool. . #1 1 On dbt; Yes, I completely agree with the author’s take that it lacks the basic functionality expected of a best-in-class tool. Airflow is obsolete. 2 I agree; permission is a mess.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Workfall

SEPTEMBER 18, 2023

Benefits of Airbyte Best Practices for Using Airbyte with Snowflake 5 Real-world Use Cases for Using Airbyte with Snowflake Hands-On Conclusion What is Airbyte? This means that business owners, analysts, and even beginners can wield its power without a steep learning curve. Clone the repo: git clone [link].

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

dbt Developer Hub

JULY 19, 2022

Stored procedures are widely used throughout the data warehousing world. However, as teams continue building their transformation logic using the stored procedure approach, we see more data downtime, increased data warehouse costs, and incorrect / unavailable data in production. What are the problems with stored procedures?

Data Pipeline 52

Data Pipeline SQL Data Warehouse Coding 52

Monte Carlo

AUGUST 8, 2023

Writing tests that pass isn’t a useful process and of itself, it’s a means to an end. For example, your data consumers might need live data for an operational use case, but you chose to go with batch data ingestion. In these cases it is important to understand data lineage.

Data Validation 52

Data Validation Data Pipeline SQL Data 52

dbt Developer Hub

JANUARY 14, 2023

Finally, I will use dbt Core for transformation. A transformation layer taking the source data loaded by my dlt functions and creating the tables necessary, handled by dbt. When I'm happy with the results, I use the ready-made dlt Sheets source connector to pull the data back into BigQuery, as defined here.

Data Ingestion 52

Data Ingestion Google Cloud Python Cloud 52

Data Engineering Weekly

SEPTEMBER 24, 2023

Request Access to Data Validate with Exploration [link] Ramp: How Ramp Accelerated Machine Learning Development to Simplify Finance Ramp writes about its machine learning infrastructure and the choice of Metaflow for running the ML workload. It’s the same with machine learning.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67