Workfall

SEPTEMBER 18, 2023

In this blog, we’re diving into the world of data integration with Airbyte, unraveling the mystery behind its simplicity, and uncovering how it seamlessly connects with Snowflake to transform your data into actionable insights. Get ready to explore the realm where data dreams become reality!

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Christophe Blefari

DECEMBER 3, 2022

Using Airflow the wrong way — An experimental article I wrote where I explore Airflow as a framework rather than a all-in-one scheduler/orchestrator tool. — Brice wrote about dbt projects structure and foundations to have a working modelisation. What if we decide to schedule Airflow DAGs in Gitlab?

Kafka 130

Kafka BI Finance Data 130

dbt Developer Hub

OCTOBER 30, 2023

Hi all, I’m Kshitij, a senior software engineer on the Core team at dbt Labs. One of the coolest moments of my career here thus far has been shipping the new dbt clone command as part of the dbt-core v1.6 In this blog post, I’ll attempt to provide this guidance by answering these FAQs: What is dbt clone ?

BI 52

BI Datasets Metadata Data Warehouse 52

Data Engineering Weekly

JANUARY 21, 2024

[link] Teads: Unit testing with dbt Teads' blog post discusses unit testing with dbt, highlighting its advantages and limitations, especially regarding macros. The author engages in an interesting discussion on these shortfalls and acknowledges recent developments within the dbt community.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

FEBRUARY 18, 2024

[link] Sponsored: Data modeling and exploration in Playground 2.0 Register for our webinar to explore Cube Cloud and learn about the convenient UI for easier data modeling. A recent survey from dbt shows that 43% of the respondents answered that ambiguous data ownership was their biggest challenge.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Weekly

DECEMBER 3, 2023

Github writes an excellent blog to capture the current state of the LLM integration architecture. The blog is an excellent read to understand late-arriving data, backfilling, and incremental processing complications. The author explores the current complication of accessing blob storage and what can be improved to optimize the access.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Christophe Blefari

MAY 19, 2023

One day, I decided to save the links on a blog created for the occasion, a few days later, 3 people subscribed. As human beings, we wanted to fill the void, so we explored different things: real-time feature store, data lineage or data contracts—we call it this way today, but back in the days it was only schema management.

Data 130

Data Data Engineering Data Engineer Hadoop 130

Data Engineering Weekly

FEBRUARY 4, 2024

link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. How are you analyzing the cost of your infrastructure?

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Weekly

JUNE 18, 2023

The blog narrates LLM training options, Storage & retrieval, and the value chain to use LLM in your private data. link] Sponsored: [dbt + Monte Carlo Webinar] 5 Critical Steps for Building Impactful Data Organizations Building a data strategy from scratch? I’m super thrilled to see the blog. You're not alone!

Data Engineering 52

Data Engineering Data Engineer Engineering MySQL 52

Data Engineering Weekly

SEPTEMBER 24, 2023

Request Access to Data Validate with Exploration [link] Ramp: How Ramp Accelerated Machine Learning Development to Simplify Finance Ramp writes about its machine learning infrastructure and the choice of Metaflow for running the ML workload. The blog specifically focuses on the following areas of LLM implementations.

Data Engineering 67

Data Engineering Data Engineer Engineering Kafka 67

Towards Data Science

OCTOBER 17, 2023

We ran a $12K experiment to test the cost and performance of Serverless warehouses and dbt concurrent threads, and obtained unexpected results. In this blog we take a technical deep dive into the cost and performance of their serverless SQL warehouse product by utilizing the industry standard TPC-DI benchmark.

SQL 52

SQL Cloud Data Science AWS 52

Data Engineering Weekly

MARCH 30, 2023

We refer to a four-part article series that explores the data infrastructure landscape and the Software as a Service (SaaS) products available in data engineering, machine learning, and artificial intelligence. We contemplated how tools like DBT might extend their capabilities by tackling other aspects of the data stack, such as ingestion.

Machine Learning 52

Machine Learning Government AWS Data Engineering 52

Data Engineering Weekly

OCTOBER 23, 2022

Oh, a humble request to the Data Catalog vendors, Please abstain from the poll ❤️ Rittman Analytics: The dbt Semantic Layer, Data Orchestration, and the Modern Enterprise Data Stack It has been an eventful last week for dbt with Coalesce conference. Is all-in-one data stacks the future?

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Podcast

MAY 1, 2022

This is an excellent exploration of the decisions and tradeoffs that need to be made while building such a complex system. This is an excellent exploration of the decisions and tradeoffs that need to be made while building such a complex system.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Weekly

MARCH 5, 2023

The blog narrates how they overcome the challenge with the combination of Cassandra & ElasticSearch [link] Sponsored: [New Guide] The Ultimate Guide to Data Mesh Architecture If implementing data mesh is high on your list of priorities, you’re not alone. link] The blog highlights six key principles of the value creation of data.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

phData: Data Engineering

JULY 12, 2022

In this blog post, we’ll take a look at how data has historically been modeled by both of these types of resources, introduce the concept of analyst engineering, and explain how dbt bridges the gap between traditional IT resources and business-oriented resources. How Do You Implement a Data Model with dbt?

Data Engineering 52

Data Engineering Data Engineer SQL Data 52

Monte Carlo

MARCH 24, 2023

We’ll explore their definitions, purposes, and methods so you can ensure both data integrity and data validity in your organization. A failed Apache Airflow job or dbt model error could prevent data from continuing to flow to downstream tables. Learn more in our blog post 9 Best Practices To Maintain Data Integrity.

Data Validation 52

Data Validation Data Integration Data Cleanse Data Pipeline 52

Data Engineering Weekly

NOVEMBER 6, 2022

I plan to write a series of blogs on Schemata and Data Contract in the coming weeks. Martin kindly stepped in for me to give the update for my promised blog posts. Exploration means showing accommodations that have never been shown to the user, with the hope of finding those that will perform better than those currently shown.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Data Engineering Weekly

AUGUST 13, 2023

🔍 [link] Meta: Scaling the Instagram Explore recommendations system AI plays an important role in what people see on Meta’s platforms. link] Sponsored: Great Data Debate–The State of Data Mesh Since 2019, the data mesh has woven itself into every blog post, event presentation, and webinar.

Data Engineering 52

Data Engineering Data Engineer Engineering Food 52

Monte Carlo

JANUARY 25, 2023

In this blog post, we’ll discuss seven common data quality tests that you can perform during the ETL (Extract, Transform, Load) process to validate your data. We’ll also highlight the challenges of performing these tests and explore an alternative approach: data observability.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Monte Carlo

JULY 7, 2022

You can read more about Convoy’s approach from our blog with their Head of Product, Data Platform, Chad Sanderson, “ The modern data warehouse is broken.” There are multiple approaches to solving these issues and data engineers are still very much pioneers exploring the frontier of future best practices. More details here.

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Monte Carlo

SEPTEMBER 12, 2022

Maxime’s experience speaks for itself: He’s the creator of the popular orchestrator Apache Airflow , followed by Apache Superset , as well as CEO and co-founder of Preset , a fast-growing data exploration tool offering AI-enabled data visualization for modern companies. Plus, he was one of the first data engineers at Facebook and Airbnb.

BI 52

BI Data Governance ETL Tools Data Warehouse 52

Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. Contact Info Blog LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Christophe Blefari

MARCH 8, 2024

I mean, there is a big difference between podcast for instance and news blogging like I'm doing. The Explorer One year and half ago I had developed the Explorer. The Explorer is a search bar that let's you search over all the links that I have shared in the 132 Data News editions.

Data 130

Data Algorithm Database Education 130

Christophe Blefari

MARCH 31, 2023

We are slowly approaching the 2-years anniversary of the blog and the newsletter. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much. This newsletter is about money ( credits ) Dear readers, already 3 months done in 2023.

Bytes 130

Bytes Data Google Cloud Education 130

Christophe Blefari

JUNE 9, 2023

This way, I'll dedicate my Fridays to write original articles, explore ideas and preparing articles stock for the summer holidays. Data contracts, dbt and modeling Back to the roots, it's been a long time since I did not share dedicated stuff about dbt. that runs tests and alerts on top of dbt.

Data 130

Data Government SQL Coding 130

Christophe Blefari

JUNE 24, 2023

In the continuity to licence topic you can watch this great video about laptop-sized ML for text, with Open Source where Nick Burch explore what you can do today on a laptop and introduce greatly the Gen AI field. At the same time James from Snowflake wrote a blog post helping you to chose between Snowflake or Iceberg table format.

PostgreSQL 130

PostgreSQL Data Data Engineering Data Engineer 130

Data Engineering Weekly

DECEMBER 24, 2023

The author explores why data PMs have to be more cautious in planning and executing a rollout. I believe there is no one answer to it; the author explores the various organization model and their adoption of Mercado Libre. The blog explores how Google uses ML to improve the efficiency of ML workloads!

Data Engineering 62

Data Engineering Data Engineer Engineering Deep Learning 62

dbt Developer Hub

AUGUST 30, 2022

Semantic layer, Python model support, the new dbt Cloud UI and IDE… there’s a lot our product team is excited to share with you at Coalesce in a few weeks. But how these things fit together—because of where dbt Labs is headed—is what I’m most excited to discuss. REGISTER NOW What's new ​ dbt Core v1.3 dbt_artifacts v1.2.0:

Programming 52

Programming Consulting Python BI 52

Christophe Blefari

AUGUST 4, 2023

As a disclaimer, this may not quite make sense in a corporate context, but since this is my blog, I'll do what I want. However over the years I've met people working at these companies so I might have a few biais. Still, the idea of this post is to give you an overview of existing tools and how everything fits together.

Google Cloud 225

Google Cloud MongoDB NoSQL Data 225

dbt Developer Hub

JULY 26, 2022

At last year's Coalesce, Drew showed us the future 1 - a vision of what metrics in dbt could look like. As we evaluated the long term plans for what this part of dbt was going to become, we realized that naming it the Semantic Layer better reflected its capabilities and where we plan on taking it. What is the dbt Semantic Layer? ​

Metadata 52

Metadata BI Datasets SQL 52

phData: Data Engineering

FEBRUARY 16, 2023

At phData, we’re starting to see a sharp increase in clients who are looking to migrate to dbt Cloud from dbt Core. This is due to a number of reasons, which we covered in a previous blog post , but in short, it’s primarily due to a need to accelerate platform adoption outside of central IT teams.

Cloud 52

Cloud Database Data Ingestion Coding 52

dbt Developer Hub

MAY 2, 2022

dbt Cloud is a hosted service that many organizations use for their dbt deployments. cron schedule, API trigger), the jobs generate various artifacts that contain valuable metadata related to the dbt project and the run results. dbt Cloud provides a REST API for managing jobs, run artifacts and other dbt Cloud resources.

Cloud 52

Cloud Metadata Python Database 52

Data Engineering Weekly

DECEMBER 18, 2022

The author writes an exciting blog, Modern data stack in a Box!! The Percona blog walkthrough JSON support in the relational databases. The blog doesn’t leave any traces of the data sources they consume, but curious What are the data sources the private venture capital firms depend on? Thank you, Shopify data team.

Data Engineering 52

Data Engineering Data Engineer Engineering Relational Database 52

Data Engineering Weekly

MAY 28, 2023

link] Walmart: Exploring an Entity Resolution Framework Across Various Use Cases Entity resolution, a crucial process that identifies and links records representing the same entity across various data sources, is indispensable for generating powerful insights about relationships and identities.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

dbt Developer Hub

AUGUST 17, 2022

Here at dbt Labs, we used the Model Timing tab in our internal analytics dbt project to help us identify inefficiencies in our incremental dbt Cloud job that eventually led to major financial savings, and a path forward for periodic improvement checks. Also, this blog post represents a pretty technical deep dive.

Data Warehouse 59

Data Warehouse Datasets Cloud Coding 59

Data Engineering Weekly

JANUARY 15, 2023

” Julius Remigio - Why I moved my dbt workloads to GitHub and saved over $65,000 dbt lab’s recent abrupt pricing change triggers interesting conversations about value creation and the per-user billing model. The blog narrates the causal machine-learning approach in designing promotional artwork for their TV shows.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52