dbt Developer Hub

AUGUST 23, 2022

Those who have been building data warehouses for a long time have undoubtedly encountered the challenge of building surrogate keys on their data models. How were surrogate keys managed in the past? ​ How were surrogate keys managed in the past? How can you do this with dbt? Let’s dive in.

Datasets 52

Datasets Metadata SQL Data Warehouse 52

Data Engineering Weekly

MARCH 19, 2023

The blog discusses implementing Type-2 SCD modeling and strategies to generate surrogate keys and bridge tables to handle many-to-many relationships. Try it free Netflix: Building a Media Understanding Platform for ML Innovations Netflix recently wrote a series of blogs about its media ML platform.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Maxime Beauchemin

JANUARY 20, 2017

In larger environments, there tends to be specialization and the creation of a formal role to manage this workload, as the need for a data infrastructure team grows. It’s also fairly common for engineers to develop and manage their own job orchestrator/scheduler.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Snowflake

SEPTEMBER 11, 2023

We covered this in depth in a previous blog post. For that second bullet point, we infer the matching load dates and keys by using the Window LAG function with the “IGNORE NULLS” option, like so: , coalesce(s1.dv_loaddate Return the matched load date and surrogate key from the adjacent satellite table for a snapshot date.

SQL 68

SQL Data Raw Data Architecture 68

Maxime Beauchemin

JANUARY 7, 2018

Eliminating side effects , i.e., changes in state that do not depend on the function inputs, can make it much easier to understand and predict the behavior of a program, which is one of the key motivations for the development of functional programming. To put it simply, immutable data along with versioned logic are key to reproducibility.

Data Engineering 183

Data Engineering Data Engineer Data Process Process 183

RandomTrees

NOVEMBER 29, 2023

Auditability and traceability become more easier here facilitating business key usages and can accommodate almost a variety of data formats and standards also keeping in mind to make it easy for data profiling and cleansing methods. The dimension table usually contains multiple columns and among them it would also have a primary key column.

Data Warehouse 52

Data Warehouse Data Lake Database-centric Data Cleanse 52