Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. My advice on this point is to learn from others.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

NOVEMBER 27, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Missing data? images, documents, etc.) Atlan is the metadata hub for your data ecosystem. Stale dashboards?

Data Process 100

Data Process Process Metadata Business Intelligence 100

phData: Data Engineering

SEPTEMBER 19, 2023

Want to learn more about data governance? Check out our Data Governance on Snowflake blog! Metadata Management Data modeling methodologies help in managing metadata within the data lake. Metadata describes the characteristics, attributes, and context of the data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Christophe Blefari

JUNE 16, 2023

Data warehouses are mutable, this is one of the many root causes proposed by Lucas. The past often changes, whether because of code or data. Data Documentation 101: Why? — Marie wrote best practices for establishing complete and reliable data documentation. This is metrics drift.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

RandomTrees

FEBRUARY 6, 2024

Generative AI corrects data imbalances, ensuring fair sentiment analysis on e-commerce platforms, enriches training data for natural language processing (NLP) tasks. Ingestion: The Art of Data Assimilation: Ensuring the digital document accurately reflects the original handwritten material.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Airbnb Tech

NOVEMBER 28, 2023

However, achieving the full data quality criteria required by Midas demands significant cross-functional investment to design, develop, validate, and maintain the necessary data assets and documentation. While this made sense for our most critical data, pursuing such rigorous standards at scale presented challenges.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Monte Carlo

FEBRUARY 12, 2024

To differentiate and expand the usefulness of these models, organizations must augment them with first-party data – typically via a process called RAG (retrieval augmented generation). Today, this first-party data mostly lives in two types of data repositories.

Database 72

Database Unstructured Data Data Pipeline Metadata 72

dbt Developer Hub

NOVEMBER 13, 2023

This mostly just worked – as long as you didn't have anything more complex than a CI job mixed in with a couple of production jobs – because important constructs like deferral in CI and documentation were only ever tied to a single job. To achieve this, configure your data warehouse, git repo and dbt Cloud account as shown above.

Cloud 52

Cloud Database-centric Metadata Data Warehouse 52

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

ProjectPro

FEBRUARY 8, 2023

It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., Glue uses ETL jobs for extracting data from various AWS cloud services and integrating it into data warehouses and lakes.

AWS 98

AWS Scala Metadata Data Lake 98

Towards Data Science

FEBRUARY 6, 2024

Order snapshots are stored in my own development area (image by the author) To prevent my extractions from impacting performance on the operational side, I queried this data regularly and stored it in a persistent staging area (PSA) within my data warehouse. Metadata update Data products need to be understandable.

Systems 82

Systems Raw Data Metadata Data Cleanse 82

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Ascend.io

FEBRUARY 28, 2024

This process reduces noise in the data, which is crucial for the effectiveness of AI algorithms, especially in complex predictive models and deep learning applications. Such comprehensive metadata management is crucial in adhering to privacy and compliance standards, safeguarding AI operations against potential legal and ethical pitfalls.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

MAY 18, 2023

A question to ask is how will these data lineage source solutions account for dbt models, Fivetran connectors, and their relationship to tables? If our organization decided to pack up and migrate from one table format or data warehouse to another, would all the work we put into maintaining our open source data lineage solution for naught?

Metadata 64

Metadata Data Lake Government Data Governance 64

Towards Data Science

DECEMBER 1, 2023

This tool automates ELT (Extract, Load, Transform) process, integrating your data from the source system of Google Calendar to our Snowflake data warehouse. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 72

Data Engineering Data Engineer Project Engineering 72

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

dbt Developer Hub

DECEMBER 19, 2023

The documentation will help you familiarize yourself with how our users interact with dbt. Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. New to dbt Cloud? ​

Metadata 52

Metadata Cloud Accessible Accessibility 52

Data Engineering Weekly

JULY 9, 2023

The platform approach to enable the citizen machine learning engineers is a great perspective while building both the Data & ML platform. Architectural patterns like Lambda Architecture and Kappa Architecture emerged to bridge the gap between real-time and batch data processing. Write SQL queries without learning SQL?

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Monte Carlo

FEBRUARY 26, 2024

A data catalog is a constantly updated inventory of the universe of data assets within an organization. It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems.

Metadata 52

Metadata Government Data Data Governance 52

phData: Data Engineering

OCTOBER 12, 2022

Hosted Doc Site for Documentation One of the most powerful features of dbt can be the documentation you generate. This documentation can give different users insight into where data came from, what the profile of the data is, what the SQL looked like, and the DAG to know where the data is being used.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

Monte Carlo

MARCH 28, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Hightouch interrupted SaaS eating the world in an attempt to shift the center of gravity to the data warehouse. Other common light transformations done within the ingestion phase are data formatting and deduplication.

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

dbt Developer Hub

OCTOBER 30, 2023

However, one of the questions I’ve received most frequently is guidance around “when” to clone that goes beyond the documentation on “how” to clone. that leverages native zero-copy clone functionality on supported warehouses to copy entire schemas for free, almost instantly. How is it different from deferral? How is this possible?

BI 52

BI Datasets Metadata Data Warehouse 52

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. Snowflake is a DataLake Platform Snowflake is moving beyond a SQL data warehouse.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Cloudera

SEPTEMBER 29, 2022

Data mesh conceptual hierarchy. Instead of having a central team that manages all the data for a company, the thinking is that the responsibility of generating, curating, documenting, updating, and managing data should be distributed across the company based on whichever team is best suited to produce and own that data.

Architecture 74

Architecture Data Architecture Metadata Data Warehouse 74

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Monte Carlo

DECEMBER 8, 2023

Enter: data provenance and data lineage. Data lineage is a visual tool that tracks the movement and transformations of data through various systems, processes, and applications. Data provenance is the record of metadata from data’s original sources, providing the historical context and authenticity of data.

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

DECEMBER 8, 2023

Enter: data provenance and data lineage. Data lineage is a visual tool that tracks the movement and transformations of data through various systems, processes, and applications. Data provenance is the record of metadata from data’s original sources, providing the historical context and authenticity of data.

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

APRIL 4, 2023

Data anomalies and incidents can be introduced across any point of this pipeline, which can make root cause analysis a fragmented and frustrating multi-tab process. Now teams can instantly get full visibility into how these systems may be impacting their data assets, all in a single pane of glass. A modified dbt model?

BI 64

BI Data Ingestion Metadata Data Pipeline 64

Monte Carlo

JUNE 28, 2022

Traditionally, data lakes held raw data in its native format and were known for their flexibility, speed, and open source ecosystem. By design, data was less structured with limited metadata and no ACID properties. Since then, Databricks has aggressively moved toward allowing users to add more structure to their data.

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

Grouparoo

FEBRUARY 9, 2022

The testing process is often performed during the initial setup of a data warehouse after new data sources are added to a pipeline and after data integration and migration projects. ETL testing can be challenging since most ETL systems process large volumes of heterogeneous data. Extract data from source systems.

Process 52

Process ETL System Data Warehouse Metadata 52

Monte Carlo

JULY 25, 2023

Data curation is the process of transforming and enriching larger amounts of raw data into smaller, more widely accessible subsets of data that provide additional value to the organization or the intended use case. For smaller organizations, this may be the only level of data curation. Here are some example architectures.

Raw Data 52

Raw Data Data Warehouse Data Architecture 52

Cloudera

DECEMBER 21, 2023

In the “assumptions” field, we see how the SQL AI Assistant looked over our data model; compared to what we’re looking for, it was able to find the right tables, columns, and joins needed to provide a query that will give us the list we’re looking for. And as a bonus, we even get the query written for us, saving us even more time!

SQL 67

SQL Data Warehouse Business Analyst Data Schemas 67

Towards Data Science

APRIL 3, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Hightouch interrupted SaaS eating the world in an attempt to shift the center of gravity to the data warehouse. Other common light transformations done within the ingestion phase are data formatting and deduplication.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Warehouse 98

Rockset

NOVEMBER 28, 2022

It works with existing streaming systems like Apache Kafka, Amazon Kinesis, and Azure Events Hubs, making it easier than ever to build a real-time data pipeline. Data warehouses were built for batch jobs, so we shouldn’t be surprised by this. _id This is a document’s unique identifier in Rockset. The upshot?

Data Warehouse 52

Data Warehouse PostgreSQL MongoDB Data Pipeline 52

Monte Carlo

JANUARY 23, 2024

The data warehouse is the mirror of the organization in many ways. If people don’t agree on what they call things in the data warehouse or what the definition of a metric is, then this lack of consensus will be reflected downstream. Data debt or data pipeline chaos. But maybe that’s OK.”

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98