Christophe Blefari

MARCH 1, 2023

You can also add metadata on models (in YAML). In a nutshell the dbt journey starts with sources definition on which you will define models that will transform these sources to something else you'll need in your downstream usage of the data. You can read dbt's official definitions. examples on BigQuery ).

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. For example, a single table named ‘Customers’ is actually an aggregation of metadata that manages and references several data files, ensuring that the table behaves as a cohesive unit.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Christophe Blefari

JANUARY 20, 2024

Every company out there has his own definition for the data engineer role. What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Reddit r/dataengineering wiki a place where some data eng definitions are written. Who are the data engineers? This is not.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Netflix Tech

DECEMBER 3, 2022

Obviously not all tools are made with the same use case in mind, so we are planning to add more code samples for other (than classical batch ETL) data processing purposes, e.g. Machine Learning model building and scoring. Workflow Definitions Below you can see a typical file structure of a sample workflow package written in SparkSQL. ???

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process.

Building 147

Building Metadata Machine Learning Data Integration 147

Monte Carlo

AUGUST 30, 2023

But there’s a lot to understand in order to both create the most value from your dbt tests and avoid leaning too heavily on a time-intensive process. Custom data tests are generally run every time a dbt `run` command is executed to process your transformations. And one of the most popular approaches is dbt testing.

SQL 52

SQL Datasets Database High Quality Data 52

Lyft Engineering

OCTOBER 3, 2023

Data quality is an amorphous term, with various definitions depending on the context. In Verity, we defined data quality as follows: Verity’s Definition of Data Quality The measure of how well data can be used as intended. Five aspects of data quality with the definition in italics and an example in quotes.

Big Data 84

Big Data Metadata Data Warehouse Data 84

AltexSoft

NOVEMBER 23, 2021

The traditional way of data integration involves consolidating disparate data within a single repository — commonly a data warehouse — via the extract, transform, load (ETL) process. If the transformation step comes after loading (for example, when data is consolidated in a data lake or a data lakehouse ), the process is known as ELT.

Process 69

Process Data Lake Metadata Data Warehouse 69

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. Lack of unified metadata makes it difficult to optimize pipelines, avoid redundant processing, and quantify data engineering costs.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

SEPTEMBER 11, 2022

Ananth Packildurai created Schemata as a way to make the creation of schema contracts a lightweight process, allowing the dependency chains to be constructed and evolved iteratively and integrating validation of changes into standard delivery systems. Atlan is the metadata hub for your data ecosystem.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

JULY 24, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What was your process for determining what subject areas to include in the book? Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Knowledge Hut

MARCH 7, 2024

Foresighted enterprises are the ones who will be able to leverage this data for maximum profitability through data processing and handling techniques. The greatest data processing challenge of 2024 is the lack of qualified data scientists with the skill set and expertise to handle this gigantic volume of data.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Airbnb Tech

NOVEMBER 28, 2023

To meet this challenge, we introduced the “Midas” process to certify our data. Starting in 2020, the Midas process, along with the work to re-architect our most critical data models, has brought a dramatic increase in data quality and timeliness to Airbnb’s most critical data. Align on the input components for assessing data quality.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Marc Lamberti

SEPTEMBER 22, 2023

That cake doesn’t get magicked into existence; it involves a process – a step-by-step recipe you carefully need to follow; otherwise, you will get something different. Digging into the official definition, you will see that Airflow is noted as an open-source platform that programmatically authors, schedules, and monitors workflows.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

Data Engineering Podcast

MAY 22, 2022

What are some of the ways that that manifests in the design and feature set? How have the design and goals of the platform changed/evolved since you first started working on it? What is the process for setting up a Flyte deployment? What are the user personas that you prioritize in the design and feature development for Flyte?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Towards Data Science

NOVEMBER 4, 2023

How to Become a Data Engineer As a data engineer, I am tasked to design efficient data processes almost every day. It was created by Spotify to manage massive data processing workloads. Typical Airflow architecture includes a schduler based on metadata, executors, workers and tasks. Consider this example below.

Data Engineering 82

Data Engineering Data Engineer Engineering BI 82

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. GraphQL Architecture The GraphQL architecture we use at LinkedIn is unique due to some of the decisions we took during the design phase. specifically for GraphQL.

Architecture 109

Architecture Metadata Java Transportation 109

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. This process is cumbersome, but it is on purpose. on FIELD_DEFINITION For example, it is used in a query definition as follows - type CustomerConsent { status : String ! We will discuss this in detail in the next section.

Metadata 86

Metadata Coding Banking Designing 86

DoorDash Engineering

DECEMBER 5, 2023

Almost six billion messages are processed each day at an average rate of four million messages per minute, which sometimes peaks at double that rate. The slowest step in that process was provisioning Kafka Topics, which involved on-call engineer approval and was prone to failure, further increasing the on-call load.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

OCTOBER 7, 2021

The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., Data and Metadata: Data inputs and data outputs produced based on the application logic. Key Design Principles of a Data Mesh.

Architecture 124

Architecture Metadata Government Kafka 124

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. We will also dive deep into our design and implementation processes and the lessons we learnt.

SQL 82

SQL Metadata Raw Data Government 82

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets.

Data Governance 95

Data Governance Government Metadata Datasets 95

Data Engineering Podcast

JANUARY 7, 2022

Summary Data observability is a set of technical and organizational capabilities related to understanding how your data is being processed and used so that you can proactively identify and fix errors in your workflows. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow.

BI 100

BI Data Warehouse Metadata SQL 100

Tweag

APRIL 19, 2023

Most training pipelines and systems are designed to handle fairly small, sub-megapixel images. To allow innovation in medical imaging with AI, we need efficient and affordable ways to store and process these WSIs at scale. A single hospital makes many captures a day, producing terabytes of such data to store and process.

Medical 60

Medical Process Cloud Bytes 60

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Cloudera

FEBRUARY 8, 2024

It has been designed and developed as an open community standard to ensure compatibility across languages and implementations. We populated the tables using INSERT-SELECT statements by reading from text format source tables but they can be populated through any ETL process. Such a query pattern is quite common in BI queries.

Metadata 90

Metadata Data Warehouse BI AWS 90

Ascend.io

FEBRUARY 28, 2024

Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Cloudera

NOVEMBER 13, 2020

Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? We’ll discuss the various phases Impala takes a query through and how small query optimizations are incorporated into the design of each phase. Query Planner Design. Metadata Caching.

Metadata 143

Metadata Coding SQL Database 143

RandomTrees

FEBRUARY 6, 2024

Generative AI corrects data imbalances, ensuring fair sentiment analysis on e-commerce platforms, enriches training data for natural language processing (NLP) tasks. Data’s exponential growth : Gen AI holds the potential to address this challenge by automating data processing tasks, extracting valuable insights from the vast amounts of data.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

AltexSoft

AUGUST 22, 2022

Some would say that it’s not a big deal, however, these mixed environments have resulted in the complexities of managing disjointed data and business processes. It is a general design approach that leverages data virtualization principles. How organizations embrace multi-cloud. Source: Flexera. What is data fabric? API support.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

LinkedIn Engineering

JUNE 26, 2023

For streaming data pipelines, this involves writing stream processing jobs. A developer would need to write and operationalize a custom stream processing job to replicate their Brooklin datastream into a Kafka topic. This means that data pipelines almost always require some form of stream processing in the middle.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Data Engineering Podcast

MAY 18, 2021

Molecula is an enterprise feature store that operationalizes advanced analytics and AI in a format designed for massive machine-scale projects without having to manage endless one-off information requests. What is the current state of the ecosystem for generating and sharing metadata between systems?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Knowledge Hut

JULY 24, 2023

The ability of a DBMS to change its schema definition at one level without affecting the schema definition at the next level is called data independence. The system stores metadata about data which makes it easier to find and retrieve data. But why do we need data integrity in a DBMS? Physical Data Independence in DBMS 1.

Database Design 52

Database Design Relational Database Metadata Database 52

François Nguyen

MARCH 22, 2021

This is really for us the definition of a self serve platform. We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing. and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…).

Technology 130

Technology Architecture Google Cloud Metadata 130

ProjectPro

FEBRUARY 8, 2023

For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. It offers a simple and efficient solution for data processing in organizations. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog.

AWS 98

AWS Scala Metadata Data Lake 98

Engineering at Meta

OCTOBER 31, 2023

In our first post on automating product deprecation , we discussed the complexities of product deprecations and introduced SCARF’s workflow management tools that guide engineers through a coordinated process to safely deprecate products. If any of the assets are not in use, SCARF initiates a process to safely remove them.

Data 93

Data Metadata Coding Relational Database 93

Cloudera

AUGUST 16, 2021

Apache Nifi is a powerful tool to build data movement pipelines using a visual flow designer. Downscaling is even more involved because users have to make sure that the NiFi node they want to decommission has processed all its data and does not receive any new data to avoid potential data loss. and later). and later).

Cloud 119

Cloud Unstructured Data Utilities Metadata 119