Monte Carlo

AUGUST 30, 2023

Often, teams run custom data tests as part of a deployment pipeline, or scheduled on production systems via job schedulers like Apache Airflow, dbt Cloud, or via in-built schedulers in your data warehouse solution. With dbt tests, no news is good news. Common dbt testing use cases Time to make all these how-to instructions come to life.

SQL 52

SQL Datasets Database High Quality Data 52

Monte Carlo

JULY 11, 2023

Examples of inaccurate data Inaccurate data can arise in a few different ways, all primarily due to manual errors, system hiccups, code changes, or data pipeline breakages. Below, we list a few examples: Data entry errors: Mistakes made during the process of entering data into a system can lead to inaccuracies.

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40

Tweag

NOVEMBER 1, 2023

The last important remaining piece to explore is the merge system. In this post, we’ll see how to use the Nickel merge system to write reusable configuration modules, and why the merge approach seems to be more adapted for modular configurations than plain functions, despite Nickel being a functional language. s t r i n g.

Metadata 104

Metadata Programming Programming Language Coding 104

Data Engineering Podcast

NOVEMBER 6, 2022

He also describes the considerations involved in bringing behavioral data into your systems, and the ways that he and the rest of the Snowplow team are working to make that an easy addition to your platforms. Atlan is the metadata hub for your data ecosystem. What are some of the unique characteristics of that information?

Building 130

Building IT Metadata MongoDB 130

Lyft Engineering

OCTOBER 3, 2023

Data quality is an amorphous term, with various definitions depending on the context. In Verity, we defined data quality as follows: Verity’s Definition of Data Quality The measure of how well data can be used as intended. Five aspects of data quality with the definition in italics and an example in quotes.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 131

MySQL Metadata Data Validation Coding 131

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. As data pipelines sprawl across disparate systems, it becomes harder to trace data lineage, track data quality, and quickly pinpoint issues when they arise.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Christophe Blefari

JANUARY 20, 2024

Every company out there has his own definition for the data engineer role. What is data engineering As I said it before data engineering is still a young discipline with many different definitions. Reddit r/dataengineering wiki a place where some data eng definitions are written. Who are the data engineers? Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

FEBRUARY 26, 2023

What are the technical systems that you are relying on to power the different data domains? What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Metadata Machine Learning Data Integration 147

Monte Carlo

JULY 10, 2023

That’s where the definition of data completeness comes in. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. This can lead to data that’s incomplete in one system, even if it’s present in another. The same is true with data.

Data Collection 40

Data Collection Data Governance Government Data 40

Data Engineering Podcast

JULY 24, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the definitions shifted over the past few decades? Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. What is your working definition of "data governance" and how does that influence your product focus and priorities?

Data Governance 130

Data Governance Government Cloud Building 130

Knowledge Hut

MARCH 7, 2024

Syncing Across Data Sources Once you import data into Big Data platforms you may also realize that data copies migrated from a wide range of sources on different rates and schedules can rapidly get out of the synchronization with the originating system. Two, it creates a commonality of data definitions, concepts, metadata and the like.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. The type system directives are available for the locations listed below. on FIELD_DEFINITION For example, it is used in a query definition as follows - type CustomerConsent { status : String ! are some entities.

Metadata 86

Metadata Coding Banking Designing 86

LinkedIn Engineering

APRIL 25, 2023

Here are some of the primary differences compared to the widely used architecture in the industry: The GraphQL type system used at LinkedIn is completely autogenerated. Type System Generator The GraphQL type system used at LinkedIn is autogenerated by federating individual entity schemas from the Rest.li Given that our Rest.li

Architecture 109

Architecture Metadata Java Transportation 109

Airbnb Tech

NOVEMBER 28, 2023

We identified the following objectives for the score: Evolve our understanding of data quality beyond a simple binary definition (certified vs uncertified). Luckily, we had two existing tools that would make this much easier: Dataportal (Airbnb’s data catalog and exploration UI), and the Unified Metadata Service (UMS).

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

MAY 22, 2022

Get started with Kyligence today at dataengineeringpodcast.com/kyligence Your host is Tobias Macey and today I’m interviewing Ketan Umare and Haytham Abuelfutuh about Flyte, the open source and kubernetes-native orchestration engine for your data systems Interview Introduction How did you get involved in the area of data management?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Monte Carlo

JANUARY 23, 2024

As LLMs and new models learn better, more efficient ways of working with data, says John, “It’s going to create a whole new kind of class system of engineering versus what everyone looked to the data scientists for in the last five to ten years. Those who don’t embrace it will be left behind. Max agrees. It’s hard with RAG.

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Marc Lamberti

SEPTEMBER 22, 2023

Digging into the official definition, you will see that Airflow is noted as an open-source platform that programmatically authors, schedules, and monitors workflows. Starting with just a few tasks to thousands of tasks thanks to the support of various systems such as Dasks, Kubernetes, or Celery. ” What is Apache Airflow?

Data Pipeline 130

Data Pipeline Python Metadata Database 130

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Christophe Blefari

FEBRUARY 3, 2023

Lyft, powering millions of real-time decisions with LyftLearn Serving — Architecture of the decentralized system Lyft use to deploy and serve ml models. What's often stays is that AE is between DE and DA, so the definition is often done complementarily to other positions. In the end actually, the answer depends.

BI 130

BI Google Cloud SQL Machine Learning 130

Cloudera

MARCH 9, 2021

While Cloudera Data Platform (CDP) already supports the entire data lifecycle from ‘Edge to AI’, we at Cloudera are fully aware that enterprises have more systems outside of CDP. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets.

Data Governance 93

Data Governance Government Metadata Datasets 93

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. As data structure changes in connected systems, the changes are automatically captured and imported to the data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Ascend.io

FEBRUARY 28, 2024

The foundation of any AI system is high-quality data. Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Databand.ai

AUGUST 30, 2023

The importance of data quality cannot be overstated, as poor-quality data can result in incorrect conclusions, inefficient operations, and a lack of trust in the information provided by a company’s systems. Consistency: The uniformity of data across different sources or systems.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. This is a great exploration of what it means to treat your data platform as a living system and apply state of the art engineering to it.

BI 100

BI Data Warehouse Metadata SQL 100

Towards Data Science

MAY 26, 2023

Usually an external system is needed (e.g. The good news is, if you use Hamilton, there isn’t the need for another system to get lineage; making use of lineage is straightforward too! Taking this code and connecting with e.g. a version control system (e.g. Adding metadata to make lineage more useful!

Metadata 86

Metadata Data Science Coding Machine Learning 86

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. Lack of governance Our platform lacked governance policies for metric definitions.

SQL 82

SQL Metadata Raw Data Government 82

Cloudera

MARCH 14, 2023

For example, a developer may be asked to tap into the data of a newly acquired application, parsing and transforming it before delivering it to the business’s favorite analytical system where it can be joined with existing data sets. Attributes contain key metadata like the source directory of a file or the source topic of a Kafka message.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

ThoughtSpot

JUNE 28, 2023

Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses. Applying analytics-as-code principles, we initially developed a scriptable approach with TML to automate the lifecycle management of analytic content across environments. vcs/git/config/create REST API v2.0

Metadata 72

Metadata Accessible Accessibility Utilities 72

Monte Carlo

MARCH 12, 2024

Google Cloud’s Dataplex is a data fabric tool that enables organizations to discover, manage, monitor, and govern their data across all of their data systems, including their data lakes, data warehouses, data lakehouses, and data marts. Dataplex works with your metadata.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

LinkedIn Engineering

JUNE 26, 2023

However, we've found that this vertical self-service model doesn't work particularly well for data pipelines, which involve wiring together many different systems into end-to-end data flows. For example, Pinot ingestion by default expects records to be flat, with field names and types that are compatible with the Pinot table definition.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Engineering at Meta

OCTOBER 31, 2023

SCARF scans production data systems to identify tables or assets that are unused and safely removes them. This process can be unexpectedly difficult, because large software systems are inevitably interconnected. Meta has a separate system, DELF , for deleting individual records, rows, and objects.

Data 93

Data Metadata Coding Relational Database 93