Monte Carlo

JULY 11, 2023

Data observability solutions also take testing a step further by providing root cause analysis tools that help teams proactively identify the root cause of data inaccuracies, as they relate to code, your data infrastructure, and even the data itself.

Data Cleanse 40

Data Cleanse Datasets Data Governance Government 40

Monte Carlo

AUGUST 30, 2023

More specifically, teams use dbt to write transformations as SQL queries, version control the code, and deploy transformations incrementally. If a test fails, i.e., the `dbt test` command exits with a non-zero code, you want your pipeline to halt and notify the team. With dbt tests, no news is good news. Curious to learn more?

SQL 52

SQL Datasets Database High Quality Data 52

Towards Data Science

MAY 7, 2024

Now you have to decide what sorts of things to test in order to figure out definitively if the tool is the right commitment for you and the team. This will help create an overall picture of logging, metadata, latency, CDC, and all the other things that come with a pipeline. Gather the team together and determine what these things are.

ETL Tools 84

ETL Tools Metadata Data Integration Data Science 84

Data Engineering Podcast

NOVEMBER 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Can you share your definition of "behavioral data" and how it is differentiated from other sources/types of data?

Building 130

Building IT Metadata MongoDB 130

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly. Snowflake Dynamic Tables.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. It is easy to update the client code for web applications, but for the mobile native apps shipped to the app store, it is impossible to update the client code. We will discuss this in detail in the next section.

Metadata 86

Metadata Coding Banking Designing 86

Monte Carlo

JULY 10, 2023

That’s where the definition of data completeness comes in. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. For example, address verification tools can supply missing ZIP codes when a customer neglects to provide theirs when filling out a form. The same is true with data.

Data Collection 40

Data Collection Data Governance Government Data 40

Towards Data Science

NOVEMBER 4, 2023

This would be the right way to go for data analyst teams that are not familiar with coding. In one of my previous articles I wrote how easy it is to create a microservice that pulls data from NASA API [3]: Python for Data Engineers Consider this code snippet for app.py The downside of this approach is it’s pricing model though.

Data Engineering 86

Data Engineering Data Engineer Engineering BI 86

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. Lack of unified metadata makes it difficult to optimize pipelines, avoid redundant processing, and quantify data engineering costs.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Towards Data Science

MAY 26, 2023

Lineage as Code With Hamilton, you do not need to add anything else, and you get lineage. Write code. By virtue of writing code in the Hamiltonian way, you are defining computation within functions, and then specifying via function input arguments, how things connect, encoding lineage. Get lineage like this. Image by author.

Metadata 86

Metadata Data Science Coding Machine Learning 86

Data Engineering Podcast

JULY 24, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the definitions shifted over the past few decades? Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer Lambda Architecture Engineering 100

François Nguyen

JANUARY 18, 2021

This article was published in October 2020 with this title : “Data Governance as Code” The idea behind that is you should “actively promotes the safe use of data with automation that improves governance while freeing data analysts and scientists from manual tasks” The article is illustrated with many examples.

Data Governance 130

Data Governance Metadata Government Data Pipeline 130

Data Engineering Podcast

SEPTEMBER 11, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What are the different places in a data system that schema definitions need to be established?

Systems 100

Systems Metadata Building MongoDB 100

The Pragmatic Engineer

AUGUST 3, 2023

These links were especially helpful for: Promotions and marketing campaigns QR codes Content sharing links that “just work” Converting desktop users to mobile ones The shutdown Dynamic links powered Firebase Invites: an app invite service where users could send app invite links to their friends, to drive installation of the app.

Metadata 159

Metadata Engineering Building Technology 159

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Just like at first everyone had to code in a language, then everyone had to know how to incorporate packages from those languages — now we’re moving into, ‘ How do you incorporate AI that will write the code for you?’”

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

François Nguyen

MARCH 22, 2021

This is really for us the definition of a self serve platform. ” Code : all the code necessary to build a data product (data pipelines, API, policies). Data As Code is a very strong choice : we do not want any UI because it is an heritage of the ETL period. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

MAY 22, 2022

In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful. What are the patterns that you and the community have established to encourage discovery and reuse of granular task definitions?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

LinkedIn Engineering

APRIL 25, 2023

resource definitions, their entity schemas and their relationship definitions as input and generates a federated GraphQL type system. These data fetchers are written in a way to use the metadata available on our type system to self-configure and wire appropriately during the service startup.

Architecture 109

Architecture Metadata Java Transportation 109

DoorDash Engineering

DECEMBER 5, 2023

Key terminology Here are definitions and links to further documentation about the tools we used. Terraform : Infrastructure-as-code (IaC) platform. Exposes a unique hardware definition language (HDL) for configuring compute resources. All Topics, even those without any messages, include some metadata that is backed up to disk.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Christophe Blefari

JUNE 16, 2023

Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI. The past often changes, whether because of code or data. This is metrics drift.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. With Alteryx , you can create workflows without needing to code by using the provided automation building blocks. Atlan Atlan data catalog.

Metadata 52

Metadata Government Data Data Governance 52

ProjectPro

FEBRUARY 8, 2023

When Glue receives a trigger, it collects the data, transforms it using code that Glue generates automatically, and then loads it into Amazon S3 or Amazon Redshift. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. You can produce code, discover the data schema, and modify it. Why Use AWS Glue?

AWS 98

AWS Scala Metadata Data Lake 98

RandomTrees

FEBRUARY 6, 2024

Transformation: Shaping Data for the Future: LLMs facilitate standardizing date formats with precision and translation of complex organizational structures into logical database designs, streamline the definition of business rules, automate data cleansing, and propose the inclusion of external data for a more complete analytical view.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. How does it work?

BI 100

BI Data Warehouse Metadata SQL 100

ThoughtSpot

JUNE 28, 2023

Applying analytics-as-code principles, we initially developed a scriptable approach with TML to automate the lifecycle management of analytic content across environments. Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses.

Metadata 72

Metadata Accessible Accessibility Utilities 72

Engineering at Meta

OCTOBER 31, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. This is the third and final post in our series on Meta’s Systematic Code and Asset Removal Framework (SCARF). SCARF coordinates several kinds of tasks for each data system: metadata collection (e.g.,

Data 93

Data Metadata Coding Relational Database 93

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 89

Metadata Python AWS Government 89

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Data Engineering Podcast

MAY 18, 2021

Your host is Tobias Macey and today I’m interviewing Julien Le Dem about Open Lineage, a new standard for structuring metadata to enable interoperability across the ecosystem of data management tools. What is the current state of the ecosystem for generating and sharing metadata between systems?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Netflix Tech

NOVEMBER 13, 2023

The detailed temporal metadata SMAD provides about speech and music regions in a polyphonic audio mixture are a first step for structural audio segmentation, indexing and pre-processing audio for the following downstream tasks. The heavy usage of ambient sounds and sound effects blurs the boundaries between active music regions and non-music.

Datasets 89

Datasets Metadata Algorithm Architecture 89

DataKitchen

FEBRUARY 2, 2023

As an analogy, the DevOps space has seen consolidation in code storage, CI/CD, team workflow, value stream management, testing, and other tools into one platform. It’s code -> governance, not governance->code. The code, configuration, and metadata about your data are the Intellectual Property of your data teams.

Metadata 52

Metadata BI Government ETL Tools 52

Monte Carlo

FEBRUARY 9, 2023

architecture (with some minor deviations) to achieve their data integration objectives around scalability and use of metadata. “A The other advantage is because we follow a standard design, we are able to generate a lot of our code using code templates and metadata. This layer has minimal transformation rules.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Ascend.io

FEBRUARY 28, 2024

Metadata Management for Enhanced Machine Interpretation Data engineering extends its importance to the realm of metadata management, providing AI models with essential context about data definitions, classifications, and governance policies. ChatGPT screenshot of AI-generated Python code and an explanation of what it means.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Towards Data Science

APRIL 6, 2023

Unit tests, which assess small pieces of code for performance on discrete, deterministic tasks, get overwhelmed by the variety of acceptable shapes and values that real-world data can take. That’s possibly correct — but the size of the distributional shift should definitely lead to an anomaly alert! What does each column represent?

Unstructured Data 82

Unstructured Data Metadata Data Coding 82

LinkedIn Engineering

JUNE 26, 2023

In those cases, creating an end-to-end data pipeline involves writing custom code to bridge the gaps. For example, Pinot ingestion by default expects records to be flat, with field names and types that are compatible with the Pinot table definition. For streaming data pipelines, this involves writing stream processing jobs.

Data Pipeline 112

Data Pipeline Kafka MySQL SQL 112

Data Engineering Podcast

FEBRUARY 6, 2022

Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. – You visually design the pipelines, and Prophecy generates clean Spark code with tests on git; then you visually schedule these pipelines on Airflow. How does it work? How does it work?

Data Pipeline 100

Data Pipeline Data Metadata Software Engineer 100

dbt Developer Hub

JULY 26, 2022

With the dbt Semantic Layer, this definition would live in dbt and the logic to create the dataset for that metric would be consistent across all different consuming experiences. Best of all, definition changes would get reflected in downstream tools, so you no longer need to manually search and update every downstream dependency.

Metadata 52

Metadata BI Datasets SQL 52