KDnuggets

APRIL 25, 2022

Metadata is the data providing context about the data, more than what you see in the rows and columns. By managing your metadata, you're effectively creating an encyclopedia of your data assets.

Metadata 155

Metadata Management Data Data Science 155

Cloudyard

OCTOBER 15, 2024

When using Iceberg tables, every Data Definition Language ( DDL ) operation triggers the generation of a new metadata JSON file that captures the updated structure. This article outlines a process for efficiently tracking schema changes in Iceberg tables by leveraging Snowflake’s powerful metadata storage capabilities.

Metadata 52

Metadata Data Governance Government Data Integration 52

Ascend.io

JULY 11, 2024

Metadata is the information that provides context and meaning to data, ensuring it’s easily discoverable, organized, and actionable. This is what managing data without metadata feels like. This is what managing data without metadata feels like. Effective metadata management is no longer a luxury—it’s a necessity.

Metadata 52

Metadata IT Government High Quality Data 52

Cloudera

JANUARY 26, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 73

Metadata Data Governance Government Management 73

Cloudera

JUNE 2, 2021

As an important part of achieving better scalability, Ozone separates the metadata management among different services: . Ozone Manager (OM) service manages the metadata of the namespace such as volume, bucket and keys. Datanode service manages the metadata of blocks, containers and pipelines running on the datanode. .

Metadata 95

Metadata Hadoop Certification Algorithm 95

Cloudera

MARCH 4, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 52

Metadata Data Governance Government Management 52

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

AUGUST 24, 2020

The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub. What were you using at LinkedIn for metadata management prior to the introduction of DataHub?

Metadata 100

Metadata Management Kafka Data Engineering 100

Snowflake

JANUARY 25, 2023

Using column-level metadata to automate data pipelines I believe the best answer to these questions is that automation tools we use need to be column-aware. For the future, our automation tools must collect and manage metadata at the column level. And the metadata must include more than just the data type and size.

Metadata 84

Metadata Data Pipeline Government Data 84

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. What are some examples of automated actions that can be triggered from metadata changes? What are the available events that can be used to trigger actions?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. What are some of the types of information that you classify and collect as metadata? What are some of the challenges that are typically faced by metadata management systems?

Business Intelligence 113

Business Intelligence Metadata Management Data Governance 113

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

KDnuggets

MAY 31, 2022

Add Layer to your existing ML code and quickly get a rich model and data registry with experiment tracking!

Metadata 108

Metadata Coding Data 108

databricks

SEPTEMBER 24, 2023

Product matching is an essential function in many retail and consumer goods organizations. Incoming products are compared to items in the existing product.

Metadata 93

Metadata Retail 93

The Pragmatic Engineer

OCTOBER 17, 2024

Results are stored in git and their database, together with benchmarking metadata. Benchmarking results for each instance type are stored in sc-inspector-data repo, together with the benchmarking task hash and other metadata.  There Then we wait for the actual data and/or final metadata (e.g.

Cloud 278

Cloud AWS Metadata Cloud Computing 278

Data Engineering Podcast

AUGUST 13, 2022

Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for.

Metadata 100

Metadata MongoDB Scala MySQL 100

Start Data Engineering

NOVEMBER 21, 2024

Metadata catalog stores information about datasets 3.1.3. Most platforms enable you to do the same thing but have different strengths 3.1. Understand how the platforms process data 3.1.1. A compute engine is a system that transforms data 3.1.2. Data platform support for SQL, Dataframe, and Dataset APIs 3.1.4.

Metadata 130

Metadata Datasets SQL Database 130

dbt Developer Hub

SEPTEMBER 14, 2021

Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Assess data health with the metadata generated by recent dbt job runs Dashboard Status Tiles : Embed this tile anywhere iFrames live to quickly check data freshness New Resources ​ Things to Read ? ​

Metadata 52

Metadata Data Pipeline Python Cloud 52

Acceldata

MARCH 2, 2023

Learn how to use Acceldata's cloud data observability platform to optimize queries for query history metadata.

Metadata 40

Metadata Cloud Data 40

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata Cloud AWS Database 52

KDnuggets

NOVEMBER 17, 2021

With KNIME extracting critical pieces of information from images becomes as easy as ABC.

Metadata 149

Metadata Coding 149

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. That's why you need a catalog.

Metadata 147

Metadata Data Warehouse BI Scala 147

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Cloudyard

DECEMBER 2, 2024

This blog highlights a real-time use case where directory tables track file-level metadata, streams monitor new file uploads, and Python functions extract specific details from PDF files—all within Snowflake’s unified platform. Directory Tables for File Metadata Tracking: E nable file-level metadata monitoring.

Python 52

Python Process Insurance Metadata 52

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 83

Metadata BI Data Lake Business Intelligence 83

Data Engineering Podcast

JUNE 16, 2024

what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? What were the requirements and selection criteria that led to the selection of that combination of technologies?

Data Lake 147

Data Lake High Quality Data Metadata Machine Learning 147

dbt Developer Hub

OCTOBER 3, 2024

These formats are changing the way data is stored and metadata accessed. Iceberg Data Catalog - an open-source metadata management system that tracks the schema, partition, and versions of Iceberg tables. The metadata management and performance make them very meaningful and should be paid attention to. What is Iceberg? ​

Metadata 86

Metadata Data Lake Data Storage Accessible 86

Precisely

OCTOBER 31, 2024

While data products may have different definitions in different organizations, in general it is seen as data entity that contains data and metadata that has been curated for a specific business purpose. A data fabric weaves together different data management tools, metadata, and automation to create a seamless architecture.

Data Architecture 69

Data Architecture Architecture Metadata Government 69

ThoughtSpot

NOVEMBER 5, 2024

In the realm of modern analytics platforms, where rapid and efficient processing of large datasets is essential, swift metadata access and management are critical for optimal system performance. Any delays in metadata retrieval can negatively impact user experience, resulting in decreased productivity and satisfaction. What is Atlas?

Metadata 52

Metadata PostgreSQL Java Database 52

ThoughtSpot

OCTOBER 9, 2023

How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.

Metadata 105

Metadata Data Government Data Governance 105

Jesse Anderson

NOVEMBER 14, 2023

That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.

Metadata 147

Metadata Data Science Big Data Data 147

Netflix Tech

NOVEMBER 14, 2023

It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities. All Iceberg tables have associated metadata that provide insight into changes or updates within the data tables.

Data Engineering 107

Data Engineering Data Engineer Engineering Metadata 107

Cloudera

DECEMBER 3, 2024

REST Catalog Value Proposition It provides open, metastore-agnostic APIs for Iceberg metadata operations, dramatically simplifying the Iceberg client and metastore/engine integration. It provides real time metadata access by directly integrating with the Iceberg-compatible metastore. spark.sql(SELECT * FROM airlines_data.carriers).show()

Metadata 65

Metadata SQL Database Data Warehouse 65

Data Engineering Podcast

NOVEMBER 13, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Weekly

JUNE 16, 2024

[link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Data Engineering Podcast

FEBRUARY 5, 2023

Orchestration is now a part of most vertical tools Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything Data catalog -> data discovery -> active metadata Business intelligence Read only reports to metric/semantic layers Embedded analytics and data APIs Rise of ELT dbt Corresponding introduction (..)

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

DECEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!

Metadata 130

Metadata Business Intelligence Data Lake BI 130