Machine Learning Metadata Store
KDnuggets
AUGUST 31, 2022
In this article, we will learn about metadata stores, the need for them, their components, and metadata store management.
Metadata Related Topics 
How Metadata Improves Security, Quality, and Transparency
KDnuggets
APRIL 25, 2022
Metadata is the data providing context about the data, more than what you see in the rows and columns. By managing your metadata, you're effectively creating an encyclopedia of your data assets.
Metadata 
155
155 Join 37,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Trending Sources
Data Engineering Best Practices - #2. Metadata & Logging
Start Data Engineering
FEBRUARY 22, 2024
Metadata: Information about pipeline runs, & data flowing through your pipeline 3.2. Introduction 2. Setup & Logging architecture 3. Data Pipeline Logging Best Practices 3.1. Obtain visibility into the code’s execution sequence using text logs 3.3. Understand resource usage by tracking Metrics 3.4.
Metadata 130
130 
Level Up Your Data Platform With Active Metadata
Data Engineering Podcast
JUNE 19, 2022
Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.
Metadata 130
130 
Metadata – Data Interoperability’s Hidden Talent (Part Two)
ArcGIS
SEPTEMBER 23, 2024
Metadata, the data about your data, is incredibly important, and Data Interoperability can help you create, manage, and maintain that data.
Metadata 108
108 
Metadata – Data Interoperability’s Hidden Talent (Part One)
ArcGIS
SEPTEMBER 23, 2024
Metadata, the data about your data, is incredibly important, and Data Interoperability can help you create, manage, and maintain that data.
Metadata 96
96 
Octopai Acquisition Enhances Metadata Management to Trust Data Across Entire Data Estate
Cloudera
NOVEMBER 13, 2024
It leverages knowledge graphs to keep track of all the data sources and data flows, using AI to fill the gaps so you have the most comprehensive metadata management solution. Together, Cloudera and Octopai will help reinvent how customers manage their metadata and track lineage across all their data sources.
Metadata 90
90 
Data News — Week 24.11
Christophe Blefari
MARCH 15, 2024
Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.
Metadata 272
272 
How to get started with dbt
Christophe Blefari
MARCH 1, 2023
You can also add metadata on models (in YAML). docs — in dbt you can add metadata on everything, some of the metadata is already expected by the framework and thank to it you can generate a small web page with your light catalog inside: you only need to do dbt docs generate and dbt docs serve.
Data Warehouse 278
278 
Using Images and Metadata for Product Fuzzy Matching with Zingg
databricks
SEPTEMBER 24, 2023
Product matching is an essential function in many retail and consumer goods organizations. Incoming products are compared to items in the existing product.
Metadata 104
104 
Databricks, Snowflake and the future
Christophe Blefari
JUNE 21, 2024
Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. That's why you need a catalog.
Metadata 147
147 
Being Data Driven At Stripe With Trino And Iceberg
Data Engineering Podcast
JUNE 16, 2024
what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? What were the requirements and selection criteria that led to the selection of that combination of technologies?
Data Lake 147
147 
Reflecting On The Past 6 Years Of Data Engineering
Data Engineering Podcast
FEBRUARY 5, 2023
Orchestration is now a part of most vertical tools Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything Data catalog -> data discovery -> active metadata Business intelligence Read only reports to metric/semantic layers Embedded analytics and data APIs Rise of ELT dbt Corresponding introduction (..)
Data Engineering 147
147 
Ready-to-go sample data pipelines with Dataflow
Netflix Tech
DECEMBER 3, 2022
This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps. DDL Often, the first step in a data pipeline is to define the target table structure and column metadata via a DDL statement. For the workflow orchestration we use Netflix homegrown Maestro scheduler.
Data Pipeline 144
144 
Build Data Products Without A Data Team Using AgileData
Data Engineering Podcast
NOVEMBER 13, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Building 130
130 
The Data Discovery Team
Jesse Anderson
NOVEMBER 14, 2023
That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.
Metadata 147
147 
Combining The Simplicity Of Spreadsheets With The Power Of Modern Data Infrastructure At Canvas
Data Engineering Podcast
JUNE 19, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Metadata 130
130 
What do Snowflake, Databricks, Redshift, BigQuery actually do?
Start Data Engineering
NOVEMBER 21, 2024
Metadata catalog stores information about datasets 3.1.3. Most platforms enable you to do the same thing but have different strengths 3.1. Understand how the platforms process data 3.1.1. A compute engine is a system that transforms data 3.1.2. Data platform support for SQL, Dataframe, and Dataset APIs 3.1.4.
Metadata 130
130 
A Look At The Data Systems Behind The Gameplay For League Of Legends
Data Engineering Podcast
NOVEMBER 20, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. From analyzing your metadata, query logs, and dashboard activities, Select Star will automatically document your datasets.
Systems 130
130 
Bring Geospatial Analytics Across Disparate Datasets Into Your Toolkit With The Unfolded Platform
Data Engineering Podcast
JUNE 26, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Datasets 130
130 
Expanding The Reach of Business Intelligence Through Ubiquitous Embedded Analytics With Sisense
Data Engineering Podcast
OCTOBER 30, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Build Better Data Products By Creating Data, Not Consuming It
Data Engineering Podcast
NOVEMBER 6, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Building 130
130 
Making Sense Of The Technical And Organizational Considerations Of Data Contracts
Data Engineering Podcast
DECEMBER 18, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!
Metadata 130
130 
Announcing Nickel 1.0
Tweag
MAY 16, 2023
Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.
MySQL 135
135 
Making The Total Cost Of Ownership For External Data Manageable With Crux
Data Engineering Podcast
JULY 17, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Data Management 130
130 
Increase Your Odds Of Success For Analytics And AI Through More Effective Knowledge Management With AlignAI
Data Engineering Podcast
DECEMBER 29, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!
Management 130
130 
How To Bring Agile Practices To Your Data Projects
Data Engineering Podcast
OCTOBER 23, 2022
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!
Project 130
130 
The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse
Data Engineering Podcast
FEBRUARY 19, 2023
Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform. Acryl]([link] The modern data stack needs a reimagined metadata management platform.
IT 147
147 
Build an Open Data Lakehouse with Iceberg Tables, Now in Public Preview
Snowflake
DECEMBER 4, 2023
With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?
Building 113
113 
Increase data literacy and trust with Alation data catalog integration
ThoughtSpot
OCTOBER 9, 2023
How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.
Metadata 105
105 
Snowflake Invests in Metaplane for Deep, End-to-End Observability in the Data Cloud
Snowflake
MAY 15, 2024
By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly. Metaplane ensures that every company can trust the data that powers their business.
Cloud 123
123 
Accelerate Your Machine Learning Workflows in Snowflake with Snowpark ML
Snowflake
JANUARY 23, 2024
Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.
Machine Learning 133
133 
Logarithm: A logging engine for AI training workflows and services
Engineering at Meta
MARCH 18, 2024
Users can query using regular expressions on log lines, arbitrary metadata fields attached to logs, and across log files of hosts and services. Each log line can have zero or more metadata key-value pairs attached to it. The extracted key-value pairs are added to the log line’s metadata. in PyTorch). Multimodal data (e.g.,
Engineering 107
107 
Data Engineering Best Practices - #1. Data flow & Code
Start Data Engineering
JULY 20, 2023
Know the when, how, & what (aka metadata) of pipeline runs for easier debugging 3. Ensure data is valid before exposing it to its consumers (aka data quality checks) 3.3. Avoid data duplicates with idempotent pipelines 3.4. Write DRY code & keep I/O separate from data transformation 3.5.
Coding 130
130 
Why Spatial Data Governance is Critical to Your Business Strategy
Precisely
NOVEMBER 14, 2023
This journey must include a strong data governance framework to align people, processes, and technology, and enable them to understand and trust their data and metadata to achieve their business objectives. Does our organization’s data governance service include visibility and transparency of our spatial data and their metadata?
Data Governance 119
119 
1. Streamlining Membership Data Engineering at Netflix with Psyberg
Netflix Tech
NOVEMBER 14, 2023
It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities. All Iceberg tables have associated metadata that provide insight into changes or updates within the data tables.
Data Engineering 107
107 
Video Multiplexer Tips and Tricks
ArcGIS
APRIL 16, 2024
Tips to properly format your metadata for the video multiplexer tool so you can geoenable video data for the Full Motion Video player.
Metadata 109
109 
Modern Data Architecture: Data Mesh and Data Fabric 101
Precisely
OCTOBER 31, 2024
While data products may have different definitions in different organizations, in general it is seen as data entity that contains data and metadata that has been curated for a specific business purpose. A data fabric weaves together different data management tools, metadata, and automation to create a seamless architecture.
Data Engineering Weekly #176
Data Engineering Weekly
JUNE 16, 2024
[link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.
Data Engineer 104
104 
Snowflake Cortex Search: State-of-the-Art Hybrid Search for RAG Applications
Snowflake
JULY 25, 2024
It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. For document- or chunk-level access controls, you can use metadata filtering to ensure that the service only returns the results that the client is authorized to view.
Announcing New Innovations for Snowflake Horizon
Snowflake
NOVEMBER 2, 2023
To give customers flexibility for how they fit Snowflake into their architecture, Iceberg Tables can be configured to use either Snowflake or an external service like AWS Glue as the catalog to track metadata.
Metadata 120
120 
Azure Data Factory: Fail Activity
Azure Data Engineering
NOVEMBER 21, 2021
An example could be when we want to check the existence of a file or folder using Get Metadata activity. During some scenarios in Azure Data Factory, we may want to intentionally stop the execution of the pipeline. We may want to fail the pipeline if the file/folder does not exist. To achieve this, we could use the Fail Activity.
Metadata 130
130 
Ensuring the Successful Launch of Ads on Netflix
Netflix Tech
JUNE 1, 2023
It also included metadata about ads, such as ad placement and impression-tracking events. A Kafka consumer retrieved the playback manifests with ad metadata and simulated a device playing the content and triggering the impression-tracking events. We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch.
Algorithm 138
138 
Building A Data Governance Bridge Between Cloud And Datacenters For The Enterprise At Privacera
Data Engineering Podcast
MARCH 27, 2022
Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform. Acryl]([link] The modern data stack needs a reimagined metadata management platform.
Data Governance 130
130 
Let's personalize your content