Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

AUGUST 24, 2020

The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub. What were you using at LinkedIn for metadata management prior to the introduction of DataHub?

Metadata 100

Metadata Management Kafka Data Engineering 100

Snowflake

JANUARY 25, 2023

Using column-level metadata to automate data pipelines I believe the best answer to these questions is that automation tools we use need to be column-aware. For the future, our automation tools must collect and manage metadata at the column level. And the metadata must include more than just the data type and size.

Metadata 81

Metadata Data Pipeline Government Data 81

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. What are some examples of automated actions that can be triggered from metadata changes? What are the available events that can be used to trigger actions?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. What are some of the types of information that you classify and collect as metadata? What are some of the challenges that are typically faced by metadata management systems?

Business Intelligence 113

Business Intelligence Metadata Management Data Governance 113

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

KDnuggets

MAY 31, 2022

Add Layer to your existing ML code and quickly get a rich model and data registry with experiment tracking!

Metadata 108

Metadata Coding Data 108

Data Engineering Podcast

AUGUST 13, 2022

Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for.

Metadata 100

Metadata MongoDB Scala MySQL 100

databricks

SEPTEMBER 24, 2023

Product matching is an essential function in many retail and consumer goods organizations. Incoming products are compared to items in the existing product.

Metadata 78

Metadata Retail 78

dbt Developer Hub

SEPTEMBER 14, 2021

Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Assess data health with the metadata generated by recent dbt job runs Dashboard Status Tiles : Embed this tile anywhere iFrames live to quickly check data freshness New Resources ​ Things to Read ? ​

Metadata 52

Metadata Data Pipeline Python Cloud 52

Christophe Blefari

JUNE 21, 2024

Below a diagram describing what I think schematises data platforms: Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. That's why you need a catalog.

Metadata 147

Metadata Data Warehouse BI Scala 147

Acceldata

MARCH 2, 2023

Learn how to use Acceldata's cloud data observability platform to optimize queries for query history metadata.

Metadata 40

Metadata Cloud Data 40

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata AWS Cloud Database 52

Data Engineering Podcast

JUNE 16, 2024

what kinds of questions are you answering with table metadata what use case/team does that support comparative utility of iceberg REST catalog What are the shortcomings of Trino and Iceberg? What were the requirements and selection criteria that led to the selection of that combination of technologies?

Data Lake 147

Data Lake High Quality Data Metadata Python 147

KDnuggets

NOVEMBER 17, 2021

With KNIME extracting critical pieces of information from images becomes as easy as ABC.

Metadata 140

Metadata Coding 140

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Snowflake

JULY 25, 2024

It supports “fuzzy” search — the service takes in natural language queries and returns the most relevant text results, along with associated metadata. For document- or chunk-level access controls, you can use metadata filtering to ensure that the service only returns the results that the client is authorized to view.

Unstructured Data 104

Unstructured Data Metadata Government SQL 104

Data Engineering Weekly

JUNE 16, 2024

[link] Picnic: Open-sourcing dbt-score: lint model metadata with ease! The more metadata there is, the more readability of the model. It is often challenging as developers are not incentivized to produce quality metadata.

Data Engineering 104

Data Engineering Data Engineer Engineering Kafka 104

Jesse Anderson

NOVEMBER 14, 2023

That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.

Metadata 147

Metadata Data Science Big Data Data 147

Cloudera

JULY 10, 2024

Observability for your most secure data For your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment.

Metadata 81

Metadata Cloud Management IT 81

Netflix Tech

NOVEMBER 14, 2023

It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities. All Iceberg tables have associated metadata that provide insight into changes or updates within the data tables.

Data Engineering 107

Data Engineering Data Engineer Engineering Metadata 107

KDnuggets

NOVEMBER 17, 2021

With KNIME extracting critical pieces of information from images becomes as easy as ABC.

Metadata 82

Metadata Coding 82

Ascend.io

JULY 23, 2024

Metadata Access: This level involves granting AI systems access to operational metadata, which includes information related to the day-to-day data operation. Learn how we use metadata to automate 90% of manual data pipeline maintenance.]

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

JULY 22, 2024

Maintaining metadata about each version. Implement Robust Metadata Management : Effective metadata management is crucial for our data versioning. We ensure that each version is accompanied by comprehensive metadata describing the changes and context, including details of which ETL processes were applied.

Metadata 52

Metadata Datasets ETL Tools Data 52

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Precisely

JUNE 24, 2024

They are working through organizational design challenges while also establishing foundational data management capabilities like metadata management and data governance that will allow them to offer trusted data to the business in a timely and efficient manner for analytics and AI.”

Food 93

Food Pharmaceutical Data Analytics Consulting 93

Data Engineering Podcast

NOVEMBER 13, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Building 130

Building Metadata MongoDB MySQL 130

Engineering at Meta

MARCH 18, 2024

Users can query using regular expressions on log lines, arbitrary metadata fields attached to logs, and across log files of hosts and services. Each log line can have zero or more metadata key-value pairs attached to it. The extracted key-value pairs are added to the log line’s metadata. in PyTorch). Multimodal data (e.g.,

Engineering 94

Engineering Metadata Architecture Designing 94

Snowflake

MAY 15, 2024

By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly. Metaplane ensures that every company can trust the data that powers their business.

Cloud 109

Cloud Metadata Data Pipeline Government 109

Data Engineering Podcast

FEBRUARY 5, 2023

Orchestration is now a part of most vertical tools Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything Data catalog -> data discovery -> active metadata Business intelligence Read only reports to metric/semantic layers Embedded analytics and data APIs Rise of ELT dbt Corresponding introduction (..)

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 98

Building Metadata Cloud Storage AWS 98

Data Engineering Podcast

DECEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Start Data Engineering

JULY 20, 2023

Know the when, how, & what (aka metadata) of pipeline runs for easier debugging 3. Ensure data is valid before exposing it to its consumers (aka data quality checks) 3.3. Avoid data duplicates with idempotent pipelines 3.4. Write DRY code & keep I/O separate from data transformation 3.5.

Coding 130

Coding Data Engineering Data Engineer Engineering 130

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

ThoughtSpot

OCTOBER 9, 2023

How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.

Metadata 86

Metadata Government Data Data Governance 86

Data Engineering Podcast

NOVEMBER 20, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. From analyzing your metadata, query logs, and dashboard activities, Select Star will automatically document your datasets.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Precisely

NOVEMBER 14, 2023

This journey must include a strong data governance framework to align people, processes, and technology, and enable them to understand and trust their data and metadata to achieve their business objectives. Does our organization’s data governance service include visibility and transparency of our spatial data and their metadata?

Data Governance 118

Data Governance Government Metadata Retail 118