Medium Data Engineering

AUGUST 14, 2023

Learn how Data Catalog can enable and activate your metadata platform. Continue reading on Medium »

Metadata 52

Metadata Data Data Engineering Data Engineer 52

Data Engineering Podcast

AUGUST 24, 2020

The key to those solutions is a robust and flexible metadata management system. LinkedIn has gone through several iterations on the most maintainable and scalable approach to metadata, leading them to their current work on DataHub. What were you using at LinkedIn for metadata management prior to the introduction of DataHub?

Metadata 100

Metadata Management Data Engineer Data Engineering 100

Medium Data Engineering

MAY 5, 2023

Metadata describes the data’s characteristics, quality, and lineage that flows through a data pipeline. Metadata can help data engineers… Continue reading on Medium »

Data Pipeline 52

Data Pipeline Metadata Data Data Engineering 52

Snowflake

JANUARY 25, 2023

Using column-level metadata to automate data pipelines I believe the best answer to these questions is that automation tools we use need to be column-aware. For the future, our automation tools must collect and manage metadata at the column level. And the metadata must include more than just the data type and size.

Metadata 85

Metadata Data Data Pipeline Coding 85

Medium Data Engineering

JUNE 18, 2023

With growing amount of data in a large organization, there is a dire need of a central metadata storage for storing useful information… Continue reading on Medium »

Metadata 40

Metadata Data Engineering Data Engineer Engineering 40

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. What are some examples of automated actions that can be triggered from metadata changes? What are the available events that can be used to trigger actions?

Metadata 100

Metadata Data Data Warehouse SQL 100

Medium Data Engineering

FEBRUARY 20, 2023

Part 2: How does metadata management work and how does it contribute to the overall data quality Continue reading on re_data »

Metadata 52

Metadata Data Management Data Engineering 52

Medium Data Engineering

MARCH 8, 2023

Part 3: Challenges, final thoughts, and metadata management recommendations Continue reading on re_data »

Metadata 52

Metadata Management Data Data Engineering 52

Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. What are some of the types of information that you classify and collect as metadata? What are some of the challenges that are typically faced by metadata management systems?

Metadata 113

Metadata Business Intelligence Management Data Engineering 113

Medium Data Engineering

FEBRUARY 17, 2023

Part 1: Establishing the baseline and what is metadata exactly. Continue reading on re_data »

Metadata 52

Metadata Data Management Data Engineering 52

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Data Transportation 110

Medium Data Engineering

APRIL 25, 2023

Most of the data companies work with is related to their products and customers. Tens of thousands of terabytes of data about customers… Continue reading on Alvin »

Metadata 52

Metadata Data Data Engineering Data Engineer 52

databricks

SEPTEMBER 24, 2023

Product matching is an essential function in many retail and consumer goods organizations. Incoming products are compared to items in the existing product.

Metadata 86

Metadata Retail 86

Medium Data Engineering

NOVEMBER 11, 2023

Software engineering is fundamentally a discipline dedicated to abstraction. Rather than writing binary, write assembly; rather than write… Continue reading on Medium »

Data Engineering 52

Data Engineering Data Engineer Metadata Engineering 52

Acceldata

MARCH 2, 2023

Learn how to use Acceldata's cloud data observability platform to optimize queries for query history metadata.

Metadata 40

Metadata Data Cloud 40

Netflix Tech

NOVEMBER 14, 2023

It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities. All Iceberg tables have associated metadata that provide insight into changes or updates within the data tables.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Medium Data Engineering

JUNE 16, 2023

Organizations have evolved substantially, either being data driven or having their entire business models based on capture and utilization… Continue reading on Litmus7 Systems Consulting »

Metadata 52

Metadata Engineering Data Utilities 52

KDnuggets

MAY 31, 2022

Add Layer to your existing ML code and quickly get a rich model and data registry with experiment tracking!

Metadata 98

Metadata Coding Data 98

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata Data AWS Cloud 52

Jesse Anderson

NOVEMBER 14, 2023

That is done via a careful examination of all metadata repositories describing data sources. Once those repositories have been carefully studied, the identified data sources must be scanned by a data catalog, so that a metadata mirror of these data sources are made discoverable for the operations team.

Data 147

Data Data Science Metadata Data Engineering 147

ThoughtSpot

OCTOBER 9, 2023

How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.

Metadata 81

Metadata Data Government Data Governance 81

Data Engineering Podcast

AUGUST 13, 2022

Summary Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for.

Metadata 100

Metadata Data Python Data Engineering 100

dbt Developer Hub

SEPTEMBER 14, 2021

Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Assess data health with the metadata generated by recent dbt job runs Dashboard Status Tiles : Embed this tile anywhere iFrames live to quickly check data freshness New Resources ​ Things to Read ? ​

Metadata 52

Metadata Data Pipeline Python Cloud 52

Precisely

NOVEMBER 14, 2023

This journey must include a strong data governance framework to align people, processes, and technology, and enable them to understand and trust their data and metadata to achieve their business objectives. Does our organization’s data governance service include visibility and transparency of our spatial data and their metadata?

Data Governance 117

Data Governance Government Data Metadata 117

Start Data Engineering

JULY 20, 2023

Know the when, how, & what (aka metadata) of pipeline runs for easier debugging 3. Ensure data is valid before exposing it to its consumers (aka data quality checks) 3.3. Avoid data duplicates with idempotent pipelines 3.4. Write DRY code & keep I/O separate from data transformation 3.5.

Data Engineering 130

Data Engineering Data Engineer Coding Data 130

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 98

Metadata Data Pipeline Data Process Scala 98

Netflix Tech

NOVEMBER 14, 2023

Using the summary column in snapshot metadata [see the Iceberg Metadata section in post 1 for more details], we parse out the partition information for each Iceberg snapshot of the source table. This information and other calculated metadata are stored in the psyberg_session_f table.

Data Process 90

Data Process Process Data Metadata 90

Medium Data Engineering

OCTOBER 16, 2023

INTRODUCTION Continue reading on Medium »

Metadata 52

Metadata Data Engineering Data Engineer Engineering 52

Snowflake

NOVEMBER 29, 2023

Also, the associated business metadata for omics, which make it findable for later use, are dynamic and complex and need to be captured separately. A sample representation of the business or functional metadata for omics type called RNA-seq is provided in Figure 1 below.

Metadata 61

Metadata Data Storage Python Cloud 61

Medium Data Engineering

OCTOBER 9, 2023

Have you ever build an ingest pipeline in Synapse only to rebuild it later? Continue reading on Medium »

Metadata 52

Metadata Building Data Engineering Data Engineer 52

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Java 73

Java Metadata Data Warehouse SQL 73

Medium Data Engineering

JUNE 3, 2023

Auto Loader is an awesome Databricks feature where you can efficiently process data files as soon as they arrive in cloud storage without… Continue reading on Medium »

Metadata 52

Metadata Cloud Storage Cloud Data 52

Data Engineering Podcast

NOVEMBER 13, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data 130

Data Building Metadata Data Engineer 130

Data Engineering Podcast

FEBRUARY 5, 2023

Orchestration is now a part of most vertical tools Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything Data catalog -> data discovery -> active metadata Business intelligence Read only reports to metric/semantic layers Embedded analytics and data APIs Rise of ELT dbt Corresponding introduction (..)

Data Engineering 147

Data Engineering Data Engineer Engineering Data 147

Data Engineering Podcast

DECEMBER 18, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don't forget to thank them for their continued support of this show!

Data 130

Data Metadata Data Engineering Data Engineer 130

KDnuggets

NOVEMBER 17, 2021

With KNIME extracting critical pieces of information from images becomes as easy as ABC.

Metadata 123

Metadata Coding 123

Medium Data Engineering

MARCH 26, 2023

In an ideal world, your SaaS provider, e.g. Xero allows you to export your data or has some systems in place that push data to your… Continue reading on Medium »

Metadata 52

Metadata Data Systems Data Engineering 52