Snowflake

JANUARY 25, 2023

Gone are the days when a few expert data engineers could manage the influx of new data and data types, and quickly apply complex business rules to deliver it to their business consumers. Translate that to database terms, and that means tens or hundreds of millions of columns that the organization needs to understand and manage.

Metadata 97

Metadata Data Pipeline Government Data 97

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. What are the common challenges faced by engineers and data practitioners in organizing the metadata for their systems? What are the goals of the project?

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. They also share their ambitions for the near future of adding data observability and data quality management features. When is Acryl the wrong choice?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

JULY 17, 2022

Crux was built to reduce the total cost of acquisition and ownership for integrating external data, offering a fully managed service for delivering those data assets in the manner that best suits your infrastructure. Atlan is the metadata hub for your data ecosystem. Can you describe what Crux is and the story behind it?

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

DECEMBER 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Atlan is the metadata hub for your data ecosystem.

Management 130

Management Metadata Business Intelligence Data Lake 130

Netflix Tech

MARCH 10, 2023

By Meenakshi Jindal Overview At Netflix, we built the asset management platform (AMP) as a centralized service to organize, store and discover the digital media assets created during the movie production. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). While functional, our current setup for managing tables is fragmented.

Big Data 78

Big Data Data Management Management Metadata 78

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Metadata 100

Metadata MongoDB Scala MySQL 100

Christophe Blefari

MARCH 15, 2024

Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. Understand how BigQuery inserts, deletes and updates — Once again Vu took time to deep dive into BigQuery internal, this time to explain how data management is done. This is Croissant.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Engineering at Meta

FEBRUARY 20, 2024

This new convergence helps Meta and the larger community build data management systems that are unified, more efficient, and composable. Meta’s Data Infrastructure teams have been rethinking how data management systems are designed. An introduction to Velox Velox is the first project in our composable data management system program.

Data Management 97

Data Management Bytes Management Datasets 97

dbt Developer Hub

SEPTEMBER 14, 2021

So let's get a win for someone else -- like Jeremy Cohen, the dbt Core product manager. Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Course you do it was two days ago. Well that's a win in your bucket and the day's barely begun!

Metadata 52

Metadata Data Pipeline Python Cloud 52

Data Council

MARCH 25, 2021

As Fishtown Analytics' community manager for the last 2.5 Here's our March 2021 roundup of links from across the web that we selected for you: 1. How to Build a Community (Fishtown Analytics) Claire Carroll's first personal blog post on community-building is a must-read.

Metadata 52

Metadata Management Building IT 52

Snowflake

JANUARY 23, 2024

We have been making it easier and faster to build and manage ML models with Snowpark ML , the Python library and underlying infrastructure for end-to-end ML workflows in Snowflake. For Snowpark ML Operations, the Snowpark Model Registry allows customers to securely manage and execute models in Snowflake, regardless of origin.

Machine Learning 135

Machine Learning Metadata Python Telecommunication 135

Snowflake

MAY 15, 2024

By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly. Metaplane ensures that every company can trust the data that powers their business.

Cloud 122

Cloud Metadata Data Pipeline Government 122

Data Engineering Podcast

FEBRUARY 27, 2022

Summary There are a wealth of options for managing structured and textual data, but unstructured binary data assets are not as well supported across the ecosystem. As organizations start to adopt cloud technologies they need a way to manage the distribution, discovery, and collaboration of data across their operating environments.

Unstructured Data 100

Unstructured Data Cloud Management Metadata 100

The Modern Data Company

JANUARY 22, 2024

In response, The Modern Data Company emerged, driven by a clear mission: to revolutionize data management and address challenges posed by a diverse and rapidly evolving data environment. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Zalando Engineering

JANUARY 30, 2023

In this post, we describe how we structured incident playbooks, and how we manage these across 100+ on-call teams. To be consistent with our UI-driven application review workflow, we intended to manage playbooks in the same way. Managing structured data in markdown is not ideal, despite the ability to use front matter for metadata.

Management 57

Management Metadata Software Engineer Software Engineering 57

Data Engineering Podcast

JANUARY 23, 2022

In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. You can observe your pipelines with built in metadata search and column level lineage. Closing Announcements Thank you for listening!

Building 100

Building Management Data Pipeline Metadata 100

Ascend.io

MARCH 20, 2023

These reports rely on queries on Ascend-managed cost metadata. Additional Reading and Resources Expanded Cost Metrics, Manage Connections via SDK Ascend Spotlight: Stop Data Problems in Real-Time Link Multiple Data Clouds to Ascend The post Spotlight: Managing Storage, Reducing Costs appeared first on Ascend.io.

Management 52

Management AWS Cloud Storage Data Pipeline 52

Ascend.io

MARCH 20, 2023

These reports rely on queries on Ascend-managed cost metadata. Additional Reading and Resources Expanded Cost Metrics, Manage Connections via SDK Ascend Spotlight: Stop Data Problems in Real-Time Link Multiple Data Clouds to Ascend The post Spotlight: Managing Storage, Reducing Costs appeared first on Ascend.io.

Management 52

Management AWS Cloud Storage Data Pipeline 52

Precisely

SEPTEMBER 5, 2023

Master Data Management systems (MDM) play an important role in harmonizing data assets across large and midsize enterprises. Although MDM serves a vitally important function by ensuring the shared master data is managed correctly and is fit for purpose, it does not offer the same benefits as a full-blown data governance program.

Data Governance 52

Data Governance Government Data Management Management 52

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 114

Building Metadata Cloud Storage AWS 114

Knowledge Hut

APRIL 25, 2024

Kubernetes (sometimes shortened to K8s with the 8 standing for the number of letters between the “K” and the “s”) is an open-source system to deploy, scale, and manage containerized applications anywhere. Kubernetes is a container-centric management software that allows the creation and deployment of containerized applications with ease.

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Data Engineering Podcast

JULY 5, 2021

Summary At the core of every data pipeline is an workflow manager (or several). Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need. What do you have planned for the future of SaaSGlue?

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Who is the target audience for this product?

Building 130

Building Metadata MongoDB MySQL 130

The Modern Data Company

JANUARY 22, 2024

In response, The Modern Data Company emerged, driven by a clear mission: to revolutionize data management and address challenges posed by a diverse and rapidly evolving data environment. Data products contain data and its metadata, code, and infrastructure so that each product is self-contained and usable independently.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Engineering Podcast

FEBRUARY 5, 2023

Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? What do you have planned for the future of the podcast?

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Netflix Tech

NOVEMBER 14, 2023

In this context, managing the data, especially when it arrives late, can present a substantial challenge! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. One of the critical features of Psyberg is its ability to detect and manage late-arriving data, no matter the partition it lands in.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Jesse Anderson

NOVEMBER 14, 2023

This kind of data discovery can be subdivided into several steps, as e.g. suggested by Piethein Strengholt in Data Management at Scale. That is done via a careful examination of all metadata repositories describing data sources. But basically, data science thinks of data discovery as finding new insights inside large amounts of data.

Metadata 147

Metadata Data Science Big Data Data 147

Data Engineering Podcast

DECEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. Atlan is the metadata hub for your data ecosystem.

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Precisely

NOVEMBER 14, 2023

This journey must include a strong data governance framework to align people, processes, and technology, and enable them to understand and trust their data and metadata to achieve their business objectives. But most enterprises still consider working with spatial data to be a complex workstream, managed by few skilled experts.

Data Governance 118

Data Governance Government Metadata Retail 118

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. Using this managed, economical solution, your data can be categorized, cleaned, enhanced, and moved from source systems to target systems.

AWS 52

AWS Data Management ETL Tools Management 52

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. For example, a single table named ‘Customers’ is actually an aggregation of metadata that manages and references several data files, ensuring that the table behaves as a cohesive unit.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Metadata 112

Metadata Government Data Governance Data 112