The Modern Data Company

APRIL 15, 2024

Manages information about data access and monitors data quality. It’s a combination of the data itself, the metadata (information about the data), code (the set of instructions on how to process or analyze the data), and infrastructure (the technological framework that supports it all).

Government 52

Government Metadata Data Data Lake 52

Rockset

APRIL 18, 2023

To highlight these new capabilities, we built a search demo using OpenAI to create embeddings for Amazon product descriptions and Rockset to generate relevant search results. In the demo, you’ll see how Rockset delivers search results in 15 milliseconds over thousands of documents.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Data Engineering Podcast

MAY 18, 2021

High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data. From feature extraction to model training to production, the Molecula feature store provides continuously updated feature access, reuse, and sharing without the need to pre-process data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

ThoughtSpot

JANUARY 31, 2024

As a result, end users expect faster response times and easier access to meaningful insights. Not only does ThoughtSpot not store your sample data or metadata, or use this information for model training, but we are also investing in bring-your-own and host-your-own model capabilities for both generative AI and machine learning models.

BI 89

BI Business Intelligence Metadata Machine Learning 89

Snowflake

MARCH 12, 2024

The Mapbox Snowflake Native App extends those benefits: it brings Mapbox’s search, navigation and mapping logic to the data and makes geospatial data analysis more accessible for developers and non-developers alike. It also provides metadata on what assumptions and fixes were used to inform the address match.

Business Analyst 95

Business Analyst Retail Business Intelligence Data Analysis 95

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. For metadata management, you can configure Snowflake to manage your Iceberg data or use an external Iceberg catalog.

Metadata 83

Metadata AWS Data Lake Datasets 83

Cloudyard

DECEMBER 13, 2022

Moreover, We will use the below architecture to showcase the Demo where convert the existing Data Lake to Snowflake Deployment. DMS For this Demo we launched RDS service for ORACLE in AWS and created some tables. Kinesisdest:Bucket to hold Streaming data i.e. JSON file Externalbucketstream:Bucket to hold In frequent access data.

Data Lake 52

Data Lake AWS Metadata Data 52

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. Rockset supports streaming updates to metadata and vectors in an efficient way. Multiple virtual instances can simultaneously access the same dataset, eliminating the need for multiple replicas of data.

Cloud 64

Cloud Metadata Database SQL 64

Precisely

MAY 31, 2023

In the same way that big cloud-platform providers offer simplified access to infrastructure, and data cloud providers like Databricks and Snowflake have vastly simplified access to data and analytics, modern data integrity tools must streamline and automate data integrity processes.

Data Integration 52

Data Integration Metadata Data Governance Data 52

Monte Carlo

MAY 25, 2023

That’s why we’re excited to announce that uniting diverse data team personas to collectively ensure data quality just got easier, thanks to a new integration between Monte Carlo’s data observability platform and Atlan’s active metadata platform. The pipeline might succeed, but we wouldn’t know if the data was right, wrong, or indifferent.

Metadata 52

Metadata Data Governance Government Business Intelligence 52

The Modern Data Company

MARCH 1, 2024

Each Data Product is designed for a specific purpose, equipped with the necessary data, transformations, and metadata. Data Products simplify access and processing, empowering faster decision-making. Democratized Data Access: Empower business users with self-service access to relevant data insights, fostering a data-driven culture.

Database-centric 40

Database-centric Data Lake Government Retail 40

The Modern Data Company

MARCH 7, 2023

However, more often than not, it conjures up the idea of a central authority strictly guarding against such access. A data taxonomy uses a system of unambiguous metadata terms (such as a filename or tags attached to a file) that allow an enterprise to classify a file or dataset into important business categories.

Data Governance 52

Data Governance Government Metadata Datasets 52

Snowflake

FEBRUARY 29, 2024

They need an easier access path to convert that latitude and longitude into something more powerful. This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Caption: A demo Streamlit app visualizes H3 hexagons at different resolutions for the same data.

Insurance 84

Insurance SQL BI Consulting 84

The Modern Data Company

JANUARY 17, 2023

Likewise, the enterprise as a whole has no access to the data in the CDP. Without lineage or governance metadata, the quality of this data is anyone’s guess. It can apply metadata and track the lineage of CDP data to improve data quality. Collecting more data isn’t always a good thing.

Metadata 59

Metadata Government Data Lake Data 59

Precisely

JULY 21, 2023

Customize the Replication Script: Use the scripting language provided by Precisely to define variables, specify metadata replication preferences, and map Cobol copybook descriptions for VSAM files. The Confluent and Precisely partnership helps make this possible and addresses the challenges of tomorrow – watch the demo now.

Data Integration 52

Data Integration Kafka Bytes Banking 52

Cloudera

MARCH 17, 2023

The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Cloudera

JULY 14, 2023

They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Snowflake

OCTOBER 2, 2023

Access to a complete view of the customer is incredibly powerful. The Snowflake Data Cloud offering enables access to all of your customer data alongside infinite compute power to model it. Accelerate AI projects with historical snapshots and metadata Operationalizing AI is quickly becoming a competitive requirement for businesses.

Cloud 100

Cloud Building Insurance Data Engineering 100

Ascend.io

DECEMBER 19, 2022

Improved Support for Databricks To highlight our improved Databricks capabilities, our re:Invent booth was next to theirs, and we chose to power our demos with their Lakehouse. Instead, it is a Sankey diagram driven by the same dynamic metadata that runs the Ascend control plane. If you were wondering why, here is your answer.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Rockset

SEPTEMBER 26, 2023

AI-powered applications almost always need access to real-time data to deliver accurate results in a responsive user experience that the market has come to expect. Metadata filtering is a useful, perhaps even essential, companion to vector search that restricts nearest-neighbor matches based on specific criteria.

Metadata 64

Metadata Kafka Cloud Database 64

Monte Carlo

MARCH 1, 2022

Our Python SDK gives data engineers programmatic access to Monte Carlo to augment our data observability platform’s lineage, cataloging, and monitoring functionalities. Now, data engineers can use the Python SDK to more conveniently access all of Monte Carlo’s APIs. Aggregating metadata. What does Pycarlo support?

Python 52

Python Metadata Business Intelligence Data Engineering 52

Monte Carlo

JANUARY 23, 2024

As the machine became smarter with the next generation of data access, we flattened everything to features and columns. It can show me how it built that chart, which dataset it used, and show me the metadata.” For Max, building trust in AI and data both tie back to metadata. Who built what for this number to get here? “We

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Ascend.io

MAY 23, 2023

They are blindsided when they read from a table they’ve been granted access to, only to find out it had been abandoned by the data engineers weeks ago. However, Ascend always keeps a record of every change in its metadata, so the tables themselves are not needed to look back at schemas, when changes were made, who made them, etc.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Databand.ai

JUNE 28, 2023

More people accessing the data and running their own pipelines and their own transformations causes errors and impacts data stability. Request a demo from one of our experts. To learn more about Databand and how our platform helps data engineers with their data pipelines, request a demo ! It isn’t enough to monitor anymore.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Silectis

FEBRUARY 3, 2021

With access to data lineage tooling, a data producer is empowered to quickly identify each of the impacted downstream pipelines and notify the owners of the error. Backed by a robust metadata repository, Magpie leverages activity tracking and data governance to offer a comprehensive suite of automated data lineage features.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

APRIL 24, 2023

However, one of the biggest trends in data lake technologies, and a capability to evaluate carefully, is the addition of more structured metadata creating “lakehouse” architecture. Amazon S3 and/or Lake Formation Amazon S3 is a popular storage platform to build and store data lakes thanks to its high availability and low latency access.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Cloudera

DECEMBER 10, 2020

In a multi-tenant environment, many users access the same data sources. The very high degree of compression built into these file formats reduces storage space and time to access storage. These techniques range from distributing concurrent query for overall throughput to metadata caching, data caching, and results caching.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Workfall

AUGUST 23, 2023

Hands-On Required installations: To perform the demo, you require the following installations: Visual Studio Code (IDE): It is a feature-rich source code editor designed to enhance coding speed and efficiency by offering a wide range of useful tools. Once prepared, we will establish a dedicated directory for our extension project.

Metadata 83

Metadata Coding Project Utilities 83

Snowflake

DECEMBER 7, 2023

retail media market is surging as retailers capitalize on the consumer shift to ecommerce while offering advertisers access to their unique audiences and data insights. Known data can serve as metadata around customers’ purchasing history, affinities and buying intent. And for good reason: the $45 billion U.S.

Media 83

Media Retail Digital Media Data 83

Data Engineering Weekly

JULY 23, 2023

Atlan AI leverages metadata that Atlan captures across the data stack to make AI part of your data stack. Access the guide today to see how your peers are boosting data adoption and maximizing the return on their data products. Write SQL queries without learning SQL? Find the right, trusted data by simply asking a question.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Data Engineering Podcast

SEPTEMBER 23, 2018

Summary As your data needs scale across an organization the need for a carefully considered approach to collection, storage, organization, and access becomes increasingly critical. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Data Engineering Weekly

JULY 9, 2023

ByteDance open sources ByConity, a scalable cloud-native data warehouse built on top of ClickHouse using FoundationDB for its metadata storage. Atlan AI leverages metadata that Atlan captures across the data stack to make AI part of your data stack. Write SQL queries without learning SQL?

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Rockset

AUGUST 28, 2023

We’ll create a query lambda with the following SQL query on our ShopEvents collection: SELECT COUNT(*) As ItemsSold FROM "Demo-Ecommerce".ShopEvents We can do this by creating a SQL query that will get all Checkout events for the previous day: INSERT INTO "Demo-Ecommerce".OrdersAnalytics ShopEvents e INNER JOIN "Demo-Ecommerce".Products

AWS 52

AWS SQL Datasets Metadata 52

Silectis

DECEMBER 15, 2020

We built Magpie with integrated security and data governance features including data lineage, detailed activity tracking, object-level access control, metadata management, scheduling and dependency management, comprehensive reporting, and DevOps support. Request your demo below and we’ll be in touch to get you set up.

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Cloudera

AUGUST 31, 2021

It has a consistent framework that secures and provides governance for all data and metadata on private clouds, multiple public clouds, or hybrid clouds. The data from your existing data warehouse is migrated to the storage option you choose, and all the metadata is migrated into SDX (Shared Data Experiences) layer of Cloudera Data Platform.

Data Warehouse 81

Data Warehouse Database-centric Metadata Cloud 81

dbt Developer Hub

JANUARY 24, 2023

The solution: Autoscaling CI ​ As of this writing, autoscaling CI is functionality built only within a python package, dbtc ; dbtc is an unaffiliated python interface to the dbt Cloud Metadata and Administrative APIs. A query method on the metadata property. x" - name : Trigger Autoscaling CI Job run : | pip install dbtc==0.3.3

Metadata 52

Metadata Python Cloud Data Warehouse 52

Rockset

APRIL 18, 2023

While Rockset can be used as a stand-alone feature store for real-time ML, in today’s demo we’re going to walk through how to use Rockset with the Feast Feature Store which is tailored to make machine learning feature management a breeze. We will use one of the files for the demo but the steps below can be applied to all files.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52