Precisely

JULY 21, 2023

Customize the Replication Script: Use the scripting language provided by Precisely to define variables, specify metadata replication preferences, and map Cobol copybook descriptions for VSAM files. The Confluent and Precisely partnership helps make this possible and addresses the challenges of tomorrow – watch the demo now.

Data Integration 52

Data Integration Kafka Bytes Banking 52

Cloudera

MARCH 17, 2023

The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62

Cloudera

JULY 14, 2023

They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Ascend.io

DECEMBER 19, 2022

Improved Support for Databricks To highlight our improved Databricks capabilities, our re:Invent booth was next to theirs, and we chose to power our demos with their Lakehouse. Instead, it is a Sankey diagram driven by the same dynamic metadata that runs the Ascend control plane. If you were wondering why, here is your answer.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Rockset

SEPTEMBER 26, 2023

AI-powered applications almost always need access to real-time data to deliver accurate results in a responsive user experience that the market has come to expect. Metadata filtering is a useful, perhaps even essential, companion to vector search that restricts nearest-neighbor matches based on specific criteria.

Metadata 64

Metadata Kafka Cloud Database 64

Ascend.io

MAY 23, 2023

They are blindsided when they read from a table they’ve been granted access to, only to find out it had been abandoned by the data engineers weeks ago. However, Ascend always keeps a record of every change in its metadata, so the tables themselves are not needed to look back at schemas, when changes were made, who made them, etc.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Databand.ai

JUNE 28, 2023

More people accessing the data and running their own pipelines and their own transformations causes errors and impacts data stability. Request a demo from one of our experts. To learn more about Databand and how our platform helps data engineers with their data pipelines, request a demo ! It isn’t enough to monitor anymore.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Silectis

FEBRUARY 3, 2021

With access to data lineage tooling, a data producer is empowered to quickly identify each of the impacted downstream pipelines and notify the owners of the error. Backed by a robust metadata repository, Magpie leverages activity tracking and data governance to offer a comprehensive suite of automated data lineage features.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

APRIL 24, 2023

However, one of the biggest trends in data lake technologies, and a capability to evaluate carefully, is the addition of more structured metadata creating “lakehouse” architecture. Amazon S3 and/or Lake Formation Amazon S3 is a popular storage platform to build and store data lakes thanks to its high availability and low latency access.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Cloudera

DECEMBER 10, 2020

In a multi-tenant environment, many users access the same data sources. The very high degree of compression built into these file formats reduces storage space and time to access storage. These techniques range from distributing concurrent query for overall throughput to metadata caching, data caching, and results caching.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Workfall

AUGUST 23, 2023

Hands-On Required installations: To perform the demo, you require the following installations: Visual Studio Code (IDE): It is a feature-rich source code editor designed to enhance coding speed and efficiency by offering a wide range of useful tools. Once prepared, we will establish a dedicated directory for our extension project.

Metadata 83

Metadata Coding Project Utilities 83

Snowflake

DECEMBER 7, 2023

retail media market is surging as retailers capitalize on the consumer shift to ecommerce while offering advertisers access to their unique audiences and data insights. Known data can serve as metadata around customers’ purchasing history, affinities and buying intent. And for good reason: the $45 billion U.S.

Media 85

Media Retail Digital Media Data 85

Data Engineering Podcast

SEPTEMBER 23, 2018

Summary As your data needs scale across an organization the need for a carefully considered approach to collection, storage, organization, and access becomes increasingly critical. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science.

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Rockset

AUGUST 28, 2023

We’ll create a query lambda with the following SQL query on our ShopEvents collection: SELECT COUNT(*) As ItemsSold FROM "Demo-Ecommerce".ShopEvents We can do this by creating a SQL query that will get all Checkout events for the previous day: INSERT INTO "Demo-Ecommerce".OrdersAnalytics ShopEvents e INNER JOIN "Demo-Ecommerce".Products

AWS 52

AWS SQL Datasets Metadata 52

Silectis

DECEMBER 15, 2020

We built Magpie with integrated security and data governance features including data lineage, detailed activity tracking, object-level access control, metadata management, scheduling and dependency management, comprehensive reporting, and DevOps support. Request your demo below and we’ll be in touch to get you set up.

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Cloudera

AUGUST 31, 2021

It has a consistent framework that secures and provides governance for all data and metadata on private clouds, multiple public clouds, or hybrid clouds. The data from your existing data warehouse is migrated to the storage option you choose, and all the metadata is migrated into SDX (Shared Data Experiences) layer of Cloudera Data Platform.

Data Warehouse 81

Data Warehouse Database-centric Metadata Cloud 81

dbt Developer Hub

JANUARY 24, 2023

The solution: Autoscaling CI ​ As of this writing, autoscaling CI is functionality built only within a python package, dbtc ; dbtc is an unaffiliated python interface to the dbt Cloud Metadata and Administrative APIs. A query method on the metadata property. x" - name : Trigger Autoscaling CI Job run : | pip install dbtc==0.3.3

Metadata 52

Metadata Python Cloud Data Warehouse 52

Rockset

APRIL 18, 2023

While Rockset can be used as a stand-alone feature store for real-time ML, in today’s demo we’re going to walk through how to use Rockset with the Feast Feature Store which is tailored to make machine learning feature management a breeze. We will use one of the files for the demo but the steps below can be applied to all files.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Netflix Tech

OCTOBER 27, 2021

Instead of a fancy GUI for demos and presentations, we wanted a serious productivity tool with carefully thought-out user workflows that would fit seamlessly into our toolchain of data science. We also had no need to build a model-monitoring GUI yet, which is a wholly separate problem domain. Example task-level plugin Try it at home!

Python 89

Python Machine Learning Data Science Metadata 89

Cloudera

APRIL 23, 2021

Access to HDFS data can be managed by Apache Ranger HDFS policies and audit trails help administrators to monitor the activity. However, any user with HDFS admin or root access on cluster nodes would be able to impersonate the “hdfs” user and access sensitive data in clear text. wget [link]. wget [link].

MySQL 70

MySQL Java Bytes Data 70

Monte Carlo

JULY 10, 2023

For example, let’s say a streaming provider has 10 million overall subscribers who can access its content. Data profiling Data profiling uses a tool or programming language to surface metadata about your data. Contact our team to schedule a personalized demo of the Monte Carlo data observability platform.

Data Collection 40

Data Collection Data Governance Government Data 40

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Metadata 70

Metadata Java PostgreSQL Data Warehouse 70

Snowflake

FEBRUARY 29, 2024

They need an easier access path to convert that latitude and longitude into something more powerful. This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Caption: A demo Streamlit app visualizes H3 hexagons at different resolutions for the same data.

Insurance 85

Insurance SQL BI Consulting 85

Data Engineering Podcast

MAY 18, 2021

High-growth startups use Molecula’s feature store because of its unprecedented speed, cost savings, and simplified access to all enterprise data. From feature extraction to model training to production, the Molecula feature store provides continuously updated feature access, reuse, and sharing without the need to pre-process data.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. Rockset supports streaming updates to metadata and vectors in an efficient way. Multiple virtual instances can simultaneously access the same dataset, eliminating the need for multiple replicas of data.

Cloud 64

Cloud Metadata Database SQL 64

Snowflake

JANUARY 23, 2024

Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.

Machine Learning 120

Machine Learning Metadata Python Telecommunication 120

Zalando Engineering

NOVEMBER 29, 2017

As mentioned above, in many cases applications that use Kafka Streams rely on heavy utilization of locally persisted RocksDBs for storage and quick data access. Monitoring Consumer Lag By looking at the metadata of a Kafka Consumer Group, we can determine a few key metrics. How many messages are being read from those partitions?

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. For metadata management, you can configure Snowflake to manage your Iceberg data or use an external Iceberg catalog.

Metadata 85

Metadata AWS Data Lake Datasets 85

Monte Carlo

MAY 25, 2023

That’s why we’re excited to announce that uniting diverse data team personas to collectively ensure data quality just got easier, thanks to a new integration between Monte Carlo’s data observability platform and Atlan’s active metadata platform. The pipeline might succeed, but we wouldn’t know if the data was right, wrong, or indifferent.

Metadata 52

Metadata Data Governance Government Business Intelligence 52

Snowflake

NOVEMBER 1, 2023

And because the Notebook is natively integrated into Snowflake’s role-based access controls (RBAC), it’s easy to securely share and collaborate on your code and results without compromising on any enterprise data. Check out the Snowpark ML demo from Snowday to see the latest launches in action.

Building 105

Building Python SQL Programming Language 105

Cloudera

AUGUST 4, 2021

Trends toward more self-service data access and real-time applications have made data governance more visible and critical than ever before. The data is profiled and enhanced with rich metadata—including operational, social, and business context—creating trusted and reusable data assets and making them discoverable.

Data Governance 84

Data Governance Government Metadata Data Security 84

Snowflake

OCTOBER 2, 2023

Access to a complete view of the customer is incredibly powerful. The Snowflake Data Cloud offering enables access to all of your customer data alongside infinite compute power to model it. Accelerate AI projects with historical snapshots and metadata Operationalizing AI is quickly becoming a competitive requirement for businesses.

Cloud 102

Cloud Building Insurance Data Engineering 102

The Modern Data Company

MARCH 7, 2023

However, more often than not, it conjures up the idea of a central authority strictly guarding against such access. A data taxonomy uses a system of unambiguous metadata terms (such as a filename or tags attached to a file) that allow an enterprise to classify a file or dataset into important business categories.

Data Governance 52

Data Governance Government Metadata Datasets 52

Monte Carlo

JANUARY 23, 2024

As the machine became smarter with the next generation of data access, we flattened everything to features and columns. It can show me how it built that chart, which dataset it used, and show me the metadata.” For Max, building trust in AI and data both tie back to metadata. Who built what for this number to get here? “We

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Cloudera

SEPTEMBER 15, 2022

Furthermore, data stored in Ozone can be accessed for various use cases via different protocols, eliminating the need for data duplication, which in turn reduces risk and optimizes resource utilization. Interoperability of the same data for several workloads: multi-protocol access. Diversity of workloads. OBJECT_STORE Bucket (“OBS”).

Systems 87

Systems Hadoop Metadata Telecommunication 87

Netflix Tech

NOVEMBER 23, 2020

You can access the notification with a // type assertion: leaf.Value().(*gnmipb.Notification) Although our NMS (and therefore its API) is not open source, we included a Target Loader plugin for loading devices from NetBox as well as from watched files.

Transportation 74

Transportation Coding Certification Utilities 74

The Modern Data Company

JANUARY 17, 2023

Likewise, the enterprise as a whole has no access to the data in the CDP. Without lineage or governance metadata, the quality of this data is anyone’s guess. It can apply metadata and track the lineage of CDP data to improve data quality. Collecting more data isn’t always a good thing.

Metadata 59

Metadata Government Data Lake Data 59