Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. Additionally, the PartitionAssignor interface exposes a metadata() method. they’re basically a short window of unavailability.

Kafka 111

Kafka Accessible Accessibility Metadata 111

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata AWS Cloud Database 52

Cloudera

SEPTEMBER 28, 2021

Cloudera Data platform ( CDP ) provides a Shared Data Experience ( SDX ) for centralized data access control and audit in the Enterprise Data Cloud. The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. Changes with file access control .

Accessible 62

Accessible Accessibility Cloud Cloud Storage 62

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular.

Metadata 92

Metadata Government Data Governance Data 92

Snowflake

APRIL 11, 2024

At Snowflake, we believe in making the power of data accessible to all. Not only do we have a unique vantage point into the challenges faced by data analysts, we also possess rich metadata that feeds into Snowflake’s dedicated text-to-SQL model that Copilot leverages in combination with Mistral’s technology.

SQL 134

SQL AWS Data Analysis High Quality Data 134

Data Engineering Podcast

FEBRUARY 5, 2023

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses. Go to [dataengineeringpodcast.com/materialize]([link] Support Data Engineering Podcast

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Engineering at Meta

MARCH 18, 2024

Users can query using regular expressions on log lines, arbitrary metadata fields attached to logs, and across log files of hosts and services. Each log line can have zero or more metadata key-value pairs attached to it. The extracted key-value pairs are added to the log line’s metadata. in PyTorch). Multimodal data (e.g.,

Engineering 89

Engineering Metadata Architecture Designing 89

The Modern Data Company

JANUARY 22, 2024

By humanizing data and streamlining access, we’re redefining how organizations harness their data’s power to drive innovation and achieve transformative business outcomes. Data products contain data and its metadata, code, and infrastructure so that each product is self-contained and usable independently.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 52

Database-centric Metadata MongoDB Pipeline-centric 52

Snowflake

NOVEMBER 29, 2023

In medicine, lower sequencing costs and improved clinical access to NGS technology has been shown to increase diagnostic yield for a range of diseases, from relatively well-understood Mendelian disorders, including muscular dystrophy and epilepsy , to rare diseases such as Alagille syndrome.

Healthcare 76

Healthcare Metadata Medical Data Storage 76

ThoughtSpot

OCTOBER 9, 2023

How ThoughtSpot builds trust with data catalog connectors For many, the data catalog is still the primary home for metadata enrichment and governance. Our data catalog integrations allow you to tap into this metadata wealth and surface it in the context where it’s needed most—when conducting business analytics.

Metadata 83

Metadata Government Data Data Governance 83

Cloudera

APRIL 2, 2024

We understand the importance of protecting sensitive government data from unauthorized access, and we have implemented robust security measures to safeguard this information. Purpose Built for Security Here at Cloudera, we focus on ensuring that our platform meets the highest standards of security and regulatory compliance.

Designing 96

Designing Process Government Metadata 96

Data Engineering Podcast

NOVEMBER 13, 2022

Shane Gibson co-founded AgileData to make analytics accessible to companies of all sizes. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Building 130

Building Metadata MongoDB MySQL 130

Snowflake

NOVEMBER 2, 2023

Snowflake Horizon is Snowflake’s built-in governance solution with a unified set of compliance, security, privacy, interoperability, and access capabilities. Snowflake continues to advance Snowflake Horizon with additional capabilities for compliance, security, privacy, interoperability, and access.

Metadata 97

Metadata Government AWS Medical 97

Rockset

MARCH 18, 2024

There are several challenges that emerge when building real-time RAG pipelines: Real-time delivery of embeddings & updates Real-time metadata filtering Scale and efficiency for real-time data In the following sections, we’ll discuss these challenges broadly and delve into how they apply more specifically to vector search and vector databases.

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. This opens the door for busy employees to access and analyze their company information away from their desk, but it has the more powerful effect of bringing first-class support to companies operating in mobile-first economies.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Many conversations around data and analytics are focused on self-service access. Atlan is the metadata hub for your data ecosystem.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Knowledge Hut

JANUARY 2, 2024

Monitoring had to be made more accessible, democratized and expanded to include additional stack tiers. Multi-dimensional data model Similar to how Kubernetes labels infrastructure metadata, the model's structure is built on key-value pairs. YAML files are used by Prometheus to access resources.

Architecture 77

Architecture Metadata Utilities Data Collection 77

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Support Data Engineering Podcast Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Go to [materialize.com]([link] today and get 2 weeks free!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

FEBRUARY 19, 2023

Your host is Tobias Macey and today I'm interviewing Ryan Blue about the evolution and applications of the Iceberg table format and how he is making it more accessible at Tabular Interview Introduction How did you get involved in the area of data management?

IT 147

IT Data Lake Metadata Data Warehouse 147

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 83

Metadata Python Government AWS 83

Snowflake

JANUARY 23, 2024

Snowpark ML Operations: Model management The path to production from model development starts with model management, which is the ability to track versioned model artifacts and metadata in a scalable, governed manner. The Snowpark Model Registry API provides simple catalog and retrieval operations on models.

Machine Learning 116

Machine Learning Metadata Python Telecommunication 116

Cloudera

APRIL 1, 2024

Our cutting-edge Shared data experience (SDX) service provides a unified control plane for common security, governance and metadata management on all structured and unstructured data. It gives the flexibility to teams to use their choice of tools/engines without worrying about ETL.

Cloud 101

Cloud Unstructured Data Metadata Datasets 101

François Nguyen

MARCH 22, 2021

and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…). The core idea of Data Mesh is how you can develop the data usages and remove the centralized and monolitich data warehouse where you have very less access. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

The Modern Data Company

APRIL 15, 2024

Manages information about data access and monitors data quality. It’s a combination of the data itself, the metadata (information about the data), code (the set of instructions on how to process or analyze the data), and infrastructure (the technological framework that supports it all).

Government 52

Government Metadata Data Data Lake 52

Towards Data Science

FEBRUARY 6, 2024

Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Accessibility : I could easily request access to these data products. Metadata update Data products need to be understandable. Does it sound familiar?

Systems 82

Systems Raw Data Data Cleanse Metadata 82

The Modern Data Company

JANUARY 22, 2024

By humanizing data and streamlining access, we’re redefining how organizations harness their data’s power to drive innovation and achieve transformative business outcomes. Data products contain data and its metadata, code, and infrastructure so that each product is self-contained and usable independently.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Striim

FEBRUARY 5, 2024

It is not a standalone product but comprises key elements, including data integration, ensuring the smooth merging of data; data quality, maintaining high data standards; metadata management, organizing and understanding data context; and security, safeguarding data integrity.

Pharmaceutical 52

Pharmaceutical Data Cleanse Metadata Medical 52

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Data lineage and metadata systems are a hot topic right now. The only thing worse than having bad data is not knowing that you have it.

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Snowflake

APRIL 23, 2024

The connector provides immediate access to up-to-date ServiceNow data without the need to manually integrate against API endpoints. Customers get a fully managed service with no additional access fees set by Snowflake. It’s available with no additional access fees set by Snowflake and uses Snowflake credits for compute and storage.

Data Warehouse 77

Data Warehouse Data Integration Consulting Metadata 77

Data Engineering Podcast

AUGUST 21, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Metadata 100

Metadata MongoDB MySQL Data Lake 100