Confluent

MAY 7, 2019

Although it is the simplest way to subscribe to and access events from Kafka, behind the scenes, Kafka consumers handle tricky distributed systems challenges like data consistency, failover and load balancing. The new designated leader would then rebuild the state of the consumer group by reading the records in the log.

Kafka 111

Kafka Accessible Accessibility Metadata 111

Monte Carlo

MAY 14, 2024

Depending on the quantity of data flowing through an organization’s pipeline — or the format the data typically takes — the right modern table format can help to make workflows more efficient, increase access, extend functionality, and even offer new opportunities to activate your unstructured data.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Engineering at Meta

MARCH 18, 2024

In this post, we present the design behind Logarithm, and show how it powers AI training debugging use cases. We designed the system to support service-level guarantees on log freshness, completeness, durability, query latency, and query result completeness. Each log line can have zero or more metadata key-value pairs attached to it.

Engineering 91

Engineering Metadata Architecture Designing 91

The Modern Data Company

JANUARY 22, 2024

The central piece to this narrative is DataOS®, Modern’s state-of-the-art data operating system designed to unify data across silos on a scalable level. By humanizing data and streamlining access, we’re redefining how organizations harness their data’s power to drive innovation and achieve transformative business outcomes.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Knowledge Hut

MAY 26, 2024

It was designed by Google to manage and schedule containers at scale. The system is designed to simplify life cycle management, allowing developers to focus on their application code rather than infrastructure maintenance. This means that you can always know exactly where your data is stored and how it is accessed.

Metadata 52

Metadata Certification Accessible Accessibility 52

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Cloudera

SEPTEMBER 28, 2021

Cloudera Data platform ( CDP ) provides a Shared Data Experience ( SDX ) for centralized data access control and audit in the Enterprise Data Cloud. The Ranger Authorization Service (RAZ) is a new service added to help provide fine-grained access control (FGAC) for cloud storage. Changes with file access control .

Accessible 62

Accessible Accessibility Cloud Cloud Storage 62

Snowflake

APRIL 11, 2024

At Snowflake, we believe in making the power of data accessible to all. Not only do we have a unique vantage point into the challenges faced by data analysts, we also possess rich metadata that feeds into Snowflake’s dedicated text-to-SQL model that Copilot leverages in combination with Mistral’s technology.

SQL 137

SQL AWS Data Analysis High Quality Data 137

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Many conversations around data and analytics are focused on self-service access. Atlan is the metadata hub for your data ecosystem.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 13, 2022

Shane Gibson co-founded AgileData to make analytics accessible to companies of all sizes. In this episode he explains the design of the platform and how it builds on agile development principles to help you focus on delivering value. Atlan is the metadata hub for your data ecosystem.

Building 130

Building Metadata MongoDB MySQL 130

Rockset

MARCH 18, 2024

There are several challenges that emerge when building real-time RAG pipelines: Real-time delivery of embeddings & updates Real-time metadata filtering Scale and efficiency for real-time data In the following sections, we’ll discuss these challenges broadly and delve into how they apply more specifically to vector search and vector databases.

Cloud 64

Cloud Building Metadata Kafka 64

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Support Data Engineering Podcast Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Email hosts@dataengineeringpodcast.com ) with your story.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. This opens the door for busy employees to access and analyze their company information away from their desk, but it has the more powerful effect of bringing first-class support to companies operating in mobile-first economies.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

The Modern Data Company

JANUARY 22, 2024

The central piece to this narrative is DataOS®, Modern’s state-of-the-art data operating system designed to unify data across silos on a scalable level. By humanizing data and streamlining access, we’re redefining how organizations harness their data’s power to drive innovation and achieve transformative business outcomes.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Cloudera

JULY 13, 2023

Iceberg is an emerging open-table format designed for large analytic workloads. In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine.

Metadata 70

Metadata Java PostgreSQL Data Warehouse 70

ThoughtSpot

MAY 31, 2023

The architecture is designed to be resilient against new-age attacks against LLMs like prompt injection and prompt leaks. This includes services that: Manage and monitor the tenant-specific resources—this does not include access to tenant data Maintains indexed data to serve as your application home page.

Data Security 86

Data Security Metadata Transportation Data Warehouse 86

Knowledge Hut

JANUARY 2, 2024

Monitoring had to be made more accessible, democratized and expanded to include additional stack tiers. Multi-dimensional data model Similar to how Kubernetes labels infrastructure metadata, the model's structure is built on key-value pairs. YAML files are used by Prometheus to access resources.

Architecture 98

Architecture Metadata Utilities Data Collection 98

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. How is everyone going to find the data they need, and understand it?

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Snowflake

AUGUST 22, 2023

The data product lifecycle includes the following stages: Discovery Design Development Deployment Let’s take a look at what they entail and the roles that lead and support each of them: Discovery starts the process. Design adds shape to the concept. and including data models, application design and UI/UX design if necessary.

Metadata 73

Metadata Data AWS Business Analyst 73

Precisely

MAY 23, 2024

Redeployment of applications to the cloud, likewise, should be achievable without major disruption to your integration design. Integrated data catalog for metadata support As you build out your IT ecosystem, it is important to leverage tools that have the capabilities to support forward-looking use cases.

Data Integration 62

Data Integration Metadata Telecommunication Data Governance 62

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. platform design, pipeline design, data usage, etc.) Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. ESA will designate critical ICT providers in January 2025. All designated critical ICT third-party service providers will be subject to direct oversight by ESA.

Transportation 84

Transportation Data Governance Government Consulting 84

Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Data lineage and metadata systems are a hot topic right now. How have the design and goals of the product changed or evolved?

IT 100

IT Metadata MongoDB MySQL 100

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables (not files/blobs) are the only API abstraction for end-users.

Big Data 78

Big Data Data Management Management Metadata 78

François Nguyen

MARCH 22, 2021

and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…). The way you have designed your data domains could help to handle the problem but at the end we should be very focus on the level of usage and the way these data domains interact together.

Technology 130

Technology Architecture Google Cloud Metadata 130

The Modern Data Company

APRIL 15, 2024

Manages information about data access and monitors data quality. It’s a combination of the data itself, the metadata (information about the data), code (the set of instructions on how to process or analyze the data), and infrastructure (the technological framework that supports it all).

Government 52

Government Metadata Data Data Lake 52

Cloudera

OCTOBER 7, 2021

The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., Data and Metadata: Data inputs and data outputs produced based on the application logic. Key Design Principles of a Data Mesh.

Architecture 124

Architecture Metadata Government Kafka 124

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Accessibility : I could easily request access to these data products.

Systems 79

Systems Raw Data Metadata Data Cleanse 79

Knowledge Hut

MARCH 7, 2024

Two, it creates a commonality of data definitions, concepts, metadata and the like. As far as Big Data is concerned, data security should be high on their priorities as most modern businesses are vulnerable to fake data generation, especially if cybercriminals have access to the database of a business.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Precisely

OCTOBER 16, 2023

This shift has influenced how cloud resources are designed and marketed, focusing on easy access, modularity, and straightforward deployment. As a result, there is a widespread desire for access to timely, trustworthy data and the analytical tools to interpret it. We call that “structured flexibility.”

Cloud 70

Cloud Cloud Computing Metadata Data Governance 70

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. For any features above in private preview, please contact your Snowflake account manager to apply for access.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Data Engineering Podcast

NOVEMBER 27, 2022

This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. The Arrow project is designed to eliminate wasted effort in translating between languages, and Voltron Data was created to help grow and support its technology and community. Struggling with broken pipelines? Missing data?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Tweag

MAY 16, 2023

Since the initial release (0.1), we’ve refined the semantics enough to be confident in a core design that is unlikely to radically change. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code. The | symbol attaches metadata to fields.

MySQL 131

MySQL Metadata Data Validation Coding 131

Cloudera

SEPTEMBER 15, 2022

It was designed as a native object store to provide extreme scale, performance, and reliability to handle multiple analytics workloads using either S3 API or the traditional Hadoop API. Interoperability of the same data for several workloads: multi-protocol access. Diversity of workloads. Bringing files and objects under one roof.

Systems 87

Systems Hadoop Metadata Telecommunication 87

Cloudera

SEPTEMBER 9, 2022

Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. These new formats need to accommodate the massive scale increases while shortening the response windows for accessing, analyzing, and using these data sets for business insights.

Architecture 100

Architecture Metadata Unstructured Data Machine Learning 100

Cloudera

FEBRUARY 22, 2022

It has been designed and developed as an open community standard to ensure compatibility across languages and implementations. Key Design Goals . In addition, the File I/O implementation provides a way to read / write / delete files – this is required to access the data and metadata files with a well defined API.

Metadata 107

Metadata Datasets BI SQL 107