Cloudera

AUGUST 30, 2022

Modern data architectures. To eliminate or integrate these silos, the public sector needs to adopt robust data management solutions that support modern data architectures (MDAs). Data Mesh: A type of data platform architecture that embraces the ubiquity of data in the enterprise by leveraging a domain-oriented, self-serve design.

Data Architecture 78

Data Architecture Architecture Data Lake NoSQL 78

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? A DataOps architecture is the structural foundation that supports the implementation of DataOps principles within an organization. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

Cloudera

JANUARY 21, 2022

This blog will summarise the security architecture of a CDP Private Cloud Base cluster. The architecture reflects the four pillars of security engineering best practice, Perimeter, Data, Access and Visibility. System metadata is reviewed and updated regularly. Security Architecture Improvements.

Architecture 110

Architecture Transportation Certification Government 110

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. In this blog post we’ll dive into data vault architecture; challenges and best practices for maintaining data quality; and how data observability can help.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Cloudera

JULY 15, 2021

The release of Cloudera Data Platform (CDP) Private Cloud Base edition provides customers with a next generation hybrid cloud architecture. The recently released Cloudera Ansible playbooks provide the templates that incorporate best practices described in this blog post and can be downloaded from [link] . Introduction and Rationale.

Architecture 100

Architecture Cloud Kafka Hadoop 100

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Snowflake’s support for Iceberg Tables is now in public preview, helping customers build and integrate Snowflake into their lake architecture.

Building 96

Building Metadata Cloud Storage AWS 96

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. Streaming-first architectures are a necessary foundation for the AI era. What are the challenges building RAG pipelines?

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Can you describe the current architecture of your data platform?

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. This architecture shift greatly reduced the processing latency and increased system resiliency.

Process 91

Process Pipeline-centric Media Metadata 91

ProjectPro

JANUARY 31, 2022

This blog walks you through what does Snowflake do , the various features it offers, the Snowflake architecture, and so much more. Table of Contents Snowflake Overview and Architecture What is Snowflake Data Warehouse? On the other hand, Snowflake integrates an entirely new SQL query engine with unique cloud-native architecture.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

Data Council

DECEMBER 21, 2020

The Modern Data Stack (Fishtown Analytics) This long-form post on the dbt blog is a must-read. Here's our December 2020 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata Architecture Data Technology 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Each of these architectures has its own unique strengths and tradeoffs.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Cloudera

JULY 19, 2023

It is a replicated, highly-available service that is responsible for managing the metadata for all objects stored in Ozone. In this blog post, we will highlight the work done recently to improve the performance of Ozone Manager to scale to exabytes of data. The hardware specifications are included at the end of this blog.

Management 70

Management Metadata Datasets Architecture 70

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Metadata 90

Metadata Data Warehouse BI AWS 90

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Given our role on this critical path, accuracy is paramount. Psyberg: The Game Changer!

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Weekly

DECEMBER 10, 2023

Thoughtworks discusses Anonymesh architecture with case studies on how it is implemented. The blog is an excellent comparison study of Ray vs. Dask’s performance. From a data architecture point of view, this enables a lot of flexibility in integrating multiple systems. Stores metadata to utilize later.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Monte Carlo

MARCH 18, 2021

To meet these goals, we chose a hybrid deployment architecture , a new approach that marries on-prem security with SaaS convenience. For the customer, an obvious benefit of choosing a vendor that leverages an on-prem architecture is rooted in perceived security and compliance. The question was: how are we going to build them?

Architecture 52

Architecture Metadata Software Engineer Software Engineering 52

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention. It’s time to harness the pulse of the digital era and seize the endless opportunities that lie ahead.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Podcast

MARCH 25, 2023

The peril is that building a robust and scalable streaming architecture is always more complicated and error-prone than you think it's going to be. TimeXtender Logo]([link] TimeXtender is a holistic, metadata-driven solution for data integration, optimized for agility. Can you describe how Grainite is architected?

MySQL 130

MySQL Python Architecture Machine Learning 130

Cloudera

FEBRUARY 22, 2022

The data architecture layer is one such area where growing datasets have pushed the limits of scalability and performance. Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . Multi-function analytics .

Metadata 107

Metadata Datasets BI SQL 107

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

DoorDash Engineering

DECEMBER 5, 2023

Understanding the legacy architecture As shown in Figure 1 below, DoorDash’s legacy approach to Topic creation involved several steps within a Cadence workflow. All Topics, even those without any messages, include some metadata that is backed up to disk. Chronosphere : Cloud-native observability platform. Built atop Prometheus.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Rockset

NOVEMBER 7, 2023

Earlier this year, Rockset introduced a new cloud architecture with compute-storage and compute-compute separation. As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. Rockset supports streaming updates to metadata and vectors in an efficient way.

Cloud 64

Cloud Metadata Database SQL 64

Cloudera

SEPTEMBER 15, 2022

In this blog post, we will talk about a single Ozone cluster with the capabilities of both Hadoop Core File System (HCFS) and Object Store (like Amazon S3). A unified storage architecture that can store both files and objects and provide a flexible, scalable, and high-performance system. FILE_SYSTEM_OPTIMIZED Bucket (“FSO”).

Systems 87

Systems Hadoop Metadata Telecommunication 87

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. A proper survey of these approaches would fill many blog posts of this size. The same applies to the metadata of your system.

Metadata 52

Metadata Database Algorithm SQL 52

Christophe Blefari

JANUARY 20, 2024

Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). new concepts — in today's data engineering a lot of new concepts enter the field every year like quality, lineage, metadata management, governance, privacy, sharing, etc. Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Snowflake

AUGUST 31, 2023

With this release comes performance improvements, new concepts, new features and changes to how Snowflake can fit into an architecture with Iceberg: Unifying Iceberg Tables: Instead of two separate table types for Iceberg, we are combining Iceberg External Tables and Native Iceberg Tables into one table type with a similar user experience.

Metadata 85

Metadata AWS Data Lake Datasets 85

Databand.ai

JULY 19, 2023

However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. This feature is particularly useful in complex data architectures, where data may pass through multiple systems and transformations.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Netflix Tech

JANUARY 25, 2023

For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. In Marken, an annotation is a piece of metadata which can be attached to an object from any domain. Actual data is stored in the metadata section of json. In this case it is BOUNDING_BOX.

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Cloudera

MAY 2, 2022

There are many reasons to deploy a hybrid cloud architecture — not least cost, performance, reliability, security, and control of infrastructure. But increasingly at Cloudera, our clients are looking for a hybrid cloud architecture in order to manage compliance requirements.

Cloud 101

Cloud Government Healthcare Metadata 101

Cloudera

JUNE 7, 2023

Cloudera Shared Data Experience (SDX) is a core component of Cloudera Data Platform’s architecture. It operates independently from compute and storage layers, offering integrated security and governance based on metadata. The post How Cloudera Supports Zero Trust for Data appeared first on Cloudera Blog.

Government 81

Government Metadata Data Lake Data 81

Netflix Tech

MARCH 10, 2023

We built an asset management platform (AMP), codenamed Amsterdam , in order to easily organize and manage the metadata, schema, relations and permissions of these assets. In this blog, we will be focusing on how we utilize Elasticsearch for indexing and search the assets. This is the layer we’d like to focus on in this blog.

Management 103

Management Metadata Digital Media Kafka 103

Cloudera

DECEMBER 16, 2022

Cloudera enables an open data lakehouse architecture that combines all the flexibility of the data lake with the performance of the data warehouse, so enterprises can use all data — both structured and unstructured. 4-Ready for modern data fabric architectures. 2-A truly open data lakehouse. 5-Integrated open data collection.

Database 90

Database Cloud Systems Management 90

Cloudera

MARCH 23, 2022

In fact, we recently announced the integration with our cloud ecosystem bringing the benefits of Iceberg to enterprises as they make their journey to the public cloud, and as they adopt more converged architectures like the Lakehouse. 1: Multi-function analytics . 3: Open Performance.

Metadata 90

Metadata Data Architecture BI Machine Learning 90

LinkedIn Engineering

JANUARY 16, 2024

In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In this blog post, we'll go deeper into how we manage account restrictions. These records held vital metadata linked to the restriction, including essential timestamps.

Kafka 84

Kafka Relational Database Java Architecture 84