Data Engineering Podcast

JULY 3, 2022

These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. Can you describe the system architecture that you have built at Acryl? What are some examples of automated actions that can be triggered from metadata changes?

Metadata 100

Metadata BI Data Warehouse Government 100

Cloudera

SEPTEMBER 29, 2022

They are being bombarded with literature about seemingly independent new trends like data mesh and data fabric while dealing with the reality of having to work with hybrid architectures. Each of these trends claim to be complete models for their data architectures to solve the “everything everywhere all at once” problem.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. What are some of the types of information that you classify and collect as metadata? Can you talk through the architecture of your platform?

Business Intelligence 113

Business Intelligence Metadata Management Data Governance 113

AltexSoft

AUGUST 29, 2023

This article explains what a data lake is, its architecture, and diverse use cases. Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. An example of a data lake architecture. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

AUGUST 13, 2022

Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Metadata 100

Metadata MongoDB Scala MySQL 100

Databand.ai

AUGUST 30, 2023

DataOps Architecture: 5 Key Components and How to Get Started Ryan Yackel August 30, 2023 What Is DataOps Architecture? A DataOps architecture is the structural foundation that supports the implementation of DataOps principles within an organization. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

Rockset

APRIL 26, 2023

We’ve noticed many common patterns across streaming data architectures and we’ll be sharing a blueprint for three of the most popular: anomaly detection, IoT, and recommendations. Our example architecture for anomaly detection will leverage both historical data and website activity to search for suspiciously low transaction counts.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Monte Carlo

DECEMBER 20, 2023

So what is the optimal data observability architecture or data quality coverage strategy? Data observability adds the ability to monitor metadata and logs from the data systems across your stack for full visibility into anomalous behavior. Monitoring for bad data is like fishing. And, even in the cloud, you can’t be everywhere.

Architecture 40

Architecture Data Datasets Machine Learning 40

Cloudera

JULY 15, 2021

The release of Cloudera Data Platform (CDP) Private Cloud Base edition provides customers with a next generation hybrid cloud architecture. All three will be quorums of Zookeepers and HDFS Journal nodes to track changes to HDFS Metadata stored on the Namenodes. Introduction and Rationale. Networking .

Architecture 96

Architecture Cloud Kafka Hadoop 96

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Snowflake’s support for Iceberg Tables is now in public preview, helping customers build and integrate Snowflake into their lake architecture.

Building 87

Building Metadata Cloud Storage AWS 87

Monte Carlo

APRIL 16, 2022

The challenge with passive ETL or transformations in the warehouse The modern data warehouse architecture creates problems across many layers. There are several problems with an architecture and process that heavily relies on transforming data once it has entered the data warehouse. Image courtesy of Chad Sanderson.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. Lack of unified metadata makes it difficult to optimize pipelines, avoid redundant processing, and quantify data engineering costs.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Council

DECEMBER 21, 2020

Here's our December 2020 roundup of links from across the web that could be relevant to you: 1. The Modern Data Stack (Fishtown Analytics) This long-form post on the dbt blog is a must-read.

Metadata 52

Metadata Architecture Data Technology 52

Netflix Tech

NOVEMBER 14, 2023

It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Iceberg metadata and Psyberg’s own metadata form the backbone of its efficient data processing capabilities. All Iceberg tables have associated metadata that provide insight into changes or updates within the data tables.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Vusal Dadalov explains how the platform is implemented, the motivation for a truly open architecture, and how they have invested in integrating with the broader ecosystem to make it easy for you to get started. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB AWS MySQL 100

ThoughtSpot

MAY 31, 2023

The architecture is designed to be resilient against new-age attacks against LLMs like prompt injection and prompt leaks. Architecture Let's start with the big picture and tackle how we adjusted our cloud architecture with additional internal and external interfaces to integrate LLM.

Data Security 86

Data Security Metadata Transportation Data Warehouse 86

Knowledge Hut

FEBRUARY 1, 2024

Movie Recommendation System Architecture The movie recommendation system architecture is a complex process that utilizes various algorithms to suggest movies to users based on their preferences. The significant merit of collaborative filtering is that it can eliminate the effects of limited metadata & low audience size.

Systems 98

Systems Entertainment Datasets Algorithm 98

Engineering at Meta

MARCH 6, 2024

Our technical solution builds on Meta’s existing client / server architecture We think the best way to deliver interoperability is through a solution which builds on Meta’s existing client / server architecture [Figure 1]. WhatsApp’s chat protocol uses optimized XML stanzas to communicate with our servers.

Media 131

Media Architecture Metadata Data Storage 131

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Data Engineering Podcast

OCTOBER 16, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Database 100

Database Metadata MongoDB MySQL 100

Netflix Tech

NOVEMBER 13, 2023

The detailed temporal metadata SMAD provides about speech and music regions in a polyphonic audio mixture are a first step for structural audio segmentation, indexing and pre-processing audio for the following downstream tasks. All models in our experiments were trained by minimizing binary cross-entropy (BCE) loss.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Data Engineering Podcast

AUGUST 6, 2022

Summary Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Can you describe the current architecture of your data platform? Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

DECEMBER 19, 2021

The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. What are the shortcomings in the "data catalog" approach to metadata collection and presentation? What is your philosophy around data modeling or schema standardization for metadata records?

Systems 100

Systems Building Metadata Data Warehouse 100

Netflix Tech

JANUARY 25, 2023

For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. In Marken, an annotation is a piece of metadata which can be attached to an object from any domain. Actual data is stored in the metadata section of json. In this case it is BOUNDING_BOX.

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Data Engineering Podcast

MAY 22, 2022

Data lake architectures provide the best combination of massive scalability and cost reduction, but they aren’t always the most performant option. Can you describe the architecture of Flyte? Data orchestration engines are a natural point for generating and taking advantage of rich metadata.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

LinkedIn Engineering

DECEMBER 19, 2023

The historical upgrade system couldn’t adapt to architectural changes like the introduction of an observer namenode (Now handling a massive influx of read requests – 150K QPS – from services such as Trino ), ZKFC auto-failover, HDFS federation , etc. The orchestrator’s implementation of the upgrade procedure’s logic performs most of the work.

Big Data 92

Big Data Hadoop Metadata Data 92

Christophe Blefari

JANUARY 20, 2024

new concepts — in today's data engineering a lot of new concepts enter the field every year like quality, lineage, metadata management, governance, privacy, sharing, etc. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Lyft Engineering

SEPTEMBER 6, 2023

in microservice architectures. Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to. Context propagation : routing overrides metadata is propagated throughout the service mesh. Istio, Linkerd, Envoy  — created at Lyft!)

Building 80

Building Metadata Electronics Engineering 80

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. Sample of a high-level data architecture blueprint for Azure BI programs.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Workfall

JUNE 12, 2023

Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention. In real-time data streaming and event-driven architectures, CDC captures data changes to trigger actions or workflows. CDC also shapes the future of data engineering.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Cloudera

JUNE 7, 2023

Cloudera Shared Data Experience (SDX) is a core component of Cloudera Data Platform’s architecture. It operates independently from compute and storage layers, offering integrated security and governance based on metadata. This is the balancing act of security. Understanding your data is critical to protecting the data.

Government 78

Government Metadata Data Lake Data 78