Data Engineering Podcast

JUNE 19, 2022

In this episode Ryan explains how he and his team have designed their platform to bring everyone onto a level playing field and the benefits that it provides to the organization. Atlan is the metadata hub for your data ecosystem. How have the design and goals of the product changed/evolved since you started working on it?

Unstructured Data 130

Unstructured Data Metadata MongoDB MySQL 130

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the design and goals changed since you started working on it? Atlan is the metadata hub for your data ecosystem.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

NOVEMBER 13, 2022

In this episode he explains the design of the platform and how it builds on agile development principles to help you focus on delivering value. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Building 130

Building Metadata MongoDB MySQL 130

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Psyberg: The Game Changer! This is mainly used to identify new changes since the last update.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Podcast

OCTOBER 16, 2022

Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics. Atlan is the metadata hub for your data ecosystem. What are the types of data that Pathway is designed to work with?

Database 100

Database Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 17, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the design and goals of the platform changed or evolved since you started working on it?

Data Management 130

Data Management Management Metadata MongoDB 130

Christophe Blefari

MARCH 1, 2023

You can also add metadata on models (in YAML). This is a methodology that has been designed more than 20 years ago that optimise the storage used. All your business logic will be in the model select statements. You should also know that model are defined in.sql files and that the filename is the name of the model by default.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares the design of the project and how it fits into your development practices. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. platform design, pipeline design, data usage, etc.) Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How did you approach the design of the SodaCL syntax to balance flexibility for various use cases, with structure and opinionated application?

Building 100

Building Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Data lineage and metadata systems are a hot topic right now. How have the design and goals of the product changed or evolved?

IT 100

IT Metadata MongoDB MySQL 100

The Modern Data Company

APRIL 15, 2024

It’s a combination of the data itself, the metadata (information about the data), code (the set of instructions on how to process or analyze the data), and infrastructure (the technological framework that supports it all). This “metadata” acts as the soul of your data, enriching it and making it more valuable.

Government 52

Government Metadata Data Data Management 52

Data Engineering Podcast

DECEMBER 19, 2021

The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. What are the shortcomings in the "data catalog" approach to metadata collection and presentation? How has that focus informed your priorities for user experience design and feature development?

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. What was the impact of leading multiple teams on the design of how to implement communication/contracts throughout the mesh?

Building 147

Building Metadata Machine Learning Data Integration 147

Data Engineering Podcast

DECEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. How have the design and goals of the system changed/evolved since you started working on it?

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

ThoughtSpot

MAY 31, 2023

The architecture is designed to be resilient against new-age attacks against LLMs like prompt injection and prompt leaks. This multi-tenant service isolates the tenant metadata index, authorizing and filtering the search answer requests from every tenant.

Data Security 86

Data Security Metadata Transportation Data Warehouse 86

Data Engineering Weekly

MARCH 24, 2024

The logging engine to debug AI workflow logs is an excellent system design study if you’re interested in it. The APIs support emitting unstructured log lines and typed metadata key-value pairs (per line). The extracted key-value pairs are written to the line’s metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. What are the organizational constraints that have the biggest impact on the design and usage of your data systems?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Azure Data Engineering

NOVEMBER 21, 2021

An example could be when we want to check the existence of a file or folder using Get Metadata activity. While the activity itself is very simple, the real utility comes from the way the control flow of the pipeline is designed. We may want to fail the pipeline if the file/folder does not exist.

Metadata 130

Metadata Data Utilities Coding 130

Data Engineering Podcast

MAY 22, 2022

What are some of the ways that that manifests in the design and feature set? How have the design and goals of the platform changed/evolved since you first started working on it? What are the user personas that you prioritize in the design and feature development for Flyte? Can you describe the architecture of Flyte?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Engineering at Meta

SEPTEMBER 7, 2023

However, traditional full workload benchmarking presents several challenges: Difficulty in forecasting future system performance : When designing an AI system, engineers frequently face the challenge of predicting the performance of future systems. We also plan to incorporate an obfuscation mechanism within the AI model.

Metadata 101

Metadata Utilities Systems Designing 101

Data Engineering Podcast

JULY 3, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Can you describe what you see as the dominant factors that influence a team’s approach to data architecture and design?

Architecture 100

Architecture Metadata MongoDB MySQL 100

François Nguyen

MARCH 22, 2021

and he/she has different actions to execute (reading, calling a vision API, transform, create metadata, store them, etc…). The way you have designed your data domains could help to handle the problem but at the end we should be very focus on the level of usage and the way these data domains interact together.

Technology 130

Technology Architecture Google Cloud Metadata 130

phData: Data Engineering

SEPTEMBER 19, 2023

Data modeling methodologies are systematic approaches used to design and define the structure and relationships of data within a system. Metadata Management Data modeling methodologies help in managing metadata within the data lake. Metadata describes the characteristics, attributes, and context of the data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Podcast

MARCH 27, 2022

How have the design and goals of the system changed or evolved since you started working on it? Acryl]([link] The modern data stack needs a reimagined metadata management platform. Acryl Data’s vision is to bring clarity to your data through its next generation multi-cloud metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Lyft Engineering

SEPTEMBER 6, 2023

Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to. Context propagation : routing overrides metadata is propagated throughout the service mesh. Routing overrides metadata with ProxyApp Staging overrides was well-received by our product engineers.

Building 79

Building Metadata Electronics Engineering 79

Knowledge Hut

MARCH 7, 2024

Two, it creates a commonality of data definitions, concepts, metadata and the like. Even the way Big Data is designed makes it harder for enterprises to ensure data security. This implies two things, one, the data coming from one source is out of date when compared to another source.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Ascend.io

FEBRUARY 28, 2024

Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing. This detailed metadata allows AI systems to better understand the nuances of data, including its origin, purpose, and limitations, thus improving model accuracy and applicability.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. According to this Spiceworks article, “Interoperability between components and integration readiness are defining traits of this design.” Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. According to this Spiceworks article, “Interoperability between components and integration readiness are defining traits of this design.” Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. They are designed to complement existing workflows, helping to streamline the transition of models from development to production.

Machine Learning 85

Machine Learning Metadata Cloud Project 85

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. Metadata update Data products need to be understandable. Producers need to incorporate meaningful metadata for the entities and properties contained.

Systems 82

Systems Raw Data Data Cleanse Metadata 82

Netflix Tech

JANUARY 25, 2023

For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. In Marken, an annotation is a piece of metadata which can be attached to an object from any domain. Actual data is stored in the metadata section of json. In this case it is BOUNDING_BOX.

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Knowledge Hut

FEBRUARY 1, 2024

In general, the architecture of a movie recommender system process is intricately designed to provide a seamless, enjoyable movie experience for users. Content-Based Filtering Content-based filtering utilizes the attributes & metadata of a movie to generate recommendations that share similar properties.

Systems 98

Systems Entertainment Datasets Algorithm 98