Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. platform design, pipeline design, data usage, etc.) Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

Snowflake

APRIL 29, 2024

ESA will designate critical ICT providers in January 2025. All designated critical ICT third-party service providers will be subject to direct oversight by ESA. Managing Third-Party Risk FEs shall manage ICT third-party risk as an integral component of ICT risk within their ICT risk management framework.

Transportation 82

Transportation Data Governance Government Consulting 82

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. They are designed to complement existing workflows, helping to streamline the transition of models from development to production.

Machine Learning 82

Machine Learning Metadata Cloud Project 82

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms. One of the unique features of Atlan is its human-centric design.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. Bitrate versus quality comparison HDR-VMAF is designed to be format-agnostic — it measures the perceptual quality of HDR video signal regardless of its container format, for example, Dolby Vision or HDR10. The graphic below (Fig.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. Data modeling methodologies are systematic approaches used to design and define the structure and relationships of data within a system.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? Types of CDC Audit Columns: This method involves using designated columns within tables to track incremental changes. These additional columns store metadata like timestamps, user IDs, and change types, ensuring granular change tracking and auditability.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Podcast

JANUARY 23, 2022

In this episode he shares his career journey, the challenges related to management of data professionals, and the platform design that he and his team have built to power analytics at a large company. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow.

Building 100

Building Management Data Pipeline Metadata 100

Netflix Tech

JANUARY 25, 2023

For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. In Marken, an annotation is a piece of metadata which can be attached to an object from any domain. Actual data is stored in the metadata section of json. In this case it is BOUNDING_BOX.

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Workfall

AUGUST 23, 2023

These extensions are designed to enhance and customize the browsing experience by adding new features, modifying existing ones, or integrating with other web services. This file played a crucial role in defining the metadata, permissions, and settings for our extension. We will come up with more such use cases in our upcoming blogs.

Metadata 83

Metadata Coding Project Utilities 83

Lyft Engineering

SEPTEMBER 6, 2023

Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to. Context propagation : routing overrides metadata is propagated throughout the service mesh. Routing overrides metadata with ProxyApp Staging overrides was well-received by our product engineers.

Building 80

Building Metadata Electronics Engineering 80

Tweag

MARCH 11, 2024

But CISA encouraged me to publish the answer as a separate blog post. It achieves very high levels of reproducibility and provenance tracking of software artifacts by design, utilizing a functional and declarative language to describe software builds and their dependencies.

Metadata 60

Metadata Utilities Building Designing 60

Data Engineering Podcast

MARCH 22, 2021

Fivetran/Airbyte/Meltano/custom scripts) What are the moving pieces in a CDC workflow that need to be considered as you are designing the system? How has the design evolved as you have grown the scale and sophistication of your system? e.g. APIs and third party data sources How can we integrage CDC into metadata/lineage tooling?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Christophe Blefari

JANUARY 20, 2024

Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). new concepts — in today's data engineering a lot of new concepts enter the field every year like quality, lineage, metadata management, governance, privacy, sharing, etc.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Pinterest Engineering

NOVEMBER 7, 2023

For these reasons, and others detailed in our original PubSub Client blog post , our team has decided to invest in building, productionalizing, and most recently open-sourcing PubSub Client (PSC). years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

Cloudera

AUGUST 26, 2021

Ozone natively provides Amazon S3 and Hadoop Filesystem compatible endpoints in addition to its own native object store API endpoint and is designed to work seamlessly with enterprise scale data warehousing, machine learning and streaming workloads. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 107

Data Science Cloud Hadoop Metadata 107

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. These metrics should be aligned with your organization’s goals and objectives, and should be designed to track the progress of your data quality strategy over time.

Building 52

Building Data Cleanse Data Governance Datasets 52

Data Engineering Weekly

FEBRUARY 18, 2024

I believe the data ownership problem is much deeper than simple metadata management. Yet, there is no tool for the data producers to Define, Design, Collaborate, Test, & Continuously Monitor the Data Products. Fractional factorial design selects a subset of the possible combinations of factors to run as experiments.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Lyft Engineering

JUNE 28, 2023

In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way. Illustrated ML Model Development Flow at Lyft In our prior work in building LyftLearn Serving , a key design tenet was creating a uniform training and serving environment for models (in the form of a shared Docker image).

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. From a design viewpoint, a typedef is analogous to a class definition. ETL/DB Load process.

Data Governance 93

Data Governance Government Metadata Datasets 93

AltexSoft

FEBRUARY 11, 2023

A data architect is an IT professional responsible for the design, implementation, and maintenance of the data infrastructure inside an organization. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is a data architect?

Data Architect 87

Data Architect Certification Generalist Big Data 87

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? Ability to handle schema changes on the fly - we can store additional metadata such as ETA prediction accuracy and model version in the future. Let’s take a look at components that constitute the system.

Kafka 40

Kafka Designing Systems Food 40

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. Streaming compute however, empowers more complex window queries on semantic correctness.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Databand.ai

AUGUST 30, 2023

A well-designed DataOps architecture must address the challenges of integrating data from multiple sources, ensuring that data is clean, consistent, and accurate. This requires implementing robust data integration tools and practices, such as data validation, data cleansing, and metadata management.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

LinkedIn Engineering

DECEMBER 12, 2023

We are excited to announce the various contributions we have made to provide a privacy-by-design approach to measure and mitigate reidentification risks. This metadata is determined as part of an onboarding process with each new application where we determine which algorithms should be used for each type of query.

Algorithm 77

Algorithm Metadata SQL Datasets 77

LinkedIn Engineering

DECEMBER 19, 2023

The new Rolling Upgrade framework The new RU orchestration design significantly enhanced our big data components deployment process. The new orchestrator agent design offers versatility and significantly improves the big data deployment process, making it smoother and less prone to issues.

Big Data 92

Big Data Hadoop Metadata Data 92

Workfall

AUGUST 14, 2023

In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. In this blog, we will cover: Understanding DBT and Airflow What is DBT? Initialize the Airflow metadata database by running airflow initdb in your terminal. What is Airflow?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. I won’t bore you with the importance of data quality in the blog. Event Routers can add additional metadata to the envelope of the event. How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Rockset

NOVEMBER 14, 2022

In this blog, we will describe the SSB configuration, results and performance enhancements. Row-Store Cache We designed a Multiversion Concurrency Control (MVCC) cache for the row store to reduce the overhead of meta operations and thereby query latency when the working set fits into memory.

Metadata 52

Metadata Datasets Cloud Accessible 52

Datakin

AUGUST 5, 2021

Blog A real-time approach to data lineage Written by Ross Turk on August 5, 2021 A data ecosystem that spans multiple pipelines, teams, and platforms can be overwhelming. Lineage metadata is the thread that connects it all together. However, there is more than one way to capture lineage metadata.

Metadata 52

Metadata Relational Database Datasets SQL 52

Cloudera

JUNE 7, 2022

Cloudera Data Platform (CDP) is designed to address the critical requirements for modern data architectures today and tomorrow. Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. Integration, metadata and governance capabilities glue the individual components together.” .

IT 106

IT Unstructured Data Data Architecture Government 106

Cloudera

SEPTEMBER 24, 2020

However, there is a secret I am keeping to the end of the blog, which makes the decision even easier for the user: so easy in fact, you do not even have to decide yourself. Impala was designed for speed. . Hive LLAP was designed for sophistication. However, Hive is designed to be very fault-tolerant.

Data Warehouse 68

Data Warehouse SQL Engineering Metadata 68

phData: Data Engineering

DECEMBER 16, 2021

In our previous blog in this series , we spent a lot of time exploring why a data catalog is valuable and who you might need to support it. Perhaps one of the most crucial elements in deciding a data catalog that’s right for your business is exploring how your tool is designed when it comes to infrastructure.

Metadata 52

Metadata Datasets ETL Tools Cloud 52

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. But first, let’s define the data mesh design pattern. A schema designer who moves from project to project may not understand the nuances behind the requirements of each data consumer.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98