Netflix Tech

SEPTEMBER 28, 2021

The Revenue and Growth Tools (RGT) team decided to set off on a journey to build tools in an abstract manner to have solutions readily available within our organization. The traditional approach of bundling dependencies at build time lacks the flexibility we need to empower our stakeholders.

Building 96

Building Metadata Designing Coding 96

Data Engineering Podcast

JANUARY 23, 2022

He also provides some excellent insights into the factors that play into the build vs. buy decision at different organizational sizes. You can observe your pipelines with built in metadata search and column level lineage. What have been some of the recurring themes that are discussed in those events?

Building 100

Building Management Data Pipeline Metadata 100

Data Engineering Podcast

OCTOBER 28, 2019

In an effort to create a better abstraction for building data applications Nick Schrock created Dagster. In this episode he explains his motivation for creating a product for data management, how the programming model simplifies the work of building testable and maintainable pipelines, and his vision for the future of data programming.

Building 100

Building Data Pipeline Programming Language Metadata 100

Netflix Tech

JUNE 1, 2023

To do this, we devised a novel way to simulate the projected traffic weeks ahead of launch by building upon the traffic migration framework described here. New content or national events may drive brief spikes, but, by and large, traffic is usually smoothly increasing or decreasing.

Algorithm 136

Algorithm Metadata Kafka Systems 136

The Pragmatic Engineer

AUGUST 3, 2023

” Because of how useful this product is, especially for app developers building on top of it, this announcement came as a surprise. To make this migration easier and as seamless as possible, we will give developers the ability to export their deep-link metadata.” We will announce more information in Q3 2023.”

Metadata 159

Metadata Engineering Building Technology 159

Data Engineering Podcast

MARCH 18, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year! It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

Ascend.io

APRIL 22, 2024

A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. But as they add more data sources, implement real-time streaming pipelines, and build out a modern data lake, the complexity compounds quickly.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

AUGUST 21, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents.

Transportation 79

Transportation Data Governance Government Consulting 79

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata Electronics MongoDB MySQL 100

Lyft Engineering

OCTOBER 3, 2023

For example, we can almost instantly validate that each record is well-formed and complete during event generation. Our Analytic Event Lifecycle below demonstrates the workflow of how much of our data gets to Hive. We log these events asynchronously at the order of millions per second.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. A data observability platform?

IT 100

IT Metadata MongoDB MySQL 100

Workfall

JUNE 12, 2023

Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention. In real-time data streaming and event-driven architectures, CDC captures data changes to trigger actions or workflows.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Weekly

MAY 16, 2023

Like the staging environment, Fronting Kafka receives all the events without validation. A streaming consumer, often implemented in stream processing frameworks like Flink or Spark, consumes the events from the fronting Kafka and runs through data contract validation. Event Routers typically don’t alter the payload.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

François Nguyen

MARCH 22, 2021

How do we build data products ? How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum. .” What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Netflix Tech

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure.

Cloud 73

Cloud Building Amazon Web Services Metadata 73

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What do you see as the closest alternatives?

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

APRIL 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Netflix Tech

SEPTEMBER 29, 2022

Timestone clients can create queues, enqueue messages with user-defined deadlines and metadata, then dequeue these messages in an earliest-deadline-first (EDF) fashion. messages that belong to queue X and have metadata Y”) is also supported. Allow for dequeueing and queue depth querying using filters (metadata key-value pairs) 4.

Systems 85

Systems Metadata Media Kafka 85

Cloudera

MARCH 14, 2023

Key requirements for building data pipelines Every data pipeline starts with a business requirement. Developers who are tasked with building these data pipelines are looking for tooling that: Gives them a development environment on demand without having to maintain it.

Data Pipeline 82

Data Pipeline Designing Kafka Metadata 82

Rockset

SEPTEMBER 26, 2023

As smart as ChatGPT appears to be, it can’t summarize current events accurately if it was last trained a year ago and not told what’s happening now. Models may need to know about events, computed metrics, and embeddings based on locality. Decouple data from your data science tools and production AI apps to test and build faster.

Metadata 64

Metadata Kafka Cloud Database 64

Data Engineering Podcast

MARCH 22, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How do you handle observability of CDC flows?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Data Engineering Podcast

FEBRUARY 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

JULY 5, 2021

In this episode Bart and Rich Wood explain how SaaSGlue is architected to allow for a high degree of flexibility in usage and deployment, their experience building a business with family, and how you can get started using it today. I understand that you are building this company with your 3 brothers.

Systems 100

Systems Management Data Warehouse Programming Language 100

LinkedIn Engineering

DECEMBER 19, 2023

The framework incorporates proactive measures to monitor critical events, provides deployment insights, and issues proper notifications to enable seamless deployments. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes. Prioritization was essential in building this robust framework.

Big Data 92

Big Data Hadoop Metadata Data 92

Monte Carlo

MAY 18, 2023

We’ve written extensively on how to analyze the build vs buy decision across the modern data stack, but it’s important to speak specifically about open source data lineage options. Our team attempted to leverage Spline (one of the open source data lineage tools listed here) as a building block. Ok, that is enough caveat emptor.

Metadata 64

Metadata Data Lake Government Data Governance 64

Engineering at Meta

AUGUST 29, 2023

To address this problem, we invested in building a scheduled notebooks infrastructure that fits in seamlessly with the rest of the internal tooling available at Meta. Investing in infrastructure helps ensure that privacy is inherent in everything we build. An example of this is the SQL cell.

SQL 80

SQL Metadata Coding Data Analysis 80

Data Engineering Podcast

JUNE 8, 2021

Summary The way to build maintainable software and systems is through composition of individual pieces. In this episode Matteo Merli shares the story behind the creation of BookKeeper, the various ways that it is being used today, and the architectural aspects that make it such a strong building block for projects such as Pulsar.

Data Warehouse 100

Data Warehouse Hadoop Metadata Architecture 100

Netflix Tech

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems. For many of our applications, model explainability matters.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously. Let’s imagine a “Hello, World!”

Kafka 111

Kafka Transportation Metadata Consulting 111

Cloudera

DECEMBER 9, 2022

Now, we shift focus on the needs of developers and addressing the challenges they face when building dataflows in the cloud. We’ve observed organizations using more and more data sources and destinations , as well as expecting a more diverse range of developers to build data movement flows. Enabling self-service for developers.

Designing 97

Designing Coding Google Cloud AWS 97

Data Engineering Podcast

APRIL 5, 2021

In this episode Prukalpa Sankar shares her experiences across multiple attempts at building a system that brings everyone onto the same page, ultimately bringing her to found Atlan. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Rockset

APRIL 26, 2023

To learn more about how to build fraud detection models, this is a great resource. If the RTA database uses a batch or microbatch ingest method (ClickHouse or Apache Druid, for example), there could be significant delays between events being produced and their availability for querying.

Architecture 52

Architecture Transportation Data Lake Insurance 52