Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 91

Process Pipeline-centric Media Metadata 91

François Nguyen

MARCH 22, 2021

You are starting to be an operation or technology centric data team. This is really for us the definition of a self serve platform. ” Code : all the code necessary to build a data product (data pipelines, API, policies). To get out of this, you have to move to another stage : the serverless stage.

Technology 130

Technology Architecture Google Cloud Metadata 130

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Just like at first everyone had to code in a language, then everyone had to know how to incorporate packages from those languages — now we’re moving into, ‘ How do you incorporate AI that will write the code for you?’”

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Maxime Beauchemin

JANUARY 20, 2017

Like data scientists, data engineers write code. There’s a multitude of reasons why complex pieces of software are not developed using drag and drop tools: it’s that ultimately code is the best abstraction there is for software. They’re highly analytical, and are interested in data visualization.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Airbnb Tech

JUNE 1, 2021

Specifically, we will showcase how we standardize dataset definitions through declarative configurations, explain how data versioning enables us to ensure cross-dataset consistency, and illustrate how we backfill data efficiently with zero downtime. Anyone can look up definitions without confusion.

Datasets 52

Datasets Pipeline-centric Metadata Data Science 52

DataKitchen

JUNE 24, 2021

Whether it’s streaming, batch, virtualized or not, using active metadata, or just plain old regular coding, it provides a good way for the data and analytics team to add continuous value to the organization.”. Their definition of DataOps was that we do some automation and check a record count. Be business-centric.

Pipeline-centric 84

Pipeline-centric Manufacturing Education Data Cleanse 84