Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. Our audits would detect this and alert the on-call data engineer (DE).

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Podcast

APRIL 22, 2018

For this reason metadata management systems are built to track the journey of your business data to aid in analysis, presentation, and compliance. What are some of the types of information that you classify and collect as metadata? What are some of the challenges that are typically faced by metadata management systems?

Business Intelligence 113

Business Intelligence Metadata Management Data Governance 113

Start Data Engineering

JULY 20, 2023

Know the when, how, & what (aka metadata) of pipeline runs for easier debugging 3. Ensure data is valid before exposing it to its consumers (aka data quality checks) 3.3. Avoid data duplicates with idempotent pipelines 3.4. Write DRY code & keep I/O separate from data transformation 3.5.

Coding 130

Coding Data Engineering Data Engineer Engineering 130

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Metadata 110

Metadata Big Data Transportation Data 110

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Sifflet also offers a 2-week free trial.

Metadata 100

Metadata MongoDB Scala MySQL 100

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How can data engineers harness the power of AI? And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Weekly

FEBRUARY 18, 2024

One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. See how recent product updates make it easier for developers and data engineers to debug data models, work in a BI-like UI, and fine-tune deployments for better query performance. Was this simply too ambitious?

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Podcast

AUGUST 28, 2022

Summary The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution. Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Data Engineering Podcast

JANUARY 30, 2022

As a result it has become a standard tool for data engineers for a wide range of applications. Today’s episode is Sponsored by Prophecy.io – the low-code data engineering platform for the cloud. You can observe your pipelines with built in metadata search and column level lineage.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Pipeline 100

dbt Developer Hub

SEPTEMBER 14, 2021

Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Assess data health with the metadata generated by recent dbt job runs Dashboard Status Tiles : Embed this tile anywhere iFrames live to quickly check data freshness New Resources ​ Things to Read ? Anna's latest reflection?

Metadata 52

Metadata Data Pipeline Python Cloud 52

Christophe Blefari

MARCH 1, 2023

dbt is a development framework that combines modular SQL with software engineering best practices to make data transformation reliable, fast, and fun. You can also add metadata on models (in YAML). Jinja templating — Jinja is a templating engine that seems to exist forever in Python.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. But unfortunately, very little that happens in data engineering today is observable. Most data pipelines are built to move but not monitor.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Snowflake

MAY 15, 2024

By continuously monitoring metrics, metadata, lineage, and logs from across your data infrastructure and using ML-based anomaly detection to detect issues, they help data teams know about and resolve issues quickly. Metaplane ensures that every company can trust the data that powers their business.

Cloud 106

Cloud Metadata Data Pipeline Government 106

Jesse Anderson

NOVEMBER 14, 2023

It’s a team that connects naturally into the constellation of the three data teams Operations team Data engineering team Data Science team as described in Jesse Anderson’s book Data Teams (2020) Before I explain what the data discovery team should do, it is necessary to add a bit of context on the concept of data discovery itself.

Metadata 147

Metadata Data Science Big Data Data 147

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. Photo by Gaining Visuals on Unsplash Here is a simple and fast pet project for Data/Analytics Engineers, who want to kick the tires on Modern Data Stack tools including dbt Core, Snowflake, Fivetran, and GitHub Actions.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Data Engineering Weekly

JUNE 29, 2023

But hey, I met my friends after a long time and got my copy of “ Fundamentals of Data Engineering ” signed by Joe Reis & Matt Housely. If you’re starting data engineering, I highly recommend reading it. The question remains how far the data catalog tools can go with just the metadata.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? At the same time, data engineering was the slightly younger sibling, but it was going through something similar. I wasn’t promoted or assigned to this new role.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together. See how it works today. Should you?

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 130

Building Metadata MongoDB MySQL 130

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata AWS Cloud Database 52

Data Engineering Weekly

MAY 16, 2023

Event Routers can add additional metadata to the envelope of the event. The same goes for the popular orchestration engine Airflow, where the Airflow has separate operators ( SQLCheckOperators ) for the data quality check. Event Routers typically don’t alter the payload. Now, Why is Data Quality Expensive? How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Data Engineering Weekly

OCTOBER 23, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Hello Data Friends, Welcome to another edition of Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

Monte Carlo

FEBRUARY 7, 2024

Data engineering is no exception. Tasks that were once fundamental to data engineering are now being accomplished by AI — usually faster, and sometimes with a higher degree of accuracy. As familiar workflows evolve, it naturally begs a question: will GenAI replace data engineers? At least, not anytime soon. Here’s why.

Data Engineering 59

Data Engineering Data Engineer Engineering Unstructured Data 59

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 95

Building Metadata Cloud Storage AWS 95

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Til about [link] Thank you, Anton Bryzgalov, for contributing the blog to the Data Engineering Weekly.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

MARCH 11, 2023

We are back in our Data Engineering Weekly Radio for edition #120. We will take 2 or 3 articles from each week's Data Engineering Weekly edition and go through an in-depth analysis. link] Data Engineering Weekly Data Catalog - A Broken Promise Data catalogs are the most expensive data integration systems you never intended to build.

Data Engineering 52

Data Engineering Data Engineer Engineering High Quality Data 52

Data Engineering Podcast

NOVEMBER 6, 2022

Summary A lot of the work that goes into data engineering is trying to make sense of the "data exhaust" from other applications and services. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Monte Carlo

JULY 13, 2023

To effectively manage and analyze this data, data engineering teams must navigate a wide range of challenges, including data access, security, compliance, and data observability. The burden of granting and revoking access to data typically falls to the data engineering teams. This is a guest post from our friends over at Satori Cyber.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

Ascend.io

APRIL 22, 2024

At Ascend, we believe it’s time to rethink data engineering from the ground up. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants. Read More: The State of Data Engineering in 2023: Does Your Data Program Stack Up? Let’s dive in!

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

FEBRUARY 19, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Hey there podcast listener, are you tired of dealing with the headache that is the 'Modern Data Stack'? How does that change based on the type of query/processing engine being used? We feel your pain. We feel your pain.

IT 147

IT Data Lake Metadata Data Warehouse 147

Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Management 130

Data Management Management Metadata MongoDB 130