Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. We expect complete and accurate data at the end of each run.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Are you using Snowflake on AWS and already using Glue Data Catalog for your data lake?

Building 95

Building Metadata Cloud Storage AWS 95

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

FEBRUARY 26, 2023

Summary There has been a lot of discussion about the practical application of data mesh and how to implement it in an organization. Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. Is a data mesh ever "done"? We feel your pain.

Building 147

Building Metadata Machine Learning Data Integration 147

dbt Developer Hub

SEPTEMBER 14, 2021

Give Jeremy a win and check out the blog he just posted on why this matters even more leading up to ?dbt Embedding the DAG within the IDE makes investigating project structure a lot easier The Metadata API : Now in GA! Your Data PM is not a panacea. Have you subscribed to the Analytics Engineering Roundup ?

Metadata 52

Metadata Data Pipeline Python Cloud 52

Data Engineering Weekly

DECEMBER 10, 2023

RudderStack, one of the leading alternatives to Segment , is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Lackluster AI/ML results often stem from poor data quality. Visit rudderstack.com to learn more.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Christophe Blefari

JANUARY 20, 2024

Learn data engineering, all the references ( credits ) This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾 So I wrote this special edition about: how to learn data engineering in 2024. The idea is to create a living reference about Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Airbnb Tech

NOVEMBER 28, 2023

By: Clark Wright Introduction These days, as the volume of data collected by companies grows exponentially, we’re all realizing that more data is not always better. In fact, more data, especially if you can’t rely on its quality, can hinder a company by slowing down decision-making or causing poor decisions.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Databand.ai

AUGUST 30, 2023

8 Data Quality Monitoring Techniques & Metrics to Watch Niv Sluzki August 30, 2023 What Is Data Quality Monitoring? Data quality monitoring refers to the assessment, measurement, and management of an organization’s data in terms of accuracy, consistency, and reliability.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Christophe Blefari

DECEMBER 3, 2022

This week I had a huge amount of work to deal with and we've launched the Advent of Data , your daily spark of data in December. Modern Data Modeling: Start with the End? How DoorDash secures data transfer — This is network stuff, but still interesting for a large part of data engineers.

Kafka 130

Kafka BI Finance Data 130

LinkedIn Engineering

JULY 19, 2023

Co-Authors: Sumedh Sakdeo , Lei Sun , Sushant Raikar , Stanislav Pak , and Abhishek Nath Introduction At LinkedIn, we build and operate an open source data lakehouse deployment to power Analytics and Machine Learning workloads. While functional, our current setup for managing tables is fragmented.

Big Data 78

Big Data Data Management Management Metadata 78

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Rockset

MARCH 18, 2024

While it's well-known that Flink excels at filtering, joining and enriching streaming data from Apache Kafka® or Confluent Cloud , what is less known is that it is increasingly becoming ingrained in the end-to-end stack for AI-powered applications. LLMs like ChatGPT are trained on vast amounts of text data available up to a cutoff date.

Cloud 64

Cloud Building Metadata Kafka 64

phData: Data Engineering

SEPTEMBER 19, 2023

With the amount of data companies are using growing to unprecedented levels, organizations are grappling with the challenge of efficiently managing and deriving insights from these vast volumes of structured and unstructured data. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Data Engineering Weekly

MARCH 10, 2024

Editor’s Note: Chennai Meetup Wrap-Up & Preparation work started for DEWCon I am so grateful for the enthusiastic participants who made our Chennai Data Heroes- Community for Data Folks meetup vibrant! Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Snowflake

AUGUST 22, 2023

One way Honeywell “makes the future” is by developing innovative data products. Snowflake’s World of Data Collaboration, the theme of Snowflake Summit , was a world of data products—extending beyond data to include AI models and applications. A data product is built to be used and reused.

Metadata 71

Metadata Data AWS Business Analyst 71

Databand.ai

JULY 19, 2023

Data Lineage Tools: Key Capabilities and 5 Notable Solutions Ryan Yackel July 19, 2023 What Are Data Lineage Tools? Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. Data lineage tools are not a new concept.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Cloudera

APRIL 2, 2024

It’s been said that the Federal Government is one of, if not the largest, producer of data in the United States, and this data is at the heart of mission delivery for agencies across the civilian to DoD spectrum. FedRAMP requires that we meet strict security standards to protect government data.

Designing 96

Designing Process Government Metadata 96

Data Engineering Podcast

MARCH 25, 2023

Summary The promise of streaming data is that it allows you to react to new information as it happens, rather than introducing latency by batching records together. As your business adapts, so should your data. RudderStack Transformations lets you customize your event data in real-time with your own JavaScript or Python code.

MySQL 130

MySQL Python Architecture Machine Learning 130

Engineering at Meta

OCTOBER 31, 2023

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. Each represents a class of data — not individual records.

Data 93

Data Metadata Coding Relational Database 93

Data Council

DECEMBER 21, 2020

The Modern Data Stack (Fishtown Analytics) This long-form post on the dbt blog is a must-read. Titled “The Modern Data Stack: Past, Present, and Future,” it answers the question that Tristan Handy has been asking himself for the past two years: “What happened to the massive innovation we saw from 2012-2016?”

Metadata 52

Metadata Architecture Data Technology 52

Cloudera

AUGUST 26, 2021

Ozone natively provides Amazon S3 and Hadoop Filesystem compatible endpoints in addition to its own native object store API endpoint and is designed to work seamlessly with enterprise scale data warehousing, machine learning and streaming workloads. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 109

Data Science Cloud Hadoop Metadata 109

Towards Data Science

MAY 19, 2023

Data Entropy — More Data, More Problems? How to navigate and embrace complexity in a modern data organisation. Business users are unable to find and access data assets critical to their workflows. Data engineers spend countless hours troubleshooting broken pipelines. Every “minor” change upstream results in mayhem.

Pipeline-centric 105

Pipeline-centric Data Software Engineer Software Engineering 105

Cloudera

JUNE 7, 2023

The revised ZTMM is organized by five categories or pillars: identity, devices, networks, applications and workloads, and data, and four levels of maturity: traditional, initial, advanced, and optimal. The data is protected but it is also accessible by the people who need it. How does Cloudera support the evolution to optimal?

Government 79

Government Metadata Data Lake Data 79

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Data Engineering Podcast

JANUARY 23, 2022

Summary Data engineering is a relatively young and rapidly expanding field, with practitioners having a wide array of experiences as they navigate their careers. Ashish Mrig currently leads the data analytics platform for Wayfair, as well as running a local data engineering meetup.

Building 100

Building Management Data Pipeline Metadata 100

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Arjun Mohnot , Jenchang Ho , Anthony Quigley , Xing Lin , Anil Alluri , Michael Kuchenbecker LinkedIn operates one of the world’s largest Apache Hadoop big data clusters. Historically, deploying code changes to Hadoop big data clusters has been complex.

Big Data 92

Big Data Hadoop Metadata Data 92

Tweag

MAY 16, 2023

To minimize the risk of misconfigurations, Nickel features (opt-in) static typing and contracts, a powerful and extensible data validation framework. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 131

MySQL Metadata Data Validation Coding 131

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

phData: Data Engineering

DECEMBER 16, 2021

Welcome to part two of our trilogy on data catalogs. If you missed our first blog on what a data catalog is , be sure to check it out! In this blog, we’ll explore what the ideal team to support your data catalog looks like. Who Are the Users of a Data Catalog?

Metadata 52

Metadata Datasets Data Education 52

RandomTrees

FEBRUARY 6, 2024

Data engineering, the practice of collecting, transforming, and organizing data for analysis, is poised for a significant transformation with the advent of Generative Artificial Intelligence (Gen AI). Ingestion: The Art of Data Assimilation: Ensuring the digital document accurately reflects the original handwritten material.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52