Data Engineering Podcast

FEBRUARY 26, 2023

What are the technical systems that you are relying on to power the different data domains? What is your philosophy on enforcing uniformity in technical systems vs. relying on interface definitions as the unit of consistency? What are the technical systems that you are relying on to power the different data domains?

Building 147

Building Metadata Machine Learning Data Integration 147

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Netflix Tech

MAY 4, 2023

Behind the scenes, a myriad of systems and services are involved in orchestrating the product experience. These backend systems are consistently being evolved and optimized to meet and exceed customer and product expectations. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Utilities 135

Utilities Systems Architecture Coding 135

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? It bridges gaps in data ecosystems, ensuring consistency and synchronisation across systems. Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Weekly

NOVEMBER 11, 2022

Not too long ago, I worked for a famous messaging platform; You may be typing on it as you read this blog. I took a break from data engineering and focused on system monitoring and observability for some time. Little did I know I would be building data pipelines to measure system efficiency instead of measuring business efficiency.

Systems 40

Systems High Quality Data Consulting Metadata 40

Databand.ai

JULY 19, 2023

Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. This feature is particularly useful in complex data architectures, where data may pass through multiple systems and transformations. It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. Five Core Pillars of DORA 1.

Transportation 82

Transportation Data Governance Government Consulting 82

Data Engineering Podcast

SEPTEMBER 4, 2022

Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

Cloudera

NOVEMBER 29, 2023

It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. Artifacts Management : The system efficiently handles the import and export of model artifacts in standard formats, promoting compatibility with different systems.

Machine Learning 82

Machine Learning Metadata Cloud Project 82

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. This culminated in the creation of GenOS, an operating system for developing GenAI-powered features.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

AUGUST 10, 2021

Apache Ozone enhancements deliver full High Availability providing customers with enterprise-grade object storage and compatibility with Hadoop Compatible File System and S3 API. . We expand on this feature later in this blog. Figure 8: Data lineage based on Kafka Atlas Hook metadata. x, and 6.3.x, x, and 6.3.x,

Cloud 96

Cloud Kafka Metadata SQL 96

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. and based on content characteristics and/or metadata signaled in the bitstream. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. Krasula, A. Choudhury, S. Malfait, A.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. A data lake is a centralized repository containing extensive storage for raw, unfiltered data coming into a company’s data storage system.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems. What are the alternatives to CDC?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

LinkedIn Engineering

DECEMBER 19, 2023

The historical upgrade system couldn’t adapt to architectural changes like the introduction of an observer namenode (Now handling a massive influx of read requests – 150K QPS – from services such as Trino ), ZKFC auto-failover, HDFS federation , etc.

Big Data 92

Big Data Hadoop Metadata Data 92

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 107

Data Science Cloud Hadoop Metadata 107

Christophe Blefari

DECEMBER 3, 2022

Joe Reis launched his Substack — Joe is the co-author of the great The Fundamentals of Data Engineering and his blog already have 2 articles I deeply recommend: No extra credit for complexity & Groundhog Days. Hey Snowflake, send me an email — Christmas is sometimes the time of the year where magic happens.

Kafka 130

Kafka BI Finance Data 130

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. Streaming compute however, empowers more complex window queries on semantic correctness.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Tweag

MARCH 11, 2024

But CISA encouraged me to publish the answer as a separate blog post. I would go further and claim that for any software system that will be modified and reassembled — which is basically always — precise knowledge and control over the components and how they should be put together is crucial. In fact, several projects exist (e.g.

Metadata 60

Metadata Utilities Building Designing 60

Cloudyard

DECEMBER 7, 2023

Read Time: 3 Minute, 0 Second In this blog post, we will delve into Snowflake’s Listing feature, which operates on the data sharing provider and consumer model. Review and accept the Provider Terms of Service.

Metadata 52

Metadata SQL Data Cloud 52

Christophe Blefari

JANUARY 20, 2024

Still, we can have a common ground when mixing software engineering, DevOps principles, Cloud — or on-prem — systems understanding and data literacy. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Datakin

AUGUST 5, 2021

Blog A real-time approach to data lineage Written by Ross Turk on August 5, 2021 A data ecosystem that spans multiple pipelines, teams, and platforms can be overwhelming. Lineage metadata is the thread that connects it all together. However, there is more than one way to capture lineage metadata.

Metadata 52

Metadata Relational Database Datasets SQL 52

Cloudera

MARCH 9, 2021

While Cloudera Data Platform (CDP) already supports the entire data lifecycle from ‘Edge to AI’, we at Cloudera are fully aware that enterprises have more systems outside of CDP. Atlas provides open metadata management and governance capabilities to build a catalog of all assets, and also classify and govern these assets. ip_address.

Data Governance 93

Data Governance Government Metadata Datasets 93

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. From Bad to Worse. One possible solution is to adopt a hybrid cloud strategy. .

Data Governance 95

Data Governance Government Metadata Medical 95

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? We’ll break our system into pieces such that each component is responsible for one primary job. Let’s take a look at components that constitute the system. Rockset satisfies all of them.

Kafka 40

Kafka Designing Systems Food 40

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great effect, but they are inherently limited in their utility due to their nature as storage systems.

Metadata 130

Metadata MongoDB Scala MySQL 130

Databand.ai

AUGUST 30, 2023

It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. These systems typically consist of siloed data storage and processing environments, with manual processes and limited collaboration between teams.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Operating System Disk Layouts. Networking .

Architecture 96

Architecture Cloud Kafka Hadoop 96

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. This may include tasks such as data profiling, data cleansing, and metadata management. Data profiling: Regularly analyze dataset content to identify inconsistencies or errors.

Building 52

Building Data Cleanse Data Governance Datasets 52

Cloudera

JULY 14, 2023

In this blog, I will describe a few strategies one could undertake for various use cases. They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. This will be discussed in a later blog. The underlying data file locations will not change.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Cloudera

MARCH 22, 2023

Over the past handful of years, systems architecture has evolved from monolithic approaches to applications and platforms that leverage containers, schedulers, lambda functions, and more across heterogeneous infrastructures. Each has their own stacked layers of enabling technologies, from operating systems to containers to resources.

Data Governance 84

Data Governance Government Business Analyst Metadata 84

Datakin

SEPTEMBER 13, 2021

Blog Datakin in 104 seconds Written by Ross Turk on Sep 13, 2021 Hi! Datakin observes your jobs as they run, collecting metadata that helps you understand how data flows through your ecosystem. View our instant demo system right now, or sign up for Datakin to see it with your own datasets and jobs.

Metadata 52

Metadata Datasets Data Pipeline Systems 52

Data Engineering Weekly

FEBRUARY 18, 2024

I believe the data ownership problem is much deeper than simple metadata management. link] Noah Kennedy: Reducing BigQuery Costs by 100–200x with dbt Incremental Models One of the repeating patterns I heard in many of the data engineers is how to set up incremental data processing across the system efficiently.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Cloudera

DECEMBER 10, 2020

Some data warehousing solutions such as appliances and engineered systems have attempted to overcome these problems, but with limited success. . In the above example, CDW could have cut costs by 18% for the duration that the system was idle. Its existing data warehousing service is a 40-node system and is quite static.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Zalando Engineering

JULY 13, 2015

PostgreSQL is a very flexible database system. Its flexibility derives from its way of storing metadata. Unlike many other databases, Postgres involves no “magic”: Every database object — table, type, function or cast — is described by a row in a special table, called a system catalog.

PostgreSQL 40

PostgreSQL Systems Metadata Database 40