Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. When is a data mesh the wrong choice?

Building 147

Building Metadata Machine Learning Data Integration 147

Snowflake

APRIL 29, 2024

Recover: establish processes for rapid recovery after incidents to ensure business continuity. This translates to several key requirements and their associated benefits: More stringent technical and process-oriented security measures, enhancing protection for both FEs and their ICT third-party providers. Five Core Pillars of DORA 1.

Transportation 83

Transportation Data Governance Government Consulting 83

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. The post Announcing General Availability of Model Registry appeared first on Cloudera Blog.

Machine Learning 84

Machine Learning Metadata Cloud Project 84

Databand.ai

JULY 19, 2023

This allows businesses to identify any transformations or processes that may be compromising the data’s integrity. Root cause analysis and debugging: Data lineage tools facilitate this process by providing a clear view of the data’s journey, allowing businesses to pinpoint exactly where the issue occurred.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? CDC also plays a crucial role in data integration and ETL processes. It enables automation in business processes, event-driven applications, and microservices, reacting to data changes in near real-time. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Christophe Blefari

JANUARY 20, 2024

The Rise of the Data Engineer The Downfall of the Data Engineer Functional Data Engineering — a modern paradigm for batch data processing There is a global consensus stating that you need to master a programming language (Python or Java based) and SQL in order to be self-sufficient. Here a small benchmark between some popular formats.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. Bitrate versus quality comparison HDR-VMAF is designed to be format-agnostic — it measures the perceptual quality of HDR video signal regardless of its container format, for example, Dolby Vision or HDR10. The graphic below (Fig.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Knowledge Hut

MAY 9, 2024

In this blog post, we will look at some of the world's highest paying data science jobs, what they entail, and what skills and experience you need to land them. Responsibilities Data architects assess an organization's data sources and design plans for centralized data management. They manage data storage and the ETL process.

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Lyft Engineering

JUNE 28, 2023

While several teams were using streaming data in their Machine Learning (ML) workflows, doing so was a laborious process, sometimes requiring weeks or months of engineering effort. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Pinterest Engineering

NOVEMBER 7, 2023

For these reasons, and others detailed in our original PubSub Client blog post , our team has decided to invest in building, productionalizing, and most recently open-sourcing PubSub Client (PSC). years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. I won’t bore you with the importance of data quality in the blog. Let’s talk about the data processing types. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Databand.ai

AUGUST 30, 2023

A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date. This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management.

Building 52

Building Data Cleanse Data Governance Datasets 52

Lyft Engineering

SEPTEMBER 6, 2023

Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to. Context propagation : routing overrides metadata is propagated throughout the service mesh. Routing overrides metadata with ProxyApp Staging overrides was well-received by our product engineers.

Building 80

Building Metadata Electronics Engineering 80

LinkedIn Engineering

DECEMBER 19, 2023

These clusters are the backbone for storing and processing extensive data volumes, empowering us to deliver essential features and services to members, such as personalized recommendations, enhanced search functionality, and valuable insights. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes.

Big Data 92

Big Data Hadoop Metadata Data 92

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. Processes: File transfer process. ETL/DB Load process.

Data Governance 93

Data Governance Government Metadata Datasets 93

Databand.ai

AUGUST 30, 2023

It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Workfall

AUGUST 23, 2023

These extensions are designed to enhance and customize the browsing experience by adding new features, modifying existing ones, or integrating with other web services. This file played a crucial role in defining the metadata, permissions, and settings for our extension. We will come up with more such use cases in our upcoming blogs.

Metadata 83

Metadata Coding Project Utilities 83

AltexSoft

FEBRUARY 11, 2023

A data architect is an IT professional responsible for the design, implementation, and maintenance of the data infrastructure inside an organization. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is a data architect? Feel free to enjoy it.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Weekly

FEBRUARY 18, 2024

I believe the data ownership problem is much deeper than simple metadata management. Yet, there is no tool for the data producers to Define, Design, Collaborate, Test, & Continuously Monitor the Data Products. Fractional factorial design selects a subset of the possible combinations of factors to run as experiments.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Cloudera

AUGUST 26, 2021

Ozone natively provides Amazon S3 and Hadoop Filesystem compatible endpoints in addition to its own native object store API endpoint and is designed to work seamlessly with enterprise scale data warehousing, machine learning and streaming workloads. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 109

Data Science Cloud Hadoop Metadata 109

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. These flow through Kafka , our event streaming platform, before being processed by Flink , our streaming compute framework.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Workfall

AUGUST 14, 2023

In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. In this blog, we will cover: Understanding DBT and Airflow What is DBT? Together, they form a dynamic duo that simplifies and streamlines the entire data pipeline process.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

LinkedIn Engineering

DECEMBER 12, 2023

We are excited to announce the various contributions we have made to provide a privacy-by-design approach to measure and mitigate reidentification risks. This metadata is determined as part of an onboarding process with each new application where we determine which algorithms should be used for each type of query.

Algorithm 77

Algorithm Metadata SQL Datasets 77

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? Orders The orders placed by a customer are stored in DynamoDB for further processing. Most of these data applications provide a near real-time tracking of the ETA once you place the order.

Kafka 40

Kafka Designing Systems Food 40

Ripple Engineering

MARCH 30, 2023

This blog outlines Ripple’s general design and approach for machine learning model lifecycle management using Databricks. Challenges and Assumptions We need to track model files and all related metadata for models like training metrics, date trained, etc. and store this information for a long time.

Machine Learning 98

Machine Learning AWS Metadata Data Collection 98

Cloudera

SEPTEMBER 24, 2020

However, there is a secret I am keeping to the end of the blog, which makes the decision even easier for the user: so easy in fact, you do not even have to decide yourself. Impala was designed for speed. . Hive LLAP was designed for sophistication. However, Hive is designed to be very fault-tolerant.

Data Warehouse 68

Data Warehouse SQL Engineering Metadata 68

Snowflake

NOVEMBER 30, 2023

Business contextual frameworks and design patterns are tightly bound to existing data models, and regulatory requirements may demand that historical data be stored as is and remain readily available for auditing. A “lift and shift” migration approach handles all of these requirements by moving the business workload and data together.

Data Warehouse 98

Data Warehouse Banking Data Cloud 98

Confluent

MARCH 26, 2019

how long it takes to execute an operation) and additional metadata like annotations and tags. For the translation service to correlate its spans with the caller service traces, it receives tracing metadata via context propagation (i.e., reading metadata injected into HTTP headers). Let’s imagine a “Hello, World!”

Kafka 111

Kafka Transportation Metadata Consulting 111

Rockset

NOVEMBER 14, 2022

In this blog, we will describe the SSB configuration, results and performance enhancements. A predicate pushdown uses the predicate to filter the data in the query, moving query processing closer to the storage layer. The metadata lookup operation takes a significant proportion of the working set memory, impacting query latency.

Metadata 52

Metadata Datasets Cloud Accessible 52

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. But first, let’s define the data mesh design pattern. It’s no fun working in data analytics/science when you are the bottleneck in your company’s business processes.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Cloudera

JUNE 7, 2022

Cloudera Data Platform (CDP) is designed to address the critical requirements for modern data architectures today and tomorrow. Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. Integration, metadata and governance capabilities glue the individual components together.” .

IT 108

IT Unstructured Data Data Architecture Government 108

Datakin

AUGUST 5, 2021

Blog A real-time approach to data lineage Written by Ross Turk on August 5, 2021 A data ecosystem that spans multiple pipelines, teams, and platforms can be overwhelming. Lineage metadata is the thread that connects it all together. However, there is more than one way to capture lineage metadata.

Metadata 52

Metadata Relational Database Datasets SQL 52

Netflix Tech

DECEMBER 8, 2021

We currently process tens of millions of log records every minute and analyze these events to perform in-house custom detections. Once ingested, findings are then enriched and processed with additional metadata collected from Netflix’s internal data sources. Make sure to check out some of our other recent blog posts!

AWS 78

AWS Metadata Cloud Utilities 78

Cloudera

DECEMBER 10, 2020

Drawing on more than a decade of experience in building and deploying massive scale data platforms on economical budgets, Cloudera has designed and delivered a cost-cutting cloud-native solution – Cloudera Data Warehouse (CDW), part of the new Cloudera Data Platform (CDP). CDW was designed to aggressively keep cloud costs under control.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Cloudera

JANUARY 19, 2022

These integrated data services provide fit-for-purpose solutions for different data workloads including – advanced analytics, streaming, Machine Learning and transaction processing, which will provide an end to end automated data lifecycle.

Database 87

Database Cloud Data Warehouse Data Lake 87

phData: Data Engineering

DECEMBER 16, 2021

In our previous blog in this series , we spent a lot of time exploring why a data catalog is valuable and who you might need to support it. Perhaps one of the most crucial elements in deciding a data catalog that’s right for your business is exploring how your tool is designed when it comes to infrastructure.

Metadata 52

Metadata Datasets ETL Tools Cloud 52