Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. What was the impact of leading multiple teams on the design of how to implement communication/contracts throughout the mesh? When is a data mesh the wrong choice? Closing Announcements Thank you for listening!

Building 147

Building Metadata Machine Learning Data Integration 147

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. They also need to be designed for real-time updates. These additional inputs are referred to as metadata filtering.

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Weekly

DECEMBER 10, 2023

link] Evidently: ML system design - 300 case studies to learn from An amazing compilation of ML system design articles from various companies. The blog is an excellent comparison study of Ray vs. Dask’s performance. Stores metadata to utilize later.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Netflix Tech

JANUARY 10, 2024

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. This architecture shift greatly reduced the processing latency and increased system resiliency.

Process 91

Process Pipeline-centric Media Metadata 91

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Given our role on this critical path, accuracy is paramount.

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Rockset

JULY 8, 2020

Building a scalable, distributed, and real-time ETA prediction system is a tough task, but what if we could simplify its design? We’ll break our system into pieces such that each component is responsible for one primary job. Let’s take a look at components that constitute the system. Rockset satisfies all of them.

Kafka 40

Kafka Designing Systems Food 40

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. Since the initial release (0.1), we’ve refined the semantics enough to be confident in a core design that is unlikely to radically change.

MySQL 131

MySQL Metadata Data Validation Coding 131

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications. Custom lint rules on Rest.li

Architecture 109

Architecture Metadata Java Transportation 109

Data Engineering Podcast

SEPTEMBER 4, 2022

Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. ESA will designate critical ICT providers in January 2025.

Transportation 83

Transportation Data Governance Government Consulting 83

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. They are designed to complement existing workflows, helping to streamline the transition of models from development to production.

Machine Learning 84

Machine Learning Metadata Cloud Project 84

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. Vector search ≠ vector database Vector search and all the associated clever algorithms are the central intelligence of any system trying to leverage vectors.

Metadata 52

Metadata Database Algorithm SQL 52

Databand.ai

JULY 19, 2023

Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. This feature is particularly useful in complex data architectures, where data may pass through multiple systems and transformations. It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Airbnb Tech

NOVEMBER 28, 2023

However, achieving the full data quality criteria required by Midas demands significant cross-functional investment to design, develop, validate, and maintain the necessary data assets and documentation. In this blog post, we share our innovative approach to scoring data quality, Airbnb’s Data Quality Score (“DQ Score”).

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? It bridges gaps in data ecosystems, ensuring consistency and synchronisation across systems. Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. We have a separate blog post explaining the details of how Zalando uses persisted queries and how we think about schema stability and granular control. The @draft and @allowedFor directives are used for this purpose.

Metadata 86

Metadata Coding Banking Designing 86

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. A data lake is a centralized repository containing extensive storage for raw, unfiltered data coming into a company’s data storage system.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Knowledge Hut

MAY 9, 2024

In this blog post, we will look at some of the world's highest paying data science jobs, what they entail, and what skills and experience you need to land them. Responsibilities Data architects assess an organization's data sources and design plans for centralized data management. What is Data Science?

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems. What are the alternatives to CDC?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

LinkedIn Engineering

JANUARY 16, 2024

We've been on a journey to strengthen our platform against abuse by continuously improving our account restriction systems. In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. This powerful system is our first line of defense against bad actors and adversarial attacks.

Kafka 84

Kafka Relational Database Java Architecture 84

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Snowflake

AUGUST 31, 2023

Because of its leading ecosystem of diverse adopters, contributors and commercial offerings, Iceberg helps prevent storage lock-in and eliminates the need to move or copy tables between different systems, which often translates to lower compute and storage costs for your overall data stack.

Metadata 83

Metadata AWS Data Lake Datasets 83

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. In this blog, we’ll dig into how Rockset has fully integrated vector search into its search and analytics database. Rockset supports streaming updates to metadata and vectors in an efficient way.

Cloud 64

Cloud Metadata Database SQL 64

DoorDash Engineering

MARCH 14, 2023

When dealing with failures in a microservice system, localized mitigation mechanisms like load shedding and circuit breakers have always been used, but they may not be as effective as a more globalized approach. Each section is accompanied by real outages drawn from our past blog posts that can be explored in greater detail.

Metadata 135

Metadata Java Database Systems 135

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. Bitrate versus quality comparison HDR-VMAF is designed to be format-agnostic — it measures the perceptual quality of HDR video signal regardless of its container format, for example, Dolby Vision or HDR10. The graphic below (Fig.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Tweag

MARCH 11, 2024

But CISA encouraged me to publish the answer as a separate blog post. I would go further and claim that for any software system that will be modified and reassembled — which is basically always — precise knowledge and control over the components and how they should be put together is crucial. In fact, several projects exist (e.g.

Metadata 60

Metadata Utilities Building Designing 60

Engineering at Meta

OCTOBER 31, 2023

SCARF scans production data systems to identify tables or assets that are unused and safely removes them. This process can be unexpectedly difficult, because large software systems are inevitably interconnected. Meta has a separate system, DELF , for deleting individual records, rows, and objects.

Data 93

Data Metadata Coding Relational Database 93

LinkedIn Engineering

DECEMBER 19, 2023

The historical upgrade system couldn’t adapt to architectural changes like the introduction of an observer namenode (Now handling a massive influx of read requests – 150K QPS – from services such as Trino ), ZKFC auto-failover, HDFS federation , etc.

Big Data 92

Big Data Hadoop Metadata Data 92

Data Engineering Weekly

FEBRUARY 18, 2024

I believe the data ownership problem is much deeper than simple metadata management. Yet, there is no tool for the data producers to Define, Design, Collaborate, Test, & Continuously Monitor the Data Products. Fractional factorial design selects a subset of the possible combinations of factors to run as experiments.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Systems that generate data add new columns to accommodate additional information, which requires downstream tables to evolve accordingly.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Christophe Blefari

JANUARY 20, 2024

Still, we can have a common ground when mixing software engineering, DevOps principles, Cloud — or on-prem — systems understanding and data literacy. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill). Is it really modern?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

LinkedIn Engineering

JULY 27, 2023

In this blog post, we will share our progress, challenges, and lessons learned from our Costwiz journey. Costwiz application and workflow management system For the initial rollout, we built a single-page app for users to perform actions on their recommendations (as seen in Figure 1).

Metadata 75

Metadata Utilities Cloud Data Lake 75

Netflix Tech

MAY 5, 2021

Netflix Drive relies on a data store that will be the persistent storage layer for assets, and a metadata store which will provide a relevant mapping from the file system hierarchy to the data store entities. 2 , are the file system interface, the API interface, and the metadata and data stores. A sample manifest file.

Metadata 69

Metadata Bytes Media Cloud Storage 69