Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive? How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Confluent

MAY 21, 2019

Having well-defined schemas that are documented, validated and managed across the entire architecture will help integrate data and microservices —a notoriously challenging problem that we discussed at some length in the past. Note that the same definitions of fields and types that once defined the REST API are now part of the event schema.

Kafka 110

Kafka Insurance Architecture Database 110

DataKitchen

DECEMBER 29, 2022

The Terms and Conditions of a Data Contract are Automated Production Data Tests. A data contract is a formal agreement between two parties that defines the structure and format of data that will be exchanged between them. One of the primary benefits of using data contracts is that they help to ensure data integrity and compatibility.

Data Validation 52

Data Validation Data Data Integration SQL 52

Confluent

AUGUST 27, 2019

We are excited to announce the release of Confluent Cloud Schema Registry in general availability (GA), available in Confluent Cloud , our fully managed event streaming service based on Apache Kafka ®. Before we dive into Confluent Cloud Schema Registry, let’s recap what Confluent Schema Registry is and does.

Cloud 18

Cloud Kafka Electronics Metadata 18

Zalando Engineering

APRIL 3, 2019

In our case, it is the OpenAPI standard for synchronous REST interface specification and JSON Schema for asynchronous events. contract) and contributes greatly to the API and overall system design quality. API-as-a-Product and API First Principle Zalando is customer-obsessed. We believe in the API First principle and always follow it.

Scala 40

Scala Software Engineer Software Engineering Java 40

Confluent

FEBRUARY 22, 2019

By strictly enforcing a requirement of using Protobuf messages on all Kafka topics, our chosen design guarantees reliable and consistent data on each topic, and provides a way for schemas to evolve without breaking downstream systems. The Need for a Structured Message Format. Deciding on an Encoding Format.

Kafka 80

Kafka Programming Language Metadata Data 80

Data Engineering Weekly

NOVEMBER 13, 2022

Meta writes about its internal implementation of the Schema management system at scale. link] Luke Lin: A PM's thoughts on data contracts Data Contract is a hot topic in data engineering. The author walks through various strategies a data contract platform can adopt to simplify the adoption.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52