Striim

NOVEMBER 8, 2023

The Data Mesh architecture is based on four core principles: scalability, resilience, elasticity, and autonomy. Data mesh technology also employs event-driven architectures and APIs to facilitate the exchange of data between different systems.

Architecture 52

Architecture Generalist Government Datasets 52

Pinterest Engineering

MAY 2, 2023

So our user sequence real-time indexing pipeline is composed of a Flink job that reads the relevant events as they come into our Kafka streams, fetches the desired features for each event from our feature services, and stores the enriched events into our KV store system. We need each dataset to store the last N events for a user.

Lambda Architecture 66

Lambda Architecture Datasets Software Engineer Software Engineering 66

Monte Carlo

MARCH 2, 2023

Otherwise you may produce more data anomalies than you prevent. Data Contracts Image courtesy of Andrew Jones. You can think of data contracts as circuit breakers, but for data schemas instead of the data itself. These ownership lines need to be traced back to owners of the business processes they support.

Food 52

Food Data SQL Data Pipeline 52

Confluent

JUNE 13, 2019

This data pipeline is a great example of a use case for Apache Kafka ®. Observational astronomers study many different types of objects, from asteroids in our own solar system to galaxies that are billions of lightyears away. The technology underlying the ZTF system should be a prototype that reliably scales to LSST needs.

Kafka 101

Kafka Bytes Data Pipeline Python 101

Tweag

FEBRUARY 22, 2023

I talked about how to write modular and customizable configuration using Nickel’s merging system. I’ve rather heard a call for simpler, safer, and better automated processes. Viktor Kleen presented his work on Terraform-Nickel , a library to use Nickel in place of HCL for Terraform. But something is in the air. isn’t sustainable.

Management 52

Management Programming Language Data Schemas Programming 52

Cloud Academy

JANUARY 27, 2022

Data engineering is the process of designing and implementing solutions to collect, store, and analyze large amounts of data. This process is generally called “Extract, Transfer, Load” or ETL. The format of the data will be different depending on the intended audience.

Data Engineering 52

Data Engineering Data Engineer Certification Engineering 52

Monte Carlo

JANUARY 25, 2023

It can be challenging when a team is expected to take full responsibility for a key data product when there are no guarantees around the upstream data quality. Without clear management of each transformation step stretching back to source systems, teams may be unwilling to bear the responsibility of contracts.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Zalando Engineering

SEPTEMBER 28, 2022

In this post, George and Daniel describe the product that was built to serve this purpose - its problem space, the solution design process, the technological context and how the product evolved to include new use-cases, such as the Zalando Sustainability topic. Our goal was to streamline this into an efficient, user-friendly process.

Certification 52

Certification Architecture Data Schemas Metadata 52

dbt Developer Hub

FEBRUARY 7, 2022

All of these questions require aggregating + syncing data from application usage, workspace information, and orders into the CRM for the sales team to have at their fingertips. Below, I’ll break down the DAG at each step of our process so you can see how it builds all together.

Data Warehouse 52

Data Warehouse Datasets Data SQL 52

Rockset

OCTOBER 6, 2022

Consequently, we needed a data backend with the following characteristics: Scale With ~50 commits per working day (and thus at least 50 pull request updates per day) and each commit running over one million tests, you can imagine the storage/computation required to upload and process all our data.

AWS 52

AWS Data Schemas Accessible Accessibility 52

Monte Carlo

APRIL 24, 2023

As we saw in Part 2 of our series , the definition of “building” and “buying” can change based on what layer of the data stack we’re considering. This data ingestion process can be accomplished by either querying the source directly, using upstream systems to publish events, or some combination of the two.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Confluent

MAY 28, 2019

Over time, organizations restructure, project scopes change, and an end system that was used by one application may now be used by multiple applications. If that happens and schema IDs are no longer globally unique, there may be collisions between schema IDs. Since Confluent Platform 4.0,

Management 82

Management Kafka Java Certification 82

Netflix Tech

OCTOBER 31, 2018

the Media Timeline Data Model In the previous post in this series, we described some important Netflix business needs as well as traits of the media data system?—?called The curious reader might have noticed that a majority of these characteristics relate to properties of the data managed by NMDB.

Media 54

Media Metadata Data MongoDB 54

ProjectPro

APRIL 26, 2016

Pig vs Hive Criteria Pig Hive Type of Data Apache Pig is usually used for semi structured data. Used for Structured Data Schema Schema is optional. Hive requires a well-defined Schema. Language It is a procedural data flow language. Hcatalog can be used to share data structures with external systems.

Hadoop 40

Hadoop Metadata SQL Database 40

ProjectPro

AUGUST 21, 2015

Hadoop vs RDBMS Criteria Hadoop RDBMS Datatypes Processes semi-structured and unstructured data. Processes structured data. Schema Schema on Read Schema on Write Best Fit for Applications Data discovery and Massive Storage/Processing of Unstructured data.

Hadoop 40

Hadoop Big Data Metadata Java 40