Pinterest Engineering

NOVEMBER 22, 2023

Initial Architecture For Goku Short Term Ingestion Figure 1: Old push based ingestion pipeline into GokuS At Pinterest, we have a sidecar metrics agent running on every host that logs the application system metrics time series data points (metric name, tag value pairs, timestamp and value) into dedicated kafka topics.

Database 95

Database Bytes Kafka Architecture 95

AltexSoft

SEPTEMBER 21, 2023

Elasticsearch’s true power lies in its ability to partition indices into smaller units known as shards, facilitating data distribution across multiple servers. It also employs replicas to duplicate these shards to ensure data reliability and availability. Having replica shards ensures your data is not lost if a node fails.

Engineering 52

Engineering NoSQL Programming Language Java 52

Netflix Tech

SEPTEMBER 29, 2022

All data structures related to a queue — from the messages it contains to the in-memory secondary indexes needed to support dequeue-by-filter — are placed in the same Redis shard. We then codify this prefix as a Redis hash tag. We want to support arbitrarily large queues, which has us building support for queue sharding.

Systems 85

Systems Metadata Media Kafka 85

Data Engineering Podcast

DECEMBER 17, 2017

In the documentation it mentions needing to specify the retention period for the shards when creating a database. How are metrics identified in Siri and is there any support for tagging? In the documentation it mentions needing to specify the retention period for the shards when creating a database.

Database 100

Database Data Pipeline Data Engineering Data Engineer 100

Knowledge Hut

OCTOBER 27, 2023

Fake product identification is a blockchain practice project that uses a particular tag to track the origin of a product. The tag contains all the information about its origin and even has an electronic signature that can be verified using blockchain technology. It could be used to verify that a diamond, for example, is not fake.

Coding 52

Coding Project Electronics Retail 52

Robinhood

FEBRUARY 7, 2024

We decide to shard the operators to better balance the load on individual operator deployment. Ensure Spark job resiliency on scale down: To enable the aggressive auto-scale down without concern, we moved the Spark driver to a separate instance group and tagged the AC to treat driver pod as in-churnable.

Process 75

Process Hadoop Architecture Accessible 75

Rockset

JULY 22, 2022

NoSQL is about tuning the data models for specific access patterns, removing the JOINs, replacing them with indexes across items on a table that sharded or partitioned and documents in a collection that share indexes because those index lookups have low time complexity, which satisfies your high velocity patterns.

NoSQL 52

NoSQL SQL Database AWS 52