Blog - Data Engineering Digest

PinCompute: A Kubernetes Backed General Purpose Compute Platform for Pinterest

Pinterest Engineering

OCTOBER 31, 2023

It is integrated with Statsboard, Pinterest’s native metrics dashboard, allowing users to configure application-level metrics with desired thresholds to trigger scaling along with scaling safeguards, such as a cool down window and replica min/max limitations. Figure 4: PinCompute Primitives: PinPod, PinApp, and PinScaler.

Architecture

Architecture Pipeline-centric Accessible Accessibility

Reducing Apache Spark Application Dependencies Upload by 99%

LinkedIn Engineering

MARCH 9, 2023

Co-authors: Shu Wang , Biao He , and Minchu Yang At LinkedIn, Apache Spark is our primary compute engine for offline data analytics such as data warehousing, data science, machine learning, A/B testing, and metrics reporting. These applications rely heavily on dependencies ( JAR files ) for their computation needs.

Hadoop

Hadoop Machine Learning Designing Data Pipeline

Data Engineering Digest

PinCompute: A Kubernetes Backed General Purpose Compute Platform for Pinterest

Reducing Apache Spark Application Dependencies Upload by 99%

Webinars

Stay Connected