Blog - Data Engineering Digest

PinCompute: A Kubernetes Backed General Purpose Compute Platform for Pinterest

Pinterest Engineering

OCTOBER 31, 2023

Harry Zhang, Jiajun Wang, Yi Li, Shunyao Li, Ming Zong, Haniel Martino, Cathy Lu, Quentin Miao, Hao Jiang, James Wen, David Westbrook | Cloud Runtime Team Image Source: [link] Overview Modern compute platforms are foundational to accelerating innovation and running applications more efficiently.

Architecture

Architecture Pipeline-centric Accessible Accessibility

Reducing Apache Spark Application Dependencies Upload by 99%

LinkedIn Engineering

MARCH 9, 2023

Co-authors: Shu Wang , Biao He , and Minchu Yang At LinkedIn, Apache Spark is our primary compute engine for offline data analytics such as data warehousing, data science, machine learning, A/B testing, and metrics reporting. These applications rely heavily on dependencies ( JAR files ) for their computation needs.

Hadoop

Hadoop Machine Learning Designing Data Pipeline

Data Engineering Digest

PinCompute: A Kubernetes Backed General Purpose Compute Platform for Pinterest

Reducing Apache Spark Application Dependencies Upload by 99%

Webinars

Stay Connected