Pinterest Engineering

OCTOBER 31, 2023

Harry Zhang, Jiajun Wang, Yi Li, Shunyao Li, Ming Zong, Haniel Martino, Cathy Lu, Quentin Miao, Hao Jiang, James Wen, David Westbrook | Cloud Runtime Team Image Source: [link] Overview Modern compute platforms are foundational to accelerating innovation and running applications more efficiently.

Architecture 95

Architecture Pipeline-centric Accessible Accessibility 95

LinkedIn Engineering

MARCH 9, 2023

Co-authors: Shu Wang , Biao He , and Minchu Yang At LinkedIn, Apache Spark is our primary compute engine for offline data analytics such as data warehousing, data science, machine learning, A/B testing, and metrics reporting. These applications rely heavily on dependencies ( JAR files ) for their computation needs.

Hadoop 124

Hadoop Machine Learning Designing Data Pipeline 124

LinkedIn Engineering

MARCH 23, 2023

Co-Authors: Yuhong Cheng , Shangjin Zhang , Xinyu Liu, and Yi Pan Efficient data processing is crucial in reducing learning curves, simplifying maintenance efforts, and decreasing operational complexity. In this blog post, we will share our progress, challenges, and lessons learned from implementing Apache Beam.

Process 97

Process Lambda Architecture Kafka Datasets 97

Airbnb Tech

SEPTEMBER 28, 2021

Consolidating long-running, lightweight tasks for improved resource utilization By: Yingbo Wang , Kevin Yang Introduction Airflow is a platform to programmatically author, schedule, and monitor data pipelines. All company, product and service names used in this website are for identification purposes only.

Utilities 52

Utilities Database Data Warehouse Data Pipeline 52