article thumbnail

Introducing WorkflowGuard: The Workflow Governance and Observability System That Oversees over 120,000 Data Workflows

Uber Engineering

Our Data Workflow Platform team introduces WorkflowGuard: a new service to govern executions, prioritize resources, and manage life cycle for repetitive data jobs. Check out how it improved workflow reliability and cost efficiency while bringing more observability to users.

article thumbnail

Managing Uber’s Data Workflows at Scale

Uber Engineering

At Uber’s scale, thousands of microservices serve millions of rides and deliveries a day, generating more than a hundred petabytes of raw data. Internally, engineering and data teams across the company leverage this data to improve the Uber experience.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

5 Free Courses to Master Data Engineering

KDnuggets

Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.

article thumbnail

KDnuggets News, December 6: GitHub Repositories to Master Machine Learning • 5 Free Courses to Master Data Engineering

KDnuggets

This week on KDnuggets: Discover GitHub repositories from machine learning courses, bootcamps, books, tools, interview questions, cheat sheets, MLOps platforms, and more to master ML and secure your dream job • Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company • And much, (..)

article thumbnail

How to Automate PySpark Pipelines on AWS EMR With Airflow

Towards Data Science

Optimising big data workflows orchestration Continue reading on Towards Data Science »

AWS 89
article thumbnail

The Scoop: Tech Layoffs in 2022

The Pragmatic Engineer

7 November Domino Data Lab (data workflow platform, Series E). In Mexico, the company had a large initiative with private drivers and a fleet of Tesla cars. That operation is in talks of selling to other investors. Meta - 13% layoffs. Widely-reported, e.g. here. 25% of their workforce laid off. Verified.

article thumbnail

Top Data Python Packages to Know in 2023

KDnuggets

These Python packages would improve your data workflow.

Python 114