article thumbnail

Introducing WorkflowGuard: The Workflow Governance and Observability System That Oversees over 120,000 Data Workflows

Uber Engineering

Our Data Workflow Platform team introduces WorkflowGuard: a new service to govern executions, prioritize resources, and manage life cycle for repetitive data jobs. Check out how it improved workflow reliability and cost efficiency while bringing more observability to users.

article thumbnail

Managing Uber’s Data Workflows at Scale

Uber Engineering

At Uber’s scale, thousands of microservices serve millions of rides and deliveries a day, generating more than a hundred petabytes of raw data. Internally, engineering and data teams across the company leverage this data to improve the Uber experience.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Automate data workflows using AWS Data Pipeline

Medium Data Engineering

If you’re looking to ETL your data using AWS, one option is to use their Data Pipeline service. Data Pipeline is a web service that helps… Continue reading on Medium »

article thumbnail

Want to streamline your data workflows and improve collaboration across teams?

Medium Data Engineering

DataOps combines the best practices of Agile, DevOps, and Lean to manage data-driven workflows and deliver high-quality data products and… Continue reading on Medium »

article thumbnail

Revolutionize Data Workflow with Airflow ETL Pipeline in Python

Medium Data Engineering

Explore how to design scalable workflows with Airflow in Python to manage the ETL pipeline. Continue reading on Simform Engineering »

Python 52
article thumbnail

DBT vs. Airflow-Which is The Best Tool for Your Data Workflows?

ProjectPro

Airflow and DBT both have the overall purpose of helping teams in providing reliable data to the users with whom they interact by using a standard interface.

article thumbnail

A Must-Have Tool: AWS Wrangler to Simplify Big Data Workflows on the Cloud

Medium Data Engineering

Big data processing is a complex task that often requires specialized skills and infrastructure. AWS Wrangler is a powerful tool that… Continue reading on Medium »

AWS 40