Uber Engineering

JANUARY 16, 2023

Our Data Workflow Platform team introduces WorkflowGuard: a new service to govern executions, prioritize resources, and manage life cycle for repetitive data jobs. Check out how it improved workflow reliability and cost efficiency while bringing more observability to users.

Data Workflow 98

Data Workflow Government Systems Data 98

ProjectPro

JUNE 6, 2025

Airflow and DBT both have the overall purpose of helping teams in providing reliable data to the users with whom they interact by using a standard interface.

Data Workflow 40

Data Workflow Programming Language SQL Data Pipeline 40

Data Engineering Weekly

JANUARY 15, 2025

As large language models (LLMs) and AI agents become indispensable in everything from customer service to autonomous vehicles, the ability to manage, analyze, and optimize unstructured data has become a strategic imperative. Billions of social media posts, hours of video content, and terabytes of sensor data are produced daily.

Data Engineering 64

Data Engineering Data Engineer Unstructured Data Engineering 64

Snowflake

MARCH 6, 2025

And to create significant technology and team efficiencies, organizations need to consider opportunities to integrate LLM pipelines with existing structured data workflows. This unification can also empower data engineers, who already manage structured pipelines, to easily onboard and maintain unstructured data workflows.

Unstructured Data 95

Unstructured Data Medical Media Data Workflow 95

KDnuggets

JUNE 24, 2025

By subscribing you accept KDnuggets Privacy Policy Leave this field empty if youre human: Next post => Latest Posts Make Sense of a 10K+ Line GitHub Repos Without Reading the Code Build a Data Cleaning & Validation Pipeline in Under 50 Lines of Python Building AI Agents with llama.cpp MLFlow Mastery: A Complete Guide to Experiment Tracking and (..)

Python 101

Python Building Data Science Machine Learning 101

Snowflake

MARCH 26, 2025

This means enterprises can run unstructured data workflows, powered by AI agents, without moving data out of Snowflake which enhances trust and helps support compliance. First, Snowflake has enabled us to strengthen user trust in our app. Second, were optimizing scalability.

Unstructured Data 97

Unstructured Data SQL Data Data Workflow 97

KDnuggets

APRIL 16, 2024

Bring the latest AI implementation to Pandas to improve your data workflow.

Utilities 155

Utilities Data Analysis Data Workflow Data 155

KDnuggets

JUNE 23, 2025

By subscribing you accept KDnuggets Privacy Policy Leave this field empty if youre human: Latest Posts MLFlow Mastery: A Complete Guide to Experiment Tracking and Model Management How to Learn Programming for Data Science: A Roadmap for Beginners 5 Things You Need to Know About Agentic AI Data Science, No Degree Top 5 Frameworks for Distributed Machine (..)

Management 90

Management Machine Learning Data Science Metadata 90

KDnuggets

SEPTEMBER 9, 2024

Exploring the not-so-famous data science libraries that can be useful in your data workflow.

Data Science 139

Data Science Python Data Workflow Data 139

Cloudera

DECEMBER 4, 2024

Our joint collaboration will enable the following: Seamless Data Sharing and Interoperability : The integration enables AWS customers to leverage Cloudera’s data lakehouse capabilities alongside Snowflake’s AI Data Cloud, facilitating unified data access and sharing across platforms Enhanced AI/ML Performance : The partnership optimizes data workflows (..)

AWS 89

AWS Raw Data Relational Database Government 89

KDnuggets

JUNE 27, 2025

Top 5 Frameworks for Distributed Machine Learning Get the FREE ebook The Great Big Natural Language Processing Primer and The Complete Collection of Data Science Cheat Sheets along with the leading newsletter on Data Science, Machine Learning, AI & Analytics straight to your inbox.

Data Science 101

Data Science Machine Learning Datasets Python 101

Data Engineering Podcast

APRIL 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Data Lake 162

Data Lake High Quality Data BI Data Workflow 162

Data Engineering Podcast

FEBRUARY 18, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

KDnuggets

JUNE 26, 2025

By subscribing you accept KDnuggets Privacy Policy Leave this field empty if youre human: Latest Posts Automate Data Quality Reports with n8n: From CSV to Professional Analysis 7 Popular LLMs Explained in 7 Minutes Vibe Coding a Speed Reading App with Python in Just 15 Minutes 10 FREE AI Tools That’ll Save You 10+ Hours a Week Make Sense of a 10K+ (..)

Datasets 105

Datasets Data Science Machine Learning Python 105

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Data Engineering Podcast

DECEMBER 24, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

KDnuggets

JUNE 19, 2025

By subscribing you accept KDnuggets Privacy Policy Leave this field empty if youre human: Latest Posts Forget Streamlit: Create an Interactive Data Science Dashboard in Excel in Minutes Go vs. Python for Modern Data Workflows: Need Help Deciding?

Data Science 91

Data Science Machine Learning Datasets Python 91

Snowflake

MARCH 24, 2025

DataOps.live keeps users at the forefront of data engineering DataOps.live works together with Snowflake to augment and extend native Snowflake features, resulting in advanced DataOps workflows for Snowflake customers. Snowflake and DataOps.lives integrated solutions simplify the development, testing and deployment of data workflows.

Cloud 68

Cloud Data Pipeline Data Workflow Data Engineering 68

KDnuggets

OCTOBER 22, 2024

Utilize these transformation techniques in your data workflow.

Data Workflow 117

Data Workflow Utilities Data 117

Monte Carlo

JUNE 30, 2025

For data professionals, understanding these trends determines career success and staying relevant in the field. Artificial intelligence streamlines data workflows 2. Data mesh architectures democratize information access 5. Data products generate business opportunities 6. Table of Contents 1.

Data Management 52

Data Management Amazon Web Services Management Government 52

Hevo

JULY 4, 2025

In 2025, the automation of ETL (Extract, Transform, and Load) processes is crucial for businesses to streamline data workflows, improve data integrity, and minimize processing times. Manual ETL pipelines are error-prone and time-consuming, making them difficult to scale. 76% […]

Data Workflow 40

Data Workflow Data Integration Process Data 40

KDnuggets

JANUARY 27, 2025

Master cleaner, faster code with these essential techniques to supercharge your data workflows.

Python 116

Python Data Workflow Data Coding 116

Ascend.io

DECEMBER 4, 2024

Deploy DataOps DataOps , or Data Operations, is an approach that applies the principles of DevOps to data management. It aims to streamline and automate data workflows, enhance collaboration and improve the agility of data teams. How effective are your current data workflows?

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52

KDnuggets

NOVEMBER 30, 2023

Data engineers must prepare and manage the infrastructure and tools necessary for the whole data workflow in a data-driven company.

Data Engineering 128

Data Engineering Data Engineer Engineering Data Workflow 128

KDnuggets

JUNE 17, 2024

Exploring the available text Python packages for your data workflow.

Python 123

Python Data Workflow Data 123

ProjectPro

JUNE 6, 2025

Dagster vs Airflow: Overview Dagster and Airflow are two popular open-source tools that have emerged as leaders in data orchestration. They are often compared because of their shared goal of automating data workflows and widespread adoption in the data engineering community. What is Airflow? What is Dagster?

Pipeline-centric 40

Pipeline-centric Database-centric Data Pipeline Data Workflow 40

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. Your first 30 days are free! Your first 30 days are free!

Project 130

Project Data Lake High Quality Data Data Workflow 130

KDnuggets

JANUARY 4, 2023

These Python packages would improve your data workflow.

Python 118

Python Data Workflow Data Data Science 118

Data Engineering Weekly

NOVEMBER 24, 2024

By creating custom linting rules tailored to their team's needs, Next Insurance has improved its data workflows' maintainability, scalability, and quality, making it easier for engineers to collaborate and debug issues.

Data Engineering 59

Data Engineering Data Engineer Engineering Insurance 59

ProjectPro

JUNE 6, 2025

Open Source Data Pipeline Tools Open-source data pipeline tools are pivotal in data engineering, offering organizations flexible and scalable solutions for managing the end-to-end data workflow. Here is the list of robust Data Pipeline Tools in Azure for scalable and optimized management of diverse data sources.

Data Pipeline 49

Data Pipeline Google Cloud AWS Kafka 49

Data Engineering Podcast

JUNE 23, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 162

Data Lake High Quality Data Hadoop Machine Learning 162

Data Engineering Podcast

APRIL 21, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Data Lake 182

Data Lake High Quality Data Machine Learning Data Pipeline 182

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex.

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

DECEMBER 3, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues for every part of your data workflow, from migration to deployment. Datafold has recently launched a 3-in-1 product experience to support accelerated data migrations. Datafold : ![Datafold]([link]

Systems 130

Systems Designing Data Lake SQL 130

KDnuggets

JUNE 24, 2025

By subscribing you accept KDnuggets Privacy Policy Leave this field empty if youre human: Latest Posts Make Sense of a 10K+ Line GitHub Repos Without Reading the Code Build a Data Cleaning & Validation Pipeline in Under 50 Lines of Python Building AI Agents with llama.cpp MLFlow Mastery: A Complete Guide to Experiment Tracking and Model Management (..)

Coding 69

Coding Data Science Machine Learning Python 69

ProjectPro

JUNE 6, 2025

It's like the ultimate solution for managing and automating big data workflows. Did you know 93% of seasoned Airflow users are willing to recommend this powerful data orchestration tool. Businesses from various sectors leverage it to manage and automate massive data workflows seamlessly. Crazy, right? stars and 13.4k

PostgreSQL 40

PostgreSQL Metadata MySQL Data Workflow 40