Ascend.io

NOVEMBER 14, 2024

Why Future-Proofing Your Data Pipelines Matters Data has become the backbone of decision-making in businesses across the globe. The ability to harness and analyze data effectively can make or break a company’s competitive edge. Resilience and adaptability are the cornerstones of a future-proof data pipeline.

Data Pipeline 52

Data Pipeline Amazon Web Services Data Data Integration 52

RudderStack

JUNE 23, 2025

million terabytes of data generated every day, even small data issues can scale into costly business problems. Data pipeline monitoring addresses this by providing visibility into the health, reliability, and performance of your data flows. What is data pipeline monitoring? With over 402.74

Data Pipeline 52

Data Pipeline Metadata Government Data 52

Simon Späti

JUNE 17, 2025

In Part 1, The Heartbeat of Data Engineering , we discussed the convergent orchestrator combining orchestration as code and no-code. The dual interface with an advanced UI but with the generated code as YAML helps automate and version control the data pipelines.

Data Pipeline 189

Data Pipeline Coding Data Data Engineering 189

ProjectPro

JUNE 6, 2025

Redshift Project for Data Analysis with Amazon Quicksight 2.Amazon Using Airflow for Building and Monitoring the Data Pipeline of Amazon Redshift 4. Therefore, organizations use Airflow to monitor the data pipelines and ensure seamless data transfer. Amazon Redshift Machine Learning 6.

Data Pipeline 64

Data Pipeline AWS Project Building 64

RandomTrees

JUNE 30, 2025

Blog Part 1: Social Media Data Pipeline – GCP Setup and Modeling Introduction In this blog series, I will walk you through a real-world case study I personally worked on, where we built an end-to-end social media data pipeline using Google Cloud Platform (GCP) and Apache Airflow.

Media 52

Media Data Pipeline Cloud Storage Google Cloud 52

ProjectPro

JUNE 6, 2025

It requires a skillful blend of data engineering expertise and the strategic use of tools designed to streamline this process. That’s where data pipeline tools come in. This blog is all about that—specifically, the top 10 data pipeline tools that data engineers worldwide rely on.

Data Pipeline 49

Data Pipeline Google Cloud AWS Kafka 49

RandomTrees

JULY 2, 2025

This approach simplifies transformation logic and brings automation into the data pipeline. Now in Part 2, we’ll focus on building an Apache Airflow DAG that automatically reads SQL files from Cloud Storage and executes them in BigQuery. Upload and Configure Airflow DAG 1.

Media 52

Media Data Pipeline Cloud Storage SQL 52

ProjectPro

JUNE 6, 2025

This article comprehensively looks at what is Apache Airflow and evaluates whether it's the right tool of choice for data engineers and data scientists. We know you are enthusiastic about building data pipelines from scratch using Airflow. How is Data Pipeline Flexibility Defined in Apache Airflow?

Data Pipeline 45

Data Pipeline Building Python Raw Data 45

Hevo

JUNE 20, 2025

Traditional data pipelines often come with the burden of managing a large infrastructure and the inability to scale during traffic spikes. Enter serverless data pipelines. Serverless data pipelines can eliminate the need for infrastructure management by offering a plug-and-play […]

Data Pipeline 40

Data Pipeline Data Systems Management 40

ProjectPro

JUNE 6, 2025

If someone is looking to master the art and science of constructing batch pipelines, ProjectPro has got you covered with this comprehensive tutorial that will help you learn how to build your first batch data pipeline and transform raw data into actionable insights. Table of Contents What Is A Batch Data Pipeline?

Data Pipeline 40

Data Pipeline Building Retail Data Ingestion 40

ProjectPro

JUNE 6, 2025

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 40

Data Pipeline Architecture Kafka Data Lake 40

ProjectPro

JUNE 6, 2025

This blog compares the most popular and helpful AWS ETL services in the market today- AWS Data Pipeline and AWS Glue. AWS Data Pipeline vs. AWS Glue- 5 Key Differences Between Them Here are five primary differences between AWS Data Pipeline ETL vs. Glue to help you decide on the best AWS ETL tool.

ETL Tools 40

ETL Tools AWS Data Pipeline Amazon Web Services 40

Hevo

JUNE 20, 2025

Every AI model requires an efficient data flow to fuel it. However, most teams struggle when building an efficient AI data pipeline. Traditional data pipelines were designed to move and transform data for analytics, but AI is changing this. AI […]

Data Pipeline 40

Data Pipeline Building High Quality Data Data 40

RandomTrees

JULY 9, 2025

Now in Part 3, we’ll focus on querying the transformed data in BigQuery to generate meaningful insights and visualize the final outputs using Looker Studio. In Part 2, we built a dynamic Airflow DAG that automates the execution of SQL transformations from Cloud Storage into BigQuery.

Media 52

Media Data Pipeline Cloud Storage SQL 52

Start Data Engineering

APRIL 14, 2025

API Data extraction = GET-ting data from a server 3.1. GET data 3.1.1. GET data for a specific entity 3. Introduction 2. APIs are a way to communicate between systems on the Internet 2.1. HTTP is a protocol commonly used for websites 2.1.1. Request: Ask the Internet exactly what you want 2.1.2.

Data Pipeline 130

Data Pipeline Python Data Systems 130

Start Data Engineering

FEBRUARY 3, 2025

Introduction If you’ve been in the data space long enough, you would have come across really long SQL scripts that someone had written years ago. Introduction 2. Split your SQL into smaller parts 2.1. Start with a baseline validation to ensure that your changes do not change the output too much 2.2. Conclusion 4. Required reading 1.

SQL 147

SQL Data Pipeline Data 147

Cloudera

DECEMBER 4, 2024

Our customers rely on NiFi as well as the associated sub-projects (Apache MiNiFi and Registry) to connect to structured, unstructured, and multi-modal data from a variety of data sources – from edge devices to SaaS tools to server logs and change data capture streams. Cloudera DataFlow 2.9

Data Pipeline 93

Data Pipeline Data Ingestion Data Preparation Architecture 93

KDnuggets

JUNE 24, 2025

🔗 Link to the code on GitHub Why Data Cleaning Pipelines? Think of data pipelines like assembly lines in manufacturing. Wrapping Up Data pipelines arent just about cleaning individual datasets. Each step performs a specific function, and the output from one step becomes the input for the next.

Python 101

Python Building Data Science Machine Learning 101

Analytics Vidhya

APRIL 3, 2023

Introduction Companies can access a large pool of data in the modern business environment, and using this data in real-time may produce insightful results that can spur corporate success. Real-time dashboards such as GCP provide strong data visualization and actionable information for decision-makers.

Data Engineering 258

Data Engineering Data Engineer Engineering Data 258

Start Data Engineering

JUNE 24, 2024

Features crucial to building and maintaining data pipelines 2.1. Schedulers to run data pipelines at specified frequency 2.2. Orchestrators to define the order of execution of your pipeline tasks 2.2.1. Define the order of execution of pipeline tasks with a DAG 2.2.2. Introduction 2.

Data Pipeline 208

Data Pipeline Coding Building Data 208

Start Data Engineering

JUNE 14, 2024

Run Data Pipelines 2.1. Batch pipelines 3.3. Stream pipelines 3.4. Event-driven pipelines 3.5. LLM RAG pipelines 4. Introduction Whether you are new to data engineering or have been in the data field for a few years, one of the most challenging parts of learning new frameworks is setting them up!

Data Engineering 239

Data Engineering Data Engineer Project Engineering 239

KDnuggets

JUNE 19, 2025

By Bala Priya C , KDnuggets Contributing Editor & Technical Content Specialist on June 19, 2025 in Programming Image by Author | Ideogram Youre architecting a new data pipeline or starting an analytics project, and you’re probably considering whether to use Python or Go. We compare Go and Python to help you make an informed decision.

Data Workflow 112

Data Workflow Python Data Ingestion Machine Learning 112

Confessions of a Data Guy

JANUARY 8, 2025

You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. appeared first on Confessions of a Data Guy.

Data Storage 124

Data Storage Data Pipeline Data IT 124

Pinterest Engineering

JUNE 24, 2025

However, the traditional ML infrastructure was constrained by slow data pipelines, costly feature iterations, and inefficient compute usage. Once the feature transformation is ready to launch, the Ray-based data pipeline will be reused to set up production serving and training data generation.

Software Engineering 73

Software Engineering Software Engineer Datasets Data Pipeline 73

Snowflake

NOVEMBER 11, 2024

Here’s how Snowflake Cortex AI and Snowflake ML are accelerating the delivery of trusted AI solutions for the most critical generative AI applications: Natural language processing (NLP) for data pipelines: Large language models (LLMs) have a transformative potential, but they often batch inference integration into pipelines, which can be cumbersome.

Unstructured Data 98

Unstructured Data SQL AWS Healthcare 98

Marc Lamberti

SEPTEMBER 22, 2023

Let’s imagine you have the following data pipeline: In a nutshell, this data pipeline trains different machine learning models based on a dataset and the last task selects the model with the highest accuracy. You should obtain the following DAG: The data pipeline is simple. How to use XCom in Airflow?

MySQL 246

MySQL Data Pipeline Database Python 246

Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Analytics Vidhya

AUGUST 7, 2024

Introduction Apache Airflow is a crucial component in data orchestration and is known for its capability to handle intricate workflows and automate data pipelines. Many organizations have chosen it due to its flexibility and strong scheduling capabilities.

Data Pipeline 192

Data Pipeline Data Process Data Workflow 192

Start Data Engineering

APRIL 22, 2024

Test types for data pipelines 2.2. Use fixture to create fake data for testing 2.2.4. Introduction 2. Ensure the code’s logic is working as expected with tests 2.1. pytest: A powerful Python library for testing 2.2.1. Set context, run code, check results & clean up 2.2.2. Tests are identified by their name 2.2.3.

Coding 208

Coding Data Pipeline Python Data 208

ProjectPro

JUNE 6, 2025

Managing complex data pipelines can be challenging, requiring coordination between multiple systems and teams. Apache Airflow DAGs provide a powerful tool for creating and managing data pipelines, streamlining the process of data processing and automation. This is where Apache Airflow DAGs come in.

Data Pipeline 63

Data Pipeline Python PostgreSQL Database 63

Marc Lamberti

OCTOBER 4, 2023

Let’s say you have the following data pipeline: A quick explanation about the DAG: the first three tasks train machine learning models. The BranchPythonOperator is useful as it allows solving new use cases for you data pipelines. Use case Using a use case is the best way to understand a concept. Have a great day!

Python 246

Python Machine Learning Data Pipeline IT 246

Marc Lamberti

OCTOBER 1, 2023

Airflow Sensors are one of the most common tasks in data pipelines. If you want to make complex and robust data pipelines, you have to understand how Sensors work genuinely. Back to your data pipeline: Image your DAG runs every day at midnight but the files from sources A, B, and C, never come.

Data Pipeline 243

Data Pipeline SQL Algorithm Coding 243

Christophe Blefari

MARCH 15, 2024

Forward thinking Dataviz is hierarchical — Malloy, once again, provides an excellent article about a new way to see data visualisations. Coding data pipelines is faster than renting connector catalogs — This is something I've always believed. It's inspirational.

Metadata 272

Metadata Data Warehouse Software Engineering Software Engineer 272

Data Engineering Podcast

APRIL 23, 2023

Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Ascend.io

DECEMBER 4, 2024

As we look towards 2025, it’s clear that data teams must evolve to meet the demands of evolving technology and opportunities. In this blog post, we’ll explore key strategies that data teams should adopt to prepare for the year ahead. The anticipated growth in data pipelines presents both challenges and opportunities.

Data Pipeline 52

Data Pipeline Metadata Data Workflow Data 52