Start Data Engineering

APRIL 17, 2020

When getting started with Apache Airflow , data engineers have questions similar to the two below “What are people’s opinions of Airflow?”

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Start Data Engineering

JANUARY 6, 2021

Setup Prerequisites Apache Airflow - Execution Day Backfill Conclusion Further Reading References What is backfilling ? What is backfilling ? Backfilling refers to any process that involves modifying or adding new data to existing records in a dataset. This is a common use case in data engineering.

SQL 130

SQL Datasets Data Engineering Data Engineer 130

know.bi

MAY 10, 2023

A number or questions have been raised in the Apache Hop chat about running workflows and pipelines in Apache Airflow recently. In this post, we'll start from the existing how-to guide in the Apache Hop docs, but add a bit more context and goes into a bit more detail on how to get everything going.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Start Data Engineering

MARCH 28, 2020

One of the most common use cases for Apache Airflow is to run scheduled SQL scripts. Developers who start with Airflow often ask the following questions “How to use airflow to orchestrate sql?”

SQL 130

SQL 130

Acceldata

OCTOBER 4, 2023

Learn how to apply effective data quality monitoring and alerting strategies to ensure that Apache Airflow workflows are reliable.

Data 52

Data 52

Simon Späti

NOVEMBER 28, 2018

I went with Apache Druid for data storage, Apache Superset for querying and Apache Airflow as a task orchestrator. In my recent blog, I researched OLAP technologies, for this post I chose some open-source technologies and used them together to build a full data architecture for a Data Warehouse system.

Data Warehouse 130

Data Warehouse Data Storage Data Architecture Architecture 130

ProjectPro

AUGUST 9, 2022

With over 8 million downloads, 20000 contributors, and 13000 stars, Apache Airflow is an open-source data processing solution for dynamically creating, scheduling, and managing complex data engineering pipelines. use Airflow to run their data pipelines efficiently. What is Apache Airflow Used For? via REST API).

Scala 52

Scala Banking Hadoop Machine Learning 52

Marc Lamberti

SEPTEMBER 22, 2023

What are XCOMs in Apache Airflow? In this tutorial, you will learn about XComs in Airflow. If you checked my course “Apache Airflow: The Hands-On Guide”, Aiflow XCom should not sound unfamiliar. Table of contents Use Case What is an Airflow XCOM? How to use XCom in Airflow? What is an Airflow XCOM?

MySQL 246

MySQL Data Pipeline Database Python 246

Marc Lamberti

OCTOBER 1, 2023

Airflow Sensors are one of the most common tasks in data pipelines. That’s the few possibilities of the Airflow Sensors. By the way, if you are new to Airflow, check my course here , at a special discount. It will be a nightmare to maintain and you skip what Airflow offers you. Airflow Sensors! Well, guess what?

Data Pipeline 243

Data Pipeline SQL Algorithm Coding 243

Marc Lamberti

SEPTEMBER 29, 2023

At the end of this introduction to Airflow, you will be all set for getting started with Airflow. You will start with the basics, such as what Airflow is and the essential concepts. Introduction to Airflow: What it is? That’s the purpose of an orchestrator, in this case, Airflow. An Operator is a task.

Data Pipeline 130

Data Pipeline Python AWS Project 130

Marc Lamberti

OCTOBER 2, 2023

Airflow Variables are easy to use but easy to misuse as well. In this tutorial, you will learn everything you need about variables in Apache Airflow. If you followed my course “Apache Airflow: The Hands-On Guide” variables shouldn’t sound unfamiliar. Why Airflow Variables? What is a Variable?

AWS 130

AWS Google Cloud Database Coding 130

Marc Lamberti

OCTOBER 2, 2023

Airflow Variables are easy to use but easy to misuse as well. In this tutorial, you will learn everything you need about variables in Apache Airflow. If you followed my course “Apache Airflow: The Hands-On Guide” variables shouldn’t sound unfamiliar. Why Airflow Variables? What is a Variable?

AWS 130

AWS Google Cloud Database Coding 130

Marc Lamberti

SEPTEMBER 21, 2023

Looking to create your first Airflow DAG? Wondering how to process data in Airflow? At the end of this short tutorial, you will have your first Airflow DAG! You might think starting with Apache Airflow is hard, but it is not. The truth is Airflow has so many features that it can be overwhelming. Airflow DAG?

Data Pipeline 130

Data Pipeline Python Coding Machine Learning 130

Seattle Data Guy

NOVEMBER 27, 2023

If you’re a data engineer, then you’ve likely at least heard of Airflow. Apache Airflow is one of the most popular open-source workflow orchestration solutions that gets used for data pipelines.

Data Pipeline 130

Data Pipeline Data Engineering Data Engineer Data 130

Analytics Vidhya

JUNE 20, 2023

Traditional data engineering solutions, such as Apache Airflow, have played an important role in orchestrating and controlling data operations in order to tackle these difficulties.

Data Process 217

Data Process Data Engineering Data Engineer Process 217

Marc Lamberti

OCTOBER 11, 2023

Do you wonder how to use the DockerOperator in Airflow to kick off a docker image? If you’re new to Airflow, I’ve created a course you can check out here. As you may know, Airflow has many operators to perform actions on different tools, systems, etc. Or how to run a task without creating dependency conflicts?

AWS 130

AWS Python Hadoop SQL 130

Marc Lamberti

OCTOBER 20, 2023

Templates and Macros in Apache Airflow allow passing data to your DAGs at runtime. If you want to dive deeper into Airflow, here is a course I made for you. Why Airflow templating? That tells Airflow to replace the placeholder between the curly brackets with the corresponding value at runtime. How can you do that?

SQL 130

SQL Python Coding Metadata 130

Marc Lamberti

JULY 6, 2021

Airflow on Kubernetes is quite popular isn’t it? There is a good chance that you know Kubernetes, that you even have a Kubernetes cluster and you would like to deploy and run Airflow on it. Apache Airflow just got its own official Helm chart! However, Kubernetes is hard. And guess what? Let’s get started!

Accessible 130

Accessible Accessibility Data Pipeline Building 130

Marc Lamberti

SEPTEMBER 24, 2023

An Airflow TaskGroup helps make a complex DAG easier to organize and read. Airflow taskgroups are meant to replace SubDAGs, the historical way of grouping your tasks. If you are new to Airflow, check out my course here; you will get it at a special discount. What is an Airflow Taskgroup? When to use an Airflow Taskgroup?

Coding 130

Coding Python Process Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. The implications of scaling Airflow — Sarah, who's working at Prefect, wrote a post about Airflow downsides at scale and how Prefect mitigates them.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Christophe Blefari

SEPTEMBER 25, 2023

Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. The implications of scaling Airflow — Sarah, who's working at Prefect, wrote a post about Airflow downsides at scale and how Prefect mitigates them.

Data 130

Data Data Warehouse Data Storage Data Pipeline 130

Towards Data Science

MAY 8, 2023

External Task Sensors stop bad data from trickling downstream in a data pipeline. Leverage them to create a reliable data infrastructure. Continue reading on Towards Data Science »

Data Pipeline 68

Data Pipeline Data Science Data Big Data 68

Marc Lamberti

SEPTEMBER 18, 2022

Airflow Taskflow is a new way of writing DAGs at ease. By the way, if you are new to Airflow, check my courses here ; you will get at a special discount. Before Airflow 2.0 The Airflow Taskflow API is not available before Apache Airflow 2.0. That helps to make DAGs easier to build, read, and maintain.

SQL 130

SQL Python Coding Accessible 130

Christophe Blefari

APRIL 12, 2024

I've shared my journey with Apache Superset and why I consider Superset the best open-source alternative when it comes to building BI applications. PS: Apache Superset is going 4.0 Local pipeline development with SQLMesh, Airflow, and Postgres. Which I have already condensed in a post a few weeks ago (in English).

BI 130

BI SQL Data Coding 130

Christophe Blefari

JANUARY 20, 2023

As a side note we are looking for speakers for a late February Airflow Meetup. Still open topics, so whatever you want to share—have to be related to Airflow at some point—we'll be happy to welcome you as speaker. When I think that some people are saying Airflow is complex to launch.

Google Cloud 130

Google Cloud Data Hadoop Business Intelligence 130

Towards Data Science

MARCH 15, 2023

A quick and simple demo for running DAGs on Airflow Continue reading on Towards Data Science »

Building 67

Building Data Science Data Data Engineering 67

Data Engineering Podcast

JULY 9, 2023

In this episode Maxime Beauchemin of Airflow and Superset fame shares his vision for the entity-centric data model and how you can incorporate it into your own warehouse design. The major strategies in use today were created decades ago when the software and hardware for warehouse databases were far more constrained.

Database-centric 130

Database-centric Machine Learning SQL Data Engineering 130

Christophe Blefari

MARCH 17, 2023

Airflow alternatives meetup Click on the image to go the the LinkedIn event. We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. Other few articles but with no comment: Introducing multi-modal index for the Lakehouse in Apache Hudi.

Data 130

Data SQL Deep Learning Kafka 130

Christophe Blefari

MARCH 24, 2023

👍 or 👎 Airflow alternatives meetup This Tuesday also took place the first part of the Airflow alternatives Meetup with Mage and Kestra. Which, to be honest, are not really far from Airflow. The tagline " Modern replacement for Airflow " on Mage side makes sense. Do you like all the GenAI hype?

Lambda Architecture 130

Lambda Architecture Data Pipeline Data SQL 130

Cloudera

JULY 13, 2021

Delivered through the Cloudera Data Platform (CDP) as a managed Apache Spark service on Kubernetes, CDE offers unique capabilities to enhance productivity for data engineering workloads: Visual GUI-based monitoring, troubleshooting and performance tuning for faster debugging and problem resolution. Flexible orchestration with Apache Airflow.

Data Engineering 126

Data Engineering Data Engineer Engineering Pipeline-centric 126

Towards Data Science

APRIL 20, 2023

A solution for effectively managing YouTube data with cloud services and job schedulers Continue reading on Towards Data Science »

Data Pipeline 73

Data Pipeline AWS Data Science Data 73

Christophe Blefari

DECEMBER 31, 2023

At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. Fast News ⚡️ Airflow 2.8 is out — The Airflow rhythm of release is crazy, I can't keep up with the awesome feature that have been added this year. And the weight are open-source as well.

Data 100

Data Cloud Storage Datasets Python 100

Marc Lamberti

NOVEMBER 2, 2021

Airflow Timetable. This new concept introduced in Airflow 2.2 I can tell you, the Airflow timetables give you the power and it’s truly a game changing feature. By the way, if you want to master Apache Airflow, take a look at my courses here 🤓 How Airflow DAGs are scheduled? the Data Intervals.

Data Pipeline 130

Data Pipeline Coding Process IT 130

Christophe Blefari

DECEMBER 9, 2022

Paris Airflow Meetup 🧑‍🔧 On Tuesday I organised the 4th Paris Apache Airflow Meetup. They shared with us the good practices they implemented to scale their Airflow development. As a figure at leboncoin 7 teams are using Airflow to operate more the 1000 DAGs.

SQL 130

SQL Data Data Science Metadata 130

Cloudera

SEPTEMBER 21, 2021

Many customers looking at modernizing their pipeline orchestration have turned to Apache Airflow, a flexible and scalable workflow manager for data engineers. Apache Airflow providers are a set of packages allowing services to define operators in their Directed Acyclic Graphs (DAGs) to access external systems.

Python 94

Python Cloud Data Workflow Accessible 94

Cloudera

DECEMBER 21, 2021

We also introduced Apache Airflow on Kubernetes as the next generation orchestration service. A flexible orchestration tool that enables easier automation, dependency management, and customization — like Apache Airflow — is needed to meet the evolving needs of organizations large and small. CDP Airflow Operators.

Data Engineering 116

Data Engineering Data Engineer Engineering Data Pipeline 116

Marc Lamberti

SEPTEMBER 21, 2021

In order to address a ton of use cases with your data pipelines you need to master one concept, the Airflow Trigger Rules. Airflow offers different mechanisms but the common one to react in case of failure are the callbacks. Another way that addresses those downsides is by using Airflow Trigger Rules!

Data Pipeline 130

Data Pipeline IT Management Data 130