What you'll learn

Design asset-centric DAGs using Airflow 3
Implement event-driven scheduling to trigger workflows based on external events rather than time-based schedules.
Create dynamic workflows using advanced mapping techniques to handle variable numbers of tasks efficiently.
Build AI workflows with latest Airflow updates using decorators and human in the loop operators

Course content

5 sections • 42 lectures • 4h 55m total length

Welcome!1:54
Who I am?1:42
Introduce Marco Marti, an Airflow expert and data engineer, highlighting his Udemy courses, YouTube channel data with Mark, and his role at Astronomer promoting Airflow at scale in production.
Course Goals3:38
Explore the course goals for Apache Airflow 3: advanced dag authoring, covering taskflow api foundations, dag scheduling, dynamic task mapping, and ai integration with airflow.
Setting Up Your Airflow Development Environment3:47
Set up a local Apache Airflow development environment using the astro CLI or a Docker Compose file. Install Docker, start Airflow, and access the UI at localhost:8080 with login airflow.

Why do you need creating DAGs with Taskflow4:10
Rewriting an old DAG with Taskflow8:30
Rewrite a two-task dag using the Taskflow API with dag and task decorators, showcasing data passing via returns and Xcom, and comparing simplicity to the old approach.
Access DAG information and more from your tasks10:34
Passing parameters to your DAGs6:44
Dynamically generate DAGs with different settings4:38
Mix decorators with operators9:53
The new bash operator (much better!)2:55
Switch from the traditional bash operator to the task bash decorator to run bash commands from Python functions, enabling complex logic and streamlined dependencies.
Group your tasks with TaskGroups14:39
Default arguments to DAGs and Taskgroups3:42
Apply default arguments in Airflow DAGs to set retries for all tasks, and use task group default_args with a pool named fetch crypto prices to serialize those tasks.
Running tasks on Airflow isn't unlimited7:09
Reuse tasks like a pro4:52
Correctly manage external resources within your DAGs21:01

The different ways of scheduling DAGs3:37
The basics but... important8:40
Cron or Timedelta?2:22
Is that streaming?9:46
Explore running Airflow DAGs continuously with the continuous preset, and learn how data intervals and max active runs differ from true streaming in Airflow 3.
Scheduling DAGs on Asset updates10:18
Learn to schedule dag runs in airflow 3 by modeling assets that trigger downstream dags on asset updates, enabling cross-dag dependencies without sensors or trigger operators.
Conditional Asset scheduling7:50
Use the or (pipe) and end operators with parentheses to schedule a downstream dag on multiple assets, enabling one or more assets to trigger the downstream dag.
Assets don't have to be static...8:10
Discover how to dynamically generate assets in Airflow 3 using asset aliases, replacing static assets with runtime-created files and downstream dags.
Access Asset metadata7:11
Learn to access asset metadata in downstream dags with inlets, outlets, and asset aliases, read the asset URI and content from events, and store extra information in the metadata object.
A different (better?) way to define Assets8:46
Explore a different approach in Airflow 3 by defining assets with the asset decorator, turning outputs into dependencies instead of tasks; note this method is not very mature.
Schedule your DAGs on EXTERNAL events4:53
Event driven scheduling with SQS13:12

Why static tasks is not enough5:34
The basics of dynamic task mapping9:18
Explore dynamic task mapping in Airflow 3, where the scheduler creates task copies at runtime from data inputs using expand and partial, enabling map and reduce across mapped outputs.
Fetch outputs from dynamic tasks2:59
Transform and filter dynamic task outputs12:00
Transform and filter dynamic task outputs with map in Apache Airflow 3. Learn to map over extracted file lists, apply a filter by extension, and skip zip files.
Mapping tasks on dictionaries6:34
Learn to map tasks on dictionaries with expand to give each file a distinct path by returning dictionaries from extract files and filtering by extension.
Combining dynamic task outputs6:09
discover how to combine multiple upstream outputs in Airflow using zip to create file-path tuples, by adjusting tasks to return lists and applying zip for downstream processing.
Concatenate dynamic task outputs6:45
Don't get lost with your dynamic tasks4:48
How to mix with traditional operators?8:40

Requirements

Working knowledge of Apache Airflow 2.x, including basic DAG authoring and execution
Proficiency in Python programming (intermediate level)
Experience with basic ETL/data pipeline concepts
Familiarity with command-line interfaces and basic Linux/Unix commands
Understanding of basic containerization concepts (Docker)
Access to a development environment capable of running Apache Airflow 3.x
Experience with git version control (basic)

Description

Airflow 3: Advanced DAG Authoring

Take your Apache Airflow skills to the next level. This course dives deep into the powerful features of Airflow 3 that separate beginners from production-ready data engineers.

You'll master the TaskFlow API—from the basics to advanced patterns like dynamic DAG generation, task groups, pools, and resource management. Learn how to build flexible, scalable pipelines using dynamic task mapping with advanced techniques like reduce, expand, and more. Explore modern scheduling strategies including assets, conditional scheduling, and event-driven pipelines with services like AWS SQS. Plus, discover how to integrate AI into your workflows using LLMs, the AI SDK, and human-in-the-loop approval patterns.

What you'll learn:

Write clean, Pythonic DAGs using the TaskFlow API
Generate DAGs dynamically and reuse tasks like a pro
Master dynamic task mapping for flexible, data-driven workflows
Schedule pipelines using assets, event-driven triggers, and continuous scheduling
Integrate AI and LLMs directly into your Airflow tasks
Implement human-in-the-loop workflows for AI approvals

Every video has its corresponding source code so it's easy for you to follow along.

Who this course is for: Data engineers and developers with foundational Airflow knowledge who want to write more efficient, maintainable, and production-grade DAGs using Airflow 3's latest features.

I hope you are ready for the course. Let's do it!

Marc Lamberti

Who this course is for:

This course is designed for data engineers who already work with Apache Airflow and want to elevate their DAG authoring skills to an advanced level.
Ideal participants have hands-on experience building basic data pipelines with Airflow 2.x and are looking to leverage Airflow powerful advanced features.

What you'll learn

Explore related topics

Course content

Introduction4 lectures • 11min

Create DAGs like Pro12 lectures • 1hr 39min

Schedule your DAGs like a Pro11 lectures • 1hr 25min

Create dynamic tasks like a Pro9 lectures • 1hr 3min

AI in Airflow like a Pro6 lectures • 38min

Requirements

Description

Who this course is for: