Udemy
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Apache Airflow 3: Dynamic Task Mapping & Resilience
New
100 students

Apache Airflow 3: Dynamic Task Mapping & Resilience

Master Apache Airflow 3 dynamic task mapping, deferrable operators, and fault-tolerant DAGs for enterprise pipelines.
Created byACHRAF ER-RAYA
Last updated 10/2026
English

What you'll learn

  • Build scalable, dynamic workflows in Apache Airflow 3 using Dynamic Task Mapping to handle runtime data payloads effortlessly.
  • Eliminate worker resource bottlenecks and lower cloud infrastructure costs using async Deferrable Operators and Triggers.
  • Implement enterprise-grade pipeline resilience with exponential retries, failure callbacks, circuit breakers, and dead letter queues.
  • Architect self-healing, event-driven Airflow 3 DAGs that automatically recover from network outages and downstream failures.

Course content

5 sections • 8 lectures • 1h 13m total length

  • Master Dynamic Pipeline Generation to Handle Scaling Data Workloads Seamlessly13:46
  • Master Dynamic Pipeline Generation to Handle Scaling Data (PART2)4:29

Requirements

  • Basic understanding of Python programming and foundational data engineering concepts (like DAGs and basic SQL). No prior experience with Airflow 3 is required.

Description

This course contains the use of artificial intelligence.

Stop building brittle, static data pipelines that break under changing workloads and waste expensive cluster resources.

In this hands-on, practical course, you will master the next-generation features of Apache Airflow 3.0 to construct fully dynamic, event-driven, and resilient data pipelines built for enterprise scale. You will learn how to scale tasks dynamically at runtime, leverage deferrable operators to reduce infrastructure costs, and build automated self-healing mechanisms into every DAG you deploy.

Who this course is for:

  • Data Engineers looking to upgrade their skills to Apache Airflow 3.0

  • Backend Developers and DevOps Engineers managing data orchestrations

  • Data Architects designing enterprise-grade, resilient pipeline infrastructure

  • Analytics Engineers wanting to automate scaling data workloads

What you will learn:

  • Build dynamic workflows with Airflow 3 dynamic task mapping

  • Eliminate worker bottlenecks using async deferrable operators and triggers

  • Design self-healing pipelines with exponential backoffs and callbacks

  • Architect high-concurrency event-driven DAGs in Apache Airflow 3.0

  • Optimize cloud infrastructure costs by freeing up idle worker slots

  • Implement circuit breakers, dead letter queues, and SLA monitoring

  • Handle unpredictably sized data payloads smoothly at runtime

  • Aggregate mapped task outputs efficiently across complex workflows

Requirements:

  • Fundamental knowledge of Python programming

  • Basic understanding of data engineering concepts and SQL

  • Familiarity with basic Apache Airflow concepts (DAGs, tasks, operators)

Final Project Description: For your capstone project, you will build an end-to-end, market-ready e-commerce data pipeline in Apache Airflow 3. Your pipeline will dynamically discover multi-region data stores, wait asynchronously for third-party file uploads using deferrable triggers, process store data in parallel, and automatically handle simulated network failures using advanced retry strategies and failure callbacks.

Who this course is for:

  • Data Engineers, DevOps Specialists, and Analytics Engineers looking to upgrade their skills to Apache Airflow 3 and build high-concurrency, resilient pipelines.

Report abuse