Apache Airflow 3: Advanced DAG Authoring

Use the Taskflow API to define dags and tasks with decorators, pass data between tasks, reduce boilerplate, and create cleaner, easier to read dag files.

Rewrite a two-task dag using the Taskflow API with dag and task decorators, showcasing data passing via returns and Xcom, and comparing simplicity to the old approach.

Learn to optimize Airflow communication by returning dictionaries to store multiple values, splitting them into per-key xcoms with multiple outputs, and access dag context via Taskflow or get current context.

Pass parameters to Airflow DAGs using params dictionary and the param class. Explore types such as integer, date, boolean, and enum, and see how the Dag decorator integrates with Taskflow.

Learn how to dynamically generate multiple Airflow dags with different parameter values using the dag decorator, a configs loop, and the dag factory pattern.

Mix Airflow task flow decorators and traditional operators in a dag to fetch bitcoin price, share data via xcom and dot output, and manage dependencies with jinja templating.

Switch from the traditional bash operator to the task bash decorator to run bash commands from Python functions, enabling complex logic and streamlined dependencies.

Learn how to organize complex dags with task groups, apply group-level default arguments, map dynamically, share data between tasks and groups, and reuse task groups as modules in Airflow.

Apply default arguments in Airflow DAGs to set retries for all tasks, and use task group default_args with a pool named fetch crypto prices to serialize those tasks.

Control parallelism in Airflow with pools to limit task slots, set priority weights, and run tasks like fetch Bitcoin prices sequentially while others run in parallel.

Refactor Airflow tasks into a single reusable currency fetch with a parameter, override task IDs for Bitcoin, Ethereum, and Solana, and build a shared library for maintainable DAGs.

Discover how to manage resources in Airflow DAGs with setup and teardown tasks, ensuring resources like EMR clusters are created before execution and reliably released after, even on failures.

Explore time-based, data-aware, and event-driven scheduling in Airflow 3 to automate and trigger DAGs from cron expression, timedelta objects, data updates, or external events.

Explore how Airflow 3.0 schedules data pipelines using new data interval and logical date concepts, including cron data intervals and delta behavior, and how this differs from execution date logic.

Compare cron expressions and Timedelta objects to schedule Airflow DAGs; learn when to use each for exact times versus relative intervals, with daily midnight and every three days examples.

Explore running Airflow DAGs continuously with the continuous preset, and learn how data intervals and max active runs differ from true streaming in Airflow 3.

Learn to schedule dag runs in airflow 3 by modeling assets that trigger downstream dags on asset updates, enabling cross-dag dependencies without sensors or trigger operators.

Use the or (pipe) and end operators with parentheses to schedule a downstream dag on multiple assets, enabling one or more assets to trigger the downstream dag.

Discover how to dynamically generate assets in Airflow 3 using asset aliases, replacing static assets with runtime-created files and downstream dags.

Learn to access asset metadata in downstream dags with inlets, outlets, and asset aliases, read the asset URI and content from events, and store extra information in the metadata object.

Explore a different approach in Airflow 3 by defining assets with the asset decorator, turning outputs into dependencies instead of tasks; note this method is not very mature.

Learn how Airflow 3 enables event driven scheduling by triggering pipelines from external messages in Kafka, SQS, or Redis, via triggers, assets, and the asset watcher.

Build an event-driven data pipeline with AWS SQS to trigger Airflow DAGs, including queue creation, IAM credentials, and configuring the Airflow AWS providers and connection.

Learn how dynamic task mapping generates tasks at runtime to process each file or table independently. Avoids reprocessing, scales with workload, and adapts to file processing, table migrations, and notifications.

Explore dynamic task mapping in Airflow 3, where the scheduler creates task copies at runtime from data inputs using expand and partial, enabling map and reduce across mapped outputs.

Fetch outputs from dynamically mapped tasks by using partial to fix a constant ten and map each value, then collect and sum the results for downstream use.

Transform and filter dynamic task outputs with map in Apache Airflow 3. Learn to map over extracted file lists, apply a filter by extension, and skip zip files.

Learn to map tasks on dictionaries with expand to give each file a distinct path by returning dictionaries from extract files and filtering by extension.

discover how to combine multiple upstream outputs in Airflow using zip to create file-path tuples, by adjusting tasks to return lists and applying zip for downstream processing.

Learn to concatenate outputs from multiple upstream tasks in Apache Airflow 3 using concat to form a single list, and distinguish it from zip-based tuples.

Learn to assign meaningful map index values for dynamically generated Airflow tasks by fetching the current context and templating the file path, improving task traceability in the UI.

Discover dynamic task mapping with traditional operators in Airflow by using the Python operator, partial for static parameters, and explicit dependencies via output and XCom to map results.

Explore how to use AI in Airflow with two convenient approaches: the AI SDK for calling LLMs within DAGs using decorators, and human in the loop operators for human approvals.

Build an ai-driven data pipeline in Airflow using the Airflow ai sdk to generate recipes from ingredient lists, then display them and guide the language model with prompts.

Learn to let AI choose the next task in an airflow dag using the LM branch decorator, classify a recipe by difficulty, and route to beginner, intermediate, or advanced tasks.

Explore the Airflow AI SDK and its decorators, and review the repository and DAGs folder for dag examples that illustrate the decorators you can use.

Explore the new human in the loop operators in Airflow 3.1, including the base and entry variants, which pause workflows for human review and form-based input to guard AI outputs.

Explore how the approval operator, a specialized human-in-the-loop tool in Airflow, validates AI-generated results before downstream tasks proceed, with approve or reject actions and deferrable operators.