Apache Airflow: The Operators Guide

Explore how Airflow operators empower reliable data pipelines by mastering operator capabilities, triggering behaviors, dependencies, and parallel task execution, and learn to handle failure using email.

Learn how airflow operators use execute as workhorse, with pre_execute and post_execute for setup and cleanup, on_kill for termination, and run provides context dictionary via task instance, dag, and xcoms.

Explore setting a unique, meaningful task id in a dag, limited to 250 characters and alphanumeric, dash, dot, and underscore. Discover four limitations, including no dynamic task id creation.

Version dag ids in airflow to preserve run history when you add or remove tasks, preventing missing logs and ui confusion by adopting a versioned dag approach.

Learn how the owner argument assigns task ownership in Airflow, not the dag, and how per-task ownership with default args shapes ui visibility, audit logs, and separation from execution user.

Learn how to make tasks resilient in Apache Airflow by using retries, retry_delay, and retry_exponential_backoff. The guide shows configuring defaults like default_task_retries, simulating failures, and observing retries in the UI.

Configure Airflow to send email alerts for failed or retried tasks using smtp.gmail.com, and set email on retry and on failure, then customize the subject and content templates.

Explore depends on past and wait for downstream to coordinate dag run task execution across runs. Learn about catch up and dag run timeout to avoid deadlocks.

Learn how depends on past and wait for downstream govern task execution across diagrams, preventing race conditions by requiring the current task and its direct downstream tasks to complete.

Learn how Airflow pools limit concurrency to manage resources like database connections and CPU usage. Assign tasks to pools so resource-heavy tasks run sequentially while others run in parallel.

Learn how to set task execution order in Airflow using priority weight and weight rule, while respecting dependencies and pool constraints with downstream, upstream, and absolute strategies.

Master Apache Airflow trigger rules to control task execution by direct upstream tasks. Learn the nine rules, including all underscore success and one underscore failed.

Define task expectations in Airflow using SLAs with time deltas. Configure email or callback alerts and learn SLA pitfalls, including their execution date basis and limitations.

Define a timeout to prevent deadlock in Airflow by setting execution_timeout with a time delta, so a long-running task fails; remember SLA warns, while timeout enforces failure.

Learn how to react to airflow task events using four callbacks—execute, retry, failure, and success callbacks—access the task context, and trigger alerts or actions based on task outcomes.

Learn how to limit concurrency in Airflow using pools and a task_concurrency parameter (task underscore concurrency) to constrain a task across diagrams, with examples like extract_A and extract_B during backfilling.

Configure the Kubernetes executor per task with executor_config to tailor pod resources like cpus and memory, achieving resource optimization and isolation; use pod_override or pod_template_file to define specs.

Learn how airflow providers decouple from the core and enable installing providers like AWS and great expectations. Find and pin provider versions via docs and the astronomer registry.

Explore the Python operator in Airflow, learning to execute a Python callable, pass arguments, use Jinja templating, and access task context for XComs and runtime data.

Master the TaskFlow API by decorating a Python function with @task to automatically create a Python operator, replacing the traditional approach and enabling context access via get_current_context.

Learn how the BashOperator executes bash commands in Apache Airflow, customize its environment, template runtime values, control XCom push, and manage exit codes including skip behavior.

Learn how to identify templated fields for an operator using airflow documentation, and note that bash_command and env are templated in the bash operator via template_fields.

Explore how to conditionally run tasks in a dag using branch Python, branch SQL, branch datetime, and branch day-of-week operators to handle trading days, database values, and time-frame triggers.

Explore how the branch python operator routes execution based on a python condition, returning next task ids such as accurate or inaccurate, and how trigger rules control downstream execution.

Schedule dags using the branch python operator to skip holidays. Build a yaml holidays file, parse it, and branch on the execution date to run only trading days.

Learn to use the branch sql operator to route tasks based on boolean sql result from a partners table, with postgresql, create and insert data, and define follow task IDs.

Learn to use the BranchDateTimeOperator in Apache Airflow to branch tasks based on a 10 a.m. to 11 a.m. window, using target lower and upper and force for execution-date comparison.

Apply the Branch Day of Week Operator to run daily tasks A and B while C runs only on a chosen day, with a single dag schedule and execution-date backfilling.

Learn how to create DAG dependencies in Airflow by linking partner-specific data DAGs to a common processing DAG, with methods and pitfalls explained.

Explore the sub dag operator for grouping tasks inside a parent dag, using a prefixed sub dag id and shared default arguments. Learn its drawbacks and conditions to avoid deadlock.

Learn how the sub dag operator can cause deadlocks when subdags run concurrently, and prevent them with mode to reschedule, timeout, and a shared pool.

Learn how to replace sub dag with task groups in airflow 2.0, and how to create, nest, and use task groups to organize and visualize tasks in the UI.

Explore creating dag dependencies with the TriggerDagRunOperator, triggering target dags from a parent dag and passing execution dates, with options like wait for completion and reset dag run.

Learn how to use Airflow's ExternalTaskSensor to wait for tasks in other dags before proceeding. Configure external_dag_id and external_task_id, and manage execution dates.

Learn to run a single dag with two schedules using the short circuit operator, so a, b, and c execute daily while d runs only on monday.