Apache Airflow 3 - The Complete Bootcamp [With Projects]

Learn how a task instance represents a running task within a DAG, with each task as a single unit of work, enabling easy debugging and parallel execution.

Understand the operator as the task type in Airflow, using Python, Bash, or PySpark templates, and see how DAG, task instance, and operator shape workflows.

Discover how queuing works in Airflow by distinguishing the execution queue from the actual queue, and learn to prioritize thousands of DAGs using a priority queue approach.

Learn to build and understand a modern Apache Airflow 3 architecture, from DAG creation to API-driven storage in Postgres and a multi-stage execution pipeline with scheduler, Redis queue, and workers.

Create a project folder, set up a python virtual environment, and configure the Airflow docker compose file with postgres port mapping and load_examples false; then run docker compose up -d.

Learn to create your first Apache Airflow 3 DAG using the decorator approach, install Airflow in a virtual environment, and define tasks and dependencies in the DAGs folder.

Access a Docker container terminal with docker exec -it, open a shell, and navigate with ls and cd to explore dags, bind mounting, and container IDs via docker ps.

Set up airflow with docker compose, build a dag using the bash operator, compare traditional vs modern decorator approaches, and discuss the python operator.

Pull xcom data in airflow using ti.xcom_pull with the fetch data key from the fetch data task, then process and print the results.

kwargs are keyword arguments that pass task-related data in airflow, carrying metadata like logical date, execution date, previous ds, and next ds with the task instance as the child.

Learn to reference runtime variables in bash tasks with double curly braces, avoid python variables in bash, and use f-strings to inject python variables when needed.

Explore delta trigger, an event-based trigger or special schedule for DAGs when cron cannot handle frequency-based scheduling, with examples like every 10 days.

Discover Airflow special events triggers to run DAGs only on holidays and organizational events by storing dates in a list (events timetable) and applying them to the DAG.

Learn how to perform incremental loading in Airflow through lift and shift followed by daily initial load, using interval-based scheduling with from and to dates and updated_at timestamps.

Design parallel branched dags in Airflow by creating bash and fetch tasks that run in parallel, pass data via parameters and XCOM, and manage dependencies.

Learn to create task groups in Airflow 3 to organize related tasks, define dependencies within the group, and use collapsible boxes to simplify large DAGs for easier monitoring.

Set up the Airflow project codebase by organizing folders, copying and editing docker-compose.yaml, and preparing logs, DAGs, config, and plugins, with careful YAML indentation.

Add an api service to docker compose, building the app from the current folder under api, and map port 8000 to 8000 to run the fast api alongside postgres.

Learn to deploy a FastAPI application with docker compose, fetch data via Python requests, and understand container networking and internal dns for application programming interface calls across containers.

Explore the SQL operator in Airflow, a tool that runs SQL commands directly and requires a SQL database with a connection ID to execute queries.

Learn how hooks in Airflow connect to external systems, using the postgres hook to run sql and copy data from a csv file into a new table via connection ID.

Explore an Airflow ETL pipeline with a DAG that runs extract, transform, and load steps, creates tables, and runs parallel branches for pandas and SQL approaches.