Apache Airflow Bootcamp: Hands-On Workflow Automation

Learn to install and configure Windows Subsystem for Linux (WSL) on Windows, enable virtual machine platform and WSL features, set WSL 2, install Ubuntu 20.04 LTS, and verify with commands.

Install PostgreSQL and PgAdmin on Ubuntu, configure localhost connections, restart the service, then register a localhost server in PgAdmin using the postgres user and password.

Install WinSCP and connect to your WSL Ubuntu distro over SSH. Reinstall and configure OpenSSH server, enable password authentication, and use WinSCP to access your Linux home directory.

Install Apache Airflow on WSL with Ubuntu 20.04, set up Python 3.10 and a virtual environment, initialize the database, create an admin user, and run the web server and scheduler.

Explore the airflow.cfg configuration, covering dag folders, executor options, parallelism, default retries, SQLAlchemy connection, and remote logging.

Explore the Apache Airflow Dag view, the central hub for managing DAGs, showing Dag IDs, owners, runs, schedules, last run, recent tasks, and UI views like graph and code.

Explore the grid view in Apache Airflow, a color-coded table that shows task status across diagrams, with hover details and filters to monitor, identify issues, and optimize workflows.

Explore the graph view of a dag in Apache Airflow, showing task nodes and dependencies with color-coded statuses and controls for zoom and fit to screen.

Visualize DAG executions on Airflow's calendar view, using a yearly calendar where each day shows the DAG runs and colored indicators reflect status; click a day for details.

Monitor task duration over time in the task duration view, a line graph in Apache Airflow, with hover details and date range filters for a dag.

Explore the code view in the Airflow UI to view and verify the dag's Python code, with line numbers and syntax highlighting for debugging and maintainability.

Explore Apache Airflow's variable view to manage key-value pairs for configuration and secrets across DAGs and tasks, with options to add, edit, delete, export, and import variables.

Explains the Gantt view in Apache Airflow, displaying tasks as horizontal bars on a timeline with start, end, duration, and status, plus tooltips with task details.

Explore Airflow DAGs, a directed acyclic graph that models tasks with directional dependencies and no cycles. Identify how nodes (tasks) and edges (dependencies) define upstream, downstream, or parallel work.

Create dag definition file in Airflow by importing modules, defining default arguments, creating a dag object, adding tasks with dependencies, placing the .py in dags, verify in web UI.

Learn Airflow DAG creation using instance of DAG class, with context manager, and Taskflow API. Create tasks, define dependencies with bitshift, and enable dynamic task generation and retries.

Trigger a dag run in airflow to execute all tasks, and monitor status, execution date, and duration in the UI graph view alongside task states and operators.

Configure Airflow by using default arguments in the dag to apply consistent task settings, including owner, depends_on_past, email notifications, retries, retry delay, exponential_backoff, and sla.

Master dag level parameters in Airflow to configure how dags behave, including dag id, start date, schedule interval, end date, catch up, default args, and on_success_callback and on_failure_callback.

Explore how the params dictionary in a dag defines runtime configuration accessible via Airflow context, and how dagrun overrides task, which overrides dag defaults, with UI or CLI overrides.

Define dag schedules in Apache Airflow with cron expressions, presets, and time delta, covering minute through day of week fields and examples like midnight daily and weekdays at 2:30 pm.

Learn how to pass runtime configuration from a parent dag to a child dag using TriggerDagRunOperator and the conf parameter, with a sample setup and parameter propagation.

Explore how the depends_on_past parameter in Apache Airflow enforces that a task's current run depends on the previous run's success, with default false behavior.

Shows how wait_for_downstream in Airflow controls a task’s success by downstream completion in past dag runs, with default false, and demonstrates a simple dag example.

the catchup parameter in Apache Airflow controls backfilling of dag runs on deployment or resume, with true by default and false means processing starts from the start date to now.

Backfill manually runs historical dag runs for a defined period to reprocess data or recover from outages. Use airflow dags backfill with start date, end date, and dag_id.

Master Airflow cli commands to trigger dags, pause and unpause, list tasks, and fetch config values, connections, variables, and import errors.

Learn how tasks function as the fundamental units of work in Airflow, define steps in a dag, and explore operators, sensors, and tasks decorated with @task for building workflows.

Explore task instances in Apache Airflow as dag runs, capturing execution date, state, and logs, and trace their life cycle from none to queued, running, and final success or failure.

Explore the Airflow task life cycle from none to scheduled, queued, running, and success. Understand transitions like up_for_retry, upstream_failed, up_for_reschedule, deferred, removed, and how they aid monitoring and debugging.

Learn how operators in Airflow define tasks in a dag, enabling modular, reusable, and flexible workflows, with action, transfer, and sensor operators such as BashOperator, PythonOperator, and FileSensor.

Learn how the bash operator in Airflow runs shell commands or scripts, including inline python, reading variables via jinja, copying files, and a simple linear dag.

Explore using the Postgres operator in Airflow to execute sql on PostgreSQL, create tables, and insert or delete data within a dag.

Explore how to use the PythonOperator in Airflow to execute Python callables, pass arguments with op_args and op_kwargs, access Airflow variables, and leverage the taskflow API with @task and get_current_context.

Learn to send Slack messages from an Airflow dag using the Slack webhook operator, including configuring a Slack workspace and app, and creating demo dag to post to channel.

Configure Airflow smtp settings and use the email operator to send dag task status alerts, including Gmail smtp and a 16-digit app password.

Explore how to use the SQLite operator in Airflow to execute SQL against a SQLite database, including creating tables, inserting data, querying records, and pushing results with XCom.

Explore sensors in Apache Airflow, a type of operator that waits for data availability, dependencies, or events, with file, time, external task, and SQL sensor examples.

Explore how the Apache Airflow File Sensor monitors a file or directory, using parameters like file path, fs_conn_id, poke interval, timeout, and mode to trigger downstream tasks.

Learn how to use the sql sensor in airflow to monitor a sql query and wait for a condition, such as records existing in a Postgres table, before downstream tasks.

Learn how to use the TimeDeltaSensor in Airflow to pause a dag for a defined duration, configured with delta, poke mode, poke_interval, and timeout controls.

Explore how the time sensor operator in Airflow uses poke mode to wait for the specified time of day and trigger the downstream task.

Explore BranchPythonOperator in Airflow, using a Python callable to return the next task_id for conditional downstream branching, with optional op_args and op_kwargs in a sample dag.

Explore BranchSQLOperator in airflow to route downstream tasks based on a sql query result, using follow_task_ids_if_true or if_false, with connection id, database override, and a practical dag example.

Learn how BranchDayofWeekOperator in Apache Airflow routes execution by day of week using the weekday parameter to conditionally run task_2 or task_3 via true/false follow_task_ids, in a simple dag.

Learn how the BranchDateTimeOperator in Apache Airflow branches a dag based on the current date time, using target_lower, target_upper, and date time conditions to run task_2 or task_3.

Learn how the short circuit operator in Apache Airflow conditionally executes downstream tasks via a Python callable, skipping when the condition is false, such as an empty table.