Mastering Apache Airflow! Deploy to Kubernetes in AWS

Airflow enables teams to programmatically author, schedule, and monitor workflows using DAGs, with reusable task logic and a powerful web interface for monitoring, logs, and metrics.

Explore Apache Airflow use cases, including ETL orchestration, cron job management, and data movement across services. Discover its support for ML pipelines and automation, plus streaming limitations.

Discover airflow architecture by examining the web server with gunicorn and flask, security features including authentication and a rest api, and the scheduler that triggers dag tasks on schedule.

Explore how the Airflow scheduler monitors DAGs, triggers runs by start date and schedule interval, uses an OrderedDict queue, and supports sequential, local, celery, and Kubernetes executors.

Master the sequential executor in Airflow, which runs one task at a time with no parallelism. Utilize it for debugging and testing, not for production.

Explore the local executor in Airflow, which runs multiple subprocesses to execute tasks concurrently on a single host. It scales vertically, but it lacks horizontal scalability and fault tolerance.

Learn how the Celery executor horizontally scales an Airflow cluster with a pool of workers and a broker, using RabbitMQ or Redis in production to delegate tasks.

The Kubernetes executor lets Airflow run each task as a Kubernetes pod, launched via the Kubernetes API and terminated after completion to achieve elastic scaling, parallelism, and resource optimization.

Explore the Apache Airflow web server, powered by Flask and Gunicorn, delivering a dashboard and enabling pause, unpause, trigger, view running DAGs, and restart of failed DAGs with scalable workers.

Explore the Apache Airflow web server's DAGs view, including the table of DAGs, on/off toggle, schedule interval (cron), owner, recent tasks and DAG runs, plus manual trigger and delete options.

Explore Apache Airflow's web server graph view to visualize dag dependencies and task states for a specific dag run, with configurable layout and filtering.

Explore Apache Airflow's tree view, a time-spanning DAG representation with color-coded operators and statuses, filtering by base date and runs, and viewing task details, logs, and metrics.

Explore the task duration view in Apache Airflow's web server to debug performance, analyze task instance durations and execution dates, and identify outliers and blocking tasks in a DAG.

Explore the Gantt view in Apache Airflow's web server to analyze task duration and overlap, identify bottlenecks, and debug performance across DAG runs.

View the code that generates a DAG in real time from GitHub to gain context, debug specific cases, and leverage the admin panel for deeper insights.

Explore Airflow configuration by examining sections like core, web server, scheduler, and kubernetes, and note that changes require restarting the web server or scheduler.

Explore Airflow's core configuration, including dags_folder, base_log_folder, remote logging options such as AWS S3, Google Cloud Storage, and Elastic Search, executors, sql_alchemy_conn, parallelism, dag_concurrency, max_active_runs_per_dag, load_examples, plugins_folder, and fernet_key.

Explore the scheduler configuration in Airflow, including scheduler_heartbeat_sec, dag_dir_list_interval, and catchup_by_default. Understand how catch_up and default false affect dag runs and where to set child_process_log_directory.

Configure web_server_host (default 0.0.0.0) and web_server_port, enable https with ssl_certificate and ssl_key, set access_logfile and error_logfile paths, adjust Gunicorn workers, enable rbac and page_size.

Explore the Kubernetes configuration section, covering worker_container properties, docker image repo and tag, delete_worker_pods, the dags_in_image and dags_volume_claim options, logs_volume_claim, env_from_configmap_ref, env_from_secret_ref, in_cluster = True, and kubernetes_labels.

Override airflow.cfg properties at runtime with environment variables following the AIRFLOW__CORE__EXECUTOR pattern to switch executors and prioritize environment values over the config file.

Learn the DAG concept and its core parameters—start_date, dag_id, and schedule_interval—and how cron-based timing governs execution. Explore max_active_runs, catchup, default_args, and how tasks are defined and sequenced within a DAG.

Explore Apache Airflow plugins to customize and extend your workflow, grouping features in a plugin, defining operators and hooks, and applying changes by restarting the webserver and scheduler.

Learn how Airflow operators define what work is executed in a DAG, with examples like Python, Postgres, Bash, and Hive operators, and how reuse and customization power your tasks.

Hooks are interfaces to external platforms and services such as S3, MySQL, Postgres, and HDFS, used by operators through Airflow's connection model to retrieve credentials from the metadata database.

Explore how Airflow sensors, derived from BaseSensorOperator, monitor long-running tasks with the poke function, waiting for conditions like S3KeySensor or SQLSensor results.

Explain XCom, a cross-communication mechanism for tasks in a DAG to exchange key, value, and timestamp data via xcom_push and xcom_pull, including automatic push on execute return.

Discover airflow variables as a global key-value store for configuration settings, accessible via the admin panel, definable in code with the Variable class, and supporting crud operations.

Verify your Python version, install and use a per-project virtual environment with virtualenv, and manage Airflow dependencies cleanly without global clutter.

Install and run Docker on Mac using Docker Desktop, establishing Docker as part of the developer tech stack and preparing to run PostgreSQL in the next lecture.

Set up PostgreSQL as a Docker container for an Apache Airflow deployment, creating an airflow database and admin user with password 1q2w3e, expose port 5416, and verify the connection.

Explore how to set up a local Kubernetes environment for Apache Airflow using minikube on a Mac, including installing VirtualBox, kubectl, and starting a single-node cluster.

Create a dedicated code folder in your home directory for the Airflow project, and set the AIRFLOW_HOME environment variable to that location to initialize the Airflow setup.

Create and activate a Python virtual environment, then install dependencies from requirements.txt for Apache Airflow, psycopg2 binary, werkzeug, flask-bcrypt, Kubernetes, statsd, crypro, and set locale variables.

Initialize the Airflow project with airflow initdb, review generated files like airflow.cfg and airflow.db, switch to PostgreSQL, and run the scheduler and web server with the local executor, port 8080.

Create your first dag with an operator, a sensor, and a plugin to grasp Airflow's core components; learn to define a simple element and extend it with resource-specific logic.

Create your first dag by adding a dags folder and my_first_dag.py. Import the dag class, configure dag_id, max_active_runs, schedule_interval, and default_args (owner, start_date, retries, retry_delay), and set catchup false.

Create your first Airflow plugin by adding a plugin file in the plugins directory. Define MyFirstPlugin that extends AirflowPlugin with a name and empty operators and hooks.

Extend BaseOperator to create your first operator, implement an execute function that logs a parameter, decorate with apply_defaults, and wire the operator into a DAG and plugin.

Learn to build a simple Airflow sensor by extending BaseSensorOperator, implement a poke function with a 30-second poke_interval, and attach it to a DAG to gate a downstream task.

Implement xcom in Airflow by pushing a minute value from a sensor to xcom and pulling it in an operator using task_instance.xcom_push and xcom_pull, then verify via DAG run logs.

Explore dynamic DAGs with Apache Airflow using the BranchPythonOperator to conditionally branch execution based on a Python callable, selecting one of multiple pipelines and skipping others.

Manage credentials for databases and services with Airflow connections using a connection_id, encrypting sensitive data via the crypto library and a fernet key. Rotate keys with airflow rotate_fernet_key for security.

Explore Apache Airflow logs, configure base_log_folder and remote logging with Amazon S3, Azure Blob Storage, and Google Cloud Storage, and customize log formats for easier debugging.

Discover how Apache Airflow's experimental API exposes web endpoints at api/experimental for creating and listing DAG runs, pausing and unpausing DAGs, with Kerberos or password authentication.

Master how to monitor Apache Airflow with a statsd–graphite stack by enabling metrics in Airflow, sending them over UDP to statsd, and visualizing them in graphite (with Grafana dashboards possible).

Build an Apache Airflow Docker image from a Dockerfile by defining a base image, Airflow home, dependencies, and an entrypoint to run webserver or scheduler.

Configure and run a multi-container Airflow setup with docker compose by defining docker-compose.yaml, building the Airflow image, and coordinating web server, scheduler, and postgres, using local.env and volumes.