Apache Airflow 2.0 using Docker, Docker Swarm

Explore how Docker creates lightweight, isolated containers to deploy and run applications, manage images on Docker Hub, and scale with multi-container setups using Docker Compose.

Acquire an AWS EC2 instance, install Docker and Docker Compose on CentOS 7 or 8 using the convenience script, start Docker, and verify installations with Docker and Docker Compose commands.

Learn to run MySQL 8 in Docker with Docker run, including container naming, port mapping, and volume mounts for persistent data access from the host.

Execute core docker commands to run a mysql container, pull images, view real-time logs, exec into the container, and verify the airflow database.

Explore Airflow 2.0 architecture, including the web server, scheduler, and metadata store (MySQL or Postgres), with Redis or RabbitMQ brokers, and how salary executor runs DAG tasks.

Install MySQL and RabbitMQ with Docker Compose version 3, defining services, images, environment variables, ports, volumes, and health checks for a single VM demo.

Configure the Celery master for Airflow 2.0 using Docker, setting up a web server and scheduler with common Airflow config, Fernet key, and MySQL and RabbitMQ connections.

Configure celery workers in a dockerized airflow setup, review bridge and host networks, run celery worker with docker-compose, monitor tasks via RabbitMQ, and adjust logs and tz for local time.

Explore the Airflow Flower service for celery workers, revealing which DAG runs on which worker and enabling load monitoring via docker compose on the master node.

Airflow 2.x supports multiple schedulers; install a second scheduler on a separate machine, run with a dedicated scheduler.yml, and verify high availability as load distributes with zero downtime.

Configure Google SMTP email alerts in Airflow with Jinja templates and per-DAG priority, restart salary workers, and verify failure notifications are sent on DAG failures.

Discover how Docker Swarm creates and manages a multi-node cluster, schedules services with replicas or global mode, and handles scaling, rolling updates, and failover across swarm managers and workers.

Design a highly available Airflow 2.0 architecture with Docker Swarm, detailing master web server and scheduler, distributed workers, salary workers, and MySQL and RabbitMQ services.

Learn how an overlay network in Docker Swarm enables containers on multiple hosts to communicate via a private, shared network with overlay IPs.

Install MySQL and RabbitMQ with Docker Swarm, create a swarm cluster and overlay network, translate compose to swarm services, configure mounts, ports, and env vars, and verify with logs.

Install the Airflow web server on Docker Swarm by converting the web server compose to a Docker service, using Swarm images, environment variables, volumes, and ports; verify logs.

Install the Airflow scheduler with Docker Swarm in global mode across master nodes, monitor scheduled tasks via the web server and the Revit MQ console, then prepare the salary workers.

install the Airflow Flower component with docker compose, convert to a docker service, and access the UI on the manager node at IP:5555; Flower requires RabbitMQ and a MySQL connection.

Deploy Apache Airflow 2.0 celery workers with Docker Swarm using a global mode service, host-network port mappings, and bind volumes with log permissions, then monitor tasks via the Flower UI.