Apache Airflow on AWS EKS: The Hands-On Guide

Define a monthly budget in AWS Budgets to monitor costs and avoid overspending, set a $30 cap, and enable email alerts for threshold breaches.

Create and manage IAM groups to control access, forming an admins group with administrator access and attaching policies, then assign users to the group to grant appropriate permissions.

Create a new iam user in the admins group to avoid the root account and enable both programmatic and console access, then securely store the access and secret keys.

Sign out from the root account and sign in as the admin user using your account ID. Change and confirm the password to complete the admin login.

DevOps blends software development and IT operations to shorten the delivery cycle through continuous delivery, integration, testing, and monitoring, with infrastructure as code and version control to standardize environments.

Discover how Kubernetes automates containerized app management via manifests defined in json or yaml, enabling rolling updates, canary and blue-green deployments, and service-based access.

Learn infrastructure as code and how to define resources with yaml or json files. Use CloudFormation or TerraForm to provision and manage infrastructure quickly and consistently with versioned changes.

Learn how GitOps uses git as the single source of truth to automate Kubernetes deployments on AWS EKS, enabling automated rollbacks, auditing, and standardized infrastructure.

Flux uses GitOps in Kubernetes with an operator to keep the cluster state in sync with Git, deploying updates automatically when changes are merged.

Set up a browser-based Cloud9 workstation on AWS EC2 with a 30-minute auto hibernation, name the environment workstation, and use Cloud9 to write, run, and debug airflow code.

Configure the workstation to prepare a Kubernetes cluster with ETL and Cube CTL, install Helme, and manage apps via a pre-configured repo, while setting up a Python environment and git.

Learn to configure Cloud9 with the admin account using the CLI by activating the virtual environment, upgrading the CLI, installing utilities, and exporting credentials and region variables.

Understand AWS network architecture by examining regions, availability zones, and how VPCs with public and private subnets enable isolated, high-availability deployments across multiple zones.

Explore how AWS EKS manages the Kubernetes control plane across multiple availability zones, automatically scaling and repairing nodes, while you configure worker nodes and VPC resources.

Create and configure a GitHub repository to enable GitOps with Flux for managing infrastructure and application deployments from repo files, and set up an SSH key for Cloud9 access.

Understand the EKS architecture with a GitHub driven pipeline using YAML files for namespaces, Flux to apply changes, and a two-VPC layout with managed node groups and auto scaling.

Create an eksctl cluster by editing the cluster YAML for airflow, set the region and three-node worker group, then install the AWS IAM authenticator and AWS CLI v2.

Install flux to synchronize the git repository with the x cluster, enabling automatic creation and updates of namespaces by committing files to git; bootstrap flux with GitHub and token setup.

Add a structured repository layout by creating three folders—workloads, releases, and namespaces—for Kubernetes resources inside the Flux system repository, clone via SSH, add dummy files, and commit and push.

Explore how namespaces partition a Kubernetes cluster to allocate resources by team—dev, staging, prod—enabling isolation, policies, and constraints for data scientists, data engineers, and analysts, with airflow on ECS.

Create dev, staging, and prod namespaces on AWS EKS with Flux and the Git repository. Push the namespace manifests; Flux provisions the namespaces and you verify with kubectl logs.

Learn Helm, the package manager for Kubernetes, to package, configure, and deploy applications as charts. Use Helm repositories, charts, releases, and values.yaml to customize Airflow deployments on Kubernetes.

Explore the official airflow helm chart, its templates and values, and how Helm renders Kubernetes resources for the web server, airflow UI, and postgres dependency.

Explore the Kubernetes executor in Apache Airflow, creating a pod per task for dynamic resource allocation, task-level CPU and memory settings, and fault tolerance, while noting overhead and learning curve.

Install the Amazon EBS CSI driver to enable dynamic volume provisioning for Airflow on Kubernetes, following helm-based steps to add the repository, install, and verify the driver.

Create and push a repository object so Flux can locate the Airflow chart, then git add, commit, and push the sources to deploy Airflow in Kubernetes.

Deploy airflow with Flux and Helm by creating a hand release object for airflow-dev in dev. Configure Helm chart values from values.html, set environment variables, and use the Kubernetes executor.

Learn how to synchronize Airflow dags across Kubernetes pods using git sync sidecar, S3, or a shared persistent volume, and package dags in Docker images for reliable, scalable workflows.

Fetch dags from a git repository using the git sync sidecar in Airflow on AWS EKS, with regular interval pulls and configurable options.

Configure the git sync sidecar in the airflow helm deployment to pull data pipelines from a git repository on master and synchronize every 60 seconds, verifying the parallel_dag appears.

Secure sensitive data with Kubernetes secrets, storing passwords and SSH keys without exposing them. Mount secrets as volumes or use them as environment variables, with volumes recommended for security.

Synchronize dags from a private git repository to an airflow instance on a cluster by configuring ssh keys, deploy keys, and git sync, then validate with kubectl.

Learn how AWS CodePipeline automates build, test, and deploy for code changes, integrating with Git, Docker images, and AWS ECR to safely deploy Airflow data pipelines.

Create a CodePipeline and ECR-driven CI/CD pipeline to deploy a dockerized Airflow with packaged dags; automate unit and integration tests and push new images to dev, staging, and prod.

Push an Airflow image to ECR via the CI/CD pipeline and deploy it to an EKS cluster using Helm. Verify pods, dags, and image tag, then sync with Flux.

Learn to implement unit testing in Airflow with pytest, covering dag validation, dag pipeline definitions, unit tests, integration tests, and end-to-end pipeline tests to safely push dags to production.

Learn practical unit testing of Airflow dags using validation and definition tests, fixtures, and pytest, with docker-based workflows to verify imports, defaults, and task dependencies.

The lecture shows building the ci/cd pipeline in dev with unit tests, including docker image creation, pytest execution, and codepipeline updates via cloudformation.

Execute integration tests by triggering a dag to verify tasks work together, monitoring run status in Airflow with Docker. Clean metadata and pause the dag for CI readiness.

Automatically build Airflow Docker images on each Git push, run integration tests in staging, and update Airflow via Helm in the ECS cluster, using CodePipeline and CloudFormation.

Run an automated cleanup script to remove permissions, roles, helm charts, pipelines, registries, and the EKS cluster; delete CloudFormation stacks, Cloud9 resources, and EFS storage to avoid ongoing billing.

Set up the aws eks cluster, code pipelines, and ecr by running the setup script, configuring values, and redeploying resources, then verify pipelines and pods in dev and staging.

Explore how Kubernetes services expose pods to external traffic using cluster IP, nodeport, and load balancer, with label-based routing to web server pods on port 8080.

Expose the Airflow web UI from a Kubernetes cluster with an AWS elastic load balancer on port 88, enabling internet access.

An ingress sits in front of Kubernetes services and routes traffic to the right app by path, enabling a single IP exposure for Airflow dev and staging.

Route dev and staging airflow UIs via a single IP using the AWS ALB ingress controller in EKS. Create NodePort services and ingress rules for path-based routing.

Expose the airflow ui using the aws alb ingress controller after removing the elb and syncing changes, then deploy the nodeport service and ingress in dev for airflow web server.

Showcases exposing the airflow staging environment via a nodeport service and ingress rule, updating the ingress controller to monitor all namespaces, and deploying changes through a helm-based CI/CD pipeline.

Enable https with ssl for the airflow ui exposed to the internet to prevent man-in-the-middle attacks; use public and private keys and certificates to encrypt and verify communications.

Create a domain for Airflow with AWS Route 53 and External DNS, deploy a service account with an IAM policy, and update ingress to route traffic.

Enable https for the airflow ui by creating a tls certificate and private key, storing them in a kubernetes secret, and updating the ingress to redirect http to https.

Export and commit the airflow ssl secret as a yaml file, remove unnecessary metadata, and automate its deployment with flux for repeatable secret management.

Upgrade the staging Airflow deployment by converting the dev release to staging, updating registries, namespace, and ingress, and streamline ci/cd to run integration tests with docker-compose and ssl secret.

Configure subdomains to access the Airflow UIs on AWS EKS via external DNS and ingress, linking dev and staging environments through custom domains and load balancer DNS.

Define role or cluster role permissions to control access in Kubernetes. Bind them to a service account via cluster role binding or role binding for a namespace.

Configure a persistent volume on AWS EFS to store Airflow task logs when using the Kubernetes executor, then mount it in the web server to enable log access.

Learn how to configure Apache Airflow on AWS to store task logs remotely in S3, including creating an S3 bucket, an IAM user, and remote logging settings for Airflow.

Learn to use a secret backend to store credentials securely, replacing hardcoded values, with secret manager handling HTTPS access, rotation, and versioning.

Learn how to configure Airflow on AWS to store and retrieve connections using AWS Secrets Manager, including creating an IAM user, secret backend setup, and testing with an S3 connection.

Export the secret object from AWS Secret Manager to a Flux-ready YAML file, remove unnecessary metadata, and commit and push to Git for automatic Flux deployment on cluster recreation.

Set up the production environment for airflow on Kubernetes, creating the prod namespace, production release, and https access with a certificate and secrets. Enable the production CI/CD pipeline with CodePipeline.

Identify single points of failure in production, such as web server and database, and learn to make them highly available, noting the scheduler isn’t highly available in this Airflow version.

Learn to make the Airflow web server highly available by deploying three web server pods across multiple nodes and availability zones, using kubectl, helm, and flux.

Switch from self-managed Postgres pods to AWS RDS for Airflow's metastore, leveraging a managed, scalable, highly available relational database that supports Postgres, MySQL, Oracle, and more.

Set up Airflow's metastore on AWS RDS for high availability with a multi-AZ Postgres, aligned to the same VPC as your EKS cluster. Replace the helm default Postgres during deployment.

Explore how dag serialization accelerates airflow by storing serialized dags as json in the metastore and loading them on demand, with update and fetch intervals tuning performance.

Enable dag serialization in airflow on aws eks by setting environment variables to serialize dags and expose code view, and configure the scheduler to serialize every 30 seconds.

Learn to deploy and manage Apache Airflow on AWS EKS with Helm, ECS, and GitOps, configure multi-environment pipelines, secret backends, remote logging, and DAG realization.