Apache Airflow on AWS EKS: The Hands-On Guide

Name: Apache Airflow on AWS EKS: The Hands-On Guide
Rating: 4.3 (422 reviews)

How to set up Apache Airflow on AWS EKS with the Kubernetes Executor and be production-ready!

Last updated 3/2023

English

What you'll learn

10 sections • 89 lectures • 7h 40m total length

[Practice] Creating the cloud9 environment for the workstation1:42
[Practice] Configuring the workstation3:51
[Practice] Configuring Cloud9 with the Admin account2:53
AZs, VPCs and Subnets in AWS2:47
What is AWS EKS?2:14
Explore how AWS EKS manages the Kubernetes control plane across multiple availability zones, automatically scaling and repairing nodes, while you configure worker nodes and VPC resources.
[Practice] Creating and configuring the Git repository for GitOps2:51
Create and configure a GitHub repository to enable GitOps with Flux for managing infrastructure and application deployments from repo files, and set up an SSH key for Cloud9 access.
Architecture Overview EKS3:15
[Practice] Creating the Cluster with eksctl6:50
[Practice] Installing and Configuring Flux4:22
Install flux to synchronize the git repository with the x cluster, enabling automatic creation and updates of namespaces by committing files to git; bootstrap flux with GitHub and token setup.
[Practice] Add a bit of structure1:41
Namespaces in Kubernetes3:25
[Practice] Creating dev, staging and prod namespaces4:11

What's Helm?2:49
Learn Helm, the package manager for Kubernetes, to package, configure, and deploy applications as charts. Use Helm repositories, charts, releases, and values.yaml to customize Airflow deployments on Kubernetes.
[Practice] Overview of the Airflow Helm chart4:45
Scaling with the Kubernetes Executor3:39
[Practice] Installing the EBS Driver1:32
Install the Amazon EBS CSI driver to enable dynamic volume provisioning for Airflow on Kubernetes, following helm-based steps to add the repository, install, and verify the driver.
[Practice] Setting up the HelmRepo1:50
[Practice] Deploying Airflow with Flux5:14
Troubleshooting deployments with Flux0:05
Synchronizing DAGs in Kubernetes4:26
[Practice] Fetching DAGs with Git-Sync1:58
Fetch dags from a git repository using the git sync sidecar in Airflow on AWS EKS, with regular interval pulls and configurable options.
[Practice] Running DAGs with Git-Sync4:05
Secrets in Kubernetes2:54
[Practice] Fetching DAGs with Git-Sync from a private repository6:39
Synchronize dags from a private git repository to an airflow instance on a cluster by configuring ssh keys, deploy keys, and git sync, then validate with kubectl.

Set up1:15
[Practice] Creating the production environment12:19
Set up the production environment for airflow on Kubernetes, creating the prod namespace, production release, and https access with a certificate and secrets. Enable the production CI/CD pipeline with CodePipeline.
Identifying single point of failures2:20
[Practice] Making the Airflow UI highly available9:07
AWS Relational Database Service2:05
[Practice] Airflow with AWS RDS11:38
DAG Serialization4:10
[Practice] Making the web server stateless with DAG Serialization6:56
Clean Up0:44
Congratulations!6:50

Struggling to set up Airflow on AWS EKS?

You are at the right place!

With more than 15,000 students, I got many feedbacks about how difficult it is to configure Airflow on AWS with the official Helm chart.

Guess what? You are about to learn everything you need to set up a production-ready architecture for Apache Airflow on AWS EKS

This course is designed to guide you through the different steps of creating a real world architecture:

Configuring the EKS cluster following best practices
Deploying automatically changes with GitOps
Using Helm to configure and set up Airflow on Kubernetes
Configuring the official Helm chart of Airflow to use the Kubernetes Executor and many different features
Deploying DAGs in Airflow with Git-Sync and AWS EFS
Deploying DAGs/Airflow through CI/CD pipelines with AWS CodePipeline
Testing your DAGs automatically
Securing your credentials and sensitive data in a Secret Backend
Enabling remote logging with AWS S3
Creating 3 different environments dev/staging and prod
Making the production environment scalable and highly available
and more!

WARNING:

The course is not meant to learn the basic of Airflow, you must be already familiar with it.
If you already know Kubernetes/Docker/AWS your learning will be easier, but no worries I explain everything you need.
YOU WON'T learn how to interact with AWS in your DAGs. This course is about designing an architecture not about DAGs.
The course is NOT free-tier eligible as we are going to use many AWS services and set up a real world architecture.