Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
A.B.C.D Apache Airflow 2.x on AWS - EKS | Minikube | Helm
Highest Rated
Rating: 5.0 out of 5(14 ratings)
275 students

A.B.C.D Apache Airflow 2.x on AWS - EKS | Minikube | Helm

Any Body Can Deploy Airflow. Be it Local Development or Production Deployment on AWS - EKS. It's as simple as it can be.
Highest Rated
Created byFaizan Qazi
Last updated 8/2023
English

What you'll learn

  • Create your own Helm Chart
  • Deploy Airflow on AWS EKS
  • Create a Scalable Cluster
  • Create Pipeline for Deployment
  • Configure airflow using templates

Course content

9 sections • 20 lectures • 3h 14m total length

  • Welcome3:29

    Explore deploying a highly scalable Apache Airflow 2.x on AWS with EKS, and test locally with Minikube, using Helm for production-grade configurations and CeleryKubernetes executor.

  • Introduction6:20

    Please share your Github account id in a Message to get access to the Project Repo.

  • Project Repo0:02
  • Setting up our Environment9:29

    Fork and clone the airflow project, set up Visual Studio Code with WSL, install Minikube, kubectl, and Helm, then create a virtual environment and templated config files.

Requirements

  • Little Knowledge around Shell/ Python Scripting
  • Basic Knowledge about Kubernetes
  • Basic Knowledge about Airflow Components

Description

Note: As of 2026 Airflow 2.x has reached End of Life and no longer would be deployed in Production. This course can be used as a learning for deploying Helm Applications in a CI/CD manner.
In this course, you will learn how to:


  • Create a Script to seamlessly deploy Airflow for Local Development

  • Create your own Helm Chart

  • Template Scripts and YAML files

  • Configure Airflow using Helm

  • Create a Scalable EKS Cluster using eksctl

  • Deploy an ALB Ingress Controller for Load Balancing and accessing the Airflow UI

  • Mounting EFS for Persisting Kubernetes Executor Logs

  • Creating a Pipeline to Deploy Airflow using AWS Code Pipeline


Prerequisites:


  • Basic Knowledge about Airflow Components

  • Basic Knowledge about Kubernetes

  • Basic Knowledge about AWS

  • Familiar with working on an IDE


Dependencies: Linux OS, Windows, AWS Account (For Non-Local Deployment )


This course is lined up with the Production Guide of Apache Airflow to deploy a Highly Scalable Airflow on EKS and also follows official Documentation of AWS while deploying Services making sure you always stay up to date and acquire more detailed information whenever you want to. 


Who this course is for:


  • If you are a DevOps Engineer and want to know the technical dependencies for deploying Airflow such as using EFS for using a Kubernetes Executor.

  • If you are a Data Engineer and want to use Airflow for Development but don't want to spend a huge amount of time learning how to configure it.

  • If you are a Full Stack Engineer and want to learn about various frameworks revolving around Airflow such as Helm, AWS EKS.

  • If you want to focus on development and get rid of all the frustration coming from trying to set up Airflow with all the core components.

Who this course isn't for: If you want to know what Airflow is or learn how to create DAGs or pipelines.


Note: This course includes using AWS resources such as EKS which is not free tier eligible.

Who this course is for:

  • Airflow beginners who wants to deploy it in a few commands
  • Devops, Platform Engineers
  • Data Engineers looking to leverage scalability of Airflow

Report abuse