Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Software Development Tools No-Code Development
Business
Entrepreneurship Communication Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certifications Network & Security Hardware Operating Systems & Servers Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Paid Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement & Gardening Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition & Diet Yoga Mental Health Martial Arts & Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Learning Teacher Training Test Prep Other Teaching & Academics
Web Development JavaScript React CSS Angular Node.Js Typescript HTML5 PHP
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Amazon AWS Cisco CCNA CompTIA Security+ Microsoft AZ-900
Microsoft Power BI SQL Tableau Data Modeling Business Analysis Data Analysis Data Warehouse Business Intelligence Blockchain
Unity Unreal Engine Game Development Fundamentals C# 3D Game Development C++ Unreal Engine Blueprints 2D Game Development Mobile Game Development
Google Flutter iOS Development Android Development Swift React Native Dart (programming language) Kotlin SwiftUI Mobile App Development
Graphic Design Photoshop Adobe Illustrator Drawing Canva Digital Painting InDesign Design Theory Procreate Digital Illustration App
Life Coach Training Neuro-Linguistic Programming Personal Development Personal Transformation Life Purpose Mindfulness Sound Therapy Emotional Intelligence Coaching
Business Fundamentals Entrepreneurship Fundamentals Freelancing Business Strategy Online Business Startup Business Plan Blogging Amazon Kindle Direct Publishing (KDP)
Digital Marketing Social Media Marketing Marketing Strategy Internet Marketing Copywriting Email Marketing Google Analytics Startup Advertising Strategy

DevelopmentSoftware EngineeringApache Airflow

Apache Airflow on AWS EKS: The Hands-On Guide

How to set up Apache Airflow on AWS EKS with the Kubernetes Executor and be production-ready!
Rating: 4.5 out of 54.5 (241 ratings)
2,976 students
Created by Marc Lamberti
Last updated 8/2021
English
English [Auto]

What you'll learn

  • How to Set Up a Production Ready Architecture for Airflow on AWS EKS From A-Z
  • How to deploy DAGs from Git (public and private)
  • How to Create CI/CD Pipelines with AWS CodePipeline Deploy DAGs
  • How to Share DAGs and Store Logs with AWS EFS
  • How to Enable Remote Logging with AWS S3 in EKS
  • How to Test your DAGs in CI/CD pipelines
  • How to Store Sensitive Data in AWS Secret Manager

Requirements

  • A good knowledge of Apache Airflow
  • A intermediate knowledge of AWS
  • A intermediate knowledge of Docker and Kubernetes
  • AWS SERVICES USED ARE NOT FREE TIER ELIGIBLE

Description

Struggling to set up Airflow on AWS EKS?

You are at the right place!

With more than 15,000 students, I got many feedbacks about how difficult it is to configure Airflow on AWS with the official Helm chart.

Guess what? You are about to learn everything you need to set up a production-ready architecture for Apache Airflow on AWS EKS

This course is designed to guide you through the different steps of creating a real world architecture:

  1. Configuring the EKS cluster following best practices

  2. Deploying automatically changes with GitOps

  3. Using Helm to configure and set up Airflow on Kubernetes

  4. Configuring the official Helm chart of Airflow to use the Kubernetes Executor and many different features

  5. Deploying DAGs in Airflow with Git-Sync and AWS EFS

  6. Deploying DAGs/Airflow through CI/CD pipelines with AWS CodePipeline

  7. Testing your DAGs automatically

  8. Securing your credentials and sensitive data in a Secret Backend

  9. Enabling remote logging with AWS S3

  10. Creating 3 different environments dev/staging and prod

  11. Making the production environment scalable and highly available

  12. and more!

WARNING:

  • The course is not meant to learn the basic of Airflow, you must be already familiar with it.

  • If you already know Kubernetes/Docker/AWS your learning will be easier, but no worries I explain everything you need.

  • YOU WON'T learn how to interact with AWS in your DAGs. This course is about designing an architecture not about DAGs.

  • The course is NOT free-tier eligible as we are going to use many AWS services and set up a real world architecture.

Who this course is for:

  • Data engineers
  • Software Engineers
  • DevOps

Instructor

Marc Lamberti
Apache Airflow Expert, Big Data Engineer
Marc Lamberti
  • 4.5 Instructor Rating
  • 9,095 Reviews
  • 45,342 Students
  • 4 Courses

Hi there,

My name is Marc Lamberti, I'm 27 years old and I'm very happy to arouse your curiosity! I'm currently working as Big Data Engineer in full-time for the biggest online bank in France, dealing with more than 1 500 000 clients. For more than 3 years now, I created different ETLs in order to address the problems that a bank encounters everyday such as, a platform to monitor the information system in real time to detect anomalies and reduce the number of client's calls, a tool detecting  in real time any suspicious transaction or potential fraudster, an ETL to valorize massive amount of data into Cassandra and so on.

The biggest issue when you are a Big Data Engineer is to deal with the growing number of available open source tools. You have to know how to use them, when to use them and how they connect to each other in order to build robust, secure and performing systems solving your underlying business needs.

I strongly believe that the best way to learn and understand a new skill is by taking a hands-on approach with just enough theory to explain the concepts and a big dose of practice to be ready in a production environment. That's why in each of my courses you will always find practical examples associated with theoric explanations.

Have a great learning time!

Top companies choose Udemy Business to build in-demand career skills.
NasdaqVolkswagenBoxNetAppEventbrite
  • Udemy Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Investors
  • Terms
  • Privacy policy
  • Sitemap
  • Accessibility statement
Udemy
© 2022 Udemy, Inc.