Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA Amazon AWS CompTIA Security+ Microsoft AZ-900
Graphic Design Photoshop Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Personal Development Personal Transformation Mindfulness Life Purpose Meditation CBT Emotional Intelligence
Web Development JavaScript React CSS Angular PHP Node.Js WordPress Vue JS
Google Flutter Android Development iOS Development React Native Swift Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Retargeting
Microsoft Power BI SQL Tableau Business Analysis Data Modeling Business Intelligence MySQL Data Analysis Blockchain
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Business Plan Startup Freelancing Online Business Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee
Development Software Engineering Apache Airflow

The Complete Hands-On Introduction to Apache Airflow

Learn to author, schedule and monitor data pipelines through practical examples using Apache Airflow
Bestseller
Rating: 4.6 out of 54.6 (2,705 ratings)
14,239 students
Created by Marc Lamberti
Last updated 3/2021
English
English [Auto], French [Auto], 
30-Day Money-Back Guarantee

What you'll learn

  • Create plugins to add functionalities to Apache Airflow.
  • Using Docker with Airflow and different executors
  • Master core functionalities such as DAGs, Operators, Tasks, Workflows, etc
  • Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.
  • The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.
  • Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc.
  • Install and configure Apache Airflow
  • Think, answer and implement solutions using Airflow to real data processing problems
Curated for the Udemy for Business collection

Course content

9 sections • 67 lectures • 5h 49m total length

  • Preview01:21
  • Preview03:50
  • Preview02:33
  • Development Environment
    02:34

  • Preview01:07
  • Why Airflow?
    01:58
  • Preview09:24
  • How Airflow works?
    06:02
  • [Practice] Installing Airflow 2.0
    08:43
  • [Practice] The CLI in Airflow 2.0
    07:57
  • [Practice] What you need to know about the UI
    08:40
  • Quiz Time!
    6 questions
  • Key Takeaways
    00:34

  • Introduction
    01:30
  • Preview01:05
  • [Practice] Time to code your first DAG
    02:45
  • [Practice] DAG Skeleton
    07:37
  • What is an Operator?
    02:20
  • [Practice] Creating Table
    09:35
  • [Practice] The secret weapon!
    01:45
  • [Practice] Is API available?
    06:02
  • [Practice] Extracting users
    04:58
  • [Practice] Processing users
    12:38
  • [Practice] Storing users
    05:54
  • [Practice] Order matters!
    03:51
  • [Practice] You data pipeline in action!
    03:56
  • Dag scheduling
    02:59
  • [Practice] Backfilling and catchup
    10:06
  • Quiz Time!
    7 questions

  • Introduction
    01:49
  • Preview07:49
  • [Practice] Start scaling with the Local Executor
    09:52
  • Scale to the infinity with the Celery Executor
    08:50
  • [Practice] Scaling Airflow with the Celery Executor in action!
    12:31
  • [Practice] Changing the executor
    00:34
  • Concurrency, the parameters you must know!
    14:43
  • [Practice] Concurrency in practice
    07:05
  • Wait a second!
    00:12
  • Quiz Time!
    5 questions

  • Introduction
    02:20
  • Adios repetitive patterns
    01:37
  • [Practice] Minimising DAGs with SubDAGs
    13:34
  • [Practice] Adios SubDAGs, Welcome TaskGroups!
    06:27
  • Sharing data between tasks with XComs
    03:22
  • [Practice] XComs in action!
    13:08
  • Choosing a specific path in your DAG
    02:12
  • [Practice] Executing a task according to a condition
    09:55
  • Trigger rules or how tasks get triggered
    05:59
  • [Practice] Changing the way your tasks are triggered
    11:48
  • Quiz Time!
    4 questions

  • Introduction
    02:05
  • Installing Elasticsearch
    01:06
  • How the plugin system works?
    05:01
  • [Practice] Creating a hook interacting with Elasticsearch
    13:27
  • [Practice] Creating the PostgresToElasticOperator
    16:28
  • Quiz Time!
    4 questions

  • Introduction
    01:35
  • Quick Reminder About Docker
    14:11
  • [Practice] Running Airflow on Docker with the Celery Executor
    09:52
  • [Practice] Running Airflow on Docker with the Local Executor
    03:09
  • Quiz Time!
    5 questions
  • Recap
    01:51

  • What to expect from Airflow 2.0?
    10:41

  • [BLOG POST] How to use the DockerOperator with Templating and Apache Spark
    00:31
  • [BLOG POST] Apache Airflow with Kubernetes Executor
    00:31
  • [BLOG POST] How to use templates and macros in Apache Airflow
    00:41
  • [BLOG POST] How to use timezones in Apache Airflow
    00:40
  • [BLOG POST] How to use the BashOperator
    00:13
  • [BLOG POST] Variables in Apache Airflow: The Guide
    00:14
  • [BLOG POST] Best Practices in Apache Airflow (part 1)
    00:23
  • [VIDEO] Running Apache Airflow on a multi-nodes Kubernetes cluster locally
    00:21
  • [BLOG POST] The PostgresOperator: All you need to know
    00:20
  • COUPON FOR MY OTHER COURSES!
    00:12

Requirements

  • VirtualBox must be installed - A VM of 3Gb will have to be downloaded
  • At least 8 gigabytes of memory
  • Some prior programming or scripting experience. Python experience will help you a lot but since it's a very easy language to learn, it shouldn't be too difficult if you are not familiar with.

Description

Apache Airflow is an open-source  platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

In this course you are going to learn everything you need to start using Apache Airflow through theory and pratical videos. Starting from very basic notions such as, what is Airflow and how it works, we will dive into advanced concepts such as, how to create plugins and make real dynamic pipelines.

Who this course is for:

  • People being curious about data engineering.
  • People who want to learn basic and advanced concepts about Apache Airflow.
  • People who like hands-on approach.

Featured review

Christian Grant
Christian Grant
25 courses
4 reviews
Rating: 5.0 out of 5a year ago
Quick and concise with queues to stop the video at some good points. It's definitely necessary to watch the video a couple of time when you do the application parts, but that's the benefit of videos. I also like the theory parts before the practice parts. Thank you.

Instructor

Marc Lamberti
Apache Airflow Expert, Big Data Engineer
Marc Lamberti
  • 4.6 Instructor Rating
  • 3,736 Reviews
  • 19,504 Students
  • 3 Courses

Hi there,

My name is Marc Lamberti, I'm 27 years old and I'm very happy to arouse your curiosity! I'm currently working as Big Data Engineer in full-time for the biggest online bank in France, dealing with more than 1 500 000 clients. For more than 3 years now, I created different ETLs in order to address the problems that a bank encounters everyday such as, a platform to monitor the information system in real time to detect anomalies and reduce the number of client's calls, a tool detecting  in real time any suspicious transaction or potential fraudster, an ETL to valorize massive amount of data into Cassandra and so on.

The biggest issue when you are a Big Data Engineer is to deal with the growing number of available open source tools. You have to know how to use them, when to use them and how they connect to each other in order to build robust, secure and performing systems solving your underlying business needs.

I strongly believe that the best way to learn and understand a new skill is by taking a hands-on approach with just enough theory to explain the concepts and a big dose of practice to be ready in a production environment. That's why in each of my courses you will always find practical examples associated with theoric explanations.

Have a great learning time!

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Impressum Kontakt
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.