The Complete Hands-On Introduction to Apache Airflow
8,826 students enrolled
The Complete Hands-On Introduction to Apache Airflow
Learn to author, schedule and monitor data pipelines through practical examples using Apache Airflow
8,826 students enrolled
Created by
Marc Lamberti
Last updated 7/2020
English
English [Auto],
French [Auto],
1 more
- Portuguese [Auto]
Current price: $34.99
Original price: $49.99
Discount:
30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
- 5 hours on-demand video
- 9 articles
- 9 downloadable resources
- Full lifetime access
- Access on mobile and TV
- Certificate of Completion
Training 5 or more people?
Get your team access to 4,000+ top Udemy courses anytime, anywhere.
Try Udemy for Business
What you'll learn
- Create plugins to add functionalities to Apache Airflow.
- Using Docker with Airflow and different executors
- Master core functionalities such as DAGs, Operators, Tasks, Workflows, etc
- Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.
- The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.
- Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc.
- Install and configure Apache Airflow
- Think, answer and implement solutions using Airflow to real data processing problems
Course content
Expand all
70 lectures
05:02:10
+
–
Course Introduction
5 lectures
11:51
Preview
01:08
MUST-READ: VM
00:29
+
–
First Approach to Airflow
8 lectures
38:40
+
–
Coding Your First Data Pipeline with Airflow
16 lectures
01:11:47
Introduction
00:31
Preview
03:31
[Practice] Time to code your first DAG
01:45
[Practice] DAG Skeleton
05:17
[Practice] Checking if tweets are available - FileSensor
13:33
[Practice] Fetching and cleaning tweets - PythonOperator
07:53
[Practice] Storing tweets into HDFS - BashOperator
04:13
[Practice] Loading tweets into Hive - HiveOperator
08:41
Operator Relationships and Bitshift Composition
01:43
[Practice] Adding dependencies
03:19
[Practice] The Twitter DAG in action!
04:21
How the Scheduler Works?
03:55
[Practice] A Quick Play With Backfill and Catchup
06:38
What is a Workflow?
00:43
Quiz Time!
7 questions
Recap
02:31
+
–
Databases and Executors
7 lectures
42:33
Introduction
00:52
Local Executor with MySQL
04:23
[Practice] Configuring your DAG with the Local Executor and MySQL
11:28
Celery Executor with MySQL and RabbitMQ
05:56
[Practice] Configuring your DAG with Celery Executor, MySQL and RabbitMQ
09:51
Quiz Time!
5 questions
Recap
02:32
+
–
Implementing Advanced Concepts in Airflow
12 lectures
57:45
Introduction
00:34
Minimising Repetitive Patterns With SubDAGs
01:39
[Practice] Minimising a DAG with SubDAGs
12:07
How to Interact With External Sources Using Hooks
01:29
[Practice] Getting data from MySQL using Hooks
05:56
How to Share Data Between Your Tasks With XCOMs
02:44
[Practice] Sharing data between tasks with XComs
08:42
How to Execute Tasks According To Criteria Using Branching
03:25
[Practice] Executing a task according to a condition
06:13
Control Your Tasks With SLAs
05:42
[Practice] Defining SLAs in your DAGs
05:57
Quiz Time!
5 questions
Recap
03:17
+
–
Creating Airflow Plugins with Elasticsearch and PostgreSQL
6 lectures
33:16
Introduction
00:31
Adding Functionalities to Apache Airflow
03:54
[Practice] Creating a hook to interact with Elasticsearch
07:40
[Practice] Creating a transfer operator to move data from MySQL to Elasticsearch
09:47
[Practice] Adding a new view to the Airflow UI
10:10
Quiz Time!
5 questions
Recap
01:14
+
–
Using Apache Airflow with Docker
7 lectures
32:10
Introduction
00:47
Quick Reminder About Docker
03:17
[Practice] Building and running your first Airflow Docker image
07:07
[Practice] Running Airflow using Sequential Executor with Docker
08:24
[Practice] Running Airflow using Local Executor with Docker
04:24
[Practice] Scaling Airflow using Celery Executor with Docker
06:20
Quiz Time!
5 questions
Recap
01:51
+
–
Airflow 2.0
1 lecture
10:41
What to expect from Airflow 2.0?
10:41
+
–
BONUS - APPENDIX
8 lectures
03:26
[BLOG POST] How to use the DockerOperator with Templating and Apache Spark
00:31
[BLOG POST] Apache Airflow with Kubernetes Executor
00:31
[BLOG POST] How to use templates and macros in Apache Airflow
00:41
[BLOG POST] How to use timezones in Apache Airflow
00:40
[BLOG POST] How to use the BashOperator
00:13
[BLOG POST] Variables in Apache Airflow: The Guide
00:14
[VIDEO] Running Apache Airflow on a multi-nodes Kubernetes cluster locally
00:21
COUPON FOR MY OTHER COURSES!
00:12
Requirements
- VirtualBox must be installed - A VM of 3Gb will have to be downloaded
- At least 8 gigabytes of memory
- Some prior programming or scripting experience. Python experience will help you a lot but since it's a very easy language to learn, it shouldn't be too difficult if you are not familiar with.
Description
Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.
In this course you are going to learn everything you need to start using Apache Airflow through theory and pratical videos. Starting from very basic notions such as, what is Airflow and how it works, we will dive into advanced concepts such as, how to create plugins and make real dynamic pipelines.
Who this course is for:
- People being curious about data engineering.
- People who want to learn basic and advanced concepts about Apache Airflow.
- People who like hands-on approach.
Featured review
Christian Grant
(
21 courses,
3 reviews
)
a year ago
Quick and concise with queues to stop the video at some good points. It's definitely necessary to watch the video a couple of time when you do the application parts, but that's the benefit of videos. I also like the theory parts before the practice parts. Thank you.