What you'll learn

To get the basic understanding of Workflows, Apache Airflow basic features, Core Concepts, Familiarization with Airflow UI, XComs, Variables, Connections Hooks
You will also learn how to install Apache Airflow on Linux / Ubunto, in 2 different flavours, Install in Docker Container and Vanilla Installation
This course will help you enhance knowledge of ETL pipelines and how to orchestrate those ETL pipelines in Airflow Dags.
Dags in this course contains working examples of Apache Spark, Elasticsearch, Snowflake, Ecommerce Dag as well.

Course content

6 sections • 34 lectures • 4h 6m total length

Introduction7:23
What will be covered in this course and an introduction about my self.

Workflows5:12
Description of the workflows, What are workflows? This video also explains different types of workflows. What is the relationship of workflows with Apache Airflow?
Medallion Architecture and ETL6:01
You will find an explanation of the Medallion Architecture in this lecture.
Problem Statement10:55
I have explained the problem statement that Apache Airflow will solve for us.
Introduction To Apache Airflow7:51
First Glance of Apache Airflow.
Summary1:29
From This Section

Overview1:38
GCP - Google Cloud Platform - Creating Virtual Machine4:21
I have explained about creating a free trial account on GCP, Google Cloud Platform, and then creating a VM (Virtual Machine) and let the machine ready for next lecture in which we will install airflow.
Apache Airflow Installation on Ubuntu9:06
I have provided scripts as resources for this lecture. In this lecture, I have explained about how to install plain vanilla airflow.
Docker Installation and Using AIrflow with Docker Image10:30
I have explained how to install Apache Airflow in Docker Container. I have also attached docker-compose script along with this lecture.
Airflow Configurations - Part 15:32
This is part one of the Airflow Config file explanation.
Airflow Configuration - Part 210:17
In this lecture, I have explained about rest of the airflow configuration.
Summary1:31
4 Questions In This Quiz, All related to videos in this section.

Airflow Architecture5:53
How different components of Airflow work together, is explained in this lecture. Those components are Scheduler, Webserver and Meta Database.
Airflow Core Concepts Part 115:25
I have explained about Dags, and Control Flow in this lecture.
Airflow Core Concepts Part 211:57
Tasks, Operators, and Sensors are described in this lecture.
Airflow Core Concepts Part 312:26
Continued explanation about Scheduler, XComs, Variables, Connections, and Hooks.
Summary1:33
6 Questions from this section

Dag Page3:01
Explanation about Dag Page in Airflow UI
Top Navigation Bar13:19
I have Presented the top navigation bar explanation in this Lesson.
Dag Detail Page6:09
UI elements explained on the Dag Details Page and multiple Views are also explained in this lesson.
Branching Operator Dag7:13
Explanation of the branching operator Dag in which I have make use of Airflow Variables
XCom Example Dag6:19
I have explained how XComs can be used to transmit information between Tasks.
Summary1:32

Use Cases3:20
Medallion Airflow Dag7:45
Explanation about the Medallion Dag Code, which is written in Scala. I have attached the source code along with this vide.
Medical Devices Airflow Dag13:08
Batch Use Case of Medical Devices Dataset. I have attached the source code along with this video.
JDBC And MySQL Operator Airflow Dag12:45
I have explained how we can use JDBC and MySQL Operators which are provided in Airflow. I have attached the source code along with this video.
Custom Operators Airflow Dag9:59
You will find an explanation about custom operators in this lecture, How to write code for Custom Operators, and how to deploy Custom Operators.
Weather Source Airflow Dag10:31
Faker Python Package is being used to generate a dummy dataset, and then on top of that, I have explained how to write ETL Pipeline and orchestrate in Apache Airflow.
ECommerce Airflow Dag10:27
I have shared the code for how we can create E-commerce ETL Pipeline using Apache Spark. There is an assignment associated with this lecture.
Snowflake Dag3:52
Explanation about Snowflake Dag
Elasticsearch Dag5:25
Explanation about Elasticsearch Dag, How we can create a connection with Elasticsearch.
Summary2:59

Requirements

You should be familiar with Python, Scala Languages, Little bit of knowledge about concepts of Big Data, Some understanding about Apache Spark will be help full in understanding of this course.

Description

Airflow is an extensive tool and has many different features. I will explain in details how we can take benefit from Airflow in our Big Data-related projects. In this Apache Airflow I will start from basic concepts, then move to Installation, Later will deep dive into advanced concepts. In the end, I will create a few big data-related projects and create Airflow Dag to give you a flavor of how all components of Airflow work together.

Please note that Apache Airflow is not a Data Flow tool rather its a Workflow Orchestration tool, With such a nice and beautiful User Interface and a bundle of rich components available. When data is flowing through different processes it is difficult to monitor each and every process, if we are working in CLI based environment. Rather using Apache Airflow we can get notified by color scheming of tasks, and we can start the task right from where it has failed.

I will show you different use cases of Big Data universe, in which we can take leverage from Apache Airflow.

After completing this course Part 1, you will be find your self equipped with a new tool just to solve your Big Data use cases.

Who this course is for:

This Course is intended for Data Engineers, Big Data Architects, or for students those does not have prior knowledge of Apache Airflow.

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 7min

Getting Started5 lectures • 31min

Airflow Installation and Configuration7 lectures • 43min

Apache Airflow Architecture and Concepts5 lectures • 47min

Apache AIrflow UI Demo6 lectures • 38min

Developing Big Data Pipeline in Apache Airflow10 lectures • 1hr 20min

Requirements

Description

Who this course is for: