Apache Airflow using Google Cloud Composer: Introduction

Name: Apache Airflow using Google Cloud Composer: Introduction
Rating: 4.0 (376 reviews)

With Google Cloud composer learn Apache Airflow without making any local install. Ensures focus is on Airflow topics.

Role Play

Created byGuha Rajan M., B.Engg, MBA

Last updated 12/2025

English

What you'll learn

Understand automation of Task workflows through Airflow
Airflow Architecture - On Premise (local install), Cloud, single node, multiple node
How to use connection functionality to connect to different systems to automate data pipelines
What is Google cloud Big query and briefly how it can be used in Dataware housing as well as in Airflow DAG
Master core functionalities such as DAGs, Operators, Tasks through hands on demonstrations
Understand advanced functionalities like XCOM, Branching, Subdags through hands on demonstrations
Get an overview understanding on SLAs, Kubernetes executor functionality in Apache Airflow
The source files of Python DAG programs (9 .py files) used in demonstration are available for download towards practice for students

Course content

19 sections • 37 lectures • 3h 51m total length

Course Overview - Topics of coverage10:01
Explore Apache Airflow with Google Cloud Composer, detailing use cases, architecture, and cloud vs custom setups, plus hands-on workflow concepts like tasks, variables, and BigQuery integration.

Data pipe lines & Uses cases for Apache Airflow8:17
Explore how data pipelines coordinate tasks with dependencies—from extraction and transformation to loading—using Apache Airflow, with use cases like weather dashboards, predictions, and external deliveries.
What is Task and why Orchestration needed?3:18
Identify tasks and dependencies in a data pipeline and explain how orchestration with Apache Airflow coordinates various program units: shell scripts, Python programs, or database procedures for automated, reliable processing.
What is Apache Airflow & environment options?11:48
Understand what Apache Airflow is and how Google Cloud Composer offers environment options to run it, with Python-based directed acyclic graph workflows for automating data pipelines.

Apache Airflow architecture7:12
Explore how Apache Airflow architecture orchestrates workflows via the web server dashboard, scheduler, queue, and worker nodes, driving DAG execution and task visualization.
Apache Airflow - Single Node vs Multinode4:30
Explain the single-node and multinode Airflow architectures in Google Cloud Composer, detailing the scheduler and worker roles, and highlighting high availability and scalability.

Provisioning Google Composer - Apache Airflow environment - Part 14:21
Provision a Google Cloud Composer environment by enabling the API and creating the environment. Explore auto scaling and the differences between Composer versions 1 and 2.
Provisoning Google Composer - Apache Airflow environment - Part 25:37
Provision a Google Cloud Composer environment by configuring name, location, resources, image version, and Python version. The VM provisioning installs Python and Apache Airflow and takes about 20–25 minutes.
Navigation - Cloud composer(Apache airflow) Web UI navigation11:36
Navigate the Google Cloud Composer Apache Airflow web UI to monitor environment health, view dag runs, trigger tasks, view logs, and manage connections, dag files, and data.
Quiz time 1

Activity 2: Using Templating functionality in Apache Airflow program6:11
Master templating in Apache Airflow using macros and dynamic parameters, including the execution date, to drive task execution and control branching.
Activity 2: Using Templating functionality in Apache Airflow program - Part 24:34
Explore templating in Apache Airflow, including upstream dependencies, template IDs, and dynamic dates, and see how they drive task execution and logging in Google Cloud Composer.

What is variable in Apache Airflow and when to use them?4:08
Discover how Apache Airflow variables store configuration values in the metadata database, avoiding hard-coded settings, and learn how to define read, update, and use them in Google Cloud Composer.
Activity 3: Variables usage in DAG python program8:25
Demonstrates defining and importing Airflow variables in a DAG, retrieving values by keys, and using bash and python operators to display and compute results.

Activity 4: Calling Bash script in different folder / different machine - Part15:10
Learn how to use Airflow with the bash operator and templating to run a Python script from another folder or different machine, enabling cross‑mission scripting in Google Cloud Composer.
Activity 4: Calling Bash script in different folder / different machine - Part 26:13
Learn how to call a bash script from Airflow in Google Cloud Composer, manage script uploads, load data from the data bucket, and execute across different folders and machines.

Requirements

Google Cloud Platform Account OR even Free Trial account - NO Install required
Good understanding on Python code and some exposure to bash shell scripting will help.

Description

Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows.

Cloud Composer is a fully managed workflow orchestration service that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers. Built on the popular Apache Airflow open source project and operated using the Python programming language, Cloud Composer is free from lock-in and easy to use.

With Apache Airflow hosted on cloud ('Google' Cloud composer) and hence,this will assist learner to focus on Apache Airflow product functionality and thereby learn quickly, without any hassles of having Apache Airflow installed locally on a machine.

Cloud Composer pipelines are configured as directed acyclic graphs (DAGs) using Python, making it easy for users of any experience level to author and schedule a workflow. One-click deployment yields instant access to a rich library of connectors and multiple graphical representations of your workflow in action, increasing pipeline reliability by making troubleshooting easy.

This course is designed with beginner in mind, that is first time users of cloud composer / Apache airflow. The course is structured in such a way that it has presentation to discuss the concepts initially and then provides with hands on demonstration to make the understanding better.

Note : This course also has AI enabled role play on "Interviewing for a Data Engineering Role: Apache Airflow Proficiency" - An interactive Chat.

The python DAG programs used in demonstration source file (9 Python files) are available for download toward further practice by students.

Happy learning!!!

Who this course is for:

People interested in Data warehousing, Big data, Data engineering
People interested in Automated tools for task workflow scheduling
Student interested to know about Airflow
Professional to wish to explore as how Apache Airflow can be used in Task scheduling and building Data pipelines

Apache Airflow using Google Cloud Composer: Introduction

What you'll learn

Explore related topics

Course content

Course Overview1 lecture • 10min

Introduction3 lectures • 23min

What is Airflow - Directed Acyclic Graph (DAG) & operators?1 lecture • 6min

Apache Airflow architecture2 lectures • 12min

Google Cloud Platform: Cloud composer used as Apache Airflow3 lectures • 22min

Understanding Apache Airflow program structure1 lecture • 4min

Activity 1 : Create and submit Apache airflow DAG program1 lecture • 13min

Activity 2: Using Template functionality in Apache Airflow program2 lectures • 11min

Using Variables in Apache Airflow2 lectures • 13min

Activity 4: Calling Bash script in different folder / different machine.2 lectures • 11min

Requirements

Description

Who this course is for: