Apache Airflow: Complete Hands-On Beginner to Advanced Class

Learn Airflow’s four components—the web server, scheduler, executer, and metadata database—and how they collaborate to monitor, trigger, and execute workflows with proper resource allocation and DAGs and connections.

Install airflow on macOS by setting airflow home, creating a Python 3.7 virtual environment, installing airflow 1.10.10, and adding cryptography and Spark for secure connections and workflows.

Learn to run Apache Airflow locally by initializing the metadata database, launching the web server and scheduler, and validating the airflow home and working directory in your environment.

Explore the Airflow UI by starting the web server and inspecting dags, task states, and logs, with tree and grand views to trigger dags and view code and variables.

Learn to manage Apache Airflow from the command line using the airflow CLI, including triggering DAGs, checking next execution times, printing DAG structure, and listing tasks, without the web UI.

Pass a default_args dictionary to a dag to apply owner and start_date to all tasks, with start_date defaulting to midnight UTC if unspecified.

Define tasks as units of work run by workers, each governed by an operator. Learn about action, transfer, and sensor operators, the base operator class, and built‑in and provider operators.

Define dependencies between tasks in Airflow using big shift operators and relationship builders, with set downstream and set upstream, and practice with chain and cross for complex, multi-task graphs.

Apply airflow to a real use case by loading sensor data from google cloud storage into Begbie, creating a table for the latest vehicle state and historical analysis.

Configure Airflow connections to external resources, using the Google Cloud Default connection, download and store the service account key JSON, place it in secrets, and provide the project ID.

Load csv data from cloud storage into BigQuery using the Google Cloud Storage to BigQuery operator, configuring source bucket, destination project, dataset, and table, plus write options.

Use the Google Cloud Storage hook in Airflow to list landing-bucket objects, enforce a single active run, and move processed files to a backup bucket to prevent duplicates.

Learn cross-task communication with xcoms by pushing and pulling messages as a shared state, use provide_context on python operators, and pass data between tasks.

Explore jinja templating and macros in Apache Airflow, learning to prefix objects with the execution timestamp, use templated fields, and define user-defined macros for consistent parameterization.

Manage airflow variables for dev, staging, and prod with the UI or CLI. Import them from a file and reference them in DAGs using variable.get for project and bucket names.

Create a Google Cloud Storage bucket with a globally unique name, upload the Facebook folder, and enable the Cloud Data API to prepare for Airflow job orchestration.

Create a dataproc hadoop cluster using the dataproc cluster create operator, with a timestamped name, two workers, and a shared storage bucket in Europe, scheduled daily at eight pm.

Explore branching in Airflow using the BranchPythonOperator to decide between weekday and weekend tasks by evaluating the execution date, returning 'big data analytics' or 'weekend analytics' as the next task.

Submit a PySpark job with Airflow's Spark operator on a Google Cloud Dataprep cluster, wiring GCS paths, cluster name, and jar for weekend analytics.

Learn how subdags embed a dag inside a dag to group related tasks, enforce a consistent schedule, and manage dependencies; avoid deadlocks when running many operators in parallel.

Use cluster delete operator to remove data after bespoke jobs, and configure a trigger rule so downstream tasks run when all upstream tasks are done using the Google cloud connection.

Define and attach dag and task documentation in the Airflow UI using markdown or json-like strings. View titles, descriptions, and instance details to verify the documentation.

Develop a custom Airflow operator by extending the base operator, register it in a plug-ins folder, and implement a data validation operator with optional ui color and templated parameters.

Create a custom sensor in the big plug-in to wait for a data set to exist in Google Cloud, returning true when the data set is found and false otherwise.

Create and run a dag using custom operators and a custom sensor to validate vehicle analytics data, including templated sql and dataset checks, and monitor results in the web ui.

Learn how to validate and load dags in Airflow, run loader tests via CLI, inspect active dags and task dependencies, and verify error-free execution.

Master unit testing for Apache Airflow dags and operators. Create tests using a cheat sheet of attributes, properties, and methods, and verify upstream and downstream relationships and the dag structure.

Test the custom data validation operator by creating a unit test that mocks its data retrieval, asserts a failure on empty results, and demonstrates operator testing.

Explore the four Airflow components—web UI, metadata database, scheduler, and executors—and compare sequential, debugging, local, celery, and distributed executors, with production and setup notes.

Define and monitor task deadlines with service level agreements, configure per-task or default SLA, and receive one-time notifications and UI logs when misses occur; distinguish SLA from execution timeouts.

Enable remote logging in Airflow, set the remote log connection ID to the Google Cloud default, and specify a base folder to store task logs in a remote bucket.

Learn to integrate Sentry with Airflow to monitor and fix crashes in real time, capture the full stack trace, and browse, resolve errors, and configure alerts in a centralized dashboard.