Practical Apache Airflow

Explore airflow architecture, including the relational database backend, the web server ui, and the scheduler, and how dags and tasks run with local, celery, or dask executors across workers.

Explore airflow architecture by examining database options, MariaDB as a meta database, web server connections to the SM database, and local versus salary executer for parallel processing.

Install airflow on Linux via VirtualBox, with emphasis on version 1.9 or 1.10. Windows installation is similar but not covered; use a VirtualBox image to follow along.

Configure the airflow environment by setting airflow home, install airflow with python 3.4+, initialize the database, and start the web server and scheduler for production using postgres or mysql.

Export AIRFLOW_HOME, install airflow, and initialize the database, then fix a Fernet key error by installing the cryptography module and generating or configuring a Fernet key.

Learn to generate a Fernet key to encrypt Airflow connection secrets, replacing the config with the new key. Start the web server and scheduler to verify the secured UI.

Engage in a hands-on quiz session that tests knowledge of airflow installation environment variables, Python 3.6 compatibility with PIP, common pitfalls, and default run settings.

Explore airflow configuration in your home directory, configure dag and log paths (including remote cloud storage), and choose an executor from sequential to local, dask, or celery for scalable workflows.

Airflow uses a metadata database via SQLAlchemy; start with sqlite locally, then switch to a scalable database, and tune pool size, pool recycle, and dag concurrency.

explore how maximum active runs limits how many dag runs execute at once, with a code example of default arguments and a dag definition, and per-dag overrides during backfilling.

Learn how Airflow uses backfilling and the maximum active runs parameter to limit concurrent dag runs. Choose the source from the configuration or the DAG definition to apply the limit.

Explore additional Airflow configuration settings, including pickling and security options, SMTP server setup, and executor options like Celery and Dask to tailor your pipeline environment.

Answer quiz questions on Apache Airflow concepts, including SQLite local executor, concurrency versus maximum active tasks, disabling loading examples, authentication and authorization, and pickling considerations.

Build a simple python and air flow data pipeline to retrieve daily CSI files, store in PostgreSQL, then aggregate and insert into a new table.

Set up a practical Apache Airflow project by creating a structured workspace with file and code directories, a Python package named movie, and data retrieval and database connection components.

Retrieve data from movie, ratings, and tags files using pandas, load them into data frames, and perform a movie_id join to explore genres and associated tags.

Learn to merge multiple data frames from csp files using inner joins, create a unified dataset like movie_readings, and drop duplicates to produce a single, ready-to-use data frame.

Perform pandas-based aggregation of ratings by movie id and title, and by tag and genre, using group by and mean to sort results, integrated in an Airflow project.

Connect to a PostgreSQL database using SQLAlchemy to create an engine, insert data from a data frame, and verify tables like aggregated movie rating and movie data.

Create airflow pipelines by building core and dag packages, wiring tasks with the python operator, and using XCom to pass dataframes between tasks while configuring schedule, backfill, retries, and alerts.

Explore how to create various workflow patterns in Apache Airflow, building on prior coverage of installation, configuration caveats, and a simple data pipeline, while comparing Airflow with similar tools.

Explore dynamic flow patterns in Apache Airflow with pattern 1 and pattern 2, comparing single-task execution to downstream dependencies, and discuss how parallelism varies with local and Celery executors.

Explore pattern three by outlining A before B, C, and D; B and C run in parallel, E depends on B and C, and G depends on E and F.

Show how A, B, and C run in parallel; D depends on E, which depends on B; F depends on C; and G runs downstream of D, C, and F.

Engage with a quiz on dynamic flow patterns in Apache Airflow, exploring downstream and upstream connections and using the big shift operator to validate patterns.

Discover how Airflow hooks connect to external platforms, create and reuse connections (like PostgreSQL), and wire hooks into dags to securely manage credentials and run pipelines.

Explore branching with the Branch Python operator to route a workflow by a callable that returns a task ID, selecting between aggregation and end based on data.

Explore hands-on use of XCOM in Apache Airflow, pushing and pulling data between tasks with key-value pairs, and implement a Python demo with data and data_received functions.

Explore how Airflow behaves differently under Python 2 and Python 3, revealing compatibility issues and unexpected behavior through a hands-on demo of importing Airflow and observing data type changes.

Learn to enable authentication in Apache Airflow across Python versions by updating the configuration file, enabling web server authentication, and selecting an authentication backend to protect the UI in production.

Set up authentication for Apache Airflow by editing the configuration file, handling version differences, restarting Airflow to apply changes, and verifying the application page requires user credentials.

Explore hands-on methods to create users in Apache Airflow using Python 3, illustrating SQLAlchemy ORM interactions, UI login, and cross-version differences with Python 2.

Practice creating a user with python 2 by copying the slide code, run it, and supply a username and password to complete the user setup.

Explore writing Airflow logs to S3 or cloud storage, compare default local logs with cloud options, and understand the execution date and task-based directory structure for storing logs.

Learn how to configure and store Airflow task logs in S3 by creating python package directories, setting up credentials, and wiring base log folders to your bucket.

Demonstrates configuring and triggering Airflow runs to publish execution logs to S3. Verify that logs, run directories, and execution dates appear under the main bucket.

Explore how containerized airflow simplifies setup by pulling a prebuilt docker image, installing docker and compose, and running airflow with minimal environment changes across Mac and Linux.

Install docker on Mac or Windows, verify it runs automatically, and test the installation by running a hello message to confirm proper setup.

Build and run an Apache Airflow container with Docker, clone the repo, use Docker Compose for a local executor, then start and monitor DAGs and backfills.

Build Airflow containers with local and celery executors, starting with correct docker compose and properly configured volumes and dags paths. Troubleshoot missing dags and scale celery workers as needed.

Explore how to build a docker container using a Python 3.6 image, configure airflow, install dependencies with pip, and manage docker compose, entrypoints, and port mappings.

Tackle the assignment by running Airflow with the Celery executor in a container, adjust executor settings and volumes per your environment, and troubleshoot missing DAGs with the comments for help.

Explore Airflow containers by entering the web server and PostgreSQL containers, inspect configuration, dags, logs, and XCOM data, and edit files with bash and vi inside the container.

Test the Airflow rest API server by querying get, post, and delete endpoints, review the Flask API in the codebase, and verify an ok status via the experimental test.

Explore how to use the Airflow rest api from a web browser to check endpoint status, fetch latest run details, and retrieve specific task information via get requests.

Learn how Apache Airflow's REST API uses post to trigger DAG runs and call endpoints for results. Understand version-specific endpoint availability and RBAC limitations.