
Master Apache Airflow by exploring DAGs, operators, and executors, installing with pip or docker, configuring the web interface, writing and managing DAGs, monitoring, scaling, and building a real-world workflow.
Explore core Apache Airflow concepts and architecture, learn to write DAGs with a Pythonic workflow, and perform hands-on installation, deployment, monitoring, and scaling of data pipelines.
Install Apache Airflow via Docker using docker-compose, configure Postgres and Redis, and understand core components like the scheduler, web server, worker, and trigger.
Install Apache Airflow using Docker with Redis and Postgres images, run containers in the background, and access the web server at localhost:8080 with the default Airflow credentials.
Learn how to write dag definitions in Apache Airflow by creating Python operators, setting default arguments, start date, and dependencies, including parallel tasks and a csv modification task.
Apply modular, well-named dag structures that follow best practices, with clear dependencies and default arguments, leveraging templates, parallelism control, robust logging, and unit tests for reliable Airflow workflows.
Learn how to write tags for Airflow. Complete an assignment to edit two CSV files in separate input and output folders using simple, logical steps.
Explore Apache Airflow executors, including sequential, local, and Celery; learn how the scheduler and metadata database coordinate task status, parallelism, and production versus testing setups.
Discover how Apache Airflow uses operators to define DAGs, with BashOperator for shell scripts and PythonOperator for custom Python logic. Learn to set default arguments, schedule intervals, and build tasks with meaningful names, dependencies, and logging to manage and troubleshoot workflows.
Scale Apache Airflow horizontally and vertically to boost capacity and reliability by adding workers, schedulers, and web servers behind a load balancer, while adopting high availability and performance optimization practices.
Learn to build a three-layer ETL with Airflow by adding a load script, configuring with config.yaml, and defining extract, transform, and load tasks in a DAG.
Learn to integrate Airflow with AWS by defining connections, using sensors and hooks to process btc usd csv data, generate daily reports, and upload them to an S3 bucket.
Enforce security and deployment best practices for Apache Airflow with access control, RBAC, OAuth or LDAP authentication, encrypted data in transit, monitoring, backups, and scalable dockerized deployments on EC2.
Complete the practical assignment by writing a unit test to validate your load transformation and dak script, following instructions and using online help if needed.
Deploy a final project with Apache Airflow that integrates real-time data from Kafka and batch data from SQL Server, MySQL, and CSV files, transforming and storing results in AWS S3.
Demonstrates building an end-to-end data pipeline by configuring a kafka flow, extracting csv and mysql data with a csv extractor and odbc, and outlining transformation and loading steps.
Build and deploy an end-to-end ETL workflow in Apache Airflow, integrating CSV and Kafka data extraction, transformation, and uploading to AWS S3, with a final MySQL extractor enhancement.
Master the Apache Airflow workflow orchestration by exploring directed acyclic graphs, architecture, installation, and DAG creation and management, then scale for high availability in real-world workflows for data engineering.
Become an Apache Airflow professional and learn one of employer's most requested skills nowadays!
This comprehensive course is designed so that Data Engineers, Data Scientists, DevOps, Software Engineers, IT Professionals, Students... can learn Apache Airflow from scratch to use it in a practical and professional way. Never mind if you have no experience in the topic, you will be equally capable of understanding everything and you will finish the course with total mastery of the subject.
After several years working in data engineering, we have realized that nowadays mastering Apache Airflow for efficient and maintainable orchestration of complex workflows is very necessary across diverse systems. Knowing how to use this tool can give you many job opportunities and many economic benefits, especially in the world of IT.
The big problem has always been the complexity to perfectly understand Apache Airflow requires, since its absolute mastery is not easy. In this course we try to facilitate this entire learning and improvement process, so that you will be able to carry out and understand your own projects in a short time, thanks to the step-by-step, detailed and hands-on examples of every concept.
With almost 7 exclusive hours of video, this comprehensive course leaves no stone unturned! It includes both practical exercises and theoretical examples to master Apache Airflow. The course will teach you how to proficiently deploy, manage, and optimize workflows, ensuring streamlined automation and robust data pipeline in a practical way, from scratch, and step by step.
We will start with the setup and installation of the needed work environment on your computer, regardless of your operating system and computer.
Then, we'll cover a wide variety of topics, including:
Introduction to Apache Airflow and course dynamics
Master installing and configuring Apache Airflow
Create and organize DAGs effectively
Different operators and executors to define and run tasks
Monitor workflow execution and task statuses
Scale Airflow installations and ensure availability
Explore advanced features and integrate with external systems
Test, debug, and deploy Airflow workflows
Mastery and application of absolutely ALL the functionalities of Apache Airflow
Quizzes, Practical exercises, complete projects and much more!
In other words, what we want is to contribute our grain of sand and teach you all those things that we would have liked to know in our beginnings and that nobody explained to us. In this way, you can learn to build and manage a wide variety of projects and make versatile and complete use of Apache Airflow. And if that were not enough, you will get lifetime access to any class and we will be at your disposal to answer all the questions you want in the shortest possible time.
Learning Apache Airflow has never been easier. What are you waiting to join?