Apache Airflow 2.0 using Docker, Docker Swarm
What you'll learn
- Apache Airflow 2.0 Architectures and Enhancements
- Docker and Docker Swarm commands
- Setting up airflow 2.0.1 using Docker Compose and Docker Swarm
- Familiar with Docker concepts and commands
- Good to have knowledge on Apache Airflow 1.x
Why Am I conducting this course ?
When I started configuring airflow in my organisation, I spent many weeks on writing Docker Compose files for each airflow component.
Airflow community provides a single docker compose file which installs all the components in a single machine. It works fine for practising purposes. But in production, we setup each component on different machines. Also, there is no docker image available on Docker registry to start airflow through Docker Swarm.
Overall, I contributed many sleepless nights to achieve fault tolerant, resilient, distributed, Highly Available airflow using Docker Swarm. I consolidated all my learnings, knowledge into this course. So that others don't need to struggle like mine.
What do you Learn From this Course ?
The primary objective of this course is to achieve resilient airflow using the Docker and Docker Swarm. I am using the latest stable airflow (2.0.1) throughout this course.
At first, we study all the required docker concepts. Don't worry if you have no prior experience on docker. I cover all docker necessary concepts which are used in this course. In the second module, we investigate Airflow 2.0 and understand the additional advantage over airflow 1.x. We discover the airflow HA architecture and discuss each system requirement.
After this, we acquire these machines from AWS and start containerising one by one applications using docker compose.
At the end, we run multiple airflow schedulers and benchmark it.
Last module is on docker swarm and we witness how easy it is to setup the entire airflow just by running a few swarm commands.
In the beginning, we learn all the swarm concepts, architecture, commands and networking. We translate the docker compose file into the docker service command. Subsequently, We create a Swarm Cluster and trigger one by one airflow services on it.
Who this course is for:
7 years working experience from many fast growing companies like Myntra Designs Pvt. Ltd, Swiggy, Lendingkart Technologies Private Limited, Sprinklr. Currently working as Senior Data Engineer at Delivery Hero, Berlin. Highly dedicated, self-motivated and confident person. Hands-on experience in developing large scale web application, writing Spark Jobs, developing Api’s etc.
Coding is my passion and I love exploring new programming methodologies.
I am a self motivated, highly enthusiastic and dedicated person. I keep myself occupied with learning and practising it. I own a blog and publish wonderful articles on this. I worked on multiple tech stacks. I love to conduct experiments by comparing each other and publish the results on my blog.
I excel in Apache Airflow, Spark, Hadoop, Hive, Presto, Kafka, Spark Streaming etc Big data technologies. I worked on Mysql, MongoDB, Memsql, PostgreSQL, Redis databases.