DevOps CI/CD Project: Jenkins, Ansible and Kubernetes

An overview of what this section covers. With the Java project already in GitHub, we set up Jenkins and Maven, then integrate GitHub and Maven with Jenkins so Jenkins can pull the code and build it using Maven. Any source code changes are made locally with Git and committed to GitHub. We begin by setting up the Jenkins server.

Learn how to set up a Jenkins server on AWS. We launch an Amazon Linux 2 EC2 instance (t2.micro) with a security group that opens port 8080, create a key pair, and connect to the instance using MobaXterm. After adding the Jenkins repository, we use Amazon Linux Extras to install the EPEL packages and Java OpenJDK 11, then install and start Jenkins. Finally, we access the Jenkins web UI on port 8080, unlock it with the default administrator password, skip plugin installation, and log in to the Jenkins dashboard.

Create and run your first Jenkins job. We create a freestyle project called Hello World Job and explore its options, such as source code management, build triggers, build and post-build actions, noting that most features come with plugins. Since Jenkins runs on Linux, we use the Execute shell build step to run echo Hello World and uptime. After saving and building the job, we check the build history and console output to confirm the commands ran and the build succeeded.

Learn how to integrate GitHub with Jenkins in three steps. We first change the Jenkins server hostname using the /etc/hostname file, then install Git on the Jenkins instance through the CLI. Next, we install the GitHub plugin from Manage Plugins in the Jenkins GUI, along with its dependency plugins. Finally, we configure Git under Global Tool Configuration, setting the Git executable path so Jenkins can pull code from GitHub.

Create a Jenkins job that pulls code from GitHub. We set up a freestyle project, select Git under Source Code Management and provide the Hello World repository's HTTPS URL, with a demo of forking the repository into your own GitHub account. Since the repository is public, no credentials are needed. After building the job, we review the console output to confirm the clone succeeded and explore the Jenkins workspace at /var/lib/jenkins/workspace, where the cloned code is stored in a folder named after the job.

Learn how to integrate Maven with Jenkins, using the Jenkins server as the build server. Following the official Maven installation guide, we download and extract Apache Maven into /opt and rename the directory to maven. We then set the JAVA_HOME, M2_HOME and M2 environment variables in the root user's .bash_profile, load them with the source command and verify the setup with mvn -v. Finally, we install the Maven Integration plugin and configure the JDK and Maven home paths under Global Tool Configuration in the Jenkins GUI.

Create a Jenkins build job to pull code from GitHub and build it with Maven. Using the Maven project option now available from the Maven plugin, we point the job to the GitHub repository, set the root pom.xml and use the clean install goals, with a brief look at the Maven build lifecycle. During the build, Maven downloads dependencies, runs test cases and builds the server and webapp modules. We then locate the generated webapp.war artifact in the target directory of the Jenkins workspace, from both the CLI and the Jenkins GUI.

An overview of this section, where we deploy the built code to a target environment. We set up an EC2 instance, install Tomcat on it, and create a Jenkins job to deploy the code to the Tomcat server.

Learn how to set up a Tomcat server, following the steps documented in the Simple DevOps Project repository. We launch an Amazon Linux 2 EC2 instance with a new DevOps security group opening port 8080, install Java OpenJDK 11 using Amazon Linux Extras, then download and extract Tomcat 9 into /opt and start it with startup.sh. To access the Manager App from outside the server, we comment out the localhost restriction in the context.xml files and add users with manager roles in tomcat-users.xml. We also create tomcatup and tomcatdown link files under /usr/local/bin for starting and stopping Tomcat, then log in to the Tomcat Manager App.

Learn how to integrate Tomcat with Jenkins to deploy code on the Tomcat server. After changing the default Jenkins admin password, we install the Deploy to container plugin and add the Tomcat deployer user as credentials under Manage Credentials. We then create a Maven build and deploy job that pulls code from the master branch, runs clean install and uses the post-build action to deploy the WAR file to Tomcat, using **/*.war as the path and Tomcat 8.x as the container with the server's URL. After running the job, we confirm the WAR file is deployed to the webapps directory and view the webapp application in the browser.

Update the application source code and redeploy it to the Tomcat server. We clone the Hello World repository to the local workstation using Git Bash, replace the content of index.jsp with a simple HTML registration form, then add, commit and push the changes to GitHub. Since GitHub now uses personal access token authentication, we clear old credentials from the Windows Credential Manager and create a personal access token in the GitHub developer settings to push the code. Finally, we rerun the Jenkins job to build a new artifact and deploy it, and see the updated registration form in the browser.

Automate the build and deployment so Jenkins runs the job whenever code changes in GitHub. We explore the build trigger options in the job configuration and compare Build periodically, which runs regardless of code changes, with Poll SCM, which runs only when the repository has changed, along with the cron schedule format. After enabling Poll SCM to check every minute, we update index.jsp locally and push the changes to GitHub, triggering the job automatically as an SCM change. We also troubleshoot a failed build, rerun it successfully and see the updated form on the Tomcat server.

Learn how to set up a Docker host to deploy code on a Docker container instead of a VM. Following the Docker installation steps in the Simple DevOps Project repository, we launch an Amazon Linux 2 EC2 instance (t2.micro) as the Docker host, connect to it using MobaXterm, install Docker and start the Docker service. We then validate the setup with basic commands such as docker images, docker ps and docker ps -a, and understand the difference between listing running containers and all containers.

Learn how to create a Docker container from a Docker image. We cover the two ways to get an image, pulling from Docker Hub or building your own with a Dockerfile, and explore the official Tomcat image and its tags on Docker Hub. After changing the Docker host's hostname and restarting the Docker service, we pull the latest Tomcat image with docker pull and create a container with docker run in detached mode, mapping internal port 8080 to external port 8081. We then open ports 8081 to 9000 in the security group and access the container from the browser, where a 404 error appears, which we fix in the next lecture.

Fix the HTTP 404 error seen in the Tomcat container, a known issue in Tomcat images after version 9. We log in to the running container with docker exec -it and /bin/bash, and find that the default applications are in webapps.dist while the webapps directory is empty. Copying the content from webapps.dist to webapps makes the Tomcat page accessible in the browser. We then stop the container, launch a new one on port 8082 and see the error return, since changes made inside a container do not affect the image. The next lecture fixes this permanently using a Dockerfile.

Learn how to write your first Dockerfile. We go through important Dockerfile instructions, including FROM, RUN, CMD, ENTRYPOINT, WORKDIR, COPY, ADD, EXPOSE and ENV, and map them to the steps for installing Tomcat on CentOS. We then write a Dockerfile that pulls the CentOS image, installs Java, creates and switches to /opt/tomcat, downloads and extracts the Tomcat package, renames the directory, exposes port 8080 and starts Tomcat using catalina.sh with CMD. After fixing a typo, we build the image with docker build, create a container on port 8083 and access Tomcat from the browser.

Create a simpler, customized Dockerfile using the official Tomcat image instead of installing Tomcat on a base OS. We write a Dockerfile that pulls tomcat:latest and uses RUN to copy the content of webapps.dist into webapps under the CATALINA_HOME directory, /usr/local/tomcat, as noted in the image documentation. After building the demo-tomcat image with docker build, we create a container on port 8085 and access the default Tomcat page in the browser without the 404 error, making the container ready to host our application.

Learn how to integrate the Docker host with Jenkins. We create a dedicated dockeradmin user, set its password and add it to the docker group with usermod. To allow password-based login, we update the sshd_config file to enable password authentication, reload the SSH service and log in as dockeradmin. In Jenkins, we install the Publish Over SSH plugin to copy artifacts, then add the Docker host under Configure System using its private IP, the dockeradmin user and password-based authentication, with a brief look at generating SSH keys using ssh-keygen. A successful test configuration confirms Jenkins can connect to the Docker host.

Create a Jenkins job that pulls code from GitHub, builds it with Maven and copies the artifact to the Docker host. We create the job by copying the configuration of the existing build and deploy job, keeping the repository URL, Poll SCM schedule and clean install goals, and replacing the Tomcat deployment step with Send build artifacts over SSH. We configure the source files as **/*.war, remove the webapp/target prefix and fix the remote directory so the WAR file lands in the dockeradmin home directory. After running the job, we confirm webapp.war is on the Docker host, ready to be included in a container through the Dockerfile in the next lecture.

Update the Tomcat Dockerfile so new containers come up with the application. We create a dedicated /opt/docker directory, move the Dockerfile into it and give ownership to the dockeradmin user. We then update the Jenkins job's remote directory to //opt/docker so the WAR file is copied there. Next, we add a COPY instruction to the Dockerfile to copy the WAR file into the Tomcat webapps directory, build a tomcat:v1 image and create a container on port 8086. Accessing /webapp in the browser confirms the application runs from the container, with automating container creation through Jenkins covered in the next lecture.

Automate the end-to-end pipeline so Jenkins builds the image and creates the container after copying the artifact. We update the job's Send build artifacts over SSH step with commands to switch to /opt/docker, build a regapp:v1 image with docker build and create a registerapp container on port 8087 with docker run. After clearing old containers and images using docker container prune and docker image prune -a, we run the job and confirm the image, container and application in the browser. We then push an index.jsp change to GitHub, which triggers the job automatically but fails because a container with the same name already exists, fixed in the next lecture.

Fix the container name conflict and complete the automated CI/CD pipeline for Docker. We update the Jenkins job to stop and remove the existing registerapp container with docker stop and docker rm before creating a new one. After a successful manual run, we add a name field to index.jsp and commit it with the single git commit -am command, which triggers the job automatically and shows the updated form in the browser. We also note that running multiple commands in the Jenkins job is not ideal, and how Ansible can take over deployment and target environment configuration in the next section.

Understand why Ansible is introduced as a deployment tool so Jenkins can focus on building code. In the updated pipeline, Jenkins pulls code from GitHub, builds the artifacts and copies them to the Ansible server. Ansible then creates a Docker image using the Dockerfile, pushes it to Docker Hub as the image repository, and deploys containers on the Docker host, which pulls the image from Docker Hub. In this section, we set up Ansible, create a Docker Hub account, integrate Ansible with Jenkins and manage the Docker host using Ansible.

Learn how to set up the Ansible control node. We launch an Amazon Linux 2 EC2 instance using the existing security group, since Ansible works on port 22, and change its hostname to ansible-server. We then create an ansadmin user, add it to the sudoers file for administrative privileges and enable password-based authentication in the sshd_config file. After logging in as ansadmin, we generate SSH keys with ssh-keygen for key-based authentication. Finally, we install Ansible using Amazon Linux Extras and verify the Python and Ansible versions.

Add the Docker host to Ansible as a managed node so Ansible can instruct it to create containers. On the Docker host, we create an ansadmin user, add it to the sudoers file and confirm password-based authentication is already enabled. On the Ansible server, we add the Docker host's private IP to the default inventory file, /etc/ansible/hosts, and copy the ansadmin public key to the Docker host using ssh-copy-id, which adds it to the authorized_keys file. Finally, we test passwordless connectivity with the Ansible ping module and check the Docker host's uptime using the command module.

Integrate Ansible with Jenkins so Jenkins handles the build and Ansible takes care of deployment. Since the ansadmin user, password-based authentication and the Publish Over SSH plugin are already in place, we only add the Ansible server under Configure System using its private IP and the ansadmin password, then run a successful test configuration. Next, we create a job by copying the build and deploy on container job, disable Poll SCM for now, select the Ansible server and remove the Docker commands, which will be handled by an Ansible playbook. After creating /opt/docker on the Ansible server and giving ownership to ansadmin, we run the job and confirm the WAR file is copied to the Ansible server.

Build a Docker image from the copied artifacts on the Ansible server. We install Docker, add the ansadmin user to the docker group with usermod and start the Docker service. We then create a Dockerfile in /opt/docker by copying the content from the Docker host, and fix a permission issue on the file with chmod. After building a regapp:v1 image with docker build, we create a container on port 8081 using docker run with the -t option and access the application at /webapp in the browser. The next lecture automates these manual steps using an Ansible playbook.

Create an Ansible playbook to build the Docker image on the Ansible server. We first see how Docker Hub lets the Docker host pull the image created by Ansible and run containers. We then add the Ansible server's private IP to the /etc/ansible/hosts inventory under an ansible group, create a dockerhost group, and copy SSH keys to the Ansible server itself so it can manage itself. Next, we write the regapp.yml playbook targeting the ansible group, using the command module to run docker build with args and chdir set to /opt/docker. After validating it with --check, we run the playbook and confirm the new regapp:latest image.

Learn how to push a Docker image to Docker Hub. We log in to the Docker Hub account, look at public and private repositories, and log in to Docker Hub from the Ansible server with the same credentials, noting that private registries such as JFrog Artifactory or AWS and Azure registries can be used in real-world projects. Since pushing directly fails without the account name, we use docker tag to add the Docker Hub username as a prefix to the image name. We then push the tagged image with docker push and confirm the new repository in the Docker Hub account.

Automate building and pushing the Docker image through Ansible and Jenkins. We add two tasks to the regapp.yml playbook to tag the image with the Docker Hub username and push it to Docker Hub, noting that you must log in to Docker Hub as ansadmin and that the playbook runs only on the ansible group, with the --limit option as an alternative. We then add the ansible-playbook command to the Jenkins job and enable Poll SCM. After fixing an unstable build by providing the playbook's full path in /opt/docker, a code change pushed to GitHub automatically triggers the job, copies the artifact, builds the image and pushes it to Docker Hub.

Write an Ansible playbook to create a container on the Docker host. We create the deploy_regapp.yml playbook targeting the dockerhost group, using the command module with docker run to pull the regapp:latest image from Docker Hub and create a container on port 8082. After validating it with --check, we remove all existing images on the Docker host, run the playbook and fix a permission issue on /var/run/docker.sock with chmod. The Docker host pulls the image without logging in, since the repository is public, and we access the updated application in the browser. Running the playbook again fails because the container name is already in use, fixed in the next lecture.

Update the deployment playbook so containers can be redeployed without errors. We add tasks to stop and remove the existing container and remove the existing image, so the Docker host always pulls the latest image from Docker Hub before creating a new container. After fixing a typo in the docker rmi command, we add ignore_errors to the stop, remove container and remove image tasks so the playbook doesn't fail when they don't exist. Running the playbook then removes the old container and image, pulls the latest image and creates a new container. We also briefly look at Ansible Docker modules as an alternative to the command module.

Automate the full CI/CD pipeline to deploy on a Docker container using Ansible. We update the Jenkins job to run the deployment playbook after the image playbook, separated by a semicolon, with its full path in /opt/docker and a 10-second wait between them. After pushing an index.jsp change to GitHub, the job automatically builds the code, updates the image on the Ansible server and Docker Hub, and recreates the container on the Docker host, with the change visible in the browser. We also disable Poll SCM on the older container job and discuss the downtime and lack of recovery when containers are replaced, which a container management system addresses in the next section.

Understand why Kubernetes is introduced into the CI/CD pipeline. Since there is no way to recover when a Docker container goes down, a container management service is needed. Of the options, Docker Swarm and Kubernetes, we choose Kubernetes for its advantages and deploy the application as a pod on a Kubernetes environment instead of a Docker container. The next lecture covers setting up Kubernetes.

Explore the different ways to set up Kubernetes using the official Kubernetes documentation. For production environments, we look at deployment tools such as kubeadm, kops and Kubespray, and turnkey cloud solutions from providers such as Microsoft Azure, Alibaba Cloud and AWS. We choose to set up Kubernetes on AWS using EKS with the eksctl utility, following the setup guide in the GitHub repository, which also includes documentation for other methods such as kubeadm. Whichever method you use, managing Kubernetes remains the same.

Walk through the procedure to set up Kubernetes on AWS using EKS. Instead of the more complex AWS Management Console, we use the eksctl command line utility, following the setup guide in the Simple DevOps Project repository, prepared from the AWS official documentation. The steps include using an EC2 instance as a bootstrap image with the latest AWS CLI, installing kubectl (version 1.21) and eksctl into /usr/local/bin, creating an IAM role with the required privileges, and creating the cluster with eksctl. We also cover validating the cluster and deleting it when not in use, since most EKS services are not free tier eligible, and note that EKS shows only worker nodes, not the master node.

Set up the bootstrap server used to create the EKS cluster. We launch an Amazon Linux EC2 instance (t2.micro) as the EKS bootstrap server and connect to it. Following the AWS documentation, we upgrade the AWS CLI to the latest version 2, then install kubectl (version 1.21), give it execution permission and move it to /usr/local/bin. We also download eksctl, move it to /usr/local/bin and verify both versions. Finally, we create an IAM role for EC2 named eksctl_role, using administrative access to avoid permission issues though minimum policies are recommended for real-world environments, and attach it to the bootstrap server.

Create the Kubernetes cluster on EKS using eksctl. We run the eksctl create cluster command with the cluster name, the us-east-1 region and t2.small as the node type, keeping the default of two nodes. While the cluster is being created, we track the CloudFormation stack, which takes around 20 to 25 minutes to complete. Once ready, we look at the configuration file used to access the cluster and confirm the two t2.small worker nodes in the EC2 console. We then validate the cluster with kubectl get nodes and kubectl get all, see the command to delete the cluster, and create a test pod using kubectl run with the httpd image.

Run basic Kubernetes commands by deploying an Nginx application. We create a deployment named demo-nginx with kubectl create deployment, using the Nginx image from Docker Hub, port 80 and two replicas, and understand how a deployment creates a replica set, which in turn creates the pods. We check these resources using kubectl get deployments, get replicaset, get pods and get all. We then expose the deployment with kubectl expose using the LoadBalancer service type, which creates a load balancer on AWS with the worker nodes attached. Finally, we access the Nginx default page in the browser through the load balancer.

Learn how to create a pod using a manifest file. We first clean up the cluster by deleting the demo-nginx deployment, which also removes its replica set and pods, and its service, which removes the load balancer on AWS. We then write a pod.yml file, referring to the Kubernetes API documentation and a sample pod manifest from the official documentation. We define apiVersion, kind, metadata with a pod name and labels, and a spec with the Nginx container, its name and containerPort 80, along with the naming and capitalization conventions for keys and values. Creating a service for this pod is covered in the next lecture.

Create a service manifest file to expose the pod. Referring to sample service manifests in the Kubernetes documentation, we write a service.yml file with apiVersion, kind as Service, a service name in metadata, and a spec with port 80, targetPort 80 and the LoadBalancer type. With the pod label commented out, we create the pod and the service using kubectl apply -f and check them with kubectl get all. The load balancer's instances show as out of service, because the service doesn't know which pod to forward requests to. We then understand how labels on pods and selectors on services connect them, which we add in the next lecture.

Connect the service to the pod using labels and selectors. Since the load balancer's backend instances are out of service, the service doesn't know where to forward requests. We add the label to the pod manifest and the same label under selector in the service manifest, then update both with kubectl apply -f, which updates the existing configuration without recreating the resources. Using kubectl describe service, we confirm the service now shows the pod as its endpoint, and verify the pod IP with kubectl get pod -o wide. Finally, we access the application in the browser through the load balancer.

Understand why deployments are used instead of pods. After deleting the demo pod, we see that Kubernetes doesn't recreate it by default, then clean up the cluster by deleting the demo service. We then walk through the deployment file written for the register app, which uses apps/v1 and kind Deployment, with a pod template and container definition that always pulls the latest image, exposes the container port and uses matchLabels to create the replica set. We also cover the RollingUpdate strategy, which replaces pods one at a time with the new image to avoid downtime. Finally, we review the service file, making sure its selector matches the deployment label and its target port matches the container port.

Create and access the register app on Kubernetes using the deployment and service files from the Simple DevOps Project repository. We update the replicas to three, apply the deployment with kubectl apply -f and confirm the pods, deployment and replica set are running, checking which node each pod runs on with kubectl get pod -o wide. We then apply the service file, whose selector matches the deployment label, and use kubectl describe to verify the load balancer and pod endpoints. After accessing the application at port 8080 /webapp through the load balancer, we delete a pod and watch the replica set create a new one to keep three pods running.

Integrate the Kubernetes bootstrap server with Ansible so Ansible can manage deployments. On the bootstrap server, we create an ansadmin user, add it to the sudoers file, enable password-based authentication and set its password. On the Ansible server, we rename the existing playbooks in /opt/docker to create_image_regapp.yml and docker_deployment.yml for clarity. We then create a local hosts inventory file with a kubernetes group for the bootstrap server, an ansible group for the Ansible server and localhost, and copy the SSH keys to the bootstrap server with ssh-copy-id. Finally, we test connectivity by checking the uptime of all systems using ansible -i hosts.

Create Ansible playbooks to apply the Kubernetes deployment and service files. On the Ansible server, we write kube_deploy.yml and a service playbook targeting the kubernetes group, each using the command module to run kubectl apply -f on the respective manifest file on the bootstrap server. After deleting the existing deployment and service, we run the playbook and find it can't locate the file, so we update both playbooks to run as the root user instead of become. We then set a password for the root user on the bootstrap server and copy the ansadmin SSH keys to it with ssh-copy-id. Running the playbooks again creates the pods and the load balancer service on the Kubernetes cluster.

Create a Jenkins job to run the Ansible playbooks that deploy on Kubernetes. We create a freestyle project named Deploy on Kubernetes that skips source code management and uses Send build artifacts over SSH to the Ansible server, running the kube_deploy and kube_service playbooks with their full paths, separated by a semicolon. After deleting the existing deployment and service, we build the job and confirm the deployment, service and three pods are created. We then merge the service task into the deployment playbook, update the job to run a single playbook and rebuild it successfully, with a note that pods are not replaced when the files are unchanged.

Create a CI job that builds the code and creates the latest image for Kubernetes deployments. We rename the existing deployment job as the RegApp CD Job, then create a RegApp CI Job by copying the copy artifacts onto Ansible job. The CI job pulls code from GitHub, builds it with Maven, copies the WAR file to /opt/docker on the Ansible server and runs the create_image_regapp.yml playbook to build the image and push it to Docker Hub, with the Docker deployment steps removed. We disable Poll SCM on the older job, push an index.jsp change to GitHub and confirm the CI job updates the WAR file, image and Docker Hub repository. Linking the CI and CD jobs is covered in the next lecture.

Integrate the CI and CD jobs and enable rolling updates on Kubernetes. After recapping the pipeline, we add a Build other projects post-build action to the RegApp CI Job to trigger the RegApp CD Job only when the build is stable. Pushing an index.jsp change to GitHub triggers both jobs and pushes the latest image to Docker Hub, but the pods are not replaced since the deployment and service files are unchanged. To fix this, we add a task using the kubectl rollout command, referenced from the Simple DevOps Project repository, to the deployment playbook with the correct deployment name, so pods are updated when a new image is available. Testing the complete pipeline with a new commit is covered in the next lecture.

Validate the complete CI/CD pipeline on Kubernetes with the final commits. We walk through what to monitor as the pipeline runs: the new commit in GitHub, the RegApp CI Job replacing the WAR file on the Ansible server and pushing the latest image to Docker Hub, and the RegApp CD Job replacing the pods. After pushing a header change to index.jsp, both jobs run automatically and the rolling update replaces the pods with the new version, visible in the browser. A final footer change confirms the rolling update replaces pods one at a time while the remaining pods keep serving the application.

Learn how to clean up the environment after completing the course. From the bootstrap server, we delete the deployment and service using kubectl delete, then delete the EKS cluster with the eksctl delete cluster command and region, as documented in the Simple DevOps Project repository, which terminates the worker nodes. We then check volumes in the AWS console and terminate the remaining EC2 instances directly from the console.