
This is an introduction to this course and I will briefly go through the topics which will be covered throughout this course.
I will shortly explain how to start Kubernetes cluster in AWS. I will provide an example how to start Kubenretes cluster with kops binary. There are now other ways how to start your K8s cluster for example by using kubeadm or some oter ways. However, from the simplicity perspective I believe that using a combination of:
1) kops which generates terraform files to a folder
2) and then starting up your K8s cluster by running terraform init && terraform apply
In case you want to stop your cluster throughout the night or whatever simply run: terraform destroy
If you again need to start your cluster then simply run terraform apply and your cluster is again up.
Hosted Zone is a prerequisite for staring up Kubernetes cluster in AWS and naturally it's is also one of the flags you need to specify when using kops command to generate terraform code. New Hosted Zone can be easily created in Route53 section in AWS.
I will explain the importance of aws binary when we are setting up Kubernetes cluster via kops and terraform and why it is actually needed in the initial phase in this lecture.
I will guide you through the process of installing kops binary in this lecture. You can go at the kops web page and follow the instructions how to install kops or you can use a simple shell function I have prepared in form of materials for this lecture.
I will explain why we need to create S3 bucket in AWS when we want to use kops and terraform to start/stop and take care about our Kubernetes cluster.
The main goal of this lecture is to show you guys how to install terraform binary which is in fact very easy. I have also prepared some trivial bash function to install terraform if you are using LINUX OS.
I will explain how to install kubectl binary to your PC and I will also explain the importance of this binary when you want to orchestrate/control your Kubernetes cluster. This command is actually used for orchestrating Kubernetes cluster in general and it is not something dedicated to AWS. You need to use kubectl every time you want to interact with your Kubernetes cluster no matter which cloud you or on-premise solution you are using.
This lecture explains how to start your Kubernetes cluster in AWS for the first time by running kops and terraform binary. We also need to generate SSH keys as a last part before starting up our Kubernetes cluster.
The purpose of this lecture is to start a Jupyter Notebooks as a single Docker image.
This will help you in future to understand what is the difference between running Docker containers "manually"
and when we let run the same Docker container to Kubernetes orchestration platform.
If your use case is simply running one Docker image in your infrastructure then you can write some
custom scripts to take care about some critical scenarios. However, most of us have much complex infrastructural and we most likely need to run tens of containers. In such a case Kubernetes will take care about your containers much better than taking care about then ourselves.
In this lecture I will describe Jupyter Notebooks deployment in a simple yaml file. More specifically, I will talk about Deployment and Service Kubernetes objects. These will be created in K8s cluster and as a result we will be able to access and use Jupyter Notebooks in AWS.
Install helm v3
curl --output /tmp/helm-v3.1.1-linux-amd64.tar.gz -L https://get.helm.sh/helm-v3.1.1-linux-amd64.tar.gz
sudo tar -xvf /tmp/helm-v3.1.1-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin/ linux-amd64/helm
sudo mv /usr/local/bin/helm /usr/local/bin/helm3
sudo chmod +x /usr/local/bin/helm3
# In case you have no helm chart repository added
helm3 repo add stable https://kubernetes-charts.storage.googleapis.com/
Verify your helm chart repository repo helm v3
helm3 repo update
helm3 repo list
Install helmfile
sudo curl -L --output /usr/bin/helmfile https://github.com/roboll/helmfile/releases/download/v0.104.0/helmfile_linux_amd64
sudo chmod +x /usr/bin/helmfile
# Create symbolic link from helm3 to helm
ln -s /usr/local/bin/helm3 /usr/bin/helm
!!! This section is only applicable if you want to use helm v2 Install helm v2
curl -L --output /tmp/helm-v2.16.5-linux-amd64.tar.gz https://get.helm.sh/helm-v2.16.5-linux-amd64.tar.gz
sudo tar -xvf /tmp/helm-v2.16.5-linux-amd64.tar.gz --strip-components=1 -C /usr/bin/ linux-amd64/helm
sudo chmod +x /usr/bin/helm
!!! This section is only applicable if you want to use helm v2
helm version
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller --upgrade
helm ls
In this lecture I will explain some basics of HELM charts and I will briefly describe why we are using helm charts. I will show you guys how to install helm binary which is essential if we want to be able to use HELM CHARTS and process deployments to our Kubernetes cluster. I will also explain tiller pod deployment to kube-system namespace in our Kubernetes cluster.
Try to close SSH tunnel
ssh -L31412:127.0.0.1:31412 \
-i ~/.ssh/udemy_devopsinuse \
admin@18.184.212.193
Determine the IP addresses/names/roles of your physical EC2 instances (servers) in AWS
kubectl get nodes -o wide | awk -F" " '{print $3"\t"$1"\t"$7}'
ROLES NAME EXTERNAL-IP
master ip-172-20-34-241.eu-central-1.compute.internal 18.184.212.193
node ip-172-20-50-50.eu-central-1.compute.internal 3.120.179.150
node ip-172-20-52-232.eu-central-1.compute.internal 18.196.157.47
List all deployments
helm3 ls -A
Before upgrade
Do some changes in example/values.yaml file in HTML section
Upgrade deployment
helm3 upgrade example helm-charts/example \
--set service.type=NodePort \
--set service.nodePortValue=31412
Explore configmap for example helm chart
kubectl get cm example -o yaml
apiVersion: v1
data:
index.html: "<!DOCTYPE html>\n\n<html lang=\"en\">\n<head>\n ... </p>\n </div>\n </div>\n </div>\n</div>\n</body>\n</html>"
kind: ConfigMap
metadata:
creationTimestamp: "2020-04-03T19:21:26Z"
labels:
app.kubernetes.io/instance: example
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: example
app.kubernetes.io/version: 1.16.0
helm.sh/chart: example-0.1.0
name: example
namespace: default
resourceVersion: "398879"
selfLink: /api/v1/namespaces/default/configmaps/example
uid: b2e95666-8e4b-4ca5-a8ea-60fe3cc3e6fb
Delete Nginx pod to load new content from updated configmap
kubectl delete pod example-7cb6767455-f84p6
Delete helm chart example deployment
helm3 delete example
I will explore HELM CHARTS in github.com web page. I will talk about where to find the HELM CHARTS. I will also explain how to use helm binary to use HELM CHARTS to process the actual deployment. I will use a simple comparison between installing apps to Kubernetes and installing packages to OS. Gogs will be used as an example for this purpose. I will also clone all the charts from github.com
In this lecture I will explain how to understand helm Gogs deployment which was triggered in a previous lecture. Gogs has dependency at PostgreSQL so I will try to explain the deployment if one HELM CHART (Gogs) is dependent at some other HELM CHART (PostgreSQL). I will go through created kubernetes objects within this deployment. I will also set up Security Group in AWS.
In this lecture I will demonstrate how to use HELM CHART from locally downloaded HELM CHARTS. I will be creating files and directory structure for such a deployment. I will slightly customize some values in both Gogs and PostgreSQL HELM CHART and use helm binary to process deployment to our Kubernetes cluster.
# open up a new HTTP tunnel for port 30222
ssh -L30222:127.0.0.1:30222 \
-i ~/.ssh/udemy_devopsinuse admin@18.184.212.193
# open up a new SSH tunnel for port 30111
ssh -L30111:127.0.0.1:30111 \
-i ~/.ssh/udemy_devopsinuse admin@18.184.212.193
# Create SSH tunnel to avoid opening
# of an extra nodePort: 31412 for "example" helm chart
ssh -L31412:127.0.0.1:31412 \
-i ~/.ssh/udemy_devopsinuse \
admin@18.184.212.193
In case you like to set up NodePorts for your Kubernetes deploymnt better in AWS web console in the section of Security group feel free to do so. In such a case you can skip pretty much all SSH tunnels and instead please us one of the IP Addresses of your Kubenretes nodes.
To retrieve IP Addresses of your physical EC2 instances within your Kubenretes cluster - run this command:
kubectl get nodes -o wide | awk -F" " '{print $3"\t"$1"\t"$7}'
ROLES NAME EXTERNAL-IP
master ip-172-20-34-241.eu-central-1.compute.internal 18.184.212.193
node ip-172-20-50-50.eu-central-1.compute.internal 3.120.179.150
node ip-172-20-52-232.eu-central-1.compute.internal 18.196.157.47
Search for MySQL helm chart (helm v3):
helm3 repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com/
incubator https://kubernetes-charts-incubator.storage.googleapis.com/
helm3 search repo stable/mysql -l | head
NAME CHART VERSION APP VERSION DESCRIPTION
stable/mysql 1.6.2 5.7.28 Fast, reliable, scalable, and easy to use open-..
Try to use helm3 template command to see what you are about to be deploying to your Kubernetes cluster in AWS
# template your mysql helm chart
# before you going to deploy it to Kubenretes cluster
helm3 template \
mysql \
--set persistence.enabled=true \
--set persistence.size=1Gi \
--set mysqlRootPassword=Start123 \
stable/mysql | less
Deploy MySQL helm chart from a "stable" helm chart repository
helm3 install \
mysql \
--set persistence.enabled=true \
--set persistence.size=1Gi \
--set mysqlRootPassword=Start123 \
stable/mysql
...
To get your root password run:
MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)
To connect to your database:
1. Run an Ubuntu pod that you can use as a client:
kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il
2. Install the mysql client:
$ apt-get update && apt-get install mysql-client -y
3. Connect using the mysql cli, then provide your password:
$ mysql -h mysql -p
To connect to your database directly from outside the K8s cluster:
MYSQL_HOST=127.0.0.1
MYSQL_PORT=3306
# Execute the following command to route the connection:
kubectl port-forward svc/mysql 3306
mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}
aws ec2 describe-volumes --profile terraform | jq .Volumes | jq '.[].Tags'
...
[
{
"Key": "kubernetes.io/created-for/pv/name",
"Value": "pvc-1b0192ec-8504-4742-9e78-fbae63a7a1e3"
},
{
"Key": "kubernetes.io/created-for/pvc/name",
"Value": "mysql"
},
{
"Key": "kubernetes.io/cluster/course.devopsinuse.com",
"Value": "owned"
},
{
"Key": "KubernetesCluster",
"Value": "course.devopsinuse.com"
},
{
"Key": "Name",
"Value": "course.devopsinuse.com-dynamic-pvc-1b0192ec-8504-4742-9e78-fbae63a7a1e3"
},
{
"Key": "kubernetes.io/created-for/pvc/namespace",
"Value": "default"
}
]
To get your root password run:
MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)
To connect to your database:
Run an Ubuntu pod that you can use as a client:
kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il
Install the mysql client:
$ apt-get update && apt-get install mysql-client -y
Connect using the mysql cli, then provide your password:
$ mysql -h mysql -p
$ create database devopsinuse;
Install helmfile binary if you have not done so
sudo curl -L --output /usr/bin/helmfile https://github.com/roboll/helmfile/releases/download/v0.104.0/helmfile_linux_amd64
sudo chmod +x /usr/bin/helmfile
# Create symbolic link from helm3 to helm
ln -s /usr/local/bin/helm3 /usr/bin/helm
Define your helmfile specification for "example" helm chart deployment to your Kubernetes cluster file: helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml
repositories:
# To use official "stable" charts
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# "example" helm chart release specification
- name: example
labels:
key: example
app: nginx
chart: ../helm-charts/example
version: 0.1.0
set:
- name: service.type
value: NodePort
- name: service.nodePortValue
value: 31412
# Example helm chart (Nginx Web Server)
# helm3 install example helm-charts/example \
# --set service.type=NodePort \
# --set service.nodePortValue=31412
# "Gogs" helm chart release specification
- name: test
labels:
key: gogs
app: gogs
chart: ../helm-charts/gogs
version: 0.7.11
set:
- name: service.httpNodePort
value: 30222
- name: service.sshNodePort
value: 30111
# Gogs helm chart via helm v3
# helm3 install test \
# --set service.httpNodePort=30222 \
# --set service.sshNodePort=30111 .
Do not forget to create SSH tunnel to open up NodePort values
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 31412 (Nginx Web server)
# - 30111 (SSH)
# - 30222 (HTTP)
ssh \
-L31412:127.0.0.1:31412 \
-L30111:127.0.0.1:30111 \
-L30222:127.0.0.1:30222 \
-i ~/.ssh/udemy_devopsinuse \
admin@35.158.122.228
Alternatively you can allow this port 31412 in "Security group" section in your AWS console
Deploy "example" and "gogs" helm charts via helmfile to your Kubernetes cluster in AWS
export HELMFILE_ENVIRONMENT="learning"
# deploy without usig --selector flag
helmfile \
--environment learning \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml sync
# deploy "example" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector key=example \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml sync
# deploy "example" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector app=nginx \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml sync
# deploy "Gogs" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector key=gogs \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml sync
# deploy "Gogs" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector app=gogs \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml sync
Explore helmfile template command for "example" helm chart deployment
# template "example" helm chart via helmfile
export HELMFILE_ENVIRONMENT="learning"
# template without usig --selector flag
helmfile \
--environment learning \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml template | less
# template "example" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector key=example \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml template | less
# template "example" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector app=nginx \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml template | less
# template "Gogs" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector key=gogs \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml template | less
# template "Gogs" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector app=gogs \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml template | less
Please check current releases deployed in your Kubernetes cluster in AWS
helm3 ls -A
Destroy "gogs" and "example" helm charts via helmfile from your Kubernetes cluster in AWS
export HELMFILE_ENVIRONMENT="learning"
# destroy all without usig --selector flag
helmfile \
--environment learning \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml destroy
# destroy "example" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector key=example \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml destroy
# destroy "example" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector app=nginx \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml destroy
# destroy "Gogs" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector key=gogs \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml destroy
# destroy "Gogs" helm chart via helmfile using --selector flag
helmfile \
--environment learning \
--selector app=gogs \
--file helmfiles/helmfile-for-example-and-gogs-helm-charts.yaml destroy
Explore helmfiles/helmfile-for-mysql-helm-chart.yaml helmfile for MySQL deployment to Kubernetes
repositories:
# To use official "stable" charts
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# "example" helm chart release specification
- name: mysql
labels:
key: database
app: mysql
chart: stable/mysql
version: 1.6.2
set:
- name: service.type
value: NodePort
- name: service.nodePort
value: 30333
- name: mysqlRootPassword
value: Start123
- name: persistence.enabled
value: true
- name: persistence.size
value: 1Gi
Compare it with an original helm3 command used to deploy “mysql” helm chart to your Kubernetes cluster in AWS
helm3 install mysql stable/mysql \
--set mysqlRootPassword=Start123 \
--set persistence.enabled=true \
--set persistence.size=1Gi \
--set service.type=NodePort \
--set service.nodePort=30333
Helmfile template mysql helm chart deployment
helmfile --environment learning \
--file helmfiles/helmfile-for-mysql-helm-chart.yaml template
Helmfile deploy mysql helm chart deployment
helmfile --environment learning \
--file helmfiles/helmfile-for-mysql-helm-chart.yaml sync
Helmfile destroy mysql helm chart deployment
helmfile --environment learning \
--file helmfiles/helmfile-for-mysql-helm-chart.yaml destroy
Establish SSH tunnel to open up NodePort value for MySQL
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30333 (MySQL)
ssh \
-L30333:127.0.0.1:30333 \
-i ~/.ssh/udemy_devopsinuse \
admin@35.158.122.228
Create helm v3 helm chart repository at your Github repository
git clone https://github.com/xjantoth/helmfile-course.git
cd helmfile-course
mkdir -p docs/hc-v3-repo
helm3 repo add stable https://kubernetes-charts.storage.googleapis.com/
helm3 repo update
helm3 search repo stable/jenkins -l | head -n 2
NAME CHART VERSION APP VERSION DESCRIPTION
stable/jenkins 1.11.3 lts Open source continuous integration server. It s...
# Fetch Jenkins helm chart from a "stable" helm chart repo
helm3 fetch stable/jenkins --destination docs/hc-v3-repo/
ls -l docs/hc-v3-repo/
helm3 repo index docs/hc-v3-repo
# git add .; git commit -m "..." and git push your changes to remote
git add docs/hc-v3-repo
git commit -m "Creating helm v3 chart repository docs/hc-v3-repo"
git push
Add hc-v3-repo to your local helm chart repositories
helm3 repo add hc-v3-repo https://xjantoth.github.io/helmfile-course/hc-v3-repo
helm3 repo update
helm3 repo list
helm3 search repo hc-v3-repo/
Run: export HELMFILE_ENVIRONMENT="learning"
repositories:
# To use Github helm chart repo
- name: hc-v3-repo
url: https://xjantoth.github.io/helmfile-course/hc-v3-repo
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# "jenkins" helm chart release specification
- name: jenkins
labels:
key: ci
app: jenkins
chart: hc-v3-repo/jenkins
version: 1.11.3
set:
- name: master.serviceType
value: NodePort
- name: master.nodePort
value: 30555
- name: master.adminUser
value: "devopsinuse"
- name: master.adminPassword
value: "Start123"
- name: persistence.enabled
value: true
- name: persistence.size
value: 1Gi
- name: agent.enabled
value: false
Template jenkins deployment via helmfile to your Kubernetes cluster in AWS
export HELMFILE_ENVIRONMENT="learning"
helmfile \
--environment "learning" \
-f helmfiles/helmfile-for-jenkins.yaml \
template
Establish SSH tunnel to open up NodePort value for Jenkins
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30555 (Jenkins)
ssh \
-L30555:127.0.0.1:30555 \
-i ~/.ssh/udemy_devopsinuse \
admin@35.158.122.228
Execute jenkins deployment via helmfile to your Kubernetes cluster in AWS
export HELMFILE_ENVIRONMENT="learning"
helmfile \
--environment "learning" \
-f helmfiles/helmfile-for-jenkins.yaml \
sync
Destroy jenkins deployment via helmfile from your Kubernetes cluster in AWS
export HELMFILE_ENVIRONMENT="learning"
helmfile \
--environment "learning" \
-f helmfiles/helmfile-for-jenkins.yaml \
destroy
file: helmfiles/helmfile-for-chartmuseum.yaml
repositories:
# To use official "stable" charts
# a.k.a https://github.com/helm/charts/tree/master/stable
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
# This is helm chart repository made of Chartmuseum
# which is running as regular deployment within our cluster
#- name: k8s
# url: http://127.0.0.1:30444/chartmuseum
# username: devopsinuse
# password: Start123
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# (Helm v3) Upgrade your deployment with basic auth
- name: chartmuseum
labels:
key: chartmuseum
app: chartmuseum
chart: stable/chartmuseum
version: 2.10.0
set:
- name: service.type
value: NodePort
- name: service.nodePort
value: 30444
- name: persistence.pv.enabled
value: false
- name: env.open.DISABLE_API
value: false
- name: env.open.CONTEXT_PATH
value: "/chartmuseum"
- name: ingress.enabled
value: true
- name: ingress.hosts[0].name
value: "*"
- name: ingress.hosts[0].path
value: "/chartmuseum"
- name: env.secret.BASIC_AUTH_USER
value: "devopsinuse"
- name: env.secret.BASIC_AUTH_PASS
value: "Start123"
Compare helm deployment for Chartmuseum via helm3 binary
# (Helm v3) Upgrade your deployment with basic auth
helm3 install \
chartmuseum \
--set persistence.pv.enabled=false \
--set env.open.DISABLE_API=false \
--set env.open.CONTEXT_PATH="/chartmuseum" \
--set ingress.enabled=true \
--set ingress.hosts[0].name="chartmuseum" \
--set ingress.hosts[0].path="/chartmuseum" \
--set env.secret.BASIC_AUTH_USER="devopsinuse" \
--set env.secret.BASIC_AUTH_PASS="Start123" \
--set service.type=NodePort \
--set service.nodePort=30444 \
stable/chartmuseum
Establish SSH tunnel to open up NodePort value for Chartmuseum
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30444 (Chartmuseum)
ssh \
-L30444:127.0.0.1:30444 \
-i ~/.ssh/udemy_devopsinuse \
admin@35.158.122.228
# template only chartmuseum via helmfile using --selector flag
helmfile \
--selector key=chartmuseum \
--environment learning \
--file helmfiles/helmfile-for-chartmuseum.yaml \
template
# deploy Chartmuseum via helmfile
helmfile \
--selector key=chartmuseum \
--environment "learning" \
-f helmfiles/helmfile-for-chartmuseum.yaml \
sync
# destroy Chartmuseum via helmfile if neceassary
helmfile \
--selector key=chartmuseum \
--environment "learning" \
-f helmfiles/helmfile-for-chartmuseum.yaml \
destroy
Fetch two helm charts Grafana, Prometheus to your local
helm3 fetch stable/grafana --destination docs/hc-v3-repo/
helm3 fetch stable/prometheus --destination docs/hc-v3-repo/
ls docs/hc-v3-repo/
total 108
-rw-r--r-- 1 1183 Apr index.yaml
-rw-r--r-- 1 46226 Apr jenkins-1.11.3.tgz
-rw-r--r-- 1 32899 Apr prometheus-11.0.6.tgz
-rw-r--r-- 1 19523 Apr grafana-5.0.11.tgz
Push helm chart to Chartmuseum with authentication
curl -u devopsinuse -XPOST --data-binary "@docs/hc-v3-repo/grafana-5.0.11.tgz" http://127.0.0.1:30444/chartmuseum/api/charts
{"saved":true}
curl -u devopsinuse -XPOST --data-binary "@docs/hc-v3-repo/prometheus-11.0.6.tgz" http://127.0.0.1:30444/chartmuseum/api/charts
{"saved":true}
Add Chartmuseum to the list of available helm chart repsitories
helm3 repo add k8s http://127.0.0.1:30444/chartmuseum --username devopsinuse --password Start123
helm3 repo update
helm3 search repo k8s/
# List all the helm chart present in Chartmuseum via API
curl -u devopsinuse -XGET http://127.0.0.1:30444/chartmuseum/api/charts
Delete (Helm v3) Chartmuseum deployment
helm3 delete chartmuseum
file: helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml
repositories:
# To use official "stable" charts
# a.k.a https://github.com/helm/charts/tree/master/stable
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
# This is helm chart repository made of Chartmuseum
# which is running as regular deployment within our cluster
- name: k8s
url: http://127.0.0.1:30444/chartmuseum
username: devopsinuse
password: Start123
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# (Helm v3) Upgrade your deployment with basic auth
- name: grafana
labels:
key: monitoring
app: grafana
#chart: k8s/grafana
chart: k8s/grafana
version: 5.0.11
set:
- name: service.type
value: NodePort
- name: service.nodePort
value: 30888
# Change context path for grafana to /grafana
- name: "grafana\\.ini.server.root_url"
value: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
- name: "grafana\\.ini.server.serve_from_sub_path"
value: true
# Ingress related settings
#- name: ingress.enabled
# value: true
#- name: ingress.hosts[0]
# value: "devopsinuse"
#- name: "ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target"
# value: "\\/$1"
#- name: ingress.path
# value: "/grafana/?(.*)"
# ./prometheus --config.file=prometheus.yml \
# --web.external-url http://localhost:19090/prometheus/ \
# --web.route-prefix=/prometheus
- name: prometheus
labels:
key: monitoring
app: prometheus
# chart: k8s/prometheus
chart: k8s/prometheus
version: 11.0.6
set:
# Modify service type to NodePort
- name: server.service.type
value: NodePort
- name: server.service.nodePort
value: 30999
# Disable Persistent data
- name: server.persistentVolume.enabled
value: false
# Disable extra Prometheus components
- name: pushgateway.enabled
value: false
- name: kubeStateMetrics.enabled
value: false
- name: alertmanager.enabled
value: false
# Change default / to /prometheus in runtime
- name: server.baseURL
value: "http://localhost:9090/prometheus"
- name: server.prefixURL
value: "/prometheus"
values:
- server:
extraArgs:
"web.route-prefix": "/prometheus"
# Ingress settings
#- name: server.ingress.enabled
# value: true
#- name: server.ingress.hosts[0]
# value: "devopsinuse/prometheus"
Template helm chart deployments with/without using --selectors
export HELMFILE_ENVIRONMENT="learning"
ssh \
-L30444:127.0.0.1:30444 \
-i ~/.ssh/udemy_devopsinuse \
admin@35.158.122.228
# template grafana, prometheus via helmfile
helmfile \
--environment learning \
--file helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
template
# template grafana, prometheus via helmfile
helmfile \
--selector key=monitoring \
--environment learning \
--file helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
template
# template grafana via helmfile
helmfile \
--selector app=grafana \
--environment learning \
--file helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
template
# template prometheus via helmfile
helmfile \
--selector app=prometheus \
--environment learning \
--file helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
template
Create SSH tunnel to open up NodePort values for Grafana and Prometheus deployment via helmfile
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30444 (Chartmuseum k8s helm chart repository)
# - 30888 (Grafana)
# - 30999 (Prometheus)
ssh \
-L30444:127.0.0.1:30444 \
-L30888:127.0.0.1:30888 \
-L30999:127.0.0.1:30999 \
-i ~/.ssh/udemy_devopsinuse \
admin@18.197.49.166
Alternatively you can allow this ports 30888, 30999 in "Security group" section in your AWS console
Deploy helm chart deployments with/without using --selectors
# deploy grafana, prometheus via helmfile
helmfile \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
# deploy grafana, prometheus via helmfile
helmfile \
--selector key=monitoring \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
# deploy grafana via helmfile selectively via --selector app=grafana
helmfile \
--selector app=grafana \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
syncCreate SSH tunnel to open up NodePort values for Grafana and Prometheus deployment via helmfile
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30444 (Chartmuseum k8s helm chart repository)
# - 30888 (Grafana)
# - 30999 (Prometheus)
ssh \
-L30444:127.0.0.1:30444 \
-L30888:127.0.0.1:30888 \
-L30999:127.0.0.1:30999 \
-i ~/.ssh/udemy_devopsinuse \
admin@18.197.49.166
Alternatively you can allow this ports 30888, 30999 in "Security group" section in your AWS console
Deploy helm chart deployments with/without using --selectors
# deploy grafana, prometheus via helmfile
helmfile \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
# deploy grafana, prometheus via helmfile
helmfile \
--selector key=monitoring \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
# deploy grafana via helmfile selectively via --selector app=grafana
helmfile \
--selector app=grafana \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
# deploy prometheus via helmfile selectively via --selector app=prometheus
helmfile \
--selector app=prometheus \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
# deploy prometheus via helmfile selectively via --selector app=prometheus
helmfile \
--selector app=prometheus \
--environment "learning" \
-f helmfiles/helmfile-for-grafana-prometheus-from-chartmuseum.yaml \
sync
file: helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml
repositories:
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
# This is helm chart repository made of Chartmuseum which is running as regular deployment within our cluster
- name: k8s
url: http://127.0.0.1:30444/chartmuseum
username: devopsinuse
password: Start123
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# (Helm v3) Upgrade your deployment with basic auth
- name: grafana
labels:
key: monitoring
app: grafana
#chart: k8s/grafana
chart: k8s/grafana
version: 5.0.11
set:
#- name: service.type
# value: NodePort
#- name: service.nodePort
# value: 30888
# Change context path for grafana to /grafana
- name: "grafana\\.ini.server.root_url"
value: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
- name: "grafana\\.ini.server.serve_from_sub_path"
value: true
# Ingress related settings for Grafana
- name: ingress.enabled
value: {{ index .Environment.Values "grafana" "ingress.enabled" }}
- name: ingress.hosts[0]
value: {{ index .Environment.Values "grafana" "ingress.hosts[0]" }}
- name: "ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target"
value: {{ index .Environment.Values "grafana" "ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target" }}
- name: ingress.path
value: {{ index .Environment.Values "grafana" "ingress.path" }}
# ./prometheus --config.file=prometheus.yml --web.external-url http://localhost:19090/prometheus/ --web.route-prefix=/prometheus
- name: prometheus
labels:
key: monitoring
app: prometheus
chart: k8s/prometheus
version: 11.0.6
set:
# Modify service type to NodePort
#- name: server.service.type
# value: NodePort
#- name: server.service.nodePort
# value: 30999
# Disable Persistent data
- name: server.persistentVolume.enabled
value: false
# Disable extra Prometheus components
- name: pushgateway.enabled
value: false
- name: kubeStateMetrics.enabled
value: false
- name: alertmanager.enabled
value: false
# Ingress settings for Prometheus
- name: server.ingress.enabled
value: {{ index .Environment.Values "prometheus" "server.ingress.enabled" }}
- name: server.ingress.hosts[0]
value: {{ index .Environment.Values "prometheus" "server.ingress.hosts[0]" }}
# Change default / to /prometheus in runtime
- name: server.baseURL
value: "http://localhost:9090/prometheus"
- name: server.prefixURL
value: "/prometheus"
values:
- server:
extraArgs:
"web.route-prefix": "/prometheus"
# nginx-ingress deployment
- name: nginx-ingress
labels:
key: proxy
app: nginx-ingress
#chart: k8s/grafana
chart: stable/nginx-ingress
version: 1.36.0
set:
- name: controller.service.type
value: NodePort
- name: controller.service.nodePorts.http
value: 30777
file: helmfiles/values.yaml
grafana:
# Ingress related settings for Grafana
ingress.enabled: true
ingress.hosts[0]: "devopsinuse"
"ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target": "\\/$1"
ingress.path: "/grafana/?(.*)"
prometheus:
# Ingress settings for Prometheus
server.ingress.enabled: true
server.ingress.hosts[0]: "devopsinuse/prometheus"
Make sure you have Grafana and Prometheus helm charts in Chartmuseum
curl -u devopsinuse -XPOST --data-binary "@docs/hc-v3-repo/grafana-5.0.11.tgz" http://127.0.0.1:30444/chartmuseum/api/charts
{"saved":true}
curl -u devopsinuse -XPOST --data-binary "@docs/hc-v3-repo/prometheus-11.0.6.tgz" http://127.0.0.1:30444/chartmuseum/api/charts
{"saved":true}
helm3 repo update
helm3 search repo k8s/
Create SSH tunnel to open up NodePort values for Grafana and Prometheus deployment via helmfile
export HELMFILE_ENVIRONMENT="learning"
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30444 (Chartmuseum k8s helm chart repository)
# - 30777 (Nginx Ingress Controller)
ssh \
-L30444:127.0.0.1:30444 \
-L30777:127.0.0.1:30777 \
-i ~/.ssh/udemy_devopsinuse \
admin@18.197.49.166
Alternatively you can allow this ports 30777, 30444 in “Security group” section in your AWS console
Template nginx-ingress deployment
export HELMFILE_ENVIRONMENT="learning"
# template nginx-ingress, grafana, prometheus
helmfile \
--selector app=grafana \
--selector app=prometheus \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
template
# template grafana
helmfile \
--selector app=grafana \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
template
# template prometheus
helmfile \
--selector app=prometheus \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
template
# template nginx-ingress
helmfile \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
template
Deploy nginx-ingress, grafana, prometheus deployment
# sync nginx-ingress, grafana, prometheus
helmfile \
--selector app=grafana \
--selector app=prometheus \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
sync
# sync grafana
helmfile \
--selector app=grafana \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
sync
# sync prometheus
helmfile \
--selector app=prometheus \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
sync
# sync nginx-ingress
helmfile \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-from-chartmuseum.yaml \
sync
Retrive Grafana admin password
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
cat /etc/hosts
# Static table lookup for hostnames.
# See hosts(5) for details.
#
#
...
127.0.0.1 devopsinuse
...
Take a look to Nginx Ingress controller from a running pod
kubectl get pods | grep nginx
nginx-ingress-controller-794878944b-cvp7d 1/1 Running 0 14m
nginx-ingress-default-backend-5b967cf596-qm595 1/1 Running 0 14m
kubectl exec -it \
nginx-ingress-controller-794878944b-cvp7d \
-- cat /etc/nginx/nginx.conf > nginx-ingress-cong-from-running-pod.conf
file: helmfiles/helmfile-for-grafana-prometheus-nginx-loadbalancer-from-chartmuseum.yaml
repositories:
# To use official "stable" charts
# a.k.a https://github.com/helm/charts/tree/master/stable
- name: stable
url: https://kubernetes-charts.storage.googleapis.com
# This is helm chart repository made of Chartmuseum
# which is running as regular deployment within our cluster
- name: k8s
url: http://127.0.0.1:30444/chartmuseum
username: devopsinuse
password: Start123
# Export your environment e.g "learning", "dev", ..., "prod"
# export HELMFILE_ENVIRONMENT="learning"
environments:
{{ requiredEnv "HELMFILE_ENVIRONMENT" }}:
values:
- values.yaml
releases:
# (Helm v3) Upgrade your deployment with basic auth
- name: grafana
labels:
key: monitoring
app: grafana
#chart: k8s/grafana
chart: k8s/grafana
version: 5.0.11
set:
#- name: service.type
# value: NodePort
#- name: service.nodePort
# value: 30888
# Change context path for grafana to /grafana
- name: "grafana\\.ini.server.root_url"
value: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
- name: "grafana\\.ini.server.serve_from_sub_path"
value: true
# Ingress related settings for Grafana
- name: ingress.enabled
value: {{ index .Environment.Values "grafana" "ingress.enabled" }}
- name: ingress.hosts[0]
value: {{ index .Environment.Values "grafana" "ingress.hosts[0]" }}
- name: ingress.hosts[1]
value: {{ index .Environment.Values "grafana" "ingress.hosts[1]" }}
- name: "ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target"
value: {{ index .Environment.Values "grafana" "ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target" }}
- name: ingress.path
value: {{ index .Environment.Values "grafana" "ingress.path" }}
# ./prometheus --config.file=prometheus.yml \
# --web.external-url http://localhost:19090/prometheus/ \
# --web.route-prefix=/prometheus
- name: prometheus
labels:
key: monitoring
app: prometheus
chart: k8s/prometheus
version: 11.0.6
set:
# Modify service type to NodePort
#- name: server.service.type
# value: NodePort
#- name: server.service.nodePort
# value: 30999
# Disable Persistent data
- name: server.persistentVolume.enabled
value: false
# Disable extra Prometheus components
- name: pushgateway.enabled
value: false
- name: kubeStateMetrics.enabled
value: false
- name: alertmanager.enabled
value: false
# Ingress settings for Prometheus
- name: server.ingress.enabled
value: {{ index .Environment.Values "prometheus" "server.ingress.enabled" }}
- name: server.ingress.hosts[0]
value: {{ index .Environment.Values "prometheus" "server.ingress.hosts[0]" }}
- name: server.ingress.hosts[1]
value: {{ index .Environment.Values "prometheus" "server.ingress.hosts[1]" }}
# Change default / to /prometheus in runtime
- name: server.baseURL
value: "http://localhost:9090/prometheus"
- name: server.prefixURL
value: "/prometheus"
values:
- server:
extraArgs:
"web.route-prefix": "/prometheus"
# nginx-ingress deployment
- name: nginx-ingress
labels:
key: proxy
app: nginx-ingress
#chart: k8s/grafana
chart: stable/nginx-ingress
version: 1.36.0
set:
- name: controller.service.type
value: LoadBalancer
#- name: controller.service.nodePorts.http
# value: 30777
file: helmfiles/values.yaml
grafana:
# Ingress related settings for Grafana
ingress.enabled: true
ingress.hosts[0]: "devopsinuse"
ingress.hosts[1]: "diu.course.devopsinuse.com"
"ingress.annotations.nginx\\.ingress\\.kubernetes\\.io\\/rewrite-target": "\\/$1"
ingress.path: "/grafana/?(.*)"
prometheus:
# Ingress settings for Prometheus
server.ingress.enabled: true
server.ingress.hosts[0]: "devopsinuse/prometheus"
server.ingress.hosts[1]: "diu.course.devopsinuse.com/prometheus"
Create SSH tunnel to open up NodePort values for Grafana and Prometheus deployment via helmfile
export HELMFILE_ENVIRONMENT="learning"
# Create SSH tunnel to avoid opening
# of an extra nodePorts:
# - 30444 (Chartmuseum k8s helm chart repository)
ssh \
-L30444:127.0.0.1:30444 \
-i ~/.ssh/udemy_devopsinuse \
admin@18.197.49.166
Alternatively you can allow this port 30444 in “Security group” section in your AWS console
Template grafana, prometheus and nginx-ingress controller deployments
helmfile \
--selector app=grafana \
--selector app=prometheus \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-loadbalancer-from-chartmuseum.yaml \
template
Deploy Nginx Ingress Controller as LoadBalancer type of service
helmfile \
--selector app=grafana \
--selector app=prometheus \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-loadbalancer-from-chartmuseum.yaml \
sync
Retrive Grafana Password
kubectl get secret \
--namespace default grafana \
-o jsonpath="{.data.admin-password}" \
| base64 --decode ; echo
Destroy your deployment for grafana, prometheus and nginx ingres controller
helmfile \
--selector app=grafana \
--selector app=prometheus \
--selector app=nginx-ingress \
--environment learning \
-f helmfiles/helmfile-for-grafana-prometheus-nginx-loadbalancer-from-chartmuseum.yaml \
destroy
Create A-record diu.course.devopsinuse.com in Route53 in AWS
The main motivation for this course was to provide students a comprehensive explanation of the applications deployment to Kubernetes cluster in AWS by using the standard called helm chart via helm version 3 and helmfile binaries. In order to achieve this goal, the course is using particular HELM CHARTS such as deploy:
Jupyter Notebooks as docker container locally and well as in Kubernetes in AWS,
create helm chart from scratch,
MySQL and Gogs (cool Github like web UI),
Jenkins from your own helm chart repository,
deploy Chartmuseum (helm chart repository),
Grafana and Prometheus from Chartmuseum,
Nginx Ingress Controller as LoadBalancer,
I will try to explain everything in practical context so everybody can put together thoughts accurate to her/his thinking process.