Grafana

The official documentation for this course is hosted for FREE at https://sbcode.net/grafana. I have written it specifically to be used by the students of this course. Please check the documentation links in the resources for each lesson, as there may be some extra notes added to each lesson.

In this course all my examples are executed on an unrestricted Ubuntu 24.04 LTS. You can get minimal Ubuntu 24.04 LTS server from cloud providers. In this course I used Digital Ocean, with this link https://m.do.co/c/23d277be9014, you will normally get $200 credit for 60 days, occasionally the offer changes, but can create and delete as many VMs as you wish during the initial offer period.

Also, you can use the Grafana Cloud option if you want to outsource the managemt of hosting your own Grafana server. Use this link : https://grafana.com/auth/sign-up?refCode=gr83y6yqNccjKnX

Grafana is excellent for monitoring time series data.

In this video, I will install Grafana

After the successful install, you can start the Grafana service

$ sudo service grafana-server start

Your Grafana server will be hosted at

http://[your Grafana server url]:3000

The default login,

Username : admin

Password : admin

Grafana is updated very regularly.

When using the open source version of Grafana, you need to manage updates yourself and at your own risk.

The open source version comes with no promise of backwards compatibility, so upgrading may affect how any dashboards, visualizations, data sources, etc work.

Remember that upgrading is done at your own risk. If you consider that the upgrade was detrimental, you can downgrade back to your previous version by using the same method.

To minimize potential backwards compatibility issues when upgrading, it is better to upgrade 1 patch at a time until you get to the version that you desire. That way you can look at the change log documentation at https://grafana.com/docs/grafana/latest/release-notes/ to get a clue about how to fix any errors that you may suddenly have.

Using my Domain name provider, I create an A name record pointing to my Grafana server IP address.

Now to add a proxy. I will use Nginx. The Nginx proxy will also allow us to more easily configure our Grafana servers public url and bind an SSL certificate to it. It is possible to change the grafana.ini settings to use a specific port number, SSL certificates and http protocol instead but you will also need to manage file permissions that the Grafana server process will need. It is more versatile to use a dedicated proxy and especially if your server will be hosting multiple other web applications.

I add SSL to the Grafana web server to ensure all traffic is encrypted between the server and web browser.

I use LetsEncrypt by following the Certbot instructions.

For Web Server software, I choose Nginx

For Operating system, I choose Ubuntu 24.04 LTS

I then SSH onto my new Grafana server,

I ensure snap is installed.

# sudo snap list

Make sure I have the latest version of snap

# sudo snap install core; sudo snap refresh core

I install the classic certbot

# sudo snap install --classic certbot

Prepare the command so that it can be executed from the command line

# sudo ln -s /snap/bin/certbot /usr/bin/certbot

Start the process of installing the SSL certificate for my domain name.

# sudo certbot --nginx

Follow the prompts, and enter the domain name that you want to secure.

After completion, you should then be able to now visit your Grafana server using the url

https://YOUR-DOMAIN-NAME

Note that after running Certbot, it has changed the settings of your Nginx configuration file you created earlier.

You can see those changes by using the cat command.

# cat /etc/nginx/sites-enabled/YOUR-DOMAIN-NAME

I use the 'TestData' data source. This is perfect for beginning to learn about Grafana visualisations.

We add a panel to the existing TestData dashboard, and more specifically, we first look at managing rows and the various presentation options we have.

We look at some of the presentation options for panels, such as positioning, size, keyboard shortcuts, duplication, deleting and more.

When creating and editing Dashboards, it is advisable to make regular saves in case you need to go back to a previous version or not.

I demonstrate many of the visualisation settings found in the panel tab and using different TestDataDB data source scenarios.

I demonstrate the settings found in the Overrides tabs.

I demonstrate using the Reduce and Add field from calculation transformations on the data from the TestData we have so far. These two transformations are likely to be the most common use case for using transforms within Grafana and are a good base to understand the remaining transforms.

Show a single value in a series.

We explore the Gauge panel.

We explore the Bar Gauge panel.

We explore the Table panel.

In this video, I demonstrate how to set up the MySQL Data Source with a collector and it's related dashboard.

I also demonstrate how to install a MySQL database server.

I install the MySQL database onto a new sever to also demonstrate the types of issues that you may have when connecting Grafana to another server.

Add the MySQL Data source in Grafana. If you try to save it, it will have some issues. We will next setup a MySQL server ready for the Data Source configuration.

After sourcing your new server (you can always use your existing Grafana server if you prefer) we next can install MySQL onto the new server. SSH onto your new MySQL server.

If you are using a Grafana Cloud instance, and you want to connect to your external MySQL server, then communication will travel across the public network.

There is a lot to consider to make this communication secure, and even just to get it to work.

An easy option we have to implement this, is to setup a Private Data Source Connect (PDC) agent on our MySQL server that can manage communications between our Grafana Cloud instance, and our externally hosted MySQL server.

We create a custom MySQL time series query that reads data from a our table in our MySQL database and formats the result set in a way that Grafana can use as a time series result set, and present that data in a graph along with the ability to filter that data using the Grafana user interface time filter drop down.

Grafana graphs Time Series data from many types of data sources extremely well.

But sometimes you just want to graph, simple non time series data. i.e., Data without timestamps, flat tables that show simple statistics or values.

Promtail has been deprecated. This course now teaches Alloy.

To keep this as simple as possible, we will install the Loki binary as a service on our existing Grafana server.

Now we will create the Alloy agent service that will act as the collector for Loki.

If you are doing this course using Grafana Cloud, and not creating your own self hosted Grafana infrastructure, then you will need a server to install Alloy onto in order to participate in the next many lessons.

Now that we have a loki data source we can query it with the LogQL query language.

In this video, we will try out many LogQL queries in the 'systemd-journal' stream selector that we just set up.

There are two types of LogQL queries:

1. Log queries returning the contents of log lines as streams.

2. Metric queries that convert logs into value matrixes.

A LogQL query consists of,

1. The log stream selector

2. Filter expression

We can use operations on both the log stream selectors and filter expressions to refine them.

Topics discussed in the video are,

• Log Stream Selectors

• Filter Expressions

• Expression Operators

• Range and Instance Vectors

• Aggregate Functions

• Aggregate Group

• Comparison Operators

• Logical Operators

• Arithmetic Operators

• Operator order

We can install an Alloy agent on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Alloy agents distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

If using an Alloy agent, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.

In this video, I demonstrate setting up annotation queries to help me visualize invalid user login attempts on my servers.

We will add to our Alloy config, the ability to read the Nginx access logs.

Restart the Alloy service and check its status.

If I go back into Grafana, I will see the new nginx job inside the Explore panel.

This is pretty good now, but we can make it better.

I want to be able to filter by the status code and other http properties.

We can use the loki pattern parser to dynamically create labels for query refinement.

Now I can create a dashboard for Nginx status codes.

Prometheus is already available on the default Ubuntu 20.04 repositories. So we can just install it and it will be set up as a service already.

$ sudo apt install prometheus

$ sudo service prometheus status

Test it by visiting http://[your domain or ip]:9090/graph

Restrict internet access by

$ iptables -A INPUT -p tcp -s localhost --dport 9090 -j ACCEPT

$ iptables -A INPUT -p tcp --dport 9090 -j DROP

$ iptables -L

Our Grafana Cloud instance has a Prometheus endpoint enabled, so we can push Prometheus system metrics to it from our servers where we have an alloy agent running.

Alloy can push the same metrics as a traditional Prometheus node exporter.

We will install two dashboards being one for the Prometheus service and the other for the Node Exporter.

The Prometheus dashboard can be found in the dashboards tab for the Prometheus data source configuration in the Grafana UI.

We will get the Node Exporter dashboard from the official Grafana Dashboards link.

https://grafana.com/grafana/dashboards

The dashboard Id is 11074 for the English version or Id 8919 for the Chinese version.

English Version: https://grafana.com/grafana/dashboards/11074

Chinese Version: https://grafana.com/grafana/dashboards/8919

Depending on your Grafana and Prometheus versions, the pre built Grafana Metrics dashboard may partly work or not at all.

In this video, I will show the steps that I used to get it to work.

Install the Grafana Metrics dashboard from the Prometheus Datasource --> Dashboards tab.

The Prometheus service, since it is local will, retrieve Grafana stats from the url http://127.0.0.0.1:3000/metrics

Grafana will return metrics data by default.

You can verify or change the settings in the grafana.ini file.

Go back into Grafana, and there is likely to be some issues with the visualizations.

In the video, I fix each problem in turn and demonstrate my problem solving process.

I will install a Prometheus Node Exporter on a different server and connect to it using the main Prometheus service.

It is now exposing the metrics endpoint on http://[your domain or ip]:9100

We can create a scrape config on the Prometheus server that retrieves metrics from that URL.

But since my new node exporter is accessible from the internet, I will block port 9100.

Next, Go back onto the main Prometheus server and edit the existing scrape config for node and add the new metrics endpoint for the other server.

You can connect directly to Zabbix via the API method. If you want faster performance than it is advised to also setup a MySQL data source for your Zabbix connection. The MySQL data source will instead connect directly to the Zabbix database and bypass the API layer for certain queries.

We import the 3 supplied Zabbix Dashboards

1. Zabbix System Status

2. Zabbix Template Linux Server

3. Zabbix Server Dashboard

In the next few lectures, we will look at how the create and use dashboard variables.

We will manually recreate fully a dynamic dashboard to query our SNMP devices that we've set up in the previous lectures.

We create Dynamic Tables from a Prometheus query using a dashboard variable.

We create repeating dynamic timeseries graphs from queries using our dashboard variables.

I have an SNMP enabled MikroTik 260GS switch configured as a host in my Zabbix system. I demonstrate creating a Grafana dashboard for it.

We will create an alerting rule for high CPU usage reported by any of our node exporters.

We will create alerting rules from when Prometheus detects a Node exporter is not up.

And we will create an alerting rule for when Prometheus is down.

When a node exporter is down, then Prometheus will return a 0 for the up metric. This is easy to write an alerting rule for.

But when Prometheus is down, then Grafana will get a timeout. There will be no response from Prometheus at all.

We will need to handle the symptom, rather than the metric value, as is returned when a Node exporter is down.

I want to send alerts using Grafana, but I need to first create an alert notification channel.

In this lecture I will create a new channel for email alerts.

I don't have an SMTP server available, so install a local send only SMTP server on my Grafana server.

We can also add Telegram as a Contact Point for Alerting.

To do this install the Telegram app on your phone or PC. It will be easiest to set this up on the PC first.

You will need a BOT API Token and Chat ID.

See Video for example how to do this.

We add users to our Grafana system using several different methods such as,

• Add User,

• Invite User,

• User Sign Up,

• Anonymous users

We also set the various roles of our users and manage the various permissions of our dashboards to allow certain roles to view and/or edit.

Teams help to segregate groups of users further.

You can assign team permissions to your dashboards and add/remove users from the teams instead without needing to go into each individual dashboard and reassign/add/remove particular user permissions each time.

Orgs can be used to create a new Grafana configuration with its own dashboards, data sources, users, team, alerts on an existing Grafana install.

Users can be shared between Orgs if you need.

We are going to install InfluxDB v2, the InfluxDB data source, a Telegraf agent and then collect some data.

InfluxDB is a database useful for storing large amounts of timestamped data.

Telegraf is an agent that supports plugins and it will save it's data into InfluxDB.

Now to install the Telegraf agent and configure the output plugin to save data into the InfluxDB.

Now that we have a System dashboard visible in InfluxDB, we can also re-produce this same dashboard in Grafana.

In this video, I demonstrate the process of doing that.

SNMP stands for Simple Network Management Protocol. 

We can configure Telegraf to read SNMP, save it into InfluxDB and view it in Grafana.

Common devices that support SNMP are routers, switches, printers, servers, workstations and other devices found on IP networks.

Not every network device supports SNMP, or has it enabled, and there is a good chance you don't have an SNMP enabled device available that you can use in this lecture.

So, I will also show you how to install and configure SNMP on your server, as well as read the SNMP data with Telegraf, save it into InfluxDB and view it in Grafana.

I will add several more SNMP agents to the Telegraf config.

In order for Telegraf to connect to the external SNMP agents, those other SNMP agents will need to be configured to allow my Telegraf agent on my Grafana server to connect remotely.

At the end of the lecture, I have 3 separate SNMP agents I can query for the next lecture.

We Import a SNMP Dashboard that uses the InfluxDB data source and the Telegraf collector.

I demonstrate installing and querying Elasticsearch 7.16.

Elasticsearch uses the JavaVM. So I recommend a minimum spec of 2GB RAM for the server that you use for the Elasticsearch service.

I am using Debian Package Instructions from https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

I demonstrate how to setup a Filebeat service to read systemd logs.

And then we can set up a new data source in Grafana, or modify the existing and test it using the explore tab.

I demonstrate how to setup a Metricbeat service to send to the Elasticsearch server.

Now to install an advanced dashboard that uses both the Filebeat and Metricbeat data sources at the same time.

I will set this up on my Linux server where the Filebeat process is already running.

Download Metricbeat for your OS from https://www.elastic.co/downloads/beats/metricbeat

My OS is a Debian based Ubuntu 20.04

Now we will create the Promtail service that will act as the collector for Loki.

We can install a Promtail service on other servers, and point them to an existing Loki service already running on a different server. If you have multiple Promtail services distributed around your network, and all pushing data to one main Loki service, then there are a few more considerations.

We then need to make sure that the job label in your Promtail configuration scrape_configs is unique from the perspective of the Loki service that it will be pushing to.

If using a Promtail service, or Loki service across the network, then it is important that you consider who can access it, or whether it needs to be encrypted since the transmitted data is likely to contain sensitive information about your server and other services.

We will add to our Promtail scrape configs, the ability to read the Nginx access logs.

We need to add a new job_name to our existing Promtail config_promtail.yml

Restart the Promtail service and check its status.

If I go back into Grafana, I will see the new nginx job inside the Explore panel.

This is pretty good now, but we can make it better.

I want to be able to filter by the status code and other http properties.

We can use the loki pattern parser to dynamically create labels for query refinement.

Now I can create a dashboard for Nginx status codes.