Grafana

Grafana is excellent for monitoring time series data.

In this video, I will install Grafana

After the successful install, you can start the Grafana service

$ sudo service grafana-server start

Your Grafana server will be hosted at

http://[your Grafana server url]:3000

The default login,

Username : admin

Password : admin

Random Order Grafana Questions

Using my Domain name provider, I create an A name record pointing to my Grafana server IP.

I add SSL to the Grafana web server to ensure all traffic is encrypted between the server and web browser.

I use LetsEncypt by following the Certbot instructions.

For Web Server software, I choose 'none of the above'

for Operating system, I choose 'Ubuntu 18.04 LTS'

I SSH onto the server,

I enter these commands

$ sudo apt-get update

$ sudo apt-get install software-properties-common

$ sudo add-apt-repository universe

$ sudo add-apt-repository ppa:certbot/certbot

$ sudo apt-get update

then

$ sudo apt-get install certbot

I temporarily stop the Grafana server

$ sudo service grafana-server stop

then run

$ sudo certbot certonly --standalone

I follow the prompts, and enter my domain name I want to secure,

I then copy the certificates over to the granafa folder, and edit the Grafana.ini file to use them, and then start the Grafana server again,

$ sudo service grafana-server stop

And now I visit my Grafana server URL replacing http with https.

I change the default Grafana web server port from 3000 to 443 since we now have an SSL certificate bound.

I use the iptables prerouting method

$ sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 3000

You can inspect the iptables settings using

$ iptables -t nat -L

Random Order Grafana Questions

I use the 'TestData DB' data source. This is perfect for beginning to learn Grafana.

We add a panel to the existing TestData DB dashboard, and more specifically, we first look at managing rows and the various presentation options we have.

We look at some of the presentation options for panels, such as positioning, size, keyboard shortcuts, duplication, deleting and more.

We explore the Graph panel.

Show a single value in a series.

We explore the Gauge panel.

We explore the Bar Gauge panel.

The Singlestat Panel allows you to highlight a statistic in a single series.

It reduces the series into a single number (by looking at the min, max, average, time, curent value and more in the series).

Singlestat also provides thresholds to color the statistic or the Panel background.

It can also translate the single number into a text value.

We explore the Table panel.

We explore the Text Panel and the Markdown syntax used to format the text in the panel.

We explore the Heatmap Panel

I create a new MySQL Data Source.

I also demonstrate how to install a MySQL database server in case you don't have one.

I install the MySQL database on my existing Grafana Server to make it easier to follow the lecture.

You can install MySQL on any server you like.

$ sudo apt install mysql-server

$ sudo mysql_secure_installation

$ sudo service mysql status

I also create an example database, and a specific user called grafana@localhost, thatr will have SELECT only access to the database.

$ mysqladmin create exampledb

$ mysql

  > CREATE USER 'grafana'@'localhost' IDENTIFIED BY 'password';

  > GRANT SELECT ON exampledb.* TO 'grafana'@'localhost';

  > FLUSH PRIVILEGES;

  > quit

I then continue setting up the MySQL data source using the Grafana User Interface.

Now we have a MySQL Datasource created, we will install a dashboard for it.

The dashboard will be the popular 2MySQL Simple Dashboard you can download from https://grafana.com/grafana/dashboards?dataSource=mysql

Before installing the dashboard, we need to set up a collector on our MySQL server.

The collector script is downloaded from https://github.com/meob/my2Collector

I have MySQL 5.7, so I will download the file called my2.sql

If you have MySQL 8, then download my2_80.sql

I download the script using wget,

$ wget https://raw.githubusercontent.com/meob/my2Collector/master/my2.sql

The event scheduler will also need to be enabled on your MySQL server.

You can do this by editing the my.cnf file

$ nano /etc/mysql/my.cnf

Add lines

[mysqld]

event_scheduler = on

Save, and restart,

$ sudo service mysql restart

Open the script that was downloaded,

$ nano my2.sql

Uncomment the last 2 lines.

Set the password to something you want, and save

Run the sql script,

$ mysql < my2.sql

Open MySQL and so some checks.

$ mysql

> show databases;

> show variables where variable_name = 'event_scheduler';

> use my2;

> show tables;

> select * from current;

> select * from status;

> quit

If everything is ok, continue

Import the new dashboard using the Grafana GUI as shown in the video,

I have an error after installing the Dashboard, that the 'select' command is denied for user 'grafana@localhost'

I need to grant select to the user on the new database 'my2' which was created when running the my2.sql script.

$ mysql

> GRANT SELECT ON my2.* TO 'grafana'@'localhost';

> FLUSH PRIVILEGES;

> quit

Reopen the dashboard, and you should begin to see data.

Note that there are now 2 database users,

• grafana@localhost
  Used by the Grafana dashboard, to query the collected data from the MySQL server. This user has been granted the SELECT privilege only.

• my2@localhost
  Used by the MySQL event scheduler to collect statistics and save them into the DB for use by the Grafana dashboard. This user has been granted ALL privileges.

Grafana graphs Time Series data from many types of data sources extremely well.

But sometimes you just want to graph, simple non time series data. i.e., Data without timestamps, flat tables that show simple statistics or values.

1st Create a MySQL Data Source that will connect directly to the Zabbix Servers MySQL Database.

SSH onto the Zabbix Server,

create a user in MySQL, with read only access, that the Grafana MySQL data source will use to connect with.

$ mysql

> CREATE USER 'grafana'@'YOUR GRAFANA SERVER IP ADDRESS' IDENTIFIED BY 'password';

> GRANT SELECT ON zabbix.* TO 'grafana'@'YOUR GRAFANA SERVER IP ADDRESS';

> FLUSH PRIVILEGES;

> QUIT;

Also allow remote connections on MySQL

$ nano /etc/mysql/mysql.conf.d/mysqld.cnf

Change

bind-address = 127.0.0.1

to

bind-address = 0.0.0.0

Restart MySQL

$ sudo service mysql restart

I then finish and save the MySQL data source for Zabbix.

Now it's time to install the Zabbix Data Source plugin that will connect to the Zabbix API, and also use the new MySQL data source we just created.

I SSH onto the Grafana Server and add a Zabbix Data Source plugin using the CLI

$ grafana-cli plugins install alexanderzobnin-zabbix-app

And then restart the Grafana server

$ sudo service grafana-server restart

I then use the Grafana UI to enable the Zabbix Data Soure Plugin, and then configure it.

The required API URL will be,

https://[your zabbix server ip or domain name]/zabbix/api_jsonrpc.php

Also select the 'Direct DB Access' to be the new MySQL Datasource you just created.

Save and Continue

We import the 3 supplied Zabbix Dashboards

1. Zabbix System Status

2. Zabbix Template Linux Server

3. Zabbix Server Dashboard

We create a custom dashboard using our Zabbix data source, and we use many different regex queries to select the data we wish to see.

We are going to install InfluxDB, the InfluxDB data source, a Telegraf agent and then collect data from some SNMP devices.

InfluxDB is a database useful for storing large amounts of timestamped data.

Telegraf is an agent that support plugins and it will save it's data into InfluxDB. 

Now to install the Telegraf agent and configure the output plugin to save data into the InfluxDB.

SNMP stands for Simple Network Management Protocol. 

We can configure Telegraf to read SNMP, save it into InfluxDB and view it in Grafana.

Common devices that support SNMP are routers, switches, printers, servers, workstations and other devices found on IP networks.

Not every network device supports SNMP, or has it enabled, and there is a good chance you don't have an SNMP enabled device available that you can use in this lecture.

So, I will also show you how to install and configure SNMP on your server, as well as read the SNMP data with Telegraf, save it into InfluxDB and view it in Grafana.

I will add several more SNMP agents to the Telegraf config.

In order for Telegraf to connect to the external SNMP agents, those other SNMP agents will need to be configured to allow my Telegraf agent on my Grafana server to connect remotely.

At the end of the lecture, I have 3 separate SNMP agents I can query for the next lecture.

We Import a SNMP Dashboard that uses the InfluxDB data source and the Telegraf collector.

In the next few lectures, we will look at how the create and use dashboard variables.

We will manually recreate fully a dynamic dashboard to query our SNMP devices that we've set up in the previous lectures.

We create a Dynamic Table from a query using our dashboard variables.

We create Dynamic Graphs from a query using our dashboard variables.

I want to send alerts using Grafana, but I need to first create an alert notification channel.

In this lecture I will create a new channel for email alerts.

I don't have an SMTP server available, so install a local send only SMTP server on my Grafana server.

$sudo apt install mailutils

At the prompt, I choose Internet Site

After the install is finished, I need to configure the Postfix SMTP server.

$ sudo nano /etc/postfix/main.cf

I edit 2 values at the bottom being,

...

inet_interfaces = loopback-only

inet_protocols = ipv4

Save, and then restart postfix

$ sudo systemctl restart postfix

I can test the SMTP server sends emails using

$ echo "This is the body" | mail -s "This is the subject" -a "FROM:[senders address]" [your personal email]

Next to configure Grafana SMTP settings

$ sudo nano /etc/grafana/grafana.ini

Find the SMTP specific settings, uncomment and edit

[smtp]

enabled = true

host = localhost:25

skip_verify = true

from_address = admin@[your grafana domain name or ip]

from_name = Grafana

Restart Grafana

$ sudo service grafana-server restart

Go back into Grafana, into your new Email notification channel settings and try to send a test email.

Check your SPAM folder.

We create alerts for when there is no data coming from a SNMP device.

In this video, I show you how you can create a custom alert channel webhook that sends SMS.
The SMS gateway I will use is AWS SNS.

We can also add Telegram to our list of Alerting Notification channels.

To do this install the Telegram app on your phone or PC. It will be easiest to set this up on the PC first.

You will need a BOT API Token and Chat ID.

See Video for example how to do this.

We will look at the logs panel, but we will first need to setup a data source that can read log files.

We will use the Loki Data Source with the Promtail client as the collector on the Grafana Server itself.

To keep this as simple as possible, we will install the Loki binary as a service on our existing Grafana server.

Now we will create the Promtail service that will act as the collector for Loki.

Now we will install the Loki Data Source, point it to our new Loki Service, and Explore some of the log files on the server.

In this lecture we create a Dashboard with a Graph Panel and a Log Panel, and then we automate the display of annotations by creating an Annotation Query.

Value groups/tags behave very differently depending on the Data Source used for the queries. This example is specific for when using the Zabbix Data Source.

Value groups/tags behave very differently depending on the Data Source used for the queries. This example is specific for when using the InfluxDB Data Source using Telegraf as the collector.

This example also takes Value groups/tags functionality further by adding a second custom input to the Telegraf collector to help group an existing query that contains no useable information that could already be used for grouping.

Through my Zabbix Data Source, I can access many of the properties of my Cisco Switch due to it already being setup as a SNMP device. I quickly create a Dashboard in Grafana for it.

I demonstrate creating a disk read/write rates dashboard from the Zabbix data source. This depends on data created from the Template Module Linux Block Devices by Zabbix agent template which is a standard template new on Zabbix 4.4 and is used by the Template OS Linux by Zabbix agent discovery rules.