Zabbix 5 Application and Network Monitoring
4.5 (534 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
2,823 students enrolled

Zabbix 5 Application and Network Monitoring

Learn Server, Proxy, Agents, Trappers, Items, Triggers, Graphs, Screens, LLD, SNMP, API, Grafana, Prometheus & more
Bestseller
4.5 (534 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
2,823 students enrolled
Created by Sean Bradley
Last updated 7/2020
English
Current price: $139.99 Original price: $199.99 Discount: 30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 9.5 hours on-demand video
  • 2 Practice Tests
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Installation and Configuration of Zabbix Server, Agents and Proxies on PC, Linux and MacOS
  • Create Auto Registration and Network Discovery Rules to Auto Add and Configure Discovered Hosts, Network Devices and Applications
  • Setup LLD Discovery Rules and Actions to Auto Configure SNMP Devices into specific Groups and Assign Templates
  • Understand Active Versus Passive Items
  • Construct a Reusable PCI DSS Monitoring Template for all hosts
  • Configuring Domain name and SSL for Zabbix Server
  • Install and Configure a Send Only SMTP server for emails
  • Media Type Configuration for customised Email, Slack, Telegram and SMS notifications
  • Advanced Items and Triggers on PC, Linux and MacOS
  • Create Standalone and Template Screens
  • Creating a Graphical Network Map of All Hosts Indicating Status
  • Creating Template Items for Assigning to Multiple Hosts
  • Item Preprocessing using Regex, Javascript and JSONPath
  • Item Cloning to produce a PCI DSS Template
  • Web Monitoring from Different Geographical Locations for HTTP Status Codes and Response Speeds
  • Create a LLD Graph Prototype from a File System Discovery Rule and Add it to a Template Screen
  • Create a LLD Trigger Prototype that Triggers Within a Range
  • Configure PSK Encryption between Zabbix Server, Agents and Proxies
  • Configure Trigger Ok Event Generation to Minimise Alert Flapping
  • Execute remote commands on Windows and Linux
  • Monitor SSL Certificate Expiry
  • Log File Monitoring Nginx Proxy HTTP Status Codes
  • Run Docker Commands with Administration Scripts
  • UserParameters
  • Execute Powershell Scripts to Check Windows Updates
  • Calculated Items
  • Dependent Items
  • JSON API Monitoring with the HTTP Agent Item
  • Zabbix Sender and Trapper, with Many Examples
  • Setup Grafana with the MySQL and Zabbix Datasources
  • Setup MySQL Monitoring
  • Setup SNMP Hosts and query using OIDs and MIBs
  • Setup SNMP Traps
  • Prometheus Node Exporter Introduction, Installing as a Service and Host Configuration
  • Setup LLD Discovery and Actions to Auto Configure Prometheus Node Exporters
  • Zabbix API Introduction with Examples, Python Script, User Permissions and Testing Tool
  • All Videos in one place, and with no Ads
Requirements
  • You will need access to several PCs and/or locally hosted or cloud hosted VMs and/or Rasberry PIs. In this course I demonstrate various features using Windows 10, Ubuntu, Rasberry Pi and MacOSX
  • You have the choice of using dedicated hardware, or using cloud or locally hosted VM managers such as Oracle Virtual Box. The more variation you can access, the better. Zabbix agents will run on most OSs, but Zabbix Server and Proxy will need Linux such as Ubuntu or CentOS
  • In this course I predominantly demonstrate using Ubuntu Linux, but also provide CentOS equivalent commands in accompanying documentation where applicable.
Description

Zabbix is a complete open source monitoring software solution for networks, operating systems and applications.

In this course you will install and extensively configure Zabbix Server, Zabbix Proxy, multiple Zabbix Agents on Windows, Linux and MacOS whether on the same network, or behind a firewall, on dedicated hardware or locally or cloud hosted VMs.

Zabbix can be used in the enterprise or even on you own home network where you can have much better visibility of the things connected and running on it and how they are used.

Who this course is for:
  • Network Engineers
  • IT Platform Specialists
  • System Administrators
  • DevOps Technicians
  • Enthusiasts wanting a better understanding and better visibility of their networks in the home or office
  • Someone who is curious and wants a better understanding of what Zabbix does and what Zabbix is good at
Course content
Expand 75 lectures 10:02:53
+ Zabbix
75 lectures 10:02:53

I have many options where to install Zabbix server. In this course I want the server to be on 24 hours a day, and to be easily accessible from many physical locations. I decide that hosting it using a cloud provider is best and I use digital Ocean. I could use AWS, Azure, GCP or many others.

This video contains a demonstration of installing Zabbix Server on a Digital Ocean hosted VM.

If you would also like to use Digital Ocean VM, you can create a $5 a month server using this link.

https://m.do.co/c/23d277be9014 (FREE $50 - 30 Day Credit). Note that the offer sometime fluctuates to $100 for 60 days.

The $5 a month VM is adequate for all the tasks in this course and it only takes a minute to set up.


Preview 03:06

I choose to install Zabbix Server from packages.

The download page is at

https://www.zabbix.com/download

I have the 'Install from Packages' tab active,

I then choose '5.0', 'Ubuntu', '20.04 Bionic', 'MySQL' for the database and Apache for the Web Server.


The commands I used in this video are,

# wget https://repo.zabbix.com/zabbix/5.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_5.0-1+focal_all.deb

# dpkg -i zabbix-release_5.0-1+focal_all.deb

# apt update


Download and Install The Zabbix Repository
02:11

The commands in this video are,

# apt install zabbix-server-mysql zabbix-frontend-php zabbix-apache-conf zabbix-agent

The Zabbix Agent installed at this time is an agent specific for monitoring this specific Zabbix Server. This is not necessary but it advised, and I use it within this course.

Install Zabbix Server, Frontend and Agent
01:31

The commands executed in this video are,

# mysql -uroot -p

password

mysql> create database zabbix character set utf8 collate utf8_bin;

mysql> create user zabbix@localhost identified by 'password';

mysql> grant all privileges on zabbix.* to zabbix@localhost;

mysql> quit;


Next import the schema,

# zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p zabbix


Then edit the Zabbix Server configuration file to tell it the DB password

# sudo nano /etc/zabbix/zabbix_server.conf

DBPassword=password



Create the Initial Database
03:37

Random Order Zabbix Questions

Zabbix Questions
5 questions

The commands in this video are,

# sudo nano /etc/zabbix/apache.conf


Scroll down and uncomment the line

# php_value date.timezone Europe/Riga

and edit to your time zone.

My time zone is London, so my line looks like,

php_value date.timezone Europe/London


PHP timezone strings can be found here,

https://www.php.net/manual/en/timezones.php

Configure the PHP front end
01:20

Commands entered in this video,

# systemctl restart zabbix-server zabbix-agent apache2

# systemctl enable zabbix-server zabbix-agent apache2

Start Server and Agent Processes
00:46

In this section you will need to enter the MySQL password you created earlier.

And then at the end, you will then log into the new completed install of Zabbix Server using the default credentials of,

Username : Admin

Password : zabbix

Log in and Configure Zabbix Server Front End
02:17

Random Order Zabbix Questions

Zabbix Questions
5 questions

I created an A name record at my DNS provider for zabbix.your-domain.tld and entered the ip address of the server.

Adding a domain name to your Zabbix Server is optional.


Configure a Domain name for the Zabbix Server
01:04

The Zabbix Server doesn't have any transport encryption enabled yet, so any messages passed between our browser and the server are in plain text. We should secure our server asap with an SSL certificate.

I create the certificate using options provided by LetsEncypt. This has the added benefit of being free.

So, we need to ssh onto the Zabbix Server and install certbot

$ sudo apt-get install certbot

Because Zabbix Server is using Apache, we will need an Apache plugin.

$ sudo apt-get install python-certbot-apache

then

$ sudo certbot --apache -d zabbix.seanwasere.com

Follow the prompts, and at the end, your Zabbix Server will now have an SSL certificate bound and accessed via https.



Configure SSL for Zabbix Server Front end
02:52

We look over at what we have so far, and how we got there.

Overview So Far
09:56

I install Zabbix Agent on an Ubuntu 20.04.

I first need to download and install the repository on the server.

The page with these instructions are at

https://www.zabbix.com/documentation/current/manual/installation/install_from_packages/debian_ubuntu

# wget https://repo.zabbix.com/zabbix/4.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.4-1+focal_all.deb

# dpkg -i zabbix-release_4.4-1+focal_all.deb

# apt update

# apt install zabbix-agent


Now to configure the agent,

$ sudo nano /etc/Zabbix/Zabbix_agentd.conf


Edit parameters for Server, ServerActive and Hostname and save.

Then,

$ sudo service zabbix-agent restart


Now to configure a new host on the Zabbix Server itself.

Install Zabbix Agent on Same Network as Zabbix Server
09:14

I download and Install the Zabbix agent for windows option.

I set this agents ServerActive parameter, and configure a template in Zabbix with Agent(Active) items only.

Since the template is using Agent(Active) items only, I do not need to create firewall forwarding rules.


Install Zabbix Agent (Active) on a Windows Host Behind a Firewall
09:15

Since this host is on a different network than the Zabbix-Server, and a Zabbix Proxy is not present, we will need to create a firewall rule to port forward external Zabbix Server Passive Checks internal hosts IP and port.

I get the Agents external IP address, and enter that into the Zabbix Server hosts setup for this windows host, and then setup the port forwarding rule on my router.

Zabbix server isn't on the same network as my new Windows host, so the windows hosts internal IP will not work without setting up a specific firewall forwarding rule.

After all configurations have been entered, you can open the services panel on windows and restart the Zabbix agent service so that the agent and server begin communicating.

You may also need to enable a firewall rule on Windows Defender for inbound TCP on port 10050 if it wasn't already created by the Zabbix Agent installer.

Enable Passive Checks on the Windows Host Behind the Firewall
04:58

In this video I install and configure Zabbix Agent to run on a Mac OSX.

The mac is behind a firewall, so Zabbix Server cannot connect to it directly, so I also create a specific firewall rule.

My router is already forwarding external requests to 10050 to my other internal Zabbix Agent host, so I use a new port 10052 wich will repoint to the Macs internal IP and Port to the default Zabbix Agent port 10050.

In later videos I will demonstrate using Zabbix Proxy which does not require a new firewall rule to be created for each individual new host when using Passive checks.

After installing the Zabbix Agent on Mac OSX, you will need to configure it's configuration file using

$ sudo nano /usr/local/etc/zabbix/zabbix_agentd.conf

edit the Server, ServerActive and Hostname parameters and save.

Then create the host on Zabbix Server, create the firewall rule,

then restart the Zabbix Agent process on the Mac.


The commands on the mac are,

$ sudo launchctl unload /Library/LaunchDaemons/com.zabbix.zabbix_agentd.plist

then

$ sudo launchctl load /Library/LaunchDaemons/com.zabbix.zabbix_agentd.plist


Go back to the Zabbix Server and it should now find the agent on the Mac.

Install Zabbix Agent on a Mac OSX Behind a Firewall
05:54

Zabbix agent can also be installed in VMs hosted on your local PC. In this video I am using Oracle Virtual Box on Windows 10 with an Ubuntu 20.04 VM.

An important setting for VMs regards how they can be found on the network. In order for Zabbix-Server to communicate with it, it will need an IP address on your local network. In my setup, I configure the VMs network adapter as a bridged adapter and my routers DHCP auto assigns it an IP address.

The detailed download instructions are at https://www.zabbix.com/documentation/current/manual/installation/install_from_packages/debian_ubuntu

The commands in this video are,

# wget https://repo.zabbix.com/zabbix/5.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_5.0-1+focal_all.deb

# dpkg -i zabbix-release_5.0-1+focal_all.deb

# sudo apt install zabbix-agent


then configure it,

# sudo nano /etc/zabbix/zabbix_agentd.conf


Edit parameters for Server, ServerActive and Hostname and save.

Then,

# sudo systemctl start zabbix-agent.service

# sudo systemctl status zabbix-agent.service


Then go to Zabbix-Server and create a new host.

This host is also behind a firewall so I will need to set up another port forward rule.

Since I have already set up a rule for the external IP, port 10050 and 10052, I will need to use another port. I use the external IP, and port 10053. This will then forward to the internal host with the agent running and listening on the default port of 10050.

Install Zabbix Agent on a VM Behind a Firewall
07:30

I auto register two extra Zabbix Agents. One is on the internet, which is the same network as my Zabbix Server, and the other is on my Internal Network. I do not have any firewall rules created for the internal host, or even a Zabbix Proxy running.

I configure the auto registration rule to automatically add the host to Zabbix, and add it to the Linux Servers group, and auto assign the Zabbix Agent for Linux Active template.

My Agent configurations both contain the extra parameter of linux in the HostNameItem field. I use the property to know which template and group to assign these new hosts to during the Auto Registration process.

Preview 08:32

I install Zabbix Proxy to manage communication between the Server and Agents.

The commands used in this lecture,

# wget https://repo.zabbix.com/zabbix/5.0/raspbian/pool/main/z/zabbix-release/zabbix-release_5.0-1+buster_all.deb

# dpkg -i zabbix-release_5.0-1+buster_all.deb

# apt update

# sudo apt install -y zabbix-proxy-sqlite3

# sudo nano /etc/zabbix/zabbix_proxy.conf

I edit the Server and Hostname and DBName parameters, and save it,

# sudo service zabbix-proxy start


I then create the proxy settings in the Zabbix Server user interface, and after several seconds, I refresh the proxy config screen and it will show that the proxy is now in communication with the server.

Install and Configure Zabbix Proxy
09:08

The correct Zabbix repository has already been set up on the Raspberrypi Zabbix Proxy, so I only need to run

# sudo apt install zabbix-agent


If your Zabbix Server and Proxy are communicating successfully, as can be verified in the Zabbix UI -> Admin -> proxies page, then we can now setup the agent on the proxy itself to retrieve items.

Now configure your agent to use 127.0.0.1 for both Server and Server Active

sudo nano /etc/zabbix/zabbix_agentd.conf

And also set the hostname. I used raspberrypi.

Configure Zabbix Agent on the Zabbix Proxy
07:36

I reconfigure my existing Zabbix Agents on my Centos7, Ubuntu20, Windows10 and Mac OSX to now use the new Zabbix Proxy we've just set up.

For each host that I need to be proxied, I need to edit it's agent config file to use the RaspberryPi for both it's Server and ServerActive settings.

I then go into the Zabbix UI and reconfigure the hosts to now be monitored by proxy. I also update the agent interface information to reference the servers from the perspective of the proxy. ie, the ip address needs to be the local address on the network, or use the host name which can be found in the local network.

Reconfigure Zabbix Agents to use Zabbix Proxy
09:36

On Linux hosts, to enable auto restart after reboot,

# sudo systemctl enable zabbix-agent.service

and if you are also running a Zabbix Proxy,

# sudo systemctl enable zabbix-proxy.service


Note:  If you've enabled the agent and/or proxy services to start upon reboot, and you've changed your mind, just execute,

# sudo systemctl disable zabbix-agent.service

and/or    

# sudo systemctl disable zabbix-proxy.service

Ensure Zabbix Agent and/or Proxy Auto Starts After Reboot
01:04

By default, agent communication is done in clear text.

For encryption we have an option to use PSK-based encryption.

PSK means pre shared key.

The PSK option consists of two important values, the PSK identity and the PSK Secret.

The secret should be minimum a 128-bit (16-byte PSK, entered as 32 hexadecimal digits) up to 2048-bit (256-byte PSK, entered as 512 hexadecimal digits)

You can generate a 256 bit PSK secret with openssl using the command

# openssl rand -hex 32

In this lecture, I also save it straight to a file.

I first create and navigate to a folder

/home/zabbix/

I then run,

# openssl rand -hex 32 > secret.psk

I also make sure that only the Zabbix user can read the file.

# chown zabbix:zabbix secret.psk

# chmod 640 secret.psk

I then reconfigure the Zabbix agent configuration file.

# sudo nano /etc/zabbix/zabbix_agentd.conf

and change the options near the bottom,

TLSConnect=psk
TLSAccept=psk
TLSPSKFile=secret.psk
TLSPSKIdentity=[whatever you like]

I then restart the agent

# sudo service zabbix-agent restart

I then go into the Zabbix Server User interface and configure the PSK encryption options for the host.

I select the

'Connections to host' = PSK
'Connections from host' = PSK
'PSK Identity' = [what ever you used in the Zabbix agent config]
'PSK' = [the long hex string generated from the OpenSSL command above]

After a minute or two, the Zabbix Server and Agent will successfully communicate using PSK encryption.

Enable PSK Encryption for Zabbix Agents
06:27

This time I setup PSK encryption specifically for communications between the Zabbix Server and Zabbix Proxy. Enabling PSK encryption for Agents behind a Proxy, only encrypts communications between the Agent and the Proxy. If your agents are in a DMZ then you may not desire encryption. But you should at least also encrypt the communication between the Zabbix Server and Proxy if it travels across a public network.

Enable PSK Encryption for Zabbix Proxy
07:00

I manually create 3 items on the host. 1 Passive and 2 Active.

The passive check is for agent.ping

The active checks are for disk space used and total.

Creating Host Items
11:50

In this section, we create a basic trigger.

The trigger will check for nodata from a host for 120 seconds.

Creating Host Triggers
05:18

I manually create 2 Graphs for this host. Before starting this section, you should manually create some new items for your host from the table in the documentation.

Creating Host Graphs
06:50

It is important to get email alerts when a trigger happens in Zabbix.
If you don't already have an SMTP server for sending emails, you can also install a Send Only server on the Zabbix Server itself.


The commands in this lecture are

$ sudo apt install mailutils


A postfix config screen should appear,

select 'internet site'


enter the mail servers name, it should be something you've already setup in your existing DNS providers name server elsewhere.

For example. I pre set up an A name record for smtp.seanwasere.com and it points to the IP Address of the this Zabbix Server.


Then configure postfix by calling

$ sudo nano /etc/postfix/main.cf


scroll to the bottom, and change 'inet_interfaces = all' to 'inet_interfaces = loopback-only'


restart postfix

$ sudo systemctl restart postfix


try sending an email from the command line,

$ echo "This is the body" | mail -s "This is the subject" -a "FROM:[senders address]" [your personal email]


Use The Email Media Type And Create A Send Only SMTP Server
13:44

In this section, I show you how to create a template from the new host items, triggers and graphs that we've created in the last few videos. I also assign the new template to a newly discovered Windows host, and replace the original items, triggers and graphs from an earlier host with the new template that we've just created.

It's important to note that a hosts applications will not be copied across, when you copy the items to the new destination. The new destination should already have identical applications configure if you want to utilise the original application names in your copied items. This only affects copying the information to teh new destination, and not when you assign a template to a new host. When you assign a template that already contains applications to a new host, the applications are also copied automatically.

Convert Host Items, Triggers and Graphs To A Template
07:49
Zabbix Questions
11 questions

We look at system level screens and several of the options that can be added, such as graphs, information panels and external urls.

Monitoring Screens
05:05

In this video, we add a screen to our custom template. I also show the several different ways of accessing it within the Zabbix user interface.

Template Defined Screens
04:02

I use the Monitoring/Maps to create a network map from scratch. I show how to link the map icon to the host so that the host problems also appear on the graphical representation.

Creating a Network Map
11:28

In this lecture I create an advanced item. The item reads the windows event logs and looks for the a specific windows event id 4625 which is also known as 'failed logon'.

The item type is 'Zabbix Agent (Active)'

and the key is "eventlog[Security,,,,4625,,skip]"

The type of information is 'Log'

The duration to keep the data and the frequency of checking the item is up to you.

I then log onto my windows VM and generate some failed logins.

I then see the failed login events in the Latest Data screen.




Reading Windows Event Logs
06:46
Zabbix Questions
5 questions

In this lecture, I add a pre-processing step to the item that instructs the agent to return only the 1st line of the windows failed logon event description.

The pre-processing regex value is

(.*)

Which means to find the start of the line to the end,

And the output is

\0

Which indicates the first value found by the regex.

So, all in all, it returns only the first line.

Preview 08:37

I demonstrate how to use JavaScript to pre-process incoming item information.

Item Preprocessing with JavaScript
04:46

In this lecture, I use item cloning to add many more events to my PCI DSS Windows Template.

I copy the 'Failed Logon' item and create many more.

Attached as a resource is the template that I created in this Lecture.
You can import it into Zabbix.


All the event items I added were,

EventID 4608 : Windows is starting up

EventID 4609 : Windows is shutting down

EventID 4610 : An authentication package has been loaded by the Local Security Authority

EventID 4611 : A trusted logon process has been registered with the Local Security Authority

EventID 4612 : Internal resources allocated for the queuing of audit messages have been exhausted, leading to the loss of some audits

EventID 4614 : A notification package has been loaded by the Security Account Manager

EventID 4616 : The system time was changed

EventID 4624 : Successful Logon

EventID 4625 : Failed Logon

EventID 4634 : An account was logged off

EventID 4657 : A registry value was modified

EventID 4660 : An object was deleted

EventID 4663 : An attempt was made to access an object

EventID 4670 : Permissions on an object were changed

EventID 4674 : An operation was attempted on a privileged object

EventID 4720 : A user account was created

EventID 4722 : A user account was enabled

EventID 4723 : An attempt was made to change an account's password

EventID 4725 : A user account was disabled

EventID 4726 : A user account was deleted

EventID 4727 : A security-enabled global group was created

EventID 4728 : A member was added to a security-enabled global group

EventID 4729 : A member was removed from a security-enabled global group

EventID 4730 : A security-enabled global group was deleted

EventID 4731 : A security-enabled local group was created

EventID 4732 : A member was added to a security-enabled local group

EventID 4733 : A member was removed from a security-enabled local group

EventID 4734 : A security-enabled local group was deleted

EventID 4738 : A user account was changed

EventID 4740 : A user account was locked out

EventID 4767 : A user account was unlocked

EventID 5143 : A network share object was modified

EventID 6144 : Security policy in the group policy objects has been applied successfully

Item Cloning to Create a PCI DSS Windows Template
08:32

Creating templates is hard work, and more often than not, somebody has already done the hard work, or part of it, for you. 

So, you can download templates and import them into Zabbix for use in your hosts very quickly.

The most famous source of Zabbix templates is at https://share.zabbix.com

In this example, I will download and install a Zabbix template to Automatically check your Hosts DNS name against several blacklists.

This is a good example to use, since it will also demonstrate using ExternalScripts and Value Mapping at the same time.

Importing Templates
10:50
Zabbix Questions
3 questions

We set up a Slack Media Type for Trigger alerts in Zabbix.

  1. Create or join a Workspace

  2. Create a Channel

  3. Create an App.

  4. Add the OAuth Scope chat:write

  5. Press the button Install App to Workspace

  6. Allow your APP to access your workspace so that it can send messages to your new Channel.

  7. Copy the Bot User OAuth Access Token and place it in the Zabbix Slack Media Type Bot Token field.

  8. Enable the Slack Media Type

  9. Add the Problem and Problem Recovery templates to the Slack Media Type

  10. In Slack, go to your Channel, and press the Add an app option, and Add your APP.

  11. Add a global Macro for {$ZABBIX_URL} and set its value to the full url of your Zabbix system. This will be used in the Slack messages to create a link to the problem information contained in the Zabbix UI.

  12. Add the Slack Media Type to your user, and select which severity to receive alerts for.

  13. Ensure the Trigger Action for Report problems to Zabbix administrators is enabled.

Preview 10:05

We configure a media type that pushes messages to Telegram.

Telegram is a messaging app that runs on PC and popular smart phones.

Before creating the media type, you will need to

  • Install Telegram,

  • Create a Bot,

  • Copy it's HTTP API Token,

  • Create a Group,

  • Add the Bot to it,

  • Send the Bot at least 1 message.

  • Retrieve the Chat ID from the chat data.

See the video and resources for the script and how to set it up in Zabbix.

Telegram Media Type
08:01

In this video, I will create the SMS media type using AWS SNS, and demonstrate receiving SMS alerts from the Zabbix server using the alertscripts option.

SMS Media Type using AWS SNS
11:00

Customising Trigger Alert Messages with Macros


URLS in this Lecture


Units : https://www.zabbix.com/documentation/3.0/manual/config/triggers/suffixes


Macros : https://www.zabbix.com/documentation/current/manual/appendix/macros/supported_by_location




I created the trigger with these settings,


Name : Disk Space Used Above 2GB, Current Value : {ITEM.VALUE}

Severity : Disaster (My users media type for severity Disaster is to receive alerts through multiple medias)

Expression : {Grafana:vfs.fs.size[/,used].last()}>2G


Customising Trigger Alert Messages with Macros
04:35

The default OS Linux Template Screen doesn't show a graph of disk space over time. So in this lecture I show you how it's done. You need to clone or create a new graph prototype in the Network Discovery rule of the OS Linux template. You then add the graph to the template screen by selecting it from the Graph Prototype options available in the constructor view of the screen. You can also do this for the Windows and MacOSX templates.

Add Disk Space History Graph To OS Linux Template Screen
07:06

I create a new LLD Trigger Prototype for the OS Linux Active Template, and modify it to trigger within a certain range and indicate a different severity.


Trigger Prototypes and Triggering within a Range
09:09

Triggers that alert Problems and the Resolutions too frequently can cause a symptom commonly referred to as Alert Flapping. In this lecture I edit the Ok Event Generation properties of a trigger to either delay the OK (Resolved) event for a period of time, or try disabling it and adding the option to manually close the problem. I also demonstrate the option that allow you to choose which severity of events get alerted, and to disable the trigger all together.

If you are experiencing alert flapping and you want to manage it, you will need to manage each trigger individually in order to find that balance of acceptable alert flapping.


Configure Trigger 'Ok Event Generation' to minimise Alert Flapping
11:15

In this video we look at Web Monitoring, and more specifically monitoring for HTTP Status Code 200 and the text returned from a websites http response. I also setup the monitoring from several geographical locations being New York, Amsterdam and Singapore.

Distributed HTTP Endpoint monitoring using Web Scenarios
12:18

Many devices and services now provide REST APIs that you can query and get JSON formatted data as a response. In this video I will show you how to create an Item that uses the HTTP Agent type to query an external REST API and extract it's data.

In case you don't have a REST API handy that you can query, I have created a test API the you can use. It has several methods that return test data.

JSON API Monitoring with the HTTP Agent Item
11:10

For pre Zabbix 5.02. In the zabbix_agentd.conf for the remote host, add EnableRemoteCommands=1 and then restart the agent process.

In Zabbix 5.0 and 5.01, you will also need to comment out the DenyKey parameter which blocks system.run by default, and then restart the agent process.

In Zabbix 5.02 and later, you can ignore EnableRemoteCommands=1 since it is now deprecated, and you should use a combination of DenyKey and AllowKey to fine tune the scripts you want to deny/allow.

Execute Bat File on Remote Windows Host with Zabbix Agent
08:06

I create a python script on my remote Linux host that prints a random number. I create a new item for the host and select the system.run option for the key and enter the python command needed to run the new py file on the remote host.

Remember, it's important that there is a line `EnableRemoteCommands=1` in your zabbix-agentd.conf file, and you need to restart the agent for it to take effect.

Execute Python Script on Remote Linux Host with Zabbix Agent
06:29

With Zabbix-get you can manually query keys on the remote hosts.

To install on ubuntu,

$ sudo apt install zabbix-get


For help use

$ zabbix_get -h


Try

$ zabbix_get -s <hostname or ip> -k agent.ping

$ zabbix_get -s <hostname or ip> -k agent.version

$ zabbix_get -s <hostname or ip> -k agent.hostname




Using the 'Zabbix Get' Command
07:35

In this lecture I use the agent running on my zabbix server to monitor days remaining before SSL expiry by creating a custom script and executing it using the system.run item key option.

You can use any linux agent you desire to run this script.


Since the expiry days can go negative, I advise you use the numeric(float) option in the 'type of information' drop down.


Visit 'Monitoring/Latest Data' to see the values.




Check SSL Certificate Expiry on Websites using Custom Script and system.run
06:15

Monitoring Log Files - HTTP Status Codes of a Nginx Proxy

The file I monitor is located at /var/log/nginx/access.log

The default user that the zabbix agent uses does not have read access to many log files on the system.

You can usually add the zabbix user to a group to solve this problem.

The access.log file can be read by the www-data or adm groups on Ubuntu 18, so I add the zabbix user to the www-data group.

To find out which groups a log file can be read by, for example, I typed,

$ ls -lh /var/log/nginx/

This tells me that the access.log file can be read by www-data and adm groups.

Then I check which groups the user zabbix is part of,

$ groups zabbix 

If it's not part of either group already, I then add it,

$ sudo usermod -a -G adm zabbix

and check again to confirm.

$ groups Zabbix


I can read the most recent log file entries by typing

$ tail -f /var/log/nginx/access.log


I then created an item for the host, with settings

Name: HTTP Status Codes

Type : Zabbix (active)

Key: log[/var/log/nginx/access.log,"^(\S+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(\S+)\s?(\S+)?\s?(\S+)?\" (\d{3}|-) (\d+|-)\s?\"?([^\"]*)\"?\s?\"?([^\"]*)\"",,,,\8,]

Type of Information : numeric (unsigned)

Update Interval : 5s


The regex value that I copy into regex101 is

^(\S+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(\S+)\s?(\S+)?\s?(\S+)?\" (\d{3}|-) (\d+|-)\s?\"?([^\"]*)\"?\s?\"?([^\"]*)\"

This regex can separate the values for Nginx and Apache logs.

The regex splits each row of the log into several groups. The HTTP Status code is in the 8th group.


I can also created triggers to notify on

101 Switching Protocols
301 Moved Permanently
302 Redirect
304 not modified
400 Bad Request
401 Unauthorised
403 Forbidden
404 Not found
500 Server Error

In this video I demonstrate creating triggers for HTTP 5XX errors and use **count** to detect 10 or more HTTP 404 Errors in 10 minutes.

How and whether you decide to trigger on HTTP status codes is up to you. 

The video just provides examples for you to follow.


Log File Monitoring - Nginx Proxy HTTP Status Codes
12:23

In this video I expand the log file monitoring item from the last video, into 1 master item that returns the whole log line, and then create several dependent items from the specific data contained in the log line.

Creating dependent items means that the agent doesn't need to run possibly identical queries on a host many times in order to extract parts of a value. The master item runs once on the host, and then the Zabbix server (or Zabbix proxy if host managed by proxy) updates the dependent items each time the master item gets its new values.

I also convert the new items into a template, create a graph and screen consisting of the graph created from the dependant item plus tables also showing values from the dependant items.

Dependent Items
14:22

In this lecture, I demonstrate how to use administration scripts from the Zabbix Server, Zabbix Proxy and Zabbix Agents.

Configuration depends on

  • from which process, on which server, the script will actually be executed.

  • whether remote commands are enabled for the process executing the script,

  • whether the Zabbix user will require sudo privileges for any part of the script command.

  • whether or not the agent has any working passive checks.

In this video, I demonstrate how to setup and problem solve the pre existing administration scripts ping, and detect operating system.

When calling the ping administration script for a host behind a Zabbix Proxy, the Zabbix Proxy configuration will also need to be updated to allow remote commands. This is because the ping will be executed from the Zabbix proxy server. You should also restart the Zabbix proxy after any configuration change.

For the administration script used to detect the operating system, the nmap executable will be used from either the Zabbix Server or Zabbix Proxy, and it will need to be installed on the server or proxy first, and you will also need to allow the Zabbix user sudo privileges to execute it.

# sudo visudo

Add the line, zabbix ALL=(ALL) NOPASSWD: /usr/bin/nmap

Press Ctrl-X and Y to save.

I will also demonstrate creating a script that runs from the Zabbix agents themselves. This will use the free command to list available memory.

Administration Scripts
13:13

Running Docker Commands with Zabbix Administration Scripts

In this video, I demonstrate how to use administration scripts to manage docker containers.

During the setup I encounter several issues in which I solve them all.

  1. It's important that the agent, proxy or server process that you want to use to run the script, needs EnableRemoteCommands=1 set in it's config file. Then restart the agent, proxy or server service. The specific details of enabled remote commands depend on which version of the agent you are running, and have been covered several times in the previous sections.

  2. If the command you execute needs privileged permissions, then run it with the sudo prefix.

  3. To allow the zabbix user to use the sudo prefix, add a new line to the sudoers file.

    # sudo visudo

    Add

    zabbix ALL=(ALL) NOPASSWD: /usr/bin/docker

With these 3 things, I was able to execute docker commands on my host using the administration scripts option.

Those commands were,

  • sudo docker ps -a

  • sudo docker stats -a --no-stream

  • sudo docker stop nginx2

  • sudo docker start nginx2

Running Docker Commands with Administration Scripts
08:53

User Parameters in Zabbix.

I create several examples of UserParameters in this video.

Example 1

Starting as simple as possible.

I create an item to check isalive

Inside the Zabbix agent configuration file, on the host that will run the UserParameter, I add UserParameter=isalive,echo 1

I save it, then I test it using,

$ zabbix_agentd -t isalive

I then restart the zabbix agent process,

$ sudo service zabbix-agent restart


I then add a new item to my host with

name = is alive

key = isalive

type of information = numeric

I go to Monitoring-->Latest Data and wait for it to appear.


Example 2

And now for something a bit more complicated, and that is Flexible User Parameters

Inside the conf file I add,

UserParameter=isalive[*],echo $1


I can test it using

$ zabbix_agentd -t isalive[123]


Restart the zabbix agent process,

$ sudo service zabbix-agent restart


The item inside Zabbix Server has, name = is alive key = isalive[123456] type of information = text


Example 3

And now, I convert an existing system.run command to a UserParameter.

The script called in this system.run command is outlined in my previous lecture Check SSL Certificate Expiry on Websites using Custom Script and system.run

Inside the conf file I add,

UserParameter=ssl.check[*],/home/zabbix/checkssl.sh $1 $2


I can test it using

$ zabbix_agentd -t ssl.check[example.com,443]


Restart the zabbix agent process,

$ sudo service zabbix-agent restart


The item inside Zabbix Server has,

name = Check SSL example.com

key = ssl.check[example.com,443]

type of information = numeric


UserParameters
09:25

We can execute PowerShell commands on remote hosts with Zabbix agent.

In this lesson we will install 3 scripts, that we can use to query Windows Updates status on a remote windows host.

Two of the scripts will be used as host items, and the other will be an administration script that we can call manually.

Note, If your scripts are slow running, you are likely to get timeouts when calling from Zabbix. You can set the timeout properties from the default 3 seconds to 1-30 seconds inside the Zabbix agents config, and also the Zabbix Proxy config if your agent is monitored by proxy.

Note, for Administration scripts that run on the agent, you will also need to edit the zabbix_agentd.conf. Set EnableRemoteCommands = 1, and/or modify the DenyKey/AllowKey properties depending on your version of the Zabbix agent.

Also, for Administration scripts, this will not work on agents connected to the Zabbix server as capable of Active checks only. Zabbix server will need to be able to do passive checks. So that means, a firewall rule, or Zabbix proxy setup on the same network as the agent, if the agent is not on the same network as the Zabbix Server.

Execute Powershell Scripts to Check Windows Updates
11:57

I create 3 calculated items and a graph in this lecture.


1.

name = Changes MySql Questions

type = calculated

key = mysqlchanges

formula = change(mysql.questions)


2.

name = Average of Changes of MySql Questions

type = calculated

key = avgmysqlchanges

formula = avg(mysqlchanges,#5)


3.

name = Forecast of Changes of MySql Questions

type = calculated

key = forecastmysqlchanges

formula = forecast(mysqlchanges,1h,,10m,exponential)



The useful documentation urls shown in this video are

https://www.zabbix.com/documentation/current/manual/config/items/itemtypes/calculated

https://www.zabbix.com/documentation/current/manual/appendix/triggers/functions


Calculated Items
10:39

We can also extend discovery rules with calculated items.

In this example we will add a calculated item to show the change in bytes for each #FSNAME in the Mounted Filesystem Discovery rule.

Calculated Items in Low Level Discovery Rules
08:27
Zabbix Sender and Trapper - Intro and Example 1 - Cron
10:16
Zabbix Sender and Trapper - Example 2 - Screen
05:42

Our Zabbix Server is using a MySQL database. We can monitor this database using the 'Template DB MySQL by Zabbix Agent' host template.

After linking this template to your host, you will get permission errors before you can see the data.

On my server, I need to create a new folder

$ mkdir /var/lib/zabbix


and add a file called .my.cnf with the contents

[client]

user=zbx_monitor

password=password


Save and restart the zabbix agent.

$ sudo service zabbix-agent restart


After a minute or so, your new MySQL items for your Host will start receiving data.

Setup MySQL Monitoring
08:00

Grafana Zabbix Plugin


Grafana is good for monitoring time series data, and so is Zabbix,

Grafana is another option for viewing Zabbix Data, and data from many other data sources.


In this lecture, I will install Grafana on the same server as my Zabbix Server.


$ sudo apt-get install -y libfontconfig1

$ wget https://dl.grafana.com/oss/release/grafana_7.0.5_amd64.deb

$ sudo dpkg -i grafana_7.0.5_amd64.deb


Then start the service


$ sudo service grafana-server start


Your Grafana server will be hosted at

http://[your zabbix server ip or domain name]:3000


Default login is

username : admin

password : admin


I add a MySQL data source and I also need to add a specific user which has read only access to the zabbix database.


$ mysql

CREATE USER 'grafana'@'localhost' IDENTIFIED BY 'password';

GRANT SELECT ON zabbix.* TO 'grafana'@'localhost';

FLUSH PRIVILEGES;


I then need to add a zabbix datasource plugin,


$ grafana-cli plugins install alexanderzobnin-zabbix-app


And then restart the grafana server


$ sudo service grafana-server restart


When configuring the Zabbix plugin, your API URL will be similar to this below.

https://[your zabbix server ip or domain name]/zabbix/api_jsonrpc.php


Also select the option for direct db access, and choose the MySQL data source you created a minute ago.



Grafana Zabbix Plugin
11:01

In this next few lectures we will look at SNMP

SNMP stands for Simple Network Management Protocol. 

Common devices that support SNMP are routers, switches, printers, servers, workstations and other devices found on IP networks.

Not every network device supports SNMP, or has it enabled, and there is a good chance you don't have an SNMP enabled device available that you can use in this lecture.

So, in this next few lectures, I will demonstrate installing SNMP on Ubuntu 18 server, Windows 10, Raspberry Pi and MacOSX

We will set up Zabbix to query using OIDs first.

We will manually create a few sample SNMP items.

Then demonstrate what setup and querying with MIB descriptions. MIB stands for Management Information Base.

And then use LLD to discover new SNMP devices and automatically configure them in Zabbix.

Setup SNMP Hosts in Zabbix
21:21

I configure one of my hosts to use a more sophisticated SNMP template. I will need to allow less restrictive SNMP OID prefixes in the SNMPD process, and then restart.

Query SNMP Hosts using OIDs
14:25

Querying SNMP agents with MID description is likely to fail by default.

The example below will fail if no MIB descriptions are installed on the server executing the snmpwalk command,

$ snmpwalk -v 2c -c mycommunity grafana.seanwasere.com IF-MIB::ifInOctets.1

> Cannot find module (IF-MIB)

> IF_MIB::ifInOctets.1: Unknown Object Identifier


We can enable querying by MIB descriptions by running this command on the Zabbix server itself.

$ sudo apt install snmp-mibs-downloader


Now this command will work

$ snmpwalk -v 2c -c mycommunity grafana.seanwasere.com IF-MIB::ifInOctets.1

> IF-MIB::ifInOctets.1 = Counter32: 566637161


I then update my items for the host in Zabbix, to query using MIB descriptions.

And then restart the Zabbix server process.

$ sudo service zabbix-server restart


Query SNMP Hosts using MIBs
07:54

I setup LLD Discovery Rules and Actions to find and auto configure all SNMP devices on my local network.

The rule and actions will scan all internal IP address for accessible SNMP daemon system descriptions, read the response, and then use that response to add it to a server group, and to also auto configure it the relevant template.

On my network I have,

- A Windows 10 workstation,

- A MacOSX,

- An Ubuntu 18 VM

- A centos 7 VM

- And a Raspberry Pi running Raspbian.

Setup LLD Discovery Rules and Actions to Auto Configure SNMP Devices
17:16

In this lecture I manually configure a CISCO 2950 24 port Catalyst Switch. You may not have an SNMP capable network switch or router to test with, but this demonstrates some of the aspects of adding a SNMP switch or router and some of the steps to consider. I recommend adding a SNMP network hardware device if possible.

Add a CISCO SNMP Enabled Network Switch
07:04

Receiving SNMP traps is the opposite of querying SNMP devices.

Information is sent from an SNMP device and is collected or "trapped" by Zabbix.

SNMP Traps are sent to the server on port 162 (as opposed to port 161 on the agent side that is used for queries).

So port 162 will need to be allowed on the Zabbix Server or Proxy, which ever will receive the SNMP traps.

For SNMP Traps to work, you need to configure some settings for either the Zabbix Server, or Zabbix Proxy.

Download zabbix_trap_receiver.pl

snmptrapd is an SNMP application that receives and logs SNMP TRAP and INFORM messages.

I demonstrate configuring my Cisco switch to send snmp traps to the server with snmptrapd listening. Zabbix proxy is also running on the same server, and will forward the messages onto the Zabbix server where the host is configured.

Setup SNMP Traps
18:28
Zabbix 5 Course Update Notice
00:22

MIB files can be analysed and converted to Zabbix templates containing Items, Discovery Rules and Value Types.

Before running this script, you should already have the required SNMP dependencies installed and configured correctly.

See the official course documentation for all commands.

Convert MIBs Files to Zabbix Templates
17:14
Prometheus Node Exporter Introduction and Install as a Service
05:27
Prometheus Node Exporter Manual Host Configuration
05:26
Setup LLD Discovery and Actions to Auto Configure Prometheus Node Exporters
07:28
Zabbix API Introduction and Examples
04:17
Zabbix API Test Tool
09:53
Zabbix API Python Example
03:01

We create a new user and a new user group with read only permissions, and no access to the front end, specific for when calling the api.

Zabbix API User Permissions
05:24