[Hands-On] Setting up Prometheus + Kafka Broker 1

A free video tutorial from Stephane Maarek | AWS Certified Cloud Practitioner,Solutions Architect,Developer
Best Selling Instructor, 10x AWS Certified, Kafka Guru
Rating: 4.7 out of 5Instructor rating
65 courses
2,420,393 students
[Hands-On] Setting up Prometheus + Kafka Broker 1

Learn more from the full course

Apache Kafka Series - Kafka Monitoring & Operations

Kafka Monitoring Setup with Prometheus and Grafana, Kafka Operations and Kafka Cluster Upgrades Hands-On. Setup in AWS

05:07:43 of on-demand video • Updated March 2024

Setup a Multi Broker Kafka Cluster in no-time in AWS (using CloudFormation)
Setup Administration Tools such as Kafka Manager, ZooNavigator, LinkedIn's Kafka Monitor
Setup Monitoring using Grafana and Prometheus
Learn how to perform a safe and automated Roll Restart of Kafka Brokers
Update Brokers Configurations in a safe way
Rebalance Partitions in a Kafka Cluster
Increase and Decrease the Replication Factor of Topics
Add a Broker to a Kafka Cluster
Service and Replace a Broker in a Kafka Cluster
Remove a Broker in a Kafka Cluster
Install Command Line Interface (CLI) tools to automate workflows
Upgrade a Kafka Cluster with no downtime
English [Auto]
So we have quite a lot of things to do. So let's get started and setting up Kafka with Prometheus. So the first thing we'll do is that we'll install the Gem Exporter Agent on our Kafka brokers, we'll install then Prometheus on the administration machine and set it up as a system service and then we'll view that in Prometheus that the Kafka data is indeed being pulled and then we'll have to set up as an exercise the two other brokers and zookeeper, because I want you to work, too. Okay. So let's get started. Okay. So the first thing I want to do is set up the Prometheus. So Prometheus tool and so what is Prometheus? If you go there, it says that it's a monitoring tool from matrix to insights and it's available its open source monitoring solution. And so we'll get study, we'll download it and so on. So the first thing you want to do though is set up the Prometheus, Prometheus Java gem, ex exporter agents. So this agent is basically something we want to set up and here's the GitHub project for it and it's basically a way to expose the gem x beans. So our Gem X matrix from Kafka via HTTP so that Prometheus can consume them. So that's the general idea. It's pretty simple. And so basically it gives us the command to run Java minus Java agents and this jar and this config and then our jar. So the first thing you do is download the jar from here onto our Kafka boxes and then we'll have the matrix available. So let's, let's have a look and do it step by step. The first thing you want to do is ssh into our Kafka box, so ssh into my first Kafka machine right here. So let's go to the instances and take Kafka one to get started. I'll take the public IP, which is right here and I'll do SSH. And here's the public IP. Okay. So we are in Kafka one. So if we do pseudo. System. CTL Uh, status. Kafka We can see that Kafka is running and everything is great. Okay, so all the code is, again coded by me before the course, so we don't have too much trouble. So what we need to do is create a Prometheus directory, go into the directory, then download the file they give us and then download the configuration. So let's just do this. We'll, um, copy this line. So the first thing is to make a directory and call it Prometheus. And in there we're going to download that Java agent jar that was from the GitHub page and we're done. Now there is a second thing we need to do is to set up the config. So if we go here, as you can see here, there is an example configs folder and within an example config there is Kafka 082. YAML and it's still very much valid right now. So this is the kind of YAML file we want to use to export our matrix. So we also need to download that file. And for this fairly easy, we can just do a web gets the right command. Oops. Let me copy this again and paste it. And here we go. So now if we look in our directory, we have and actually I forgot to do the right thing. So I need to move these files, obviously. Into my Prometheus directory. So I'll move this one and I'll move the next one into the Prometheus directory. So if we look at the roots, we have Kafka. Kafka on those core 2.12 Kafka and then Prometheus. And if we go to Prometheus, we can see that we have the Gen-X Java agents and the Kafka 082 YAML. If you look at what the Kafka, uh, 082. yaml is, so let's have a look. So nano Kafka. So this is basically a configuration and I wouldn't change it too much if I were you, but it basically says to export the patterns of the Kafka matrix based on these rules. And so these roles are made so that every Kafka matrix should be exported. Just have a look at for that file in case of changes. But this basically pulls everything from Kafka into a Prometheus format. So this looks good. We have our Java agents and we have our Kafka at YAML. And so what we need to do now is basically change our Kafka service definition so that it uses the Java agents to get started. So again, just a reminder. If we go here and we go to the agent, it says we need to run this minus Java agent option pointing to x prometheus java agents right here into configure YAML. And so for this, we're going to have to edit our system default. So we'll do nano. And we'll do a pseudo of this because we need to be elevated. So Etsy systemd system. Kafka wrote service. And so here we are. And so we need to change this file basically to add this Java agent line. So right above exec start all the right environment equals and there is this variable called Kafka options that you can pass options to and we'll automatically add this right line correctly. And so we need to edit this Kafka options with the line we need. So for this because I'm lazy and I already typed it just to make sure we have it right, I'll copy and paste this. Here we go. So let's have a look at what this did. So with you, Kafka options equals minus Java agents. So just like in the GitHub repository, we just saw then the full path of my Prometheus Java agent. So if you didn't download it correctly in the right path, it will air out for you 8080 which is the port. It's going to be exported as so 8080 is what we can query and then we also pass in the configuration. So the configuration is Kafka 082. YAML and this is the full path. So this looks correct. Now exit it and press. Yes. And so we are ready. And so now if I do my restart of Kafka, if things are work so pseudo system CTL restart Kafka and it says Kafka service change on disk run systems it'll daemon reload to reload units. So I'll first do this pseudo. Sys CTL Damon. And that's because we changed this file right here. Kafka at service. So I'll run this and now I can restart the Kafka properly. So I'm restarting Kafka, and this command may take a while. Okay, it's done. Now I can just do so. Still sit still. Status. Kafka and Kafka is active and running. So that means that, uh, hopefully this Java agent option get taken into account. Otherwise it would be crashed. I guess. So if we do a curl now onto local host. Part 080. We should see all the Kafka metrics. You see all these things, that's all the Kafka metrics. And so that's really good because this is what Prometheus will get from Kafka every now and then. So from this step, what we did is that we did set up Kafka broker number one, to have the Java Emacs agents directly running in the service. So this looks correct. And now in the next step, we need to go and set up the administration box. So I'm going to go on the right hand side. So here on the right hand side is going to be my administration box. And here on the left hand side is going to be my Kafka box. So let's go into the administration box so we can set up Prometheus there. So administration is right here. Here is my public IP. So that looks good. Now I'm going to ssh into it. So ssh into this. Okay. I am in my administration box. And so one thing we can do already is see if we can access the matrix of the left hand side from the right hand side. So Kafka, one matrix from the administration box. So for this we'll do a curl and the Kafka IP address. So. One 7231. One 31.800. And yeah, it works. So we get all the Kafka metrics on the right hand side and pulled it from the machine on the left hand side. So cool. So this is working. We have a connectivity working between our administration box to our Kafka machine, and now we're ready to set up Prometheus on this machine. So let's have a look at how to set up Prometheus. So let's just call that in our little tutorial. It's pretty easy. There is basically on the GitHub of Prometheus a release download. So let's have a look right here. So if we go to Prometheus and we go to the Prometheus page, here we go. So by the way, it's a project with 18,000 stars, so it's quite a popular one if we go to releases. We see that there is a version 2.3.2 at the time of recording and it basically gives us a Linux ALM 64 or whatever the the um distribution you need for your stuff. So for our machine, what we need is Linux AMD 64. So we're gonna do a web get so here in our administration machine. Okay don't be mistaken we download Prometheus so it can take a bit of time because this is quite a big file. So for me, there is about, uh, 30 seconds, so I'll just pause. Okay. The file is now downloaded, so if we look into our directory, we do see that we had, uh, Kafka monitor that we downloaded from before and now this Prometheus file. So we need to extract this file for this. We run the TR command minus x, z, f to just extract the correct file. So here we go. We run this and now if we do lx we see that now we have our LS We see that now we have a directory called Prometheus 2.3.2 Linux IMG so we'll have to rename that directory. So I'll just move this into Prometheus. Okay. So now if we do ls we have a Kafka monitor directory, a Prometheus directory and then our zip file from before and now we're ready to remove our zip file. So let me just remove the zip file. Here we go. So if we do ls we have Kafka, monitor and Prometheus, and if we look into the Prometheus directory, we see that there is prompt tool, Prometheus, YAML, etc., etc., etc. So this is good. Now we need to look at this configuration file prometheus yaml. So let's have a look at it. So nano prometheus. Prometheus. Prometheus. GMO. Okay, so let's have a look. I'll just kiss here. So this is a global configuration and basically it says that every 15 seconds you are going to look for new data. So every 15 seconds it is going to query Kafka and say Kafka, what do you have as new data? And so basically as you scroll down, you can set up different jobs and it's called scrape configs. And scrape config is basically a way to get metrics from wherever you want. So as you can see here, there is a job name named Prometheus which queries itself on localhost 8080 to see the metrics of Prometheus. But so what we need to do here is to add a scrape config so that we can scrape Kafka. Easy enough. I have everything set up for you. So if we go to Prometheus right here and go to Prometheus YAML, as you can see, I have added a scrape config job named Kafka static configs targets and here is the Kafka ip. So I need to make sure that my kafka ip is correct cuz it's different here. So I'll do. Um, from what I remember it is one 7131 131. So one. 31. So this is my Kafka one. One, 72, 31 and 31. And so that's Kafka one. Change IP for your use case. And so I need to basically paste this scrape config and even paste a global. So I'll just copy this whole thing and I'll actually remove this file and erase it. So we'll, we'll see into Prometheus, we'll remove the Prometheus, the YAML file and we'll create a new one Prometheus YAML file, paste it. And as we can see now, this job should have a target of my Kafka. So let's just verify that this is the right one. I'll exit and save. Yes. And if we do a curl again on this, it works. We get the metrics. So the Prometheus configuration file is correct. And so now that we have this, we should just start Prometheus. So for this it's just one command dot slash Prometheus. So let's have a look and start Prometheus. And as you can see now, it started really, really quickly and it says Server is ready to receive REB Web requests. So now we actually have to go ahead and open Prometheus into our web browser. And as you can see here, the address is 000 slash port 1990. So we have to go to port 1990. So let's have a look here. So now here I am in my web browser and I will go and open a new page and go to Port 9090. And if everything works. No, it doesn't work, obviously, because maybe you've guessed I need to change the security group one more time. So in the security group, we need to add an inbound rule and we'll add the port 1990 from everywhere, and we'll call this Prometheus. Pro Mathias. That looks good. So any time you get an error as in the report can't be accessed or whatever, this is the kind of things you need to look out for. Okay, so let's just refresh this page now. So copy the URL again. Port 1990. And if everything works now in the web browser, we should see Prometheus. So this is Prometheus. It doesn't look very, very good, I have to say, but it works really well. And so basically if you type any, um, any query right here on Kafka, well you get all the Kafka metrics right here. So this is really, really nice. If we get, for example, uh, Kafka cluster partition replicates, count and executes, we see all the value of this metric, this one particular metric. So it's not very usable so far. Right. But is all the metrics from Kafka pulled by Prometheus and you just get the hint that it's working because of all these things right here, all these metrics are available for us. So this is all the kind of stuff that will be displayed by Gravano. So now we're almost done. Um, Prometheus is indeed running. But we need to set it up as a service just like before. So I'm going to stop Prometheus and Prometheus and stop it says, see you next time. This is nice. See you next time, Prometheus. And so next what we have to do is go into this little directory appropriate for you and set up Prometheus as a system default. So for this, we'll create a new system D nano file. So pseudo nano ETSI system. The system. And then Prometheus. That service. And now we need you fill something in. So just like before, everything is ready for you in systemd. Right here we have Prometheus that service and we can copy this entire thing in there. So this is provided directly by Prometheus. So here's the description. It's a Prometheus server, here's the documentation and we have the same after. Before, when the network is online, the user is going to be running. This is easy to user and to start Prometheus very easy. We run the Prometheus command which is run before and we also pass the config file as an argument. So this looks good. Control X, control Y. So let's just check right now if I refresh this page, it says this site can't be reached, but now I'm going to do pseudo service start. Prometheus. And obviously it's not service. It's system. Systemd. I'm a bit. System. Ctl. Start Prometheus. And now we're going to do a status on Prometheus just to make sure that everything is working. So it says active running. So this looks good. And I refresh this page and we get our Prometheus ready available. And if I worry for any, um, command instinct replica accounts, it works. So we have our metrics and everything all working in Prometheus. So this looks nice so far. What we've did, just as a summary, we've installed the gem exporter on the Kafka brokers, we've installed the Prometheus on the administration machines as a service. We viewed in Prometheus that the Kafka data isn't being pulled out. And now in the next lecture is, as I said, you're going to have to set up the two other brokers and zookeeper, and that's a fun exercise for you. All right. So I will see you in this lecture with the solution to that.