[Hands-On] Setting up Prometheus + Kafka Broker 1

Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
A free video tutorial from Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
Best Selling Instructor, Kafka Guru, 9x AWS Certified
4.7 instructor rating • 36 courses • 536,116 students

Learn more from the full course

Apache Kafka Series - Kafka Monitoring & Operations

Kafka Monitoring Setup with Prometheus and Grafana, Kafka Operations and Kafka Cluster Upgrades Hands-On. Setup in AWS

05:04:58 of on-demand video • Updated September 2020

  • Setup a Multi Broker Kafka Cluster in no-time in AWS (using CloudFormation)
  • Setup Administration Tools such as Kafka Manager, ZooNavigator, LinkedIn's Kafka Monitor
  • Setup Monitoring using Grafana and Prometheus
  • Learn how to perform a safe and automated Roll Restart of Kafka Brokers
  • Update Brokers Configurations in a safe way
  • Rebalance Partitions in a Kafka Cluster
  • Increase and Decrease the Replication Factor of Topics
  • Add a Broker to a Kafka Cluster
  • Service and Replace a Broker in a Kafka Cluster
  • Remove a Broker in a Kafka Cluster
  • Install Command Line Interface (CLI) tools to automate workflows
  • Upgrade a Kafka Cluster with no downtime
English [Auto] OK so we have quite a lot of things to do. So let's get started in setting up a cascade with premi us. So the fish thing will do is that we'll install the J mix exporter agent on our Kavakeb brokers will install then permit us on the administration machine and set it up as is system the service and then we'll view that in premi views that the data is indeed being pulled. And then we'll have to set up an exercise to other brokers and zookeeper because I want you to work too. OK. So let's get started. OK so the first thing I want to do is set up the Prometheans. So Promethea is too. And so what is Promethea is if you go there it says that it's a monitoring tool from metrics your insights and it's available it's open source monitoring solution. And so we'll get started we'll download it and so on. So the first thing you want to do though is set up the Prometheans pro Meath thius Jether JMS exporter agents. So this agent is basically something we want to set up and here's the good heart project for it and it's basically a way to expose the Jameco beans. So RJM mix metrics from Kafka via HGP so that Prometheans can consume them. So that's the general idea. It's pretty simple. And so basically it gives us to come in to run Java minus Java agent and this jar and this config and then our jar. So the first thing to do is download the jar from here on to our Cafcass boxes and then we'll have the metrics available. So let's let's have a look and do it step by step. OK. The first thing you want to do is S-sh into our cash box. So as I said to my first cash machine right here so that's good to the instances and take one to get started. I'll take the public IP which is right here and I'll do S-sh And here is the public. OK. So we are in Cascade 1. So if we do pseudo systems it's Yelle status Kafka. We can see that Africa is running and everything is great. Okay. So all the code is again quoted by me before the quarter so we don't have too much struggle. So what we need to do is create a premise used octree going to the directory then download the file they gave us and then download the configuration foolishest do this will copy this line. So the first thing is to make a directory and call it Prometheans. OK. And in there we'll going to download that Jever agent Jarre. That was from the get help page and we're done. Now there is a second thing we need to do is to set up the config. So if we go here as you can see here there is an example conflicts folder and within an example conflicts there's Cathcart 0 8 2. YAML and it's still very much valid right now. So this is a kind of YAML file we want to use to export our metrics. So we also need to download that file. And for this fairly easy we can just do a web gets the right command AUPs cuddliest again and paste it. And here we go. So now if you look in our directory We have an actually I forgot to do the right thing so I need to move these files. Obviously into my Prometheans directory so I'll move this one and I'll move the J-m X-1 into the premise to use directory. So if you look at the roots we have Kafka Kafka on those 11:45 I've got a product and then Prometheans. And if you could you premi Yes we can see that we have the GM X Java agents and the Kavkaz you're a tutor. If you look at what the Kafka is your email is. So let's have a look so nano Kafka. So this is basically a configuration and I wouldn't change it too much if I were you. But it basically says to export the patterns of the Kathe metrics based on these rules. And so these rules are made so that every Cafcass metric should be exported and just have a look for that file in case of changes. But this basically pulls everything from Cathcart into a premise here is formats. So this looks good. We have our Java agents and we have our kaffiyeh YAML. And so what we need to do now is basically change our Cafcass service definition so that it uses the Djerba agents to get started. So again just a reminder if we go here and we go to Java agent It says we need to run this minus Jever agent option pointing to J.J M-x Prometheans Java agents right here into configure YAML. And so for this we're going to have to edit our system default. So we'll do nano we'll do suit all of this because we need to be elevated cist EDC system the system can get a service. And so here we are. And so we need to change this file basically to add this Jether agent line. So right about execs start allright and environment equals and there is this variable called Kefaya options that you can pass options to and will automatically add this right line correctly. And so we need to edit this character options with the line we need. So for this because I'm lazy and I already typed it just to make sure we have it right I'll copy and paste this. Here we go. So let's have a look at what this did. So we do Kafka. Options equals minus other agents. So just like in the get Hubbard Basri which are so then the full path of my Prometheans Java agent. So if you didn't deliver it correctly in the right path it will air out for you 8 0 0 which is the port. It's going to be exported as well. So it's is yours what we can query. And then we also pass in the configuration. So the configuration is Kavkaz you're a tool at yemo and this is the full path. So this looks correct now exit it and press yes. And so we are ready. And so now if I do my restart of Kalka things would work. So pseudo systems that Yelle restart kaffiyeh. And it says Cafcass service change on disk run systems that G.L. daemon reload to reload units so offers to this pseudo system sitio daemon. And that's because we changed this file right here. Cathcart service so I run this and now I can restart GAFCON properly. So I'm restarting Kafka and this commands may take a while. OK it's done now. I can just do cysto students Jilek status kaffiyeh and Kafka is active and running so that means that hopefully there's other agents option get taken into account. Otherwise it would be crushed I guess. So if we do occur. Now on to local host port zero zero. We should see all the calfskin metrics. You see all of these things that's all the metrics. And so that's really good because this is what Promethea will get from Kafka. Every now and then. So from this step what we did is that we did set up Kavakeb Capricorn number one to have the Java JM agents very clean running in the service. So this looks correct. And now in the next step we need to go and set up the administration box. So I'm going to go on the right hand side. So here on the right hand side is going to be my administration box in here on the left hand side is going to be my catheter box. So let's go into the administration box so we can set up Prometheans there. And Mr. Ashton is right here. Here's my public IP. So that looks good. Now I'm going to S-sh into its S-sh into this. OK. I am in my administration box. And so one thing we can do already is see if we can access the metrics of the left hand side from the right hand side. So calfskin one metrics from the administration box. So for this we'll do a curl and the IP address a 172 31 1 31 port zero zero and yet it works. So we get all the metrics on the right hand side and pulled from the machine on the left hand side. So cool. So this is working we have a connectivity working between our administration box to our cash machine and now we're ready to set up Promethea on this machine. So let's have a look at how to set up premi thius. So let's just call in our old tutorial. It's pretty easy. There is basically on the Get hub of Promethea is a release download. So let's have a look right here. So if we go to premise yes and we go to the premise yes page. Here you go. So by the way supported with 18000 star so it's quite a popular one. If we go to releases we see that there is a version 2 1 3 1 2 at the time of recording and it basically gives us a linux arm 64 or whatever distribution you need for your stuff. So for our machine what we need is Linux HMD 64 so we're going to do a web gets. So here in our administration machine. Don't be mistaken. We download premi. Yes. So it can take a bit of time because this is quite a big file. So for me there is about 30 seconds or just pause. OK the file is now downloaded. So if we look into our directory we do see that we had a calf come and ensure that we download from before and now this Prometheans file. So we need to extract this fall for this. We run the tar command minus X Zed F to just extract the correct files so here we go. We run this. And now if we do L-L we see that now we have or else we see that now we have a directory called premises two to three top two Linux HMD. So we'll have to rename that directory. So I'll just move this into the premise. Yes. OK so now if we do ls we have a cad come an entire directory a premi fuse directory and then our zip file from before. And now we're ready to remove our zip files. So let me just remove the zip file. Here we go. So unless we have Kefaya manager and premi he is. And if you look into the premier views directory we see that there is some tool from his that YAML et cetera et cetera et cetera. So this is good. Now we need to look at this configuration file for me if you start YAML. So let's have a look at it. So nano Promethea is promiscuous. Yes. Yeah. OK. So let's have a look at just this year. So this is a global configuration and basically it's saying that every 15 seconds you are going to look for new data. So every 15 seconds it is going to query Kafka and say Kafka what do you have as new data. And so basically as you scroll down you can set up different jobs and it's called scrape conflicts and great conflict is basically a way to get metrics from wherever you want so as you can see here there is a job named Name name Prometheans which carries itself on local has ADHD. Do you see the metrics of Promethea as well. So what we need to do here is to add a scrape config so that we can scrape Kafka. Easy enough. I have everything set up for you. So if we go to Prometheans right here and go to Prometheans the YAML as you can see I have added a scrape config Jubb named karega study configfs targets. And here is the kaffiyeh IP. So we need to make sure that my IP is correct because it's different here so I'll do that from what I remember. It is one. 31 131 So one 31. OK. So this is my calfskin one 172 31. And so that's Cafcass one change IP for your use case. OK. And so I need you baby excuse. Paste this great config and even a base to global. So I'll just copy this whole thing and I'll actually remove this file and erase it so will remove CD into Prometheans will remove the premise is that YAML file and will creates a new one for me. Here's the YAML file. Paste it in as you can see now. This job should have a target of my calf. So it's just very that this is the right one. Exit and save. Yes and if you do a curl again on this it works. We get the metrics so the Prometheans configuration file is correct. And so now that we have this we should just start premi this. So for this is just one command dot slash Promethea us. So let's have a look and start Promethea is. And as you can see now it started really really quickly and it says server is ready to receive read web request. So now we actually have to go ahead and open premi views into our web browser and as you can see here the address is 000 slash port 1990. So we have to go to Port 1990. So let's have a look here. So now here I am in the web browser and I will go and open a new page and go to Port 90 90 and if everything works No it doesn't work obviously because maybe you've guessed I need to change a security group one more time. So in the security group when you add an inbound rule and will add the port 1990 from everywhere and will call this Prometheans me thius that looks good. So any time you get an error as a deports can't be access to whatever. This is the kind of things you need to look out for. OK so let's just refresh Aspasia now. So Copy that. Again port 1990 and if everything works now in the web browser we should see Prometheans. So this is bear with us. It doesn't look very very good as you say but it works really well. And so basically if you type any any query right here on Kafka where you get all the Cafcass metrics right here. So this is really really nice. If we get for example Kafka cluster or partition replicas count and executes we see all the value of the metric. This one particular metric. So it's not very usable so far. Right but it is all the metrics from Cathcart pulled by Prometheans and you just get the hint that it's working because of all these things right here all these metrics are available for us. So this is all the kind of stuff that will be displayed by Gravano. So now we're almost done. Promethea is indeed running but we need to set it up as a service just like before. So when you stop permissions and permit he has a stop he says See you next time. This is nice. See you next time from with yours. And so next what we have to do is go into this little directory I prepared for you and set up Promethea as a system default. So this will create a new system. Nano also pseudo Nano. It is C system D system and then Prometheans that service. And then when you fill something in. So just like before everything is ready for you in system Deara. Here we have premi feels that service and we can copy this entire thing in there. So this is provided directly by Prometheans. So here's the description it's a premise here sir. Here's the documentation and we have the same after before when the network is online. The user is going to be running this is easy to user and to start Promethea is very easy. We run the premi his command which is run before. And we also passed the config file as an argument. So this looks good control X control y. So let's just check right now if I refer to this page it says this site can't be reached. But now I'm going to do sudo service start Promethea. And obviously it's not service it's system D. I'm a bit system seeks Yelle start Prometheans. And now we're going to do a status on premi fuse just to make sure that everything is working. So this is active running. So this looks good. And I refreshed this page and we get our premi here is really available. And if I query for any command in the fingerpick accounts it works. So we have our metrics and everything all working in for me for us. So this looks nice. So far what we used in just as a summary we saw the Jamison's border on the caf to brokers we've installed the premises on the administration machines as a service we've used in premise use that the kaffiyeh is being pulled out and now in the next lecture is as I said you're going to have to set up the two other brokers and zookeeper. And that's a fun exercise for you. All right. So I was in this lecture with a solution to that.