Setting up Elasticsearch and Kibana

-: Hi there, in this lecture, I'm gonna show you how to download and install Java on your Mac operating system and then I'm gonna show you how to download and configure Elastic Search as well as Kibana. So first things first, let's make sure that we don't have Java installed on the system already. Open up the terminal. All right, and if you don't know where that is, if you press the command key and the space key, it brings up this spotlight search and you can type in terminal, okay, and that is gonna bring up this terminal window. And in here we can type in different Linux commands, but don't get intimidated by this. All we're gonna see is whether we have Java installed on the system. So if you type in Java, just like that and hit enter. Notice that it says no Java runtime present, requesting install and so it brings up this icon here or this window rather. So click on more info and it's gonna take you to the Oracle website where you can download Java. Okay, and so that's this thing right here, Java SE downloads. This stands for Standard Edition, and at the time of this recording and the latest edition is JDK 10. This JDK stands for Java development kit. So we're gonna need to download the Java development kit for this course. So let's download this and it's gonna take you to another window. Accept the license agreement by clicking on this radio button and then choose the operating system that pertains to your, you know, your setup. So for mine it's Mac OS. If you're a Windows user, you'd have to download that. So for Mac, let's click on this file, which is gonna start downloading a DMG file. And now it's asking us where to save it. I'm just gonna save it on the desktop and just hit save. And in just a few moments, this should be done downloading, depending on your internet speed. Okay, there you go, we see the green check mark there, that means it is done downloading. Let's open that file up and this is the actual installer. Okay, so double click this icon and it's gonna launch the Java, the JDK installer. And at the time of this recording, it's version 10. So let's continue. Click next, and then we need to enter our administrator password. And then install software, and now it's beginning to install those files. And there we go. Let's hit close and let's hit close here. And now let's see if we have Java installed. Let's open up the terminal again, and if you type in Java now, let's hit enter. Notice that things pop up right, it no longer says that no Java runtime present. It actually shows a bunch of stuff and so we can check the version. If you type in Java-version, notice it's saying Java version 10 is running on this machine. Okay, so we're ready to go, we've downloaded and installed the Java development kit on our system, so let's close the terminal, we don't need that anymore. The next thing we need to do is go to the Elastic Search website and download Elastic Search. So go to Google and just type in Elastic Search and I think it's a third or fourth link down, here it is, okay, where it says open source, search and analytics, make sure you're at the correct link, elastic.co. Let's click on that and then go to the downloads link right here on the top right or you can go to products. So let's click on downloads and then you choose one of these softwares. So the main workhorse for the Elastic Search engine is, of course this, Elastic Search. This is what the entire ecosystem is based on, okay? So we of course are gonna need that in this course and then we're also gonna use Kibana. Kibana is used to interact with Elastic Search and then you have these other auxiliary tools such as Logstash, and Beats. These are used to ingest data into Elastic, if you wanna send log files or large files into Elastic Search. You can use Logstash and Beats. We're actually gonna be exploring Logstash later on in the course. So let's get started. The two software we need for now is Elastic Search and Kibana, so let's download Elastic and this is a helpful screen. It has all the different types of downloads you can use. I'm gonna click on this tar. If you're using a Linux based operating system, like the Mac or Ubuntu, just click on this tar link and that will start the download. So where do I wanna download? I'm gonna download it to my home directory, which is MT as mod and I'm gonna download in a folder which I created called Elastic Search Course. Let's click that and in this folder is where this file is gonna be downloaded, so let's hit save. And now the download is pretty much complete. It shouldn't take that long, depending on your internet speed. The next thing we need to download, let's go back, is Kibana, okay. So I would say out of all of these tools that you see here, these two are going to be involved with 80% or 90% of the work, okay, so this is very important. Both of these, the rest are just auxiliary tools. Down here you've got Elastic Cloud. This is for hosting it on the cloud. So these are other features that this website provides. So this, again, these are the two workhorses for this course. So let's download Kibana and then I'm gonna do the same thing. For this, I'm gonna download the Mac link that they have here. If you're using Linux or Ubuntu, you can use this. If you're using Windows, of course you know what to do. So let's download the Mac link and this will basically save it in the same Elastic Search_course folder that I created, okay. I recommend you'd follow the same thing. Let's follow the same convention, as you're walking along with me. So let's save it and there you go, the download is complete. So in the installation steps on down here, this is very helpful. This basically tells us the different steps that we need to take to start Elastic Search in Kibana. So the first thing we need is to make sure we install Elastic Search. Okay, so we've already done that. Let's go to the directions for how to run Elastic Search. If you go back to this link right here, click on download. Down here are the instructions for Running Elastic Search. Notice it says that we need to run this command, bin/ElasticSearch if we're using a Mac or Linux based operating system. If you're using Windows, you have to run bin\ElasticSearch.bat. So helpful screen here, make sure you take a note of that. So let's go to our folder where we downloaded Elastic Search as well as Kibana, which is right here and let me navigate to the actual folder here. So these are the two software, I'm gonna double click Elastic Search and that will like extract it into this folder. So I no longer need this guy. Let me delete this, and then I'm gonna double click Kibana and that will extract it into a Kibana folder. And there we go. We can delete this Kibana download. So now let's go back to the installation steps and notice we have to run Elastic Search. So where do we run it from? Well, we can run it from the command line on Linux. The command line or the terminal, if you wanna know where that is, you just press Command key and Space Bar and it'll bring up this Spotlight search. Okay, this is specific to the Mac users only, by the way and you can just open up Terminal. If you're using Ubuntu, you can open up terminal there too. You should see an icon for that and then for Windows, you open the command line, you know the DOS command line. So in here we need to navigate over to where we downloaded Elastic Search. So I'm gonna introduce a few commands. Don't worry, these are not Rocket Sciences, actually very simple. If you type in LS, that will list the directories in your current directory, it'll list all the files. So where we downloaded Elastic Search was in Elastic Search course folder right here. So I'm gonna switch to that directory and in Linux, the way to switch to that directory is you do a CD, which stands for Change Directory, and then you type in the directory name in which you want to go to and that is ElasticSearch_Course. So let's go there and then you do an LS to list the contents of that folder and we have of course, these two folders, the Elastic Search folder and the Kibana folder. So let's go to the Elastic Search folder and then hit enter and then do an LS again and you're gonna see all of the files that exist within the Elastic Search folder. All of these, so what is it saying? It says we need to run the Elastic Search software by doing Bin/ElasticSearch, like that. So hit enter and there you go. It's now actually started Elastic Search. So at this point, I'm not gonna close this window. Make sure you leave this running. We're gonna leave this running in the background. In case we forget if we have Elastic Search running or not, you know, this Window's proof of the fact that Elastic Search is in fact running. So if you follow the instructions, step three here is saying that we can execute this command cURL HTTP and then the URL that it shows there. So I'm gonna copy that and open a new terminal window by clicking on new window or however you navigate in your system. So there's another terminal window, I haven't closed the other one that was running, I just minimized that and now I just paced that cURL command and hit enter and boom, notice we get a response back. Okay, this is the response. And notice that it's giving us the version of the software that we're running, at which at the time of this recording is 6.3.2. Okay, so that means Elastic Search is running successfully. As a matter of fact, it's running on this URL. If I copy this URL specifically, and open in another tab in the browser, hit enter. Notice we get this response. Okay, so we're good with Elastic Search. Now let's start Kibana. Let's go back to the Elastic Search website and let's go back to the tool set here and click on the Kibana download. So we've already downloaded Kibana. Let's go to the installation steps and see what we have to do. So it's saying Navigate to this particular location and execute bin/Kibana. But before that, before that what we need to do is step number one, I forgot to mention this, it says open config/Kibana.yml in an editor. Okay, so let's do that. Let me go to the folder where we have Kibana. So here's our folder, Elastic Search in Kibana. Let's open up Kibana and see what they're talking about here. So they're saying go to config/Kibana.yml. So that's right here, config, this is the Kibana file. I'm gonna right click and open in a text editor, okay. It could be in any editor of your choice. I have Sublime installed on this machine so I'm gonna open it in Sublime Text. And so what else are they saying? They're saying that once we open it, set elasticsearch.URL property to point to your Elastic Search instance. Okay, so our Elastic Search instance is running on this particular URL. This is a default URL. So if you were to change it, you wanna make sure that your Kibana configuration is pointing to the same location. So notice that this is our Elasticsearch.URL property. It's commented out. So it's saying that the URL of Elastic Search tends to use for all queries. So I guess by default it has this set up already. So this is all comments and the comments are specified by these pound signs, and so to register a line as code, you have to remove the pound. Okay, and now this is actually meaningful, meaning that Kibana will read this configuration property, right and understand that our Elastic Search instance is running on this URL at this port number. Okay, so I uncommented out that line, I got rid of that pound sign. So this line is in the configuration. So save this file by doing a control + S and let's just close it, okay. So now that we've made the changes in the configKibana.yml file and you know we've pointed Elastic Search to the correct instance, now all we have to do is run Kibana. And the way to do that is we go to bin and then we run Kibana, so let's go there. So where am I at this current location? Let's do an LS. I'm in my home directory, okay. Another way to find out where we are in the file system is you do a PWD, which stands for present working directory, right. These commands that I'm going over have nothing to do with Elastic Search. These are Linux specific commands. So if you already know them, you can just cruise right along. But basically, this is where I have my Elastic Search course folder. It exists in Users/MTS. So let's go to Elastic Search course folder and in here, in the Kibana folder, let's do a CD Kibana and by the way, if you wanna, instead of typing out the whole thing, you just type in the first few characters and then do a tab, it fills out the rest of the folder for you, okay. So hit enter and now inside of here, we've got the bin folder, okay, let's go into the bin. Actually, we don't even have to go into the bin, we could just execute the command as they're telling us, which is bin/Kibana. So let's run that. Hit enter, and in just a few moments, we should see thumb lines being printed on the screen. And there we go. So now notice it's saying listening. All right, it has info and listening and it's a server running at this particular URL. So I'm gonna copy this URL and I'm gonna paste it in the browser like that. And here we are, we're in the Kibana dashboard. Now, depending on the version of the software that you installed for Kibana, the interface might look a little bit different. This course used to be titled Elastic Search five, but now it's Elastic Search six, so I've upgraded it and that is why you know you're gonna see a new interface for some lectures and other lectures, you might still see the old interface, doesn't make a difference. You're still gonna learn exactly what you need to know for Elastic Search version six. The interface doesn't really make a difference. So if we go to visualize here, this is where we can actually visualize some reports. You're gonna have different charts that you can use, and do all kinds of fancy stuff. We'll get into that later. But most of the work that we're gonna be doing in this course is gonna be in dev tools. So click on that and this is where we are gonna be typing our commands to interact with Elastic Search and when we type these commands here, we're gonna get a response on the right side. Okay, so I had something typed up already in this window, so you may not have this when you of course start up Kibana. So let me get rid of that. You may have some other commands as default, depending on your Kibana version, but don't run them just yet. We're gonna go step by step as to how to issue commands to Elastic Search. We're gonna be typing them up on this side of the window and we're gonna get a response on the right side of the window, okay. Another tab is the monitoring tab. So click that, and so this is saying that monitoring is currently turned off. So if we turn on monitoring, this may take a few moments. So there you go. It recognizes that we have Elastic Search running, okay and it is saying that the health is green, meaning that the health is good, it's up and running and it tells us the disc availability and the JVM heap. This is basically telling us how much memory is being used by the JVM software, right. This is all built in Java. So Java has this thing called JVM, a Java virtual machine and it has one gig pretty much allocated for the Elastic Search software to be running. So right now it's utilizing only 469 megabytes out of about a gig worth of RAM available to it. And of course, the discs availability is pretty straightforward. You should know what that is. Now, one thing I wanna make sure is that you understand, Kibana can only run if Elastic Search is running. Okay, so over here it says health is green. So Kibana is running, the health is green, Elastic Search is running, the health is green. If I turn off Elastic Search, let's open up the Elastic Search window. So this is the Kibana window and this is the Elastic Search window. If I turn off Elastic Search, I could do a control + C and that will exit the terminal, okay? It'll exit the program that was running in the terminal. So we just exited out of Elastic Search. Elastic Search is no longer running on our system and all of a sudden notice it's saying login is currently disabled. Administrator should consult Kibana logs for details. Okay, so Kibana is no longer talking with Elastic Search. It's actually indicating that there's a problem. We can't even enter any username or credentials here. Okay, so Kibana needs to be working with a live version of Elastic Search. It should be running in the background. So let's start Elastic Search again, as a matter of fact, in the logs for Kibana, right, this is where Kibana is running. Notice is giving us all kinds of warnings, okay? No living connections. So it's looking for Elastic Search to be running on that port, 9,200, but we turned off Elastic Search. So let's start it up again. You can use the up key in the terminal and that will bring up a command that you used in the past. One of them has Bin Elastic Search and I have other commands that we used. So this bin/ElasticSearch is the one we wanna run. Let's run that and now, in just a few moments, Elastic Search is gonna start running and you're gonna start seeing green letters here to indicate that you know, things are looking better. So right now, it's still in warning state. Elastic Search is initializing, it's starting. And there we go, we start seeing good info, green info and then it's giving us the status. Now, depending on your operating system, you might see different colors here but the idea is that you should see info instead of warning. Okay, and so now it's saying Elastic Search monitoring cluster status is active. So let me go back to the URL, which is localhost:5601. This is the Kibana URL, by the way. So hit enter, and Kibana is up and running again. If you go to management, excuse me, monitoring, notice it's now back to green. Okay, so there we go. We're up and running with Elastic Search and Kibana. You're ready to rock and roll on this course. Let me wrap it up here, thanks for watching. I'll see you in the next lecture.