Setting up Elasticsearch and Kibana

Imtiaz Ahmad
A free video tutorial from Imtiaz Ahmad
Senior Software Engineer & Trainer @ Job Ready Programmer
4.6 instructor rating • 12 courses • 293,485 students

Learn more from the full course

Complete Elasticsearch Masterclass with Logstash and Kibana

Learn how to build a search engine and break into big data by mastering Elasticsearch 6, Kibana and Logstash (ELK stack)

06:01:51 of on-demand video • Updated May 2021

  • Build an Elasticsearch 6 cluster from scratch
  • Perform various searches using the query DSL
  • Perform powerful realtime analytics using the Aggregations DSL
  • Combine Filters, Queries and Aggregations and understand document relevancy and searching
English [Auto] Hi there, and this lecture, I'm going to show you how to download and install Java on your Mac operating system, and then I'm going to show you how to download and configure ElasticSearch as well as Kobana. So first things first. Let's make sure that we don't have Java installed on the system already. Open up the terminal. All right. And if you don't know where that is, if you press the command key and the space key, it brings up the spotlight search and you can type in terminal, OK? And that is going to bring up this terminal window. And in here we can type in different Linux commands. But don't get intimidated by this. All we're going to see is whether we have Java installed on the system. So if you type in Java just like that and hit enter, notice that it says no Java runtime present requesting install. And so it brings up this icon here or this window, rather. So click on more info and it's going to take you to the Oracle website where you can download Java. OK, and so that's this thing right here. Java C downloads the Stanzler Standard Edition. And at the time of this recording and the latest edition is JDK 10. This JDK stands for Java Development Kit. So we're going to need to download the Java development kit for this course. So let's download this. And it's it's going to take you to another window except the license agreement by clicking on this radio button and then choose the operating system that pertains to your your setup. So for mine, it's Mac OS. If you're a Windows user, you'd have to download that. So for Mac, let's click on this file, which is going to start downloading a DMG file. And now it's asking us where to save it. I'm just going to save it on the desktop and just save. And in just a few moments, this should be done downloading depending on your Internet speed. OK, there you go, we see the green check mark there, that means it is done downloading. Let's open that file up and this is the actual installer. OK, so double click this icon and it's going to launch the Java, the JDK installer. And at the time of this recording, its version 10. So let's continue. Click next, and then we need to enter our administrator password. And then install software and now it's beginning to install those files and there we go, that's it, close and let's hit close here and now let's see if we have Java installed. Let's open up the terminal again. And if you type in Java now, lets it enter. Notice that things pop up right at it. No longer says that no Java runtime president actually shows a bunch of stuff. And so we can check the version if you type in Java Dash version. Notice saying Java version 10 is running on this machine. OK, so we're ready to go. We've downloaded and installed the Java development kit on our system, so let's close the terminal. We don't need that anymore. The next thing we need to do is go to the ElasticSearch website and download ElasticSearch. So go to Google and just type in elastic search. And I think it's a third or fourth link down here, it is OK. It says Open source research and analytics. Make sure you're at the correct link. Vlasák Tazo, click on that and then go to the downloads link right here on the top. Right. Or you can go to products. So let's click on downloads and then you choose one of these software. So the main workhorse for the ElasticSearch engine is, of course, this elasticsearch. This is what the entire ecosystem is based on. OK, so we of course, are going to need that in this course. And then we're also going to use Kobana. Kobana is used to interact with ElasticSearch and then you have these other auxillary tools such as log stasch and Beats. These are used to ingest data into elastic. If you want to send log files or large files into ElasticSearch, you can use log stasch and beats. We're actually going to be exploring Log Stachel later on in the course. So let's get started. The two software we need for now is ElasticSearch and Caivano, so let's download Lastic and this is a helpful screen. It has all the different types of downloads you can use. I'm going to click on this thaat. If you're using a Linux based operating system like the Mac or Ubuntu, just click on this Tor link and that will start the download. So where do I want to download? I'm going to download it to my home directory, which is Imtiyaz Imod and I'm going to download it in a folder which I created called ElasticSearch Course. Let's click that and in this folders where this file is going to be downloaded. So let's hit save and now the download is pretty much complete. It shouldn't take that long depending on your Internet speed. The next thing we need to download, let's go back is Kobana. OK, so I would say out of all of these tools that you see here. These two are going to be involved with 80 percent or 90 percent of the work. OK, so this is very important. Both of these the rest are just auxiliary tools down here. You've got a plastic cloud. This is for hosting it on the cloud. So these are other features that this website provides. So, again, these are the two workhorses for this course. So let's download Kobana and then I'm going to do the same thing for this. I'm going to download the Mac link that they have here. If you're using Linux or Ubuntu, you can use this. If you're using Windows, of course, you know what to do. So as download the Mac link and this will basically save it in the same elasticsearch underscore course folder that I created. OK, I recommend you pull the same thing. Let's follow the same convention as you're walking along with me, so let's save it. And there you go. The download is complete. So in the installation steps on down here, this is very helpful. This basically tells us the different steps that we need to take to start ElasticSearch and Kobana. So the first thing we need is to make sure we install ElasticSearch. OK, so we've already done that. Let's go to the the the directions for how to run ElasticSearch. We go back to this link right here. Click on Download Down. Here are the instructions for running ElasticSearch Notice. It says that we need to run this command. Ben Slash ElasticSearch if we're using a Mac or Linux based operating system. If you're using Windows, you have to run bin Backslash ElasticSearch that bat. So full screen here. Make sure you take a note of that. So let's go to our folder where we downloaded ElasticSearch as well as Kobana, which is right here. And let me navigate to the actual folder here. So these are the two software I'm going to double click ElasticSearch and that will extract it into this folder. So I no longer need this guy. Let me delete this and then I'm going to double click Kobana. And I will extract it into a Kobana folder. And there we go. We can delete this Kobana download, so now let's go back to the installation steps and notice we have to run ElasticSearch. So where do we run it from? Well, we can run it from the command line on Linux, the command line or the terminal. If you want to know where that is, you just press command key and spacebar and it'll bring up this spotlight search. OK, this is specific to the Mac users only, by the way. And you can just open up the terminal if you're using Ubuntu, you can open up terminal there, too. You should see an icon for that. And then for Windows, you open the command line, you know, the DOS command line. So in here, we need to navigate over to where we downloaded ElasticSearch. So I'm going to introduce a few commands. Don't worry, these are not rocket science is actually very simple. If you type in less, that will list the directories in your current directory to list all the files. So we're re downloaded. Classic search was in ElasticSearch course folder right here, so I'm going to switch to that directory and in Linux, the way to switch to that directory is you do a CD which stands for Change Directory and then you type in the directory name in which you want to go to. And that is ElasticSearch Underscore of course. So let's go there. And then you do allow us to list the contents of that folder. And we have, of course, these two folders, the ElasticSearch folder and the Kobana folder. So let's go to the ElasticSearch folder. And then it hit enter and then do analysis again, and you're going to see all of the files that exist within the elasticsearch folder, all of these. So what does it say? It says we need to run the ElasticSearch software by doing Behn slash elastic search like that. So hit enter. And there you go. It's now actually started ElasticSearch. So at this point, I'm not going to close this window. Make sure you leave this running. We're going to leave this running in the background in case we forget if we have ElasticSearch running or not. You know, this window is proof of the fact that ElasticSearch is, in fact, running. So if you follow the instructions, step three here is saying that we can execute this command, Colonel HTP and then the you are all that it shows there. So I'm going to copy that and open a new terminal window by clicking on new window or however you navigate in your system. So there's another terminal window. I haven't closed the other one that was running. I just minimize that. And now I just paste that kernel command and hit enter and boom notice we get a response back. OK, this is the response. And notice that it's giving us the version of the software that we're running at, which at the time of this recording is six point three point two. OK, so that means ElasticSearch is running successfully. As a matter of fact, it's running on this Eurorail. If I copy this, your URL specifically and open in another tab in the browser hit enter notice we get this response. OK, so we're good with ElasticSearch. Now let's start Kobana. Let's go back to the ElasticSearch website and let's go back to the tool set here and click on the Kobana download. So we've already downloaded Kobana. Let's go to the installation steps and see what we have to do to navigate to this particular location and execute Ben Kobana. But before that, before that, what we need to do is step number one. I forgot to mention this. It says Open config, Kobana, dot y, AML in an editor. OK, so let's do that. Let me go to the folder where we have Kobana. So here's our folder, ElasticSearch and Kobana. Let's open up Kobana and see what what they're talking about here. So they're saying go to configure Bernadotte YAML. So that's right here config. This is the Kobana file. I'm going to right. Click and open in a text editor. OK, it could be in any editor of your choice. I have sublime installed on this machine, so I'm going to open it in sublime text. And so what else are they saying? They're saying that once we open it set elasticsearch dot you url property to point to your elasticsearch instance. OK, so our elasticsearch instance is running on this particular euro. This is the default euro. So if you were to change it, you want to make sure that your Kobana configuration is pointing to the same location. So notice that this is our elasticsearch, that euro property. It's commented out. So it's saying that the Euro ElasticSearch and sense to use for all your queries. So I guess by default it has this set up already. So this is all comments and the comments are specified by this these pound signs. And so to register a line as a as a as code, you have to remove the pound. OK. Now, this is actually meaningful, meaning that Kobana will read this configuration property. Right. And understand that our elasticsearch instance is running on this euro at this point. No. OK, so I uncommented out that line. I got rid of that pound sign. So this one is is in the configuration. So save this file by doing a control s and let's just close it. OK, so now that we've made the changes in the config Kobana that YAML file and we've pointed ElasticSearch to the correct instance. Now all we have to do is run Kobana and the way to do that is we do we go to Ben and then we run Kobana. So let's let's go there. So where am I at this current location, lastone else I'm in, I'm in my home directory. OK, another way to find out where we are in the in the file system is you do a P, which stands for present working directory or these commands that I'm going over have nothing to do with ElasticSearch. These are Linux specific commands. So if you already know them, you can just cruise right along. But basically this is where I have my elasticsearch course folder. It exists in users imtiyaz. So let's go to ElasticSearch course folder and in here in the Kobana folder, let's do a CD, Kibwana. And by the way, if you want to, instead of typing out the whole thing, you just type in the first few characters and then do a tab. It fills out the rest of the folder for you. OK, so hit enter. And now inside of here, we've got the bin folder. OK, let's go into the bin. Actually, we don't even have to go into the bend, we could just execute the command, as you're telling us, which has been forward, KOBANA So let's run that hit enter. And in just a few moments, we should see some lines being printed on the screen. And there we go. So now notice it saying listening, it has info and listening and it's a server running at this particular URL. So I'm going to copy this URL and I'm going to paste it in the browser. Like that. And here we are, we're in the Kobana dashboard now, depending on the version of the software that you installed for Kobana, the interface might look a little bit different. This course used to be titled ElasticSearch five, but now it's ElasticSearch six. I've upgraded it. And that is why, you know, you're going to see a new interface for some lectures and other lectures. You might still see the old interface doesn't make a difference. You're still going to learn exactly what you need to know. For ElasticSearch Version six, the interface doesn't really make a difference. So if we go to visualize here, this is where we can actually visualize some reports. You're going to have different charts that you can use and do all kinds of fancy stuff. We'll get into that later. But most of the work that we're going to be doing in this course is going to be in dev tools or click on that. And this is where we are going to be typing our commands to interact with ElasticSearch. And when we type these commands here, we're going to get a response on the right side. OK, so I had something typed up already in the in this window. So you may not have this, uh, when you, of course, start up Kobana. So let me get rid of that. You may have some other commands as default, depending on your Kobana version, but don't run them just yet. We're going to go step by step as to how to issue commands to ElasticSearch. We're going to be typing them up on this side of the window and we're going to get a response on the right side of the window. OK, another tab is the monitoring tab. So click that. And so this is saying that monitoring is currently turned off, so if we turn on monitoring, this may take a few moments. So there you go. It recognizes that we have ElasticSearch winning, OK, and it is saying that the health is green, meaning that the health is good, it's up and running at. It tells us the availability and the JVM heap. This is basically telling us how much memory is being used by the GM software. Right. This is all built in Java. So Java has this thing called GBM, a Java virtual machine, and it has one gig pretty much allocated for this ElasticSearch software to be running. So right now it's utilizing only four hundred sixty nine megabytes out of, uh, about a gig worth of RAM available to it. And of course, the disk availability is pretty straightforward. You should know what that is. Now, one thing I want to make sure is that you understand Kobana can only run if ElasticSearch is running, OK? So over here, it says health is green. So carbon is running, the health is green. ElasticSearch is running. The health is green. If I turn off ElasticSearch, let's open up the elasticsearch window. So this is the Kobana window and this is the elasticsearch window. If I turn off ElasticSearch I could do a control, see, and that will exit the terminal, ok. It'll exit the program that was running in the terminal. So we just, we just exited out of ElasticSearch. ElasticSearch is no longer running on our system and all of a sudden notice it saying Log-in is currently disabled. Administrator should consult Kobana logs for details. OK, so Kobana is no longer talking with ElasticSearch. It's actually indicating that there's a problem. We can't even enter any any user name or credentials here. OK, so Kobana needs to be working with a live version of ElasticSearch. It should be running in the background. So let's start ElasticSearch again. As a matter of fact, in the logs for Kobana, right. This is is running. Notis giving us all kinds of warnings. OK. No living connexions. So it's it's looking for ElasticSearch to be running on that port 90 two hundred, but we turned off ElasticSearch, so let's start it up again. You can use the up key in the terminal and I will bring up a command that you used in the past. One of them has been ElasticSearch and I have other commands that we used. So this bin ElasticSearch is the one run run that's run that. And now in just a few moments, elasticsearch to start running and you're going to start seeing green letters here to indicate that, you know, things are looking better. So right now, it's still in morning state ElasticSearch is initializing. It's starting. And there we go, we start seeing good info, green info, and then it's giving us the status. Now, depending on your operating system, you might see different colors here, but the idea is that you should see info instead of warning. OK. And so now it's saying ElasticSearch monitoring Kluster status is active. So let me go back to the URL. Which is localhost Colan five, six year, and this is the Kobana Yooralla, by the way, so hit Enter and Kobana is up and running again. If you go to management excuse me, monitoring. Notice it's now back to green. OK, so there we go, we're up and running with ElasticSearch and Kobana. You're ready to rock and roll in the SCHAUS. Let me wrap it up here. Thanks for watching. I'll see you in the next lecher.