Analyzing Logs with Kibana Dashboards

Sundog Education by Frank Kane
A free video tutorial from Sundog Education by Frank Kane
Founder, Sundog Education. Machine Learning Pro
4.5 instructor rating • 22 courses • 493,977 students

Learn more from the full course

Elasticsearch 7 and the Elastic Stack: In Depth and Hands On

Complete Elasticsearch tutorial - search, analyze, and visualize big data with Elasticsearch, Kibana, Logstash, & Beats.

16:27:03 of on-demand video • Updated February 2021

  • Install and configure Elasticsearch 7 on a cluster
  • Create search indices and mappings
  • Search full-text and structured data in several different ways
  • Import data into Elasticsearch using several different techniques
  • Integrate Elasticsearch with other systems, such as Spark, Kafka, relational databases, S3, and more
  • Aggregate structured data using buckets and metrics
  • Use Logstash and the "ELK stack" to import streaming log data into Elasticsearch
  • Use Filebeats and the Elastic Stack to import streaming data at scale
  • Analyze and visualize data in Elasticsearch using Kibana
  • Manage operations on production Elasticsearch clusters
  • Use cloud-based solutions including Amazon's Elasticsearch Service and Elastic Cloud
English So now that we've actually got file beat up and running and imported some access logs into a plastic search. Let's go back and use Cabana to visualize that data using some of the dashboards that come with file beat. It'll make life a lot easier and it's pretty fun too. So let me show you how to get started so it's pretty common to see Cabana being used to analyze a patchy access logs and Ara logs and they are some really great visualizations for that but we do need to install them first. They don't just come straight out of the box. So what's the type of following to take care of that. First let's see the entire file beat lives C.D. slash user slash Cher slash file beat slash bean. And from here we're gonna say pseudo file beat setup dash dash dashboards they make it nice and easy just let that go and do its thing and I am assuming that we still have Cabana running. You certainly should be unless you just started about your machine and it hasn't spun up yet. All right. No problem there. So that's all well and good. Let's restart the cabana service just to make sure it picks everything up sudo slash Ben's last system control stop cabanas out service and then we'll start it again at this point we'll have to give Cabana a few minutes to start backup so be patient and come back in five minutes. All right. So I think we've given Cabana ample time to start back up. Let's go back to our Web browser here and make sure we can get back to the cabana home page here successfully just reload that and yeah. Looks like it's responding so that's great. All right. So now that we have our dashboards let's dig into some of that log data that we just imported through file beat and visualize it using Cabana. So this is really what you would probably be doing in the real world guys if you're using elastic search for operational purposes. So first let's go to the little gear icon here and set up an index pattern for our new index. Analysts click on that here and there we have it. Cool. All right. So this is all set up already really. So we already have a time filter field set up on an app timestamp which is exactly what we want. And you can see that file being set up a lot of fields associated with each row of that log that's actually 1300 48 of them that you can dig into and a lot of them are just automatically created. So you can look through these and there's all sorts of goodness all a whole lot of data in there that we can explore. But let's just go to the Discover tab and start discovering shall we. So click on the little compass here and wait for that to load up. Now we're still in the Shakespeare index here so let's change that to file B and you'll see that initially we have no results and this is a very common issue when you're dealing with log data. You have to change the date range to actually see data. So by default it's only looking at the last 15 minutes. And since this isn't actually live data from a live Web site there is no data in the last 15 minutes. So let's go ahead and specify an explicit date range here. I'll just click on that and say absolute and we will start from April 29 2017 was when this data started. And we will take that until not now but another absolute date will do may 6 2017. All right. And let's hit update still not getting anything. The reason is I still have this filter on here from the Shakespeare exercise. Let's get rid of that. Now we've got some results. Very cool. All right so you can see that not only are we seeing you know individual log entries here broken out by individual fields we also have a nice little graph here showing the count of traffic that we're getting broken up per three hour block of time. So a nice little histogram being generated for us automatically. So without doing any work at all I can eyeball this and say hey these are my traffic patterns on my Web site. That's kind of interesting. Lots of peaks and valleys and whatnot that vary based on time of day and maybe even some crawlers hitting me at unexpected moments. So let's play around with things here we can for example drill in on things that have a 500 response code say let's see if we can find response dot status code here somewhere. There it is and where he had just without even trying again we can see the histogram of how those tax codes break down two hundred is success and only seventy five percent of my requests had a successful response says a little bit disconcerting isn't it. We can actually visualize this call see how that breaks down. This is not a lot of five hundreds. They are they do exist though. So if you wanted to explore those five hundred errors in more detail we could go back to the Discover tab here and open up a filter on that. Let's just add one and we can select a field from here that's going to be the. Type it in status code. There it is HDP dot response dot status code is 500 that school and we can see very plainly that we had a big spike of 500 error codes at the end of the May 5th we could dig into that in more depth or just click on that and we can zoom in there and see it broken down by a finer grain of time that we're now looking at five minute intervals so yeah there's this kind of range of time here where things went bad and down below here we could actually look into individual entries and try to figure out forensically what might have happened there are they all coming from the same user agent maybe. I don't know. Probably. So so far we've just been kind of using the built in pieces of Cabana that we've used before. But let me show you something really cool it's useless dashboards that we installed so here's the dashboard icon and look at all this stuff. This wasn't here before. Let's look at access logs access and error logs check that out. That's pretty sweet. It's kind of like you know looking at Google analytics or something you know back in the old days at least we actually have it like a geographic breakdown of where traffic's coming from that's kind of awesome. So you know we got a lot traffic from the United States and specifically from these cities even and apparently a lot coming from Eastern Europe there Russia. That's one What's up with that. And here you can dig into response codes over time. You see these stacked bar charts of the various status codes and maps over those to see individual ones that you might be interested in. That's cool. So for example five hundred errors indicate a server error that might be something I want to dig into and we can also look at how that breaks down as a pie chart on various different specific your URLs. So for example I can see a lot of those five hundreds were actually coming from Excel R.P. dot P HP and that probably indicates someone's trying to hack into my web site that see a former security hole in WordPress that I think someone was trying to exploit. So that's not very friendly of them is it. We can go ahead and click on that to drill into it some more actually see it from up here. So let's say we want to like dig into where five 500 were coming from during this particular five minute block just by clicking on that stacked piece of the histogram there we can select a filter automatically and apply that to our traffic. So we're now looking at the sources of the five hundred errors. And interestingly it's coming from a pretty diverse range of places let's dig in further to a specific time there. Now we're going down to the millisecond layer and yeah I mean so that's interesting. I mean it kind of looked at first like a hack from a specific place but either it's not either it is actually a real problem with a server that's causing it to show up from different origins or whoever is doing this hacking is covering their tracks really well and making it appear that they're coming from different places. I'm not you know a security engineer so I'm not really gonna know how to dig into this in more depth myself but you can see how this is a great forensic tool for kind of like troubleshooting problems on a Web site and really diving into what's driving them in depth. So really fun stuff. Let's take that filter off and we can change the timestamp back to something more interesting if you want. Let's you know go back a couple of days now and there we have it again pretty fast. Right. I mean you know we're asking you to do a lot of slicing and dicing of the state. We're like building these histogram these pie charts browser distributions here operating system breakdown for some reason I'm really hot on Mac OS it seems and this geographical map of geographical breakdown and just within you know under a second it was able to analyze all that data through elastic search and produced this really cool visualization. So again just sort of a testament to how fast elastic search can be. So this is what an axis narrow log for a patchy dashboard looks like in Cabana. And again I just encourage you to play around with it you know click on things change settings make new filters get a feel for what it can do. This is what you would actually be using you know for operational diagnostics and potentially a real world setting. So mess around with the guys and just to force you to do so let's do an exercise next with a specific problem to drill into.