Analyzing Shakespeare with Kibana

Sundog Education by Frank Kane
A free video tutorial from Sundog Education by Frank Kane
Founder, Sundog Education. Machine Learning Pro
4.5 instructor rating • 22 courses • 454,020 students

Lecture description

Although Kibana is often used to visualize log data, it can do so much more. Let's actually turn the power of Kibana into gaining some new insights into the works of William Shakespeare.

Learn more from the full course

Elasticsearch 6 and Elastic Stack - In Depth and Hands On!

Search, analyze, and visualize big data on a cluster with Elasticsearch, Logstash, Beats, Kibana, and more.

08:03:25 of on-demand video • Updated May 2019

  • Install and configure Elasticsearch 6 on a cluster
  • Create search indices and mappings
  • Search full-text and structured data in several different ways
  • Import data into Elasticsearch using several different techniques
  • Integrate Elasticsearch with other systems, such as Spark, Kafka, relational databases, S3, and more
  • Aggregate structured data using buckets and metrics
  • Use Logstash and the "ELK stack" to import streaming log data into Elasticsearch
  • Use Filebeats and the Elastic Stack to import streaming data at scale
  • Analyze and visualize data in Elasticsearch using Kibana
  • Manage operations on production Elasticsearch clusters
  • Use cloud-based solutions including Amazon's Elasticsearch Service and Elastic Cloud
English Now that we have come down in style we can play around with it. Really that's the best way to learn it just sort of dive in and explore it's pretty intuitive to use. Let's play with the works of William Shakespeare you might recall that we installed that way back on our set up lecture and maybe we can extract some insights about the works of William Shakespeare because we can. And it's sort of a fun example to work with initially, So let's dive in. Once kibana is installed you should be seeing a screen like this under the management tab should be selected by default and the first thing we need to do is configure an index pattern. This is really the most unintuitive piece of kibana. So we just need to tell it what index we want to analyze. Given some pattern it can do more than one at a time given a pattern. But for us we want to analyze the Shakespeare index that we created earlier and we can go ahead and hit create and you can see that it is picking up all the fields in here it is good to familiarize yourself with the different fields that you have to work with here such as speaker and text underscore entries the actual lines of each line of the place that are in there. So given that we can start to mess with Shakespeare data, click on the discover tab if you just want to search and interactively mess around with that data. So this is one easy way of doing searches in a elasticsearch without having to write json queries. Hoo. And we were friendly pretty much done with that world. So let's for example search for the word die. I mean Shakespeare was a pretty morbid person a lot of his plays involve a lot of death and murder and whatnot. And you can see that he's pretty fond of the word die and it turns out that in the play A Midsummer Night's Dream is actually a line that just says now die die die died. Wow. It's kind of dark stuff but you know these were dark times. Now we can further break this down by play name or speaker or whatever you want. So let's see how that works. Let's click on Play name here. And you can see that with a click on the single button here we can see how often the word die appears in individual plays and it is most common in the play Measure for Measure, file by Richard the Third and then Henry the sex and Romeo and Juliet so you can kind of get a little taste here of what's the most morbid plays are in the Shakespeare index. How about by speaker, looks like these characters say the word die a lot. Talbot as Bela Gloucester Gloucester these guys are pretty much tied for first place. Who are these people anyway. Who is Gloucester. Well we can do a search for an individual field up here. If you do is say the field name speaker:Gloucester for example and that will search the field name in particular for the term Gloucester and we can see here that Gloucester appears most frequently in the play. Richard the Third and also in Henry the six and also in King Lear So this character appears in more than one play which is why he's so popular and apparently he talks about death a lot. So there you have it. Cool stuff. You can also visualize this stuff if you just click on this handy dandy visualized button. That's pretty awesome. So we can actually get a breakdown right there with a single click or the button of how often Gloucester appears in each individual play. So you can see that he has you know over 700 lines 743 to be precise. And Richard the Third and in King Lear 364 times and so on and so forth and just a brief cameo appearance in Henry the Fifth. Let's look at visualisations in more depth. We can do some pretty cool stuff here. I mean visualisations are usually created in this context of like analyzing law data or more structured numerical data and things like that because usually it's about you know making charts and graphs. But there are visualizations you can do strictly on text data as well. Let's click the handy dandy creative visualization button here and let's make a tag cloud. This is cool we can make a tag cloud out of all the works of William Shakespeare. And it's really easy to do. All we need do is select the Shakespeare index and go ahead and add in a bucket We'll start with the bucket for terms and select the text entry.keyword term and let's go ahead and make That's 100 hundred terms in our tag cloud and just hit play and there you get a tag cloud of how often each search term appears and the entire works of William Shakespeare. That's kind of awesome huh. So exit and exeunt apparently are the most frequently used words in the works of William Shakespeare and Antipholus I don't even know what that means. I mean I'm sure that means something very significant to a Shakespeare scholar and some interesting stuff in here. Bishop Countess these are all things that appear frequently within the works of William Shakespeare. What else can we do. Let's do it by speaker instead. See what speakers appear in Shakespeare the most often. And you can see here that overall the character Gloucester that we looked at earlier is in fact the most popular character in all of William Shakespeare's works so hey we've got some interesting insights into Shakespeare by using kibana already. Hamlet is just second place from the looks of it. And I go Falstaff. I've heard of that. King Henry the Fifth. Very cool stuff. So you can see that with just a few clicks of the mouse here you can do some pretty interesting stuff in kibana and get some very visually satisfying results so it's hard to underestimate the power of that you know I mean when you have sort of a nice UI to work with to actually analyze your data with it sort of encourages you and motivates you to explore your data more deeply. So it's not just for good looks. You know I think this does have real value in motivating people to extract more meaning and more insights out of the data that they have. And with a elasticsearch you can do it very quickly. I mean think about how quickly that came back right. Like I just hit play here and bang that that tag cloud just came right up. So the real power here is that you can do complex aggregations and visualize them in beautiful ways with kibana with just a click of a button and a few milliseconds of time. I mean it's really really awesome stuff. This gets me irrationally excited. Now I show you something else about Kibana that you're really gonna love. If you want to click on the dev tools tab here check this out. You can actually enter in json requests without dealing with the console at all. So if you want to get down and dirty. We don't have to use the console anymore and we don't want to deal with all that craziness about dealing with tabs so so we can for example just clear out all this existing stuff that we just ran and type in our own arbitrary json requests. There is a shorthand available here instead of saying you know the server name and the port name and Currall and all that we can just seven get follow with what you would have put at the end of the actual uri for your elasticsearch host. So you know already we've saved quite a few keystrokes and the beautiful thing is we can actually use the tab key as it was intended so we can just say query. And it's even automatically completing our brackets for us which saves us even more time it's automatically tabbing things for us so a lot easier to deal with it in here. Let's say that phrase and we can say title Star Wars and look at already close that off for us. And now we can set up our aggs, set of our titles aggregation which consists of two aggregations a terms and a sub aggregation called, The average rating which contains an attorney average aggregation on the field rating and looks like we are missing one closing bracket there. But hey that was a lot easier to type in than using it on the console right. So we have a somewhat complicated query here. Let's go ahead and hit the play button to execute that. And we got back the same results as before only it's a lot prettier to look at and a lot easier to type in. So in the future when you're doing json requests you don't have to use CURL anymore. We can actually use this console within kibana to make life a lot simpler. So. Oh man do you wish we had this way back I think at the beginning of the course. Anyway that's about kibana for you so I encourage you to play some more with this. You know go back to the discover tab or the and search for more specific fields or explore your data in more ways click on a visualization tab and see what other visualizations are available and just mess around and you'll quickly get the feel of what you can do with kibana and I'm going to make you mess around in our next exercise.