Mastering ElasticSearch 6.x and the Elastic Stack

We’ve seen how we can insert data into ES, but we need to understand more about that process to be effective.

   •  Documents are the foundation of data within ES

   •  Insert, update and delete documents into ES a few ways

   •  Finish up by retrieving the documents back

Understand what options do we have for storing data within a document.

   •  Review various data types within ES

   •  Configure ES to be able to index and store our docs

   •  Validate documents and mappings

Pick up on more details to how we classified data.

   •  Review a mapping file

   •  Insert mapping file into ES

   •  Verify mappings of our documents

We’ve started inserting documents, but we need to learn how we can arrange groups of documents.

   •  Find example data to insert for different use cases

   •  Setup indexes for ‘reference’ data as well as time based data

   •  Alias, reindex and delete indexes via APIs

We’re on a roll, we have all kinds of data and options to put data in, but, we need to be familiarised with how we get data back.

   •  Insert new data and perform basic queries

   •  Explore filtering and searching via different API

   •  Look at aggregations and buckets

Kibana can be overwhelming at first, there are so many components that you need to understand before you can decide how to use it.

   •  First we’ll browse through the user interface

   •  We’ll cover various components that we are going to use for searching

   •  Get ready to search

Kibana needs to know how your data is stored within ES. It can auto discover a lot of things, but you need to start by telling Kibana what indexes to use.

   •  Create an index pattern

   •  Determine what ‘regex’ to use depending on your use case

   •  See how index patterns appear in the search tab

In this video, we will understand that If you’re looking at time based data, you’ll have specific searching needs.

   •  Walk through an overview of how time ranges are selected

   •  Define different time range selection options

   •  Use the built in interface to change time ranges

Learn how Kibana provides different ways to search for specific data.

   •  First we’ll look into using the search bar

   •  Then we’ll use the top fields aggregations to discover data without typing in queries

   •  Modify and manipulate searches on the fly

Kibana provides different ways to save and share queries

   •  Learn how to Save Queries

   •  Understand all about Loading queries

   •  Share and report queries

Logstash is the primary tool for getting data into ES, we need to learn all about it.

   •  Dive into the configuration for a pipeline in ES

   •  Execute a few pipelines

   •  Determine how to use this for real logs

After we configure Logstash to send data to ES, learn how to make sure our configuration is doing what we think it’s doing.

   •  Find the logstash pipeline viewer in Kibana

   •  Explain what each component of the pipeline viewer means

   •  Figure out how we can use this in the real world

Understand how Logstash can do a lot more than just read data and write it to ES.

   •  Look at the configuration file and determine how to make changes

   •  Apply different changes to the data

   •  Verify all of our data changes in ES

It’s really easy to have Logstash read files from a local system. Learn how can we get distributed data

   •  Look at different options to receive data

   •  Configure logstash to receive data over the network

   •  Verify data was making it to ES/Kibana

Understand that Logstash is quite a heavy application, and in the shifting paradigm of micro services and Docker containers, Logstash may be too heavy.

   •  We’ll look at what Beats is and how we can use it

   •  Walk through the various beats packages

   •  Make sure they all work in our setup

We’re heavily focused on ES and Kibana (obviously, since this is an Elastic Stack class!), but now you will learn what else can Logstash do.

   •  Look through a list of potential outputs

   •  Configure and monitor logstash outputs

   •  Figure out what you need

Learn that Kibana can do a lot more than just an interactive search console.

   •  Break down different visualizations

   •  Pick the right chart for the data you have

   •  Create and save visualizations

In this video, we will learn how do we put all of our visualizations together.

   •  Create a new dashboard

   •  Add visualizations to the dashboard

   •  Rearrange them to make it tell a great story

You’ve made the best dashboard anyone has ever seen. Now learn how you can show it off to the world.

   •  Create, save and load a dashboard

   •  Share via a link

   •  Generate and download a report to email around

We setup monitoring in the very first section. Remember? Even if you do, now that you have a lot more context, let’s dig deep.

   •  Find your cluster information

   •  Look at index status

   •  Monitor all the things

Our ES and Kibana are configured how we want them. Learn to now restrict access

   •  Configure and enable security

   •  Add users and roles

   •  Authenticate

If we’re tracking events, like network events, there are tons of relationships. Learn to figure them out in this video

   •  Find the graph module

   •  Get some data visualized

   •  Pivot to other data

Computers are good at alerting when certain conditions are programmed. Learn how can the Elastic Stack automatically determine limits

   •  Configure a machine learning job

   •  Load data and back analyze it

   •  Dig into anomalies

We’ve monitored it all. Files, networks and system metrics. Learn to get this visibility into our applications

   •  Install and configure an APM node

   •  Wire up an application with the client

   •  Look at all the fantastic data in your Python or Node application

Continue your journey with APM

   •  Get a hands on feel of APM

My favorite use case for the Elastic Stack is pulling in data over time. How can we analyze it effectively?

   •  Find timelion in the menu

   •  Start using your data over time

   •  Chain, custom colors, and external data

Our Elasticsearch node has been yellow, and that’s OK, Learn how to we fix that and make our data safer.

   •  Learn about shards

   •  Learn about replica’s

   •  Learn how to plan shard distribution across Elasticsearch cluster

In this video, we add nodes to our Elasticsearch cluster

   •  Add nodes

   •  Add another node, and watch your data get spread across your cluster

   •  Realize the safety and performance increases you just unlocked

If your node is really important, let’s look at master nodes to keep them safe.

   •  Learn what a master node is

   •  Avoid split brain syndrome

   •  Size your master nodes appropriately

Learn how to configure different node types in the configuration

   •  Differentiate a master only node from an all purpose node

   •  Set up data only nodes for heavy work loads

   •  Plan a cluster with both master and data nodes

In this video, we see how Kibana provides different ways to search for specific data.

   •  First we’ll look into using the search bar

   •  Then we’ll use the top fields aggregations to discover data without typing in queries

   •  Modify and manipulate searches on the fly