Learning Elastic Stack 6.0

Before we start writing our first queries to interact with Elasticsearch, we should familiarize ourselves with a very important tool – Kibana Console.

   •  Send the query GET

   •  Continue working on it

Elasticsearch supports a wide variety of data types for supporting different scenarios. We will also look at mappings.

   •  Create an index with name catalog and define mappings for type of product

   •  Look at core, complex, and other data types

In this video, we will look at how to perform basic CRUD operations, which are the most fundamental operations required by any data store.

   •  Look at Index, Get, Update and Delete API

You would want to control how indices are created and also how mapping is created. We will see how you can take control of this process in this video.

   •  Create an index

   •  Create type mapping in an existing index

   •  Update mapping

The APIs that deal with Elasticsearch are categorized into some types. We will look at them and work with indexing.

   •  Format the JSON response

   •  Deal with multiple indices

Logstash allows us to easily build a pipeline that can help in collecting data from a wide variety of input sources, and parse, enrich, unify, and store it in a wide variety of destinations. In this video, we will look at salient features of logstash and Download and install Logstash.

   •  Look at salient features

   •  Installation and configuration

In this video, we will explore about Logstash pipeline in detail and with code example.

   •  After that we will learn several types of plugins.

   •  Understand the Logstash Architecture using the pipeline diagram

   •  Installing or updating Logstash plugins

An input plugin is used to configure a set of events to be fed to Logstash. This video will help you with some of the most commonly used input plugins in detail.

Output plugins allow one to configure single or multiple output sources. This video will walk through some of the most commonly used output plugins in detail.

In this video, we will look at the type of aggregations and learn how they work.

   •  Look at bucket, metric, matrix aggregations

Metric aggregations work with numeric data, computing one or more aggregate metrics within the given context. Let’s see more about them

   •  Work with sum, average, min, and max aggregations

Sometimes, we may need to bucket the data or segment the data based on a field that has a string datatype, typically keyword typed fields in Elasticsearch.

   •  Another common scenario is when we want to segment or slice the data into various buckets based on a numeric field. We will learn both in this video.

   •  Perform terms aggregation for string

   •  Perform histogram and range aggregation on numeric data

Elasticsearch has a very powerful Date Histogram aggregation. We will bucket on date/time data using that

   •  Create buckets across time. Use a different time zone

   •  Compute other metrics within sliced time intervals

   •  Focus on a specific day and changing intervals

Another powerful feature is the ability to do geo-spatial analysis on the data. Let’s see how to do that in this video

   •  Look at Geo distance and GeoHash grid ggregation

One of the important processes of Logstash is converting unstructured log data into structured data, which helps in searching for relevant information easily and also assists in analysis. In this video we will explore some common filter plugins used for transformation.

   •  Understand the need to parse and enrich logs using logstash

   •  Look at the types of filter plugins

Beats are lightweight data shippers that are installed as agents on edge servers to ship operational data to Elasticsearch. In this video, we will look at some of the commonly used beats by Elastic.co in detail

As Kibana is all about gaining insight from data, let's load some sample data that we will use as we follow the tutorial. Before that, we will also configure Kibana.

   •  Configure Kibana

   •  Create apache.conf. Start the Logstash

   •  Verify total number of documents indexed

Before you can start working with data and creating visualizations to analyze the data, Kibana requires you to configure the index pattern. That’s what we will see in this video.

   •  Look at time series and regular indexes

   •  Type logstash–* in index name

   •  Create @timestamp time filter field name

The Visualize page helps to create visualizations in the form of graphs, tables, and charts, thus assisting in visualizing all the data that has been stored in Elasticsearch easily.

   •  Work with Kibana aggregations

   •  Create a visualization

In this video, we will see how different visualizations are used to perform functions.

   •  Create visualization to find response codes and top 10 URLs

   •  Find bandwidth usage of top five countries over time and web traffic originating from different countries

   •  Find the most used user agent

Dashboards help one bring different visualizations into a single page.

   •  Create a dashboard

   •  Save the dashboard

   •  Clone and share the dashboard

Timelion is a visualization tool for analyzing time-series data in Kibana. Plugins are a way to enhance the functionality of Kibana. Let’s get to know them better here

   •  Understand timeline UI and timeline expressions

   •  Install and remove plugins

We have understood what the application is about and what the data represents. As we start developing the application, we will start the solution from the inside out. So, we will start defining our solution from the very heart of it by first building the data model in Elasticsearch

   •  Define an index template

   •  Understand mapping

   •  Setup metadata database

The sensor_metadata database is ready to look up the necessary sensor metadata. In this video, let us build the Logstash data pipeline by performing following steps.

   •  Accept JSON requests over the web

   •  Store resulting documents in Elasticsearch

   •  Senddata to Logstash over HTTP

We have successfully setup the Logstash data pipeline and also loaded some data using the pipeline into Elasticsearch. It is time to explore the data and build a dashboard that will help us gain some insights into the data.

   •  Set up an index pattern in Kibana

   •  Build visualizations for different scenarios