
This video will give you an overview about the course.
The purpose of this video is to better understand Elasticsearch as a technology.
Define Elasticsearch
Identify key features of Elasticsearch
Give examples of how Elasticsearch is being used in industry
In this video, we will develop a general understanding of how Elasticsearch works.
Define Elasticsearch as a search and analytics engine
Gain understanding of how Elasticsearch delivers near real-time results
Take a quick look at DSL
In this video, we will take a detailed look at how to install Elasticsearch core technologies: Elasticsearch and Kibana.
Learn how to install Elasticsearch
Learn how to install Kibana
Live walkthrough for installing Elasticsearch and Kibana
In this video, we will understand indices in Elasticsearch.
Define Elasticsearch index
Compare Elasticsearch to relational DBs
Demonstrate how to add and delete an index
In this video, we will look at documents in Elasticsearch.
Define documents in Elasticsearch
Define field and type
Live demonstration of adding and deleting documents
In this video, we will learn about clusters in Elasticsearch.
Define a cluster
Define a node/instance
Demonstrate how to start an Elasticsearch instance
In this video, we will understand shards and replicas.
Detailed explanation of shards and how they work
Detailed explanation of replicas and how they work
Demonstrate how to set shards and replicas in Elasticsearch
In this video, we will develop an understanding of Bulk API in Elasticsearch.
Define Bulk API
Show example of Bulk API
Demonstrate how to use the Bulk API in Elasticsearch
In this video, we will develop a general understanding of RESTful API.
Define RESTful API
Look at HTTP verbs used in RESTful API
Breakdown the dichotomy of a request string
In this video, we will learn how to run search queries with RESTful API.
Quickly look at the sample movie index and use _count to get number of documents
Run term query across movie index
Run match query across movie index
In this video, we will understand Domain Specific Language in Elasticsearch.
Understand leaf and compound queries
Look at different types of queries
Demonstrate queries using DSL
In this video, we will understand how context, relevancy and score work in Elasticsearch.
Understand query clauses
Understand relevancy in Elasticsearch
Learn about the _score in Elasticsearch
In this video, we will learn about exists queries in Elasticsearch.
Define exists query
View example of an exists query
Demonstrate how to run exists queries in Elasticsearch
In this video, we will develop a general understanding of RESTful API.
Define elastic stack
Detailed breakdown of the elastic stack makeup
Look at why using the elastic stack is beneficial
In this video, we will understand Kibana.
Define Kibana
Look at how Kibana fits into the Elastic Stack
Go through detailed walk through of Kibana installation and setup
In this video, we will understand Logstash.
Look at a Logstash pipeline
Learn how to configure Logstash
Demonstrate installation and configuration of Logstash
In this video, we will understand X-Pack.
Define X-Pack
Look at how to install and configure X-Pack on services
Demonstration of installing and configuring X-Pack
In this video, we will understand light-weight data shippers in Elasticsearch.
Define beats
Look at the different beats shippers
Take a quick look at how to setup/configure Filebeat
In this video, we will learn all the steps to preparing for secure log analysis.
Get a walkthrough of installing X-Pack and setting passwords
Understand how to configure Logstash and kibana yml files
Learn how to configure .conf and filebeat.yml
In this video, we will learn how to secure Elasticsearch by example.
Get a walkthrough installing X-Pack on Elasticsearch, Kibana, and Logstash
Setup password for all services using interactive or auto
Demonstrate configuring kibana.yml and logstash.yml with password information
In this video, we will learn how to configure .conf and filebeat.yml by example.
Get a walkthrough confirming log data
Learn how to configure filebeat.yml
Learn how to configure .conf logstash pipeline file by example
In this video, we will learn how to run a secure pipeline for log analysis by example.
Start up the necessary services: Elasticsearch and Kibana
Learn and use command to start Filebeat
Test and then start pipeline by executing required Logstash command
In this video, we will learn how term, range and boosting queries add power to Elasticsearch.
Define term, range and boosting queries
View examples of term, range and boosting queries
Demonstrate how to perform term, range, and boosting queries in Elasticsearch
In this video, we will learn how aggregations in Elasticsearch provide turnkey analytics.
Look at metrics and bucket aggregations
Show detailed examples of metrics aggregations
Demonstrate how to do average and extended stats aggregations in Elasticsearch
In this video, we will learn how geo aggregations work in Elasticsearch.
Define geo searching
Look at the different types of geo searches: distance, distance range and sorting
Get detailed explanations for distance, distance range and sorting in Elasticsearch
In this video, we will learn how to run geo queries by example.
Setup index template to facilitate IP searches after log analysis run
Run example of geo distance query
Run example of geo sorting query
In this video, we will learn about sorting in Elasticsearch and why it’s useful.
Look at the importance of sorting in Elasticsearch
Learn about basic sorting and the use of sort mode
Demonstrate sorting in Elasticsearch
In this video, we will learn about the power of synonyms in Elasticsearch.
Define synonyms
Look at simple expansions versus simple contractions
Understand how the process of analysis effects synonym
In this video, we will get an overview of machine learning.
Define machine learning
Learn some use cases for machine learning
Understand how machine learning works
In this video, we will learn how Elasticsearch interfaces with machine learning.
Learn how to get data into an Elasticsearch cluster for ML
Understand the different types of machine learning jobs
Learn to train machine learning models with Elasticsearch
In this video, we will perform a step-by-step walkthrough of machine learning in Elasticsearch.
Simulate data to perform machine learning jobs
Learn to do single metric machine learning job
Evaluate and understand results
In this video, we will look at how to get the data that we use throughout this course.
Learn how to restore snapshot
Simulate log data
Include access.log file
This video gives glimpse of the entire course.
Elasticsearch is at the core of Elastic Stack, playing the central role of a search and analytics engine. Elasticsearch is built on a radically different technology, Apache Lucene.
Look at key benefits of using Elasticsearch
Some Elastic stack components are general purpose and they can be used outside of Elastic Stack without using any of the other components. In this video we will look at the purpose of each component and how they fit in the stack.
Understand the function and application of each component
In this video we will downloading and installing the key components. Precisely, we will download and install Elasticsearch and Kibana
Before we start writing our first queries to interact with Elasticsearch, we should familiarize ourselves with a very important tool – Kibana Console.
Send the query GET
Continue working on it
Elasticsearch supports a wide variety of data types for supporting different scenarios. We will also look at mappings.
Create an index with name catalog and define mappings for type of product
Look at core, complex, and other data types
In this video, we will look at how to perform basic CRUD operations, which are the most fundamental operations required by any data store.
Look at Index, Get, Update and Delete API
You would want to control how indices are created and also how mapping is created. We will see how you can take control of this process in this video.
Create an index
Create type mapping in an existing index
Update mapping
The APIs that deal with Elasticsearch are categorized into some types. We will look at them and work with indexing.
Format the JSON response
Deal with multiple indices
Logstash allows us to easily build a pipeline that can help in collecting data from a wide variety of input sources, and parse, enrich, unify, and store it in a wide variety of destinations. In this video, we will look at salient features of logstash and Download and install Logstash.
Look at salient features
Installation and configuration
In this video, we will explore about Logstash pipeline in detail and with code example.
After that we will learn several types of plugins.
Understand the Logstash Architecture using the pipeline diagram
Installing or updating Logstash plugins
An input plugin is used to configure a set of events to be fed to Logstash. This video will help you with some of the most commonly used input plugins in detail.
Output plugins allow one to configure single or multiple output sources. This video will walk through some of the most commonly used output plugins in detail.
In this video, we will look at the type of aggregations and learn how they work.
Look at bucket, metric, matrix aggregations
Metric aggregations work with numeric data, computing one or more aggregate metrics within the given context. Let’s see more about them
Work with sum, average, min, and max aggregations
Sometimes, we may need to bucket the data or segment the data based on a field that has a string datatype, typically keyword typed fields in Elasticsearch.
Another common scenario is when we want to segment or slice the data into various buckets based on a numeric field. We will learn both in this video.
Perform terms aggregation for string
Perform histogram and range aggregation on numeric data
Elasticsearch has a very powerful Date Histogram aggregation. We will bucket on date/time data using that
Create buckets across time. Use a different time zone
Compute other metrics within sliced time intervals
Focus on a specific day and changing intervals
Another powerful feature is the ability to do geo-spatial analysis on the data. Let’s see how to do that in this video
Look at Geo distance and GeoHash grid ggregation
One of the important processes of Logstash is converting unstructured log data into structured data, which helps in searching for relevant information easily and also assists in analysis. In this video we will explore some common filter plugins used for transformation.
Understand the need to parse and enrich logs using logstash
Look at the types of filter plugins
Beats are lightweight data shippers that are installed as agents on edge servers to ship operational data to Elasticsearch. In this video, we will look at some of the commonly used beats by Elastic.co in detail
As Kibana is all about gaining insight from data, let's load some sample data that we will use as we follow the tutorial. Before that, we will also configure Kibana.
Configure Kibana
Create apache.conf. Start the Logstash
Verify total number of documents indexed
Before you can start working with data and creating visualizations to analyze the data, Kibana requires you to configure the index pattern. That’s what we will see in this video.
Look at time series and regular indexes
Type logstash–* in index name
Create @timestamp time filter field name
The Visualize page helps to create visualizations in the form of graphs, tables, and charts, thus assisting in visualizing all the data that has been stored in Elasticsearch easily.
Work with Kibana aggregations
Create a visualization
In this video, we will see how different visualizations are used to perform functions.
Create visualization to find response codes and top 10 URLs
Find bandwidth usage of top five countries over time and web traffic originating from different countries
Find the most used user agent
Dashboards help one bring different visualizations into a single page.
Create a dashboard
Save the dashboard
Clone and share the dashboard
Timelion is a visualization tool for analyzing time-series data in Kibana. Plugins are a way to enhance the functionality of Kibana. Let’s get to know them better here
Understand timeline UI and timeline expressions
Install and remove plugins
We have understood what the application is about and what the data represents. As we start developing the application, we will start the solution from the inside out. So, we will start defining our solution from the very heart of it by first building the data model in Elasticsearch
Define an index template
Understand mapping
Setup metadata database
The sensor_metadata database is ready to look up the necessary sensor metadata. In this video, let us build the Logstash data pipeline by performing following steps.
Accept JSON requests over the web
Store resulting documents in Elasticsearch
Senddata to Logstash over HTTP
We have successfully setup the Logstash data pipeline and also loaded some data using the pipeline into Elasticsearch. It is time to explore the data and build a dashboard that will help us gain some insights into the data.
Set up an index pattern in Kibana
Build visualizations for different scenarios
This video gives an overview of the entire course.
In this video, we’ll look at the target that we want to build within Elasticsearch and Kibana.
We need to get data and visualize it
Start installing components and configuring connectivity
Review the data
From the beginning, we don’t have an Elasticsearch node running, let’s set one up.
Cover all the information needed to download and install ES
Configure ES to be usable for our demos
Verify a running ES node
Now that we have the system to store our data, we need to be able to visualize it.
Cover the information on how to download and install Kibana
Configure Kibana to be usable and connect to ES
Verify configuration by seeing a configuration screen in Kibana
Before we start using ES and Kibana, we need to be able to validate the health of our system from the beginning.
Configure ES and Kibana by installing X-Pack
Configure ES and Kibana to use monitoring, but turn off security for now
Dig into the monitoring section
Our ES node has no other data besides monitoring, learn how to fix that.
Determine what data will look like
Use an HTTP API to insert documents
Find those documents in Kibana
We’ve seen how we can insert data into ES, but we need to understand more about that process to be effective.
Documents are the foundation of data within ES
Insert, update and delete documents into ES a few ways
Finish up by retrieving the documents back
Understand what options do we have for storing data within a document.
Review various data types within ES
Configure ES to be able to index and store our docs
Validate documents and mappings
Pick up on more details to how we classified data.
Review a mapping file
Insert mapping file into ES
Verify mappings of our documents
We’ve started inserting documents, but we need to learn how we can arrange groups of documents.
Find example data to insert for different use cases
Setup indexes for ‘reference’ data as well as time based data
Alias, reindex and delete indexes via APIs
We’re on a roll, we have all kinds of data and options to put data in, but, we need to be familiarised with how we get data back.
Insert new data and perform basic queries
Explore filtering and searching via different API
Look at aggregations and buckets
Kibana can be overwhelming at first, there are so many components that you need to understand before you can decide how to use it.
First we’ll browse through the user interface
We’ll cover various components that we are going to use for searching
Get ready to search
Kibana needs to know how your data is stored within ES. It can auto discover a lot of things, but you need to start by telling Kibana what indexes to use.
Create an index pattern
Determine what ‘regex’ to use depending on your use case
See how index patterns appear in the search tab
In this video, we will understand that If you’re looking at time based data, you’ll have specific searching needs.
Walk through an overview of how time ranges are selected
Define different time range selection options
Use the built in interface to change time ranges
Learn how Kibana provides different ways to search for specific data.
First we’ll look into using the search bar
Then we’ll use the top fields aggregations to discover data without typing in queries
Modify and manipulate searches on the fly
Kibana provides different ways to save and share queries
Learn how to Save Queries
Understand all about Loading queries
Share and report queries
Logstash is the primary tool for getting data into ES, we need to learn all about it.
Dive into the configuration for a pipeline in ES
Execute a few pipelines
Determine how to use this for real logs
After we configure Logstash to send data to ES, learn how to make sure our configuration is doing what we think it’s doing.
Find the logstash pipeline viewer in Kibana
Explain what each component of the pipeline viewer means
Figure out how we can use this in the real world
Understand how Logstash can do a lot more than just read data and write it to ES.
Look at the configuration file and determine how to make changes
Apply different changes to the data
Verify all of our data changes in ES
It’s really easy to have Logstash read files from a local system. Learn how can we get distributed data
Look at different options to receive data
Configure logstash to receive data over the network
Verify data was making it to ES/Kibana
Understand that Logstash is quite a heavy application, and in the shifting paradigm of micro services and Docker containers, Logstash may be too heavy.
We’ll look at what Beats is and how we can use it
Walk through the various beats packages
Make sure they all work in our setup
We’re heavily focused on ES and Kibana (obviously, since this is an Elastic Stack class!), but now you will learn what else can Logstash do.
Look through a list of potential outputs
Configure and monitor logstash outputs
Figure out what you need
Learn that Kibana can do a lot more than just an interactive search console.
Break down different visualizations
Pick the right chart for the data you have
Create and save visualizations
In this video, we will learn how do we put all of our visualizations together.
Create a new dashboard
Add visualizations to the dashboard
Rearrange them to make it tell a great story
You’ve made the best dashboard anyone has ever seen. Now learn how you can show it off to the world.
Create, save and load a dashboard
Share via a link
Generate and download a report to email around
We setup monitoring in the very first section. Remember? Even if you do, now that you have a lot more context, let’s dig deep.
Find your cluster information
Look at index status
Monitor all the things
Our ES and Kibana are configured how we want them. Learn to now restrict access
Configure and enable security
Add users and roles
Authenticate
If we’re tracking events, like network events, there are tons of relationships. Learn to figure them out in this video
Find the graph module
Get some data visualized
Pivot to other data
Computers are good at alerting when certain conditions are programmed. Learn how can the Elastic Stack automatically determine limits
Configure a machine learning job
Load data and back analyze it
Dig into anomalies
We’ve monitored it all. Files, networks and system metrics. Learn to get this visibility into our applications
Install and configure an APM node
Wire up an application with the client
Look at all the fantastic data in your Python or Node application
Continue your journey with APM
Get a hands on feel of APM
My favorite use case for the Elastic Stack is pulling in data over time. How can we analyze it effectively?
Find timelion in the menu
Start using your data over time
Chain, custom colors, and external data
Our Elasticsearch node has been yellow, and that’s OK, Learn how to we fix that and make our data safer.
Learn about shards
Learn about replica’s
Learn how to plan shard distribution across Elasticsearch cluster
In this video, we add nodes to our Elasticsearch cluster
Add nodes
Add another node, and watch your data get spread across your cluster
Realize the safety and performance increases you just unlocked
If your node is really important, let’s look at master nodes to keep them safe.
Learn what a master node is
Avoid split brain syndrome
Size your master nodes appropriately
Learn how to configure different node types in the configuration
Differentiate a master only node from an all purpose node
Set up data only nodes for heavy work loads
Plan a cluster with both master and data nodes
In this video, we see how Kibana provides different ways to search for specific data.
First we’ll look into using the search bar
Then we’ll use the top fields aggregations to discover data without typing in queries
Modify and manipulate searches on the fly
Elasticsearch is a powerful tool not only for powering search on big websites, but also for analyzing big data sets in a matter of milliseconds! It's an increasingly popular technology, and a valuable skill to have in today's job market. If you’re a technologist who wants to add Elasticsearch to their tool chest for searching and analyzing big data sets, then go for this Learning Path.
This comprehensive 3-in-1 course is a hands-on guide to using Elasticsearch in conjunction with Elastic Stack, to search, analyze, and visualize data with Elasticsearch, Logstash, Beats, Kibana, and more. You will gain a firm understanding of all the fundamentals of Elasticsearch 6 to build efficient search and analytics applications using Elasticsearch 6. You will also learn what Elastic stack is all about, and how to use it efficiently to build powerful real-time data processing applications. This course covers each and every concept of ElasticSearch and Elastic Stack with the help of practical examples making it easy for you to understand and implement in your own applications.
This training program includes 3 complete courses, carefully chosen to give you the most comprehensive training possible.
The first course, Learning Elasticsearch 6, begins with explaining you what is Elasticsearch, what is it used for, and why is it important. You will then be introduced to the new features of Elasticsearch 6 and its fundamental components such as indices, documents, nodes and clusters, all which form the dichotomy of Elasticsearch. You will also learn how to add more power to your searches using filters, ranges, and more. Next, you will explore how Elasticsearch can be used with the other components of the Elastic Stack such as LogStash, Kibana, and Beats, to get data into an Elasticsearch cluster. Finally, you will develop a Elasticsearch application.
In the second course, Learning Elastic Stack 6.0, after a quick overview of the newly introduced features in Elastic Stack 6, you'll learn how to set up the stack by installing the tools, and explore their basic configurations. You will then demonstrate the creation of custom plugins using Kibana. You will also get some useful tips on how to use the Elastic Cloud and deploy the Elastic Stack in production environments.
The third course, Mastering ElasticSearch 6.x and the Elastic Stack, focuses on two major use cases with Elasticsearch. The first use case is on leveraging the powerful full-text search engine ElasticSearch is built on, allowing developers to add blazingly fast search features to applications. The second use case is on leveraging different components of the Elastic Stack to continuously monitor applications, infrastructure, or even customer transactions.
By the end of this Learning Path, you will be well-versed with the concepts of Elasticsearch and Elastic Stack to build complete, open source solutions for storing, managing, analyzing, and visualizing structured and unstructured data.
Meet Your Expert(s):
We have the best work of the following esteemed author(s) to ensure that your learning journey is smooth:
Ethan Anthony is a San Francisco based Data Scientist who specializes in distributed data-centric technologies. He is also the Founder of XResults, where the vision is to harness the power of big data to deliver intuitive customer-facing solutions, largely to non-technical professionals. Ethan is Harvard-educated in the areas of data science and software engineering. He began using Elasticsearch in 2012 and delivered solutions based on the Elastic Stack to a broad range of clientele. Ethan has also consulted globally with firms in a cross-section of industry verticals, from the U.S. to the Far East.
Pranav Shukla is the founder and CEO of Valens DataLabs, a technologist, husband, and father of two. He is a big data architect and software craftsman who uses JVM-based languages. Pranav has diverse experience of over 14 years in architecting enterprise applications for Fortune 500 companies and startups. His core expertise lies in building JVM-based, scalable, reactive, and data-driven applications using Java/Scala, the Hadoop ecosystem, Apache Spark, and NoSQL databases. He is a big data engineering, analytics, and machine learning enthusiast. Pranav founded Valens DataLabs with a vision to help companies leverage data to their competitive advantage. Valens DataLabs specializes in developing next-generation, cloud-based, reactive, and data-intensive applications using big data and web technologies. The company believes in agile practices, lean principles, test-driven and behavior-driven development, continuous integration, and continuous delivery for sustainable software systems.
Sharath Kumar M N has done his masters in Computer Science at The University of Texas, Dallas, USA. He has been in the IT industry for more than ten years now and is the Elasticsearch Solutions Architect at Oracle. He is an Elastic Stack advocate, and being an avid speaker he has also given several tech talks in conferences such as the Oracle Code Event. Sharath is a certified trainer—Elastic Certified Instructor—one of the few technology experts in the world who has been certified by Elastic Inc to deliver their official from the creators of Elastic training. He is also a data science and machine learning enthusiast. In his free time, he enjoys trekking, listening to music, playing with his lovely pets Guddu and Milo and the geek in him loves exploring his Python skills for stock market analysis.
Chris Fauerbach is an active technical expert in the area of cybersecurity and the Elastic Stack. As a seasoned software engineer, he's built multiple commercial products on the Elastic Stack and has a passion for teaching. Chris continues to research new technologies and explore new ways to solve problems. As he has become an expert in his field, he has been focusing primarily on writing and teaching.