Mastering ElasticSearch

In this video, we will gain understanding of most common type of index.

• Identify and defined the two most common index patterns
• Gain deeper understanding of monolith indices
• Learn how over sharding can be achieved with monolith indices 

Learn how to optimize cluster performance with indices.

• Understand how to leverage aliases
• Dive into disk based allocation to optimize sharding
• Learn to close unused indices to preserve cluster resources 

In this video, we will look at how to optimize search performance by making indices smaller.

• Learn how to use force-merge
• Leverage rollover indices to control index size
• Learn API calls for force-merge and rollovers 

Learn How to leverage scoring to improve search results.

• Understand relevancy in ElasticSearch
• Learn about Boolean Model and TF/IDF and Vector Space Model
• Gain understanding of document scores 

Understand how boosting affects query results.

• Define Boosting in ElasticSearch
• Learn about dynamic boosting
• See examples of dynamic boosting 

The goal of this video is to gain an understanding of the often confusing concept of query rescoring.

• Learn what query rescoring is and when it is executed
• Gain understanding of query_weight versus query_rescore_weight
• See examples of query rescoring in ElasticSearch 

Setting up a production ready cluster in ElasticSearch is the most often overlooked challenge to learning ElasticSearch. Configurations aside, understanding where to start relative to your needs can be an enormous task.

• Focus on nodes in an ElasticSearch cluster
• Learn the importance of dedicated data and master nodes
• See an example of beginning infrastructure requirements for production cluster 

Proper settings in ElasticSearch can help to void split-brain or ping timeouts.

• Learn about the key production settings in ElasticSearch
• Understand split brain and how to avoid it
• Gain knowledge of ping_timeout setting and when to adjust it 

Unbeknownst to many newbies, some configurations must be made to ensure that your cluster is successfully running in production.

• Learn the five configuration settings necessary in production
• Get a detailed understanding of each
• View applicable examples 

While many engineers understand the basics of authentication and authorization, it's necessary to exceed the basics to properly implement security in an ElasticSearch cluster.

• Understand built-in users in ElasticSearch
• Learn to change and disable default passwords for built-in users
• Look at external security constructs that play well with ElasticSearch 

Of all the security constructs (realms) in ElasticSearch, arguably the most popular is the Native Realm. Understanding the native realm is critical to one's ability to effectively implement security in ElasticSearch.

• Learn about the Native Realm in User Management API
• Gain understanding of password management
• See examples of User Management API 

While authentication deals with login credentials, authorization handles what an authenticated person can and can't do. Understanding authorization is critical for cluster security and management.

• Gain understanding of authorization in ElasticSearch
• Learn about role based access control
• Get thorough authorization breakdown 

All information passed between nodes, including passwords are done in plain text. For this reason, in production, node to node encryption is required.

• Understand the requirements for node to node communication
• Learn how SSL is used to encrypt traffic between nodes
• Create a certificate signing request with certgen 

One viable security option involves controlling the IP addresses that can access a production cluster, as well as setting up nodes to communicate via private IP addresses.

• Understand how IP filtering works in ElasticSearch
• Learn options of disabling HTTP versus disabling transport
• See example of dynamically filtering a subnet of IPs 

The only way to ensure a healthy cluster is to perform monitoring; in doing so, knowing what to monitor and how to identify potential problems are very important.

• Understand monitoring in Kibana
• Learn key metrics to monitor in Kibana
• View example of metrics to monitor 6.2 Setting up a Monitoring Cluster 

While monitoring directly from the production cluster remains an option, a far more productive way to monitor in ElasticSearch is to set up a dedicated monitoring cluster. This ensures that even if your production cluster goes offline, stats will still be available.

• Walk through the steps to setup monitoring cluster
• Learn configuration requirements for monitoring cluster
• Learn how to point production cluster to monitoring cluster 

The fact remains that even the most optimized ElasticSearch cluster can go down or experience setbacks. As such, having an effective backup provides an immediate way to rollback and restore data.

• Understand how backups work in ElasticSearch
• Learn steps to setting up a backup
• Get helpful tips on backups 

Repositories and Snapshots are the two necessary components when backing up in ElasticSearch. Both require deeper understanding.

• Learn to register a repository
• Gain experience in creating snapshots
• View and understand snapshot details 

Machine Learning has become the standard in data analysis today. ElasticSearch 5.5 introduced a turnkey Machine Learning integration which adds substantial power to data analysis.

• Learn what is Machine Learning
• Understand different types of Machine Learning
• Learn steps for Machine Learning in ElasticSearch 

To implement Machine Learning in ElasticSearch it is important to understand exactly how to the interface with one another.

• View steps to prepare data for Machine Learning example
• Learn pattern to feed data after model generated
• Review data used in example 

As experience is the best teacher, the most effective way to learn how to implement machine learning in ElasticSearch is to do it.

• Learn steps in building an unsupervised learning model
• Understand how to evaluate model
• Learn to set time period and feed data to model 

The best way to close out this course is with helpful tips for production and security.

• Learn production tips
• Understand how to leverage hot-warm architecture
• Gain helpful security tips 

Here we round things up.

• Have a recap of what we learnt so far
• Some pointers about ElasticSearch
• We conclude the course