Complete Guide to Elasticsearch

OpenSearch is similar to Elasticsearch, but not identical. Let's explore what OpenSearch is, its history, and how it relates to Elasticsearch. You can choose either one for this course, although lectures are recorded using Elasticsearch and Kibana.

Let's explore which options there are for installing and hosting Elasticsearch and OpenSearch. In terms of hosting, most managed services offer a free trial which you can use for this course.

If you decide to use OpenSearch for this course, Bonsai has a great free plan that you can use. In this lecture I will showcase the platform.

See how to easily host Elasticsearch and Kibana on Elastic Cloud, which is the official Elasticsearch Service by the Elastic company.

Learn how to install both Elasticsearch and Kibana on macOS and Linux.

Learn how to install both Elasticsearch and Kibana on macOS and Linux.

Understand the basic architecture of Elasticsearch, including what nodes, indices, and documents are.

Let's take a moment to see what the cluster that we started up looks like internally.

Using Kibana's Console tool is not the only way you can send requests to Elasticsearch's REST API. Learn how to do the same with the cURL HTTP client in this lecture.

Learn what sharding is, and how it helps scale an Elasticsearch cluster, both in terms of data volume and query throughput.

Learn what replication is, and how it helps ensure high availability as well as increased query throughput.

Learn how to add more nodes to our single-node Elasticsearch cluster, forming a cluster consisting of three nodes.

Elasticsearch nodes have roles, which can be configured for each node. Get an overview of which roles are available, and when you should use them.

Let's take a quick moment to wrap up this section of the course.

Before adding some data into our cluster, let's see how to both create and delete indices. When creating an index, we will supply two settings for specifying the number of shards and replica shards for the index.

In this lecture, you will learn how to index documents in two ways; by having Elasticsearch generate an ID for documents automatically, and by specifying an ID ourselves.

Learn how to retrieve a document based on its ID.

Learn how to update an existing field's value, and how to add a new field to a document.

Learn how to update documents with scripts. Scripts enable us to update documents dynamically, such as by performing calculations on field values. You will also see how to include parameters within scripts. Lastly, you will learn how to ignore a given document during an update, how to conditionally update a document, and how to delete a document within a script.

In this lecture, we will cover so-called upserts. An upsert is a way to index a document or update an existing one, depending on whether or not a given document already exists.

Apart from updating documents, we can also replace them altogether. You will learn how to do just that in this lecture.

The last document action you need to learn, is to delete documents. Fortunately, this is extremely easy due to Elasticsearch's REST API.

How did Elasticsearch find the documents that we have indexed in the previous lectures? And how did it know where to store the documents that we indexed? That's what we will be looking at in this lecture.

Now that you know what routing is, let's take a closer look at how Elasticsearch reads data. This involves a coordinating node and Adaptive Replica Selection (ARS).

Now that you know how Elasticsearch reads data, let's talk about how it writes data. That's a more complicated subject involving something called primary terms, sequence numbers, as well as global and local checkpoints. We will also cover the basics of how data is replicated to replica shards.

Elasticsearch uses a primitive kind of document versioning. Let's take a look at what that is all about.

Documents might be overwritten inadvertently due to a web server running multiple threads in parallel, for instance. How do we deal with this problem? We first look at how we used to handle this, before diving into how primary terms and sequence numbers can be used to overcome this challenge.

Let's see how we can update multiple documents with a single query, similar to an UPDATE WHERE query for relational databases. We will also go through the inner workings of the query, including how it handles errors and performs optimistic concurrency control.

Now that you know how to update multiple documents with one query, let's do the same thing, just in regards to deleting documents.

Learn how to process multiple (potentially thousands or millions) actions within one query. This can be used to index, update, or delete many documents at the same time.

It's time to import some test data. Learn how to import data by using the Bulk API together with cURL on the command line.

Let's take a quick moment to wrap up this section of the course.

Quick introduction to what is covered in this section of the course.

Introduction to the concept of text analysis.

Learn how to use the Analyze API to test built-in and custom analyzers. It can also be used to test character filters, tokenizers, and token filters.

Learn how text values are analyzed and stored within a data structure called an inverted index.

Introduction to the concept of mapping.

An overview of the most important data types in Elasticsearch, with a focus on the "object," "nested," and "keyword" data types.

Let's take a closer look at how the "keyword" data type works and learn why it is used for exact matches, aggregations, and sorting.

Learn what type coercion is all about and how Elasticsearch can sometimes convert from one data type to another.

How can we index array values into Elasticsearch? How do arrays work, and what are their limitations? Learn the answers to those questions in this lecture.

Time to add our first mapping to an index.

Learn how to retrieve mappings for an index, and also specific field mappings.

Mapping queries can be simplified when using a dot notation syntax for nested fields. Learn how to do that in this lecture.

Sometimes you might want to add field mappings to an existing index. Learn how to do that in this lecture.

Let's take a look at how Elasticsearch handles dates. In particular, which date formats are supported, how timezones work, how Elasticsearch stores dates internally, and how dates are searched by search queries.

What happens if we leave out a field when indexing a document? And does adding a field mapping make a field required? Those are some of the questions that we will answer in this lecture.

In this lecture, we take a look at the most important mapping parameters, including "doc_values," "norms," "index," and more.

In this lecture you will learn how to update existing field mappings. There are quite a few limitations to updating mappings in Elasticsearch, so we will cover those as well.

Let's see how we can reindex documents into a new index. We will take a look at how the Reindex API works and cover a number of common use cases with examples.

Instead of renaming fields, we can make use of field aliases. You will see how to add and use field aliases in this lecture.

An Elasticsearch field can actually be mapped in multiple ways. This is very useful for querying field values in different ways.

Index templates can be used to specify field mappings and index settings. These can then automatically be applied to new indices based on index patterns. We will take a look at how index templates are commonly used.

Time to get into dynamic mapping now. We will take a look at the rules that Elasticsearch uses when dynamically mapping fields, along with a couple of examples.

Now that you know of both explicit and dynamic mapping, what about combining them? As you will see, you don't necessarily have to choose either explicit or dynamic mapping.

Let's take a look at how we can configure dynamic mapping. You will see how to disable it, how to set it to "strict," and how inheritance is supported. Apart from that, we will also look at numeric detection and date detection.

Dynamic mapping can be configured with dynamic templates. These templates define certain conditions, and when they are matched, the specified mapping will be applied.

Before moving on to analysis and analyzers, I have a few recommendations in regards to mapping.

Stemming and stop words are two common things to handle during text analysis. We will look at how words can be stemmed to their root form as a way to improve search results. We will also cover the concept of removing stop words.

If terms are changed during the analysis process (e.g. stemmed or stop words being removed), how do searches work? Find the answer in this lecture.

Let's go through the most important built-in analyzers and also cover how to configure and use them.

In this lecture, we are going to build a custom analyzer. As part of that, I will show you how to configure a built-in token filter as well.

Apart from specifying analyzers when creating a new index, sometimes we might need to add an analyzer to an existing index. Let's see how to do that in this lecture. In the process, we will look at opening and closing indices, as well as static and dynamic index settings.

Let's take a look at how to update analyzers. While doing so is easy, there is a very important thing to be aware of; failing to handle the situation correctly, can potentially lead to big problems.

A quick wrap up of this section.

Let's take a first look at how we can search for data in Elasticsearch. There are two ways; URI searches and using the Query DSL. We will use the Query DSL for searches in this course.

Let's take a look at a group of queries called term level queries. We will look at what characterizes term level queries, as well as a few do's and don'ts.

Let's see how to use term level queries in Elasticsearch, beginning with the "term" query.

If you want to retrieve a number of documents by their IDs, this can be done with the "ids" query. This lecture shows you how it works.

The range query is used to perform range searches in Elasticsearch. It can be used with both numbers and dates, both of which you will see in this lecture.

Term level queries are generally used for exact matching, but we can add a bit of flexibility by searching for prefixes, or by using wildcards or regular expressions.

Let's see how to find documents that have (or don't have) a value for a given field. There are a few gotchas - such as when a field "exists" - which we will take a look at.

Let's now talk about another group of queries, namely full text queries. They are fundamentally different than term level queries. This lecture covers the difference.

The "match" query is probably the most important full text query in Elasticsearch. It's both simple and powerful at the same time. Let's take a look.

We haven't talked about relevance scoring yet. Here is a quick introduction to how relevance scoring works in Elasticsearch.

Sometimes you might want to search multiple fields at the same time, which can be done with the "multi_match" query. There are a few things to be aware of, though, which we will cover in this lecture.

Apart from searching for individual terms, we can also search for phrases. In this case the ordering and positions of terms matter. We will take a look at how this all works under the hood.

In Elasticsearch there are two main categories of queries; leaf queries and compound queries. So what's the difference? Let's take a look.

Time to write some more complex queries. With the "bool" query, we can write both simple and complex boolean logic, while influencing relevance scores of matching documents. Let's take a look at one of the most important queries in Elasticsearch; the "bool" query.

In Elasticsearch, there are two so-called execution contexts; filter and query. Let's take a look at both and see how they differ.

The "boosting" query enables us to reduce relevance scores of documents that match a given criteria (query). This is especially powerful when combined with the "bool" query. This gives us a lot of control over relevance scoring.

Let's take a look at the "dis_max" (disjunction max) query, which is a great way of finding documents that match one or more criteria, while having an influence on relevance scoring. We will also look at how this query relates to the "multi_match" query that we previously looked at.

Remember why we need to use the "nested" data type when we need to query arrays of objects independently? We will take a closer look at that in this lecture and see how to actually query such objects.

So a document has one or more objects that matched a given query. But which objects actually matched the query? Let's see how we can figure this out with Elasticsearch.

Using nested fields is not free, so there are a couple of limitations, which we will take a look at in this lecture.

Introduction to this section on joining queries.

Before proceeding, we need to add a new index and a little bit of test data.

Learn how to map document relationships with the "join" field data type.

Learn how to add documents for indices with "join relationships."

Learn how to rRetrieve documents based on their parent's ID.

See how to search for documents whose parent match certain criteria.

See how to search for documents whose child documents match certain criteria.

Learn how to add and use multi-level relationships, also referred to as nested relations.

See how to include inner hits with queries using a "join" field.

See how to use the so-called terms lookup mechanism. Essentially a more efficient way of using the "terms" query when dealing with lots of terms.

Joining queries are great in some scenarios, but not so great in others. This lecture discusses the pros and cons of using join queries and gives you recommendations about when to use them and when to stay away from them.

Joining queries are often not very fast. This lecture discusses the performance of joining queries and teaches you when performance is going to be an issue, enabling you to determine whether or not the "join" field is the right tool for your particular use case.

A quick recommendation in regards to types.

See how to choose between JSON and YAML as the result formats, and also how to format JSON for human eyes.

Learn how to specify which fields Elasticsearch should return for each match.

See how to specify how many search results Elasticsearch should return for a request.

See how to specify a match offset for a request.

See how the result size and offset can be combined to implement pagination within an application.

This lecture shows how to sort results in ascending and descending order.

This lecture shows how to sort by multi-value fields, such as average ratings, for example.

So what are aggregations all about? Get a quick introduction in this lecture.

Metric aggregations is the simplest type of aggregations. They are simple to understand, but still very powerful. Metric aggregations can be compared to SQL aggregations.

Having covered metric aggregations, let's take a look at another group of aggregations; bucket aggregations.

There is a gotcha in regards to document counts. The document counts are approximate. Get the full explanation in this lecture.

Aggregations can be nested. You will see how in this lecture.

Aggregations work on the documents that are available within the aggregation context. Sometimes you might want to filter out documents, which you will see in this lecture.

Define rules for which buckets documents are placed into with filters.

Range aggregations are... well, about ranges. Aggregate documents and get statistics for ranges of values.

Histograms are an excellent way of understand the distribution of documents, such as total sales per month.

Aggregations work on documents based on the aggregation context. Sometimes you might want to break out of that context, which you can go with the "global" aggregation.

Documents containing NULL values or missing fields, can be aggregated with the "missing" aggregation.

Similar to the "nested" query, nested objects must be handled a little differently in the context of aggregations.

In this section, we will be looking at various ways of improving search results.

In this lecture, we will be looking at so-called proximity searches.

See how to affect relevance scores based on term proximity.

Learn how to handle typos in user input with fuzzy match queries.

Related to fuzzy match queries, is the "fuzzy" query, although it is different.

See how to add synonyms and how they are handled in searches.

An alternative way of adding synonyms, is to load them from a file, which you will see in this lecture.

See how to highlight matching parts of documents within search results.

See how to apply stemming to words and how this is handled in searches.

As a thank you for enrolling in this course, here are some coupon codes for my other courses.