Elasticsearch 9 and the Elastic Stack: In Depth and Hands On

Map and index data in elastic search by importing a dataset of movies and ratings. Practice inserting, updating, and deleting documents, and examine analysis, tokenization, and relational data modeling.

Explore the free MovieLens data set, a collection of user ratings with movie IDs, titles, and genres, and prepare for indexing in Elasticsearch by inspecting ratings.csv and movies.csv.

Master bulk import in Elasticsearch using the bulk API and JSON format to insert multiple documents at once, routing each document to its shard and validating results.

Elasticsearch updates by creating a new document with an incremented underscore version and marking the old version for deletion, using post updates or partial updates to modify fields.

Learn how optimistic concurrency control uses sequence numbers and primary terms in Elasticsearch to prevent conflicting updates, with automatic retries on conflicts.

Explore data modeling in elastic search, comparing normalized and denormalized designs, two-hop versus one-hop queries, and parent-child relationships for scalable updates.

Set up a parent-child mapping in Elasticsearch by creating a series index linking franchise and films, then use has parent and has child queries to explore relational data.

Explore how the flattened data type prevents mapping explosions by storing inner fields in a single flattened field. The trade-off is detailed inner-field analysis for keyword-only search.

Explore how elastic search helps customers find data quickly through various search interfaces, sorting, paginating, and filtering, with fuzzy queries, partial matches, and search-as-you-type features, in a hands-on section.

Learn to perform JSON body searches in Elasticsearch, using match queries, filters, and boolean logic to refine results with range, term, and missing filters.

Explore phrase matching in Elasticsearch 9, using match_phrase, slop values, and proximity queries to control term order, distance, and relevance in title searches like star wars.

Learn how to implement pagination in Elasticsearch 9 using from and size parameters, counting from zero, with uri or json body approaches, plus caveats about deep pagination and upper bounds.

Explore partial matching in Elasticsearch by performing prefix and wildcard queries on string fields, remapping the year as text, reindexing data, and leveraging RegExp queries for advanced pattern matching.

Explore query-time search as you type in Elasticsearch using match phrase prefix on the title field to auto-complete, with slope and slot value adjustments, noting it isn’t the most performant.

Explore edge n-grams for autocomplete in Elasticsearch 9, part 2, by building an autocomplete analyzer, mapping the title field, and reindexing movies data, with _analyze validation.

Learn how the search as you type data type in Elasticsearch 9 creates engrams and edge n-grams to power fast autocomplete, with multi match and prefix queries.

Discover practical methods to import diverse data into Elasticsearch, from unstructured and streaming data to sequel databases and S3, using scripts, Logstash, and integrations with Kafka, Hadoop, and Spark.

Modify a Python script to import tags from tags.csv into a new tags index in Elasticsearch, denormalizing tag data with movie titles from the movies dictionary.

Install and configure logstash with a file input and grok for apache logs, set the date format, and output to localhost:9200 and the console using ruby debug.

Learn to import data from a MySQL database into Elasticsearch with Logstash, configuring the GPS driver, connection string, and SQL query, and loading MovieLens data.

Learn to connect Logstash to MySQL via the JDBC connector, configure the driver path, and export data to Elasticsearch in a Movielens SQL index using JSON lines.

Learn how to parse json data with the Logstash json filter, map it into structured Logstash events, and index those documents into Elasticsearch, a json-based search and analytics engine.

Learn how to import data from an S3 bucket into Elasticsearch with Logstash, configure the S3 input and credentials, and use an Apache log grok filter for indexing.

Parse unstructured logs with the Logstash grok filter to extract fields for Elasticsearch analysis, using predefined patterns and the grok debugger to handle mismatches and support multiple grok filters.

Explore the heartbeat input plugin in Logstash to send periodic health messages to Elasticsearch with a configurable interval and observe health indicators in the generated documents.

Collect and centralize syslog data with Logstash by forwarding logs over TCP to Elasticsearch, or have Logstash monitor local log files and index them into Elasticsearch for rapid search.

Configure logstash with a kafka input on localhost:9092 listening to the Kafka Logs topic and output to the kafka_logs index in Elasticsearch, then publish data via the console producer.

Use Apache Spark to load a fake friends dataset, map lines to a person schema, and index the data into Elasticsearch with the sparked_friends index, demonstrating scalable parallel ingestion.

Import movie ratings from the ML latest small data set into Elasticsearch via Spark, creating a new ratings index; adapt the CSV script to map lines to a rating class.

Explore how Elasticsearch acts as a vector database by storing and querying dense_vector embeddings for semantic search and retrieval-augmented generation, using OpenAI embeddings and a Python workflow.

Explore Elasticsearch aggregations, including metrics, histograms, buckets, nested aggregations, and pipelines, enabling moving averages, cumulative sums, and matrix-based analysis.

Explore how Elasticsearch handles time series data with date histogram aggregations by hour on the at_timestamp field, using log data to analyze website traffic and Googlebot hits.

Learn to pinpoint web outages by querying logs with a date histogram, starting coarse then drilling down to the minute, filtering for 500 errors in Elasticsearch 9.

Explore Kibana management for Elasticsearch, including spaces, saved objects, and migrations; configure access, import/export of objects, and set advanced settings like default landing page and dark mode for teams.

Master data frame transforms in Elasticsearch to aggregate data by entities and create destination indices. Use pivot definitions and preview and start transform APIs for efficient, advanced analytics.

Explore how X-Pack security protects Elasticsearch data with access control, password protection, RBAC, and IP filtering, plus encryption, audit trails, and LDAP integration for role-based access control.

Install Filebeat from the Elastic repository, enable Apache access log paths, start the Filebeat service, and verify logs feed directly into Elasticsearch as a new index.

Explore Cabana to analyze a half-hour of 404 errors from 9:30 to 10 a.m. zulu time, identifying San Jose crawlers and their origins.

Wraps up by tying the elastic stack together, including Elasticsearch, Logstash, Kibana, and Beats, and shows how this simpler system prepares you to manage an elastic search cluster.

Rotate your indices with index aliases to add capacity without reindexing, searching across multiple indices like logs_current and logs_last_three_months.

Learn how Elasticsearch index lifecycle management automates aging indices through hot, warm, cold, frozen, and delete stages using size- or age-based policies and a rollover alias.

Allocate about half of your physical memory to elastic search and the rest to the os and lucene, avoiding heap sizes above 32 gigabytes for optimal performance.

Learn to monitor the elastic stack with zpack, covering basic monitoring, alerting, security, and anomaly detection via machine learning, plus Kibana encryption keys setup.

Troubleshoot common Elasticsearch issues across node setup, cluster formation, and cluster state. Learn to use bootstrap checks, discovery, log analysis, and backups for resilient operation.

Watch failover in action as Elasticsearch adds a node, loses the master, and automatically rebalances shards across three nodes on a single vm, then recovers when the original node returns.

Automate backups for Elasticsearch clusters with Snapshot Lifecycle Management, configuring repositories and policies to schedule snapshots, include all indices, and set retention and deletion rules.

Perform rolling restarts by disabling shard allocation and stopping new data indexing, then perform maintenance on each node, restart, rejoin the cluster, and re-enable allocation for green status.

Learn to implement uptime monitoring with heartbeat in the Elastic Stack by using ICMP, TCP, and HTTP monitors, configuring YAML monitors, and visualizing results with a heartbeat dashboard.

Explore cloud options for renting Elasticsearch clusters, including AWS Elasticsearch Service on EC2 and Elastic Cloud, to save time and money.

Spin up an OpenSearch domain on AWS with Elasticsearch compatibility and IAM security. Learn cost aware settings, data nodes, and AWS integrations like Kinesis and Firehose.

Explore Elastic Cloud, a hosted Elasticsearch service with automatic cloud provisioning (Google by default, Azure or AWS options) and paid add-ons; set up a free trial and deploy.

Review what you have learned in section nine and plan the next steps to complete the elastic stack course.

Wraps up by recapping installation, mapping, indexing, searching, and aggregating data with the elastic stack, plus importing data from files and streaming sources, and visualizing with Kibana.