
Learn to maximize your learning on Udemy by using the q&a, transcripts, captions, speed controls, and playback settings, and by rating the course honestly.
Set up an Ubuntu Linux environment with a virtual machine on Windows or macOS, install Elasticsearch, and index the complete works of William Shakespeare while exploring Elasticsearch architecture.
We'll talk about why Elasticsearch is important and what you can expect from this course. Then, we'll install a virtual Ubuntu machine right on your own desktop PC, install Elasticsearch on it, and search the complete works of William Shakespeare!
Let's look at the components of the Elastic Stack from a 30,000-foot level, and see how they all fit together.
Elasticsearch exposes a RESTful API, and we communicate with Elasticsearch using nothing but standard HTTP requests and responses. Let's cover the basics of how that works.
Explain elastic search basics by defining documents and indexes, using json data and unique ids, and map cluster, index, and document concepts to database analogies for hands-on exploration.
Explore how inverted indices map search terms to documents, enabling fast retrieval, and learn tf-idf as term frequency divided by document frequency to rank relevance.
Use an Elasticsearch index via RESTful APIs over http with json requests, and leverage client APIs or Kibana—the web-based graphical UI—to explore indices.
Explore Elasticsearch architecture and shard-based scaling with primary and replica shards, automatic distribution across a cluster, and failover resilience using Lucene-backed shards.
Understand how document schemas are defined by the index type mapping and document type. Review inverted indices and how shard and replica counts affect total shards in elastic search.
Celebrate the progress as you install an Ubuntu virtual machine and run Elasticsearch. Prepare to import, search, and analyze big data with Elasticsearch in hands-on sections.
Map and index data in elastic search by importing a dataset of movies and ratings. Practice inserting, updating, and deleting documents, and examine analysis, tokenization, and relational data modeling.
Connect to your elasticsearch cluster via ssh, opening port forwarding for 22, 9200, and 5601. Set up ssh on the vm and log in with putty or terminal.
Explore the free MovieLens data set, a collection of user ratings with movie IDs, titles, and genres, and prepare for indexing in Elasticsearch by inspecting ratings.csv and movies.csv.
Learn how mappings define schema and data types in Elasticsearch, and how analyzers, tokenizers, and filters shape indexing, search, and language-specific queries.
Import the Interstellar document into an Elasticsearch index using curl and a put request, with a date mapping for the year, and IMAX and sci fi genres.
Master bulk import in Elasticsearch using the bulk API and JSON format to insert multiple documents at once, routing each document to its shard and validating results.
Elasticsearch updates by creating a new document with an incremented underscore version and marking the old version for deletion, using post updates or partial updates to modify fields.
Identify the document id via a search, then issue a delete request to the movies index with id. Demonstrate a two-step deletion process using The Dark Knight as an example.
Apply your Elasticsearch basics by inserting a fake movie into the movies index, updating its genres to documentary and comedy, and deleting the document to clean up.
Learn how optimistic concurrency control uses sequence numbers and primary terms in Elasticsearch to prevent conflicting updates, with automatic retries on conflicts.
Explore how analyzers and tokenizers control text search in elasticsearch 9, choosing keyword for exact matches or text with language analyzers for partial, ranked results.
Explore data modeling in elastic search, comparing normalized and denormalized designs, two-hop versus one-hop queries, and parent-child relationships for scalable updates.
Set up a parent-child mapping in Elasticsearch by creating a series index linking franchise and films, then use has parent and has child queries to explore relational data.
Explore how the flattened data type prevents mapping explosions by storing inner fields in a single flattened field. The trade-off is detailed inner-field analysis for keyword-only search.
Explore Elasticsearch mappings, including explicit vs dynamic mapping, common mapping exceptions, and strategies like ignore malformed and dead letter queue pattern to manage limits.
Master the basics of elastic search, import data via json and rest, and perform insert, update, and delete operations while you explore concurrency, analyzers, tokenizers, and data modeling.
Explore how elastic search helps customers find data quickly through various search interfaces, sorting, paginating, and filtering, with fuzzy queries, partial matches, and search-as-you-type features, in a hands-on section.
Explore the query lite interface for quick Elasticsearch searches via the url query parameter. Learn when to use it for quick experimentation and why it isn’t recommended for production.
Learn to perform JSON body searches in Elasticsearch, using match queries, filters, and boolean logic to refine results with range, term, and missing filters.
Explore phrase matching in Elasticsearch 9, using match_phrase, slop values, and proximity queries to control term order, distance, and relevance in title searches like star wars.
Practice three Elasticsearch search methods for Star Wars movies after 1980. Use query light syntax with and without url encoding, and a json body search with a bool query.
Learn how to implement pagination in Elasticsearch 9 using from and size parameters, counting from zero, with uri or json body approaches, plus caveats about deep pagination and upper bounds.
Sort Elasticsearch results by numeric fields with simple queries, and sort by text fields using a non-analyzed raw subfield, while managing mapping and reindexing requirements.
Explore advanced filter queries in Elasticsearch, mastering boolean queries with must, must not, and range filters to find science fiction films released between 2010 and 2015, with practical curl examples.
Practice constructing a curl-based search that filters science fiction movies released before 1960 and sorts results by the raw title field, as shown with Plan Nine from Outer Space.
Explore fuzzy queries in Elasticsearch by tolerating typos and misspellings with a distance-based fuzziness parameter, handling substitutions, insertions, and deletions, including auto settings.
Explore partial matching in Elasticsearch by performing prefix and wildcard queries on string fields, remapping the year as text, reindexing data, and leveraging RegExp queries for advanced pattern matching.
Explore query-time search as you type in Elasticsearch using match phrase prefix on the title field to auto-complete, with slope and slot value adjustments, noting it isn’t the most performant.
Use n-grams and edge n-grams to enable search as you type by prefix matching, with an index-time auto complete analyzer and optional completion suggesters.
Explore edge n-grams for autocomplete in Elasticsearch 9, part 2, by building an autocomplete analyzer, mapping the title field, and reindexing movies data, with _analyze validation.
Learn how the search as you type data type in Elasticsearch 9 creates engrams and edge n-grams to power fast autocomplete, with multi match and prefix queries.
Master the basics of Elasticsearch for effective searching and become productive today; stay tuned as the course reveals more tricks and deeper capabilities.
Discover practical methods to import diverse data into Elasticsearch, from unstructured and streaming data to sequel databases and S3, using scripts, Logstash, and integrations with Kafka, Hadoop, and Spark.
Explore how to import data into Elasticsearch using code, Logstash, beats, or AWS services, and connect with Kafka, Spark, and Kinesis Firehose through existing Elasticsearch integrations.
Explore importing data into Elasticsearch with language client libraries, not json, using Python examples to perform bulk imports and denormalize ratings with titles.
Modify a Python script to import tags from tags.csv into a new tags index in Elasticsearch, denormalizing tag data with movie titles from the movies dictionary.
Logstash acts as a flexible data ingestion and transformation layer between diverse sources and destinations, enabling structuring, filtering, anonymization, geo-location, and buffering across the elastic stack.
Install and configure logstash with a file input and grok for apache logs, set the date format, and output to localhost:9200 and the console using ruby debug.
Run Logstash with a configuration file, ensure the data directory is writable, and observe Logstash parsing Apache access logs, grokking fields, and sending data to Elasticsearch, verified by index queries.
Learn to import data from a MySQL database into Elasticsearch with Logstash, configuring the GPS driver, connection string, and SQL query, and loading MovieLens data.
Learn to connect Logstash to MySQL via the JDBC connector, configure the driver path, and export data to Elasticsearch in a Movielens SQL index using JSON lines.
Study how to import and parse csv data with Logstash, handle csv delimiters and headers, map fields, and index clean data into Elasticsearch.
Learn how to parse json data with the Logstash json filter, map it into structured Logstash events, and index those documents into Elasticsearch, a json-based search and analytics engine.
Learn how to import data from an S3 bucket into Elasticsearch with Logstash, configure the S3 input and credentials, and use an Apache log grok filter for indexing.
Parse unstructured logs with the Logstash grok filter to extract fields for Elasticsearch analysis, using predefined patterns and the grok debugger to handle mismatches and support multiple grok filters.
Hands-on examples of parsing logs with Logstash into Elasticsearch using Grok patterns. Includes Apache, NGINX, MongoDB, AWS ELB, AWS ALB, MySQL slow logs, Elasticsearch logs, Elasticsearch slow logs, IIS, and AWS CloudFront logs.
Explore the heartbeat input plugin in Logstash to send periodic health messages to Elasticsearch with a configurable interval and observe health indicators in the generated documents.
Explore the Logstash generator input to produce custom test events with JSON lines, and use the dead letter queue input plugin to route and reprocess failed events into Elasticsearch.
Learn to use the http polar input plugin in logs dash to poll http endpoints, store responses in Elasticsearch, and organize data in external API and health status indices.
Collect and centralize syslog data with Logstash by forwarding logs over TCP to Elasticsearch, or have Logstash monitor local log files and index them into Elasticsearch for rapid search.
Configure logstash with a kafka input on localhost:9092 listening to the Kafka Logs topic and output to the kafka_logs index in Elasticsearch, then publish data via the console producer.
Explore integrating Elasticsearch with Apache Spark using the Elasticsearch spark connector and running a spark shell workflow to read a CSV and save to Elasticsearch.
Use Apache Spark to load a fake friends dataset, map lines to a person schema, and index the data into Elasticsearch with the sparked_friends index, demonstrating scalable parallel ingestion.
Import movie ratings from the ML latest small data set into Elasticsearch via Spark, creating a new ratings index; adapt the CSV script to map lines to a rating class.
Explore how Elasticsearch acts as a vector database by storing and querying dense_vector embeddings for semantic search and retrieval-augmented generation, using OpenAI embeddings and a Python workflow.
Wrap up section 4 by mastering ways to import data from a variety of systems and set the stage for Elasticsearch's powerful aggregation capabilities.
Explore elastic search aggregation to analyze and visualize structured data, like weblogs, with Google analytics-like dashboards at scale. It aggregates massive data sets in milliseconds, faster than Hadoop or Spark.
Explore Elasticsearch aggregations, including metrics, histograms, buckets, nested aggregations, and pipelines, enabling moving averages, cumulative sums, and matrix-based analysis.
Shows how to generate histograms in Elasticsearch by bucketing values into interval ranges and counting documents per bucket. Uses ratings and decades as practical examples.
Explore how Elasticsearch handles time series data with date histogram aggregations by hour on the at_timestamp field, using log data to analyze website traffic and Googlebot hits.
Learn to pinpoint web outages by querying logs with a date histogram, starting coarse then drilling down to the minute, filtering for 500 errors in Elasticsearch 9.
Use nested aggregations to break data into dimensions and compute the average rating for each Star Wars movie by aggregating ratings per title, noting text-field gotchas and mapping needs.
Learn how to perform nested aggregations in Elasticsearch 9 by tackling text field limitations, enabling field data, and using a raw keyword subfield to aggregate on full titles.
Explore Kibana to analyze and visualize your Elasticsearch index data without writing queries, building on low-level aggregation concepts for practical use.
Explore Kibana, a powerful web UI on top of elastic search, to analyze data interactively without json requests. See how it opens your indices to analysis by anyone with examples.
Install Kibana to visualize Elasticsearch data with a web UI, configure it for a single host, match versions, start the service, and access via port 5601 for interactive analytics.
Explore Kibana hands-on by loading the Shakespeare data, creating data views, discovering data, and building visualizations—from tag clouds to Lens charts—using the Kibana UI and dev tools.
Practice Kibana by creating a vertical bar chart in the Shakespeare index that aggregates document counts by play name, sorted by lines, revealing Hamlet as the longest.
Learn to create flexible visualizations with Kibana Lens, drag-and-drop fields, data views, and switch chart types, combining metricbeat data with Elasticsearch for dashboards.
Explore Kibana management for Elasticsearch, including spaces, saved objects, and migrations; configure access, import/export of objects, and set advanced settings like default landing page and dark mode for teams.
Explore how Elasticsearch sql lets you query an index with familiar sql, treating it as a table, translating queries to dsl and json via an execution pipeline.
See why Kibana is powerful and easy to use, enabling insights from data that were hard to obtain, and preview how filebeat streams log data into an Elasticsearch cluster.
Beats provides a lightweight on-server utility to publish log data to a separate log stash cluster, which buffers to your Elasticsearch cluster, easing setup and use.
Master data frame transforms in Elasticsearch to aggregate data by entities and create destination indices. Use pivot definitions and preview and start transform APIs for efficient, advanced analytics.
Filebeat is a lightweight ingestion layer between log files and Logstash or Elasticsearch, enabling buffering and back pressure for a scalable, reliable Elastic Stack.
Explore how X-Pack security protects Elasticsearch data with access control, password protection, RBAC, and IP filtering, plus encryption, audit trails, and LDAP integration for role-based access control.
Install Filebeat from the Elastic repository, enable Apache access log paths, start the Filebeat service, and verify logs feed directly into Elasticsearch as a new index.
Visualize Apache access logs with Filebeat dashboards in Kibana. Import data, create data views, adjust absolute date ranges, and explore status codes to identify traffic patterns and 500 errors.
Explore Cabana to analyze a half-hour of 404 errors from 9:30 to 10 a.m. zulu time, identifying San Jose crawlers and their origins.
Wraps up by tying the elastic stack together, including Elasticsearch, Logstash, Kibana, and Beats, and shows how this simpler system prepares you to manage an elastic search cluster.
Explore real-world Elasticsearch cluster design, including architectural choices, hardware selection, and scaling as data grows, with X-Pack monitoring, failover, backups, and safe restart and update practices.
Explore Elasticsearch operations and how to choose the right number of primary shards, balancing write traffic, read capacity, and replica availability through real-world experimentation and phased scaling.
Create a new index with a defined number of shards and replicas to enable scaling; use index templates to automatically apply mappings, analyzers, aliases, and settings.
Rotate your indices with index aliases to add capacity without reindexing, searching across multiple indices like logs_current and logs_last_three_months.
Learn how Elasticsearch index lifecycle management automates aging indices through hot, warm, cold, frozen, and delete stages using size- or age-based policies and a rollover alias.
Prioritize RAM when selecting hardware for an elasticsearch cluster, aiming for 64 gigabytes per machine, with 32 gigabytes for elasticsearch and 32 gigabytes for the OS and disk cache. Pair fast local solid state drives with the right io schedulers and a robust network, and choose medium to large configurations to balance performance and cost.
Allocate about half of your physical memory to elastic search and the rest to the os and lucene, avoiding heap sizes above 32 gigabytes for optimal performance.
Learn to monitor the elastic stack with zpack, covering basic monitoring, alerting, security, and anomaly detection via machine learning, plus Kibana encryption keys setup.
Troubleshoot common Elasticsearch issues across node setup, cluster formation, and cluster state. Learn to use bootstrap checks, discovery, log analysis, and backups for resilient operation.
Watch failover in action as Elasticsearch adds a node, loses the master, and automatically rebalances shards across three nodes on a single vm, then recovers when the original node returns.
Explore dynamic index settings in Elasticsearch to improve performance and resiliency by adjusting replicas, blocks, and pipelines, and learn to split, shrink, and manage shards.
Learn how to configure Elasticsearch snapshots to back up indices incrementally, store them in a repository (local filesystem, S3, Azure, or others), and restore from snapshots.
Automate backups for Elasticsearch clusters with Snapshot Lifecycle Management, configuring repositories and policies to schedule snapshots, include all indices, and set retention and deletion rules.
Perform rolling restarts by disabling shard allocation and stopping new data indexing, then perform maintenance on each node, restart, rejoin the cluster, and re-enable allocation for green status.
Learn to implement uptime monitoring with heartbeat in the Elastic Stack by using ICMP, TCP, and HTTP monitors, configuring YAML monitors, and visualizing results with a heartbeat dashboard.
Wraps up elastic search fundamentals and cluster setup, then explains renting elastic search clusters in the cloud to avoid provisioning hardware, freeing you to build applications and analyze data.
Explore cloud options for renting Elasticsearch clusters, including AWS Elasticsearch Service on EC2 and Elastic Cloud, to save time and money.
Spin up an OpenSearch domain on AWS with Elasticsearch compatibility and IAM security. Learn cost aware settings, data nodes, and AWS integrations like Kinesis and Firehose.
Launch and connect to an Amazon OpenSearch Service cluster, import data with the bulk API, query to verify, explore OpenSearch dashboards, and shut down the domain to avoid costs.
Explore Elastic Cloud, a hosted Elasticsearch service with automatic cloud provisioning (Google by default, Azure or AWS options) and paid add-ons; set up a free trial and deploy.
Review what you have learned in section nine and plan the next steps to complete the elastic stack course.
Wraps up by recapping installation, mapping, indexing, searching, and aggregating data with the elastic stack, plus importing data from files and streaming sources, and visualizing with Kibana.
Now updated for Elasticsearch 9! (Almost all lessons are compatible with Elasticsearch 8 as well.)
Elasticsearch and the Elastic Stack are important tools for managing massive data. You need to know the problems it solves and how it works to design the best systems, and be the most valuable engineer you can be.
Elasticsearch is a powerful tool for analyzing big data sets in a matter of milliseconds! It’s increasingly popular technology for powering search and analytics on big websites, and a valuable skill to have in today's job market. This course covers it all, from installation to operations. Learn how to use Elasticsearch and implement it in your work within the next few days.
We've teamed up with Coralogix to co-produce the most comprehensive Elastic Stack course we've seen— with over 100 lectures including 14 hours of video.
We'll show you how to set up search indices on an Elasticsearch 9 cluster, and query that data in many different ways. Fuzzy searches, partial matches, search-as-you-type, pagination, sorting - you name it. And it's not just theory, every lesson has hands-on examples where you'll practice each skill using a virtual machine running Elasticsearch on your own PC.
You'll learn how to use Elasticsearch as a vector database or vector store, an important component of modern generative AI applications.
We cover, in depth, the often-overlooked problem of importing data into an Elasticsearch index. Whether it's via raw RESTful queries, scripts using Elasticsearch API's, or integration with other "big data" systems like Spark and Kafka - you'll see many ways to get Elasticsearch started from large, existing data sets at scale. We'll also stream data into Elasticsearch using Logstash and Filebeat - commonly referred to as the "ELK Stack" (Elasticsearch / Logstash / Kibana) or the "Elastic Stack".
Elasticsearch isn't just for search anymore - it has powerful aggregation capabilities for structured data, which allows you to glean new insights from your indexed data. We'll bucket and analyze data using Elasticsearch, and visualize it using the Elastic Stack's web UI, Kibana and Kibana Lens.
You'll learn how to manage operations on your Elastic Stack, monitoring your cluster's health, and how to perform operational tasks like scaling up your cluster, and doing rolling restarts. We'll also spin up Elasticsearch clusters in the cloud using Amazon Opensearch Service and the Elastic Cloud.
Elasticsearch is positioning itself to be a much faster alternative to Hadoop, Spark, and Flink for many common data analysis requirements. It's an important tool to understand, and it's easy to use! Dive in with me and I'll show you what it's all about.