AI & ML Search with OpenSearch (elasticsearch + AI/ML)

Explore system and network requirements for OpenSearch, including installation options (Docker, package managers, Kubernetes), storage, Java needs, and dynamic versus static configuration.

Explore the OpenSearch plugin ecosystem, including bundled and on-demand plugins like alerting, anomaly detection, cross-cluster replication, and security analytics, and learn to manage plugins via the CLI and dashboards.

Compare and use OpenSearch Python clients, balancing the low level client for raw API control with the ML client that simplifies ML pipelines for anomaly detection and similarity search.

Learn to set up OpenSearch on an AWS EC2 instance by provisioning an Ubuntu 20.04 LTS VM, installing Docker, and enabling SSH access via Visual Studio Code.

Connect to an AWS EC2 instance running OpenSearch via remote SSH in Visual Studio Code, configure PEM key, clone the course repository from GitHub, and create a Python virtual environment.

Set up and access OpenSearch on a local machine by connecting VS Code to a remote Linux VM, running a single-node docker compose with OpenSearch and dashboard, and validating startup.

Explore how data layout, indexing, tokenizers, and analyzers shape search performance using a wardrobe analogy, showing how color, type, or size layouts affect retrieval time, redundancy, and scalability in OpenSearch.

Explore OpenSearch, an open source, enterprise-grade search and observability suite that finds meaning in data. See how its Apache 2.0 license and Elasticsearch fork enable powerful search and security analytics.

Explore OpenSearch search types, including text, neural/semantic, and hybrid searches, and learn their strengths, costs, and key concepts like exact match, full-text, phrase, wildcard, and fuzzy searches.

Explore advanced search techniques in OpenSearch, including regex, compound and nested queries, geospatial searches, multi-field and function score queries, and practical examples across JSON and architecture.

Explore OpenSearch use cases across industries like e-commerce, media, fintech, and healthcare. Learn how real-time search, personalized recommendations, anomaly detection, and log management drive engagement and security.

Compare OpenSearch and Elasticsearch against competitors, review db-engines rankings, and highlight OpenSearch advantages such as open-source licensing, security, AWS integration, and AI/ML capabilities.

Master OpenSearch core concepts such as inverted index, TF-IDF, and the BM25 ranking algorithm. Learn how term frequency, inverse document frequency, and normalization shape relevance scores for efficient search results.

Demonstrate an end-to-end ai ml search pipeline on OpenSearch and Elasticsearch. Launch a docker compose cluster, connect via Visual Studio Code, and use Kibana dev console to index and search.

Create and populate an Elasticsearch/OpenSearch index named Megacorp, explore mappings, perform document retrieval and searches with match, phrase, and boolean queries, and compare parity between Elasticsearch and OpenSearch.

Explore the OpenSearch text analysis pipeline, from character filters to tokenizers and token filters, and learn how indexing uses analyzed text to boost search relevance.

Explore analyzers in OpenSearch, from built-in to custom, and how character filters, tokenizers, and token filters form the text analysis pipeline for effective indexing and search.

Customize OpenSearch analyzers with tokenizers, token filters, and character filters, using case normalization, stemming, and synonyms to boost search relevance. Understand tokenization and single-token normalizers for keyword queries.

Extend the minimal text search workflow in OpenSearch and Elasticsearch. Demonstrate indexing, using keyword fields, running aggregations, and checking cluster health via dashboards and APIs.

Explore OpenSearch mappings in this demo, binding fields and types for efficient indexing, including dynamic formats, numeric detection, and runtime fields to enhance search.

Compare OpenSearch mappings and field data types with Elasticsearch. Map alias, completion, date, nested, and geo point fields while noting OpenSearch limits on dense vectors and flattened types.

Explore text analysis with OpenSearch by testing analyzers and tokenizers, including whitespace and standard tokens, lowercasing and ascii folding, plus custom analyzers and HTML stripping.

OpenSearch's search methods span bm25 keyword search, vector and k-nn neural, semantic dense retrieval with embedding models, and multimodal and hybrid search; explore query languages, performance, and user behavior insights.

Explore autocomplete with term and phrase suggestions using edit distance and n-gram models, and retrieve specific fields with doc value and stored fields, plus source filtering.

Explore lexical search with OpenSearch through hands-on demos, including filtering, highlighting, autocomplete, and did you mean, plus index setup, data loading, collapse, and search after patterns.

Learn to filter source fields with includes and excludes in OpenSearch, shape results with mappings and bool queries, and compare post versus get requests and pagination.

Explore kNN search and OpenSearch integration, mapping documents to vectors, measuring semantic similarity with distance functions, and choosing between approximate and exact kNN for large vs small indices.

Discover neural search in OpenSearch, turning text into embeddings, indexing with a KNN vector index, and applying semantic, hybrid, and multimodal searches for scalable results.

Explore the intuition behind large language models and machine learning through a desert survival analogy, covering embeddings, pattern matching, and agentic frameworks for problem solving with OpenSearch.

This lecture demonstrates semantic search using sentence transformers, comparing dense vs sparse embeddings, cosine similarity, and clustering to improve OpenSearch retrieval with examples in Python.

Explore retrieval augmented generation (RAG) with OpenSearch and multimodal data, using vector stores and embeddings to retrieve context and generate accurate up-to-date results.

Explore vector indexing to organize embeddings for faster similarity searches, and compare ann and knn across techniques like ivf, ivf pq, scalar quantization, and NSW.

Demonstrate kNN vector search in OpenSearch using a two-node cluster, building a kNN index with vector dimensions, euclidean distance, and inner product options, and testing with query vectors.

Explore neural search with embeddings and vectors in OpenSearch, building and deploying a semantic pipeline using a model group, text embeddings, and hybrid lexical-neural queries.

Explore neural search and hybrid search in OpenSearch using a Python client to build embedding models, ingestion pipelines, and vector indices, then perform semantic and lexical-hybrid queries.

Explore edge n-grams in OpenSearch by building an autocomplete analyzer, configuring min and max grams, indexing with product name mappings, and testing trigram and unigram matches.

Discover how the ML Commons plugin connects local, pre-trained, and externally hosted models (OpenAI, Anthropic, SageMaker) to OpenSearch for text embedding, cross-encoder, and sparse encoding with access control and connectors.

Showcases text embedding with the Msmarco Distilbert pretrained model in OpenSearch ML Commons. Demonstrates docker compose setup, model group creation, deployment, and prediction to generate a vector embedding.

Demonstrates embedding text with a local onnx sentence transformer model in OpenSearch, including downloading a pretrained model, registering, deploying, and performing text embeddings via the ML client.

Walk through registering and deploying a local torchscript sentence transformer model for OpenSearch text embedding, highlighting a torch vs onnx format deployment error and related troubleshooting.

Discover how to deploy an unregistered local Onnx sentence transformer in OpenSearch for text embedding, register and deploy the model, and perform inference with pre-trained models.

Explore sparse encoding in OpenSearch by registering, deploying, and using sparse encoding models to generate token: weight pairs in sparse vectors.

Explore how cross encoder enables query reranking for retrieval augmented generation by scoring query–document pairs. The demo registers a Hugging Face cross encoder in OpenSearch, deploys it, and predicts similarity.

Explore externally hosted models like OpenAI, cohere, bedrock, and Gemini, and connect them to OpenSearch with connectors, model groups, and deployment workflows for embeddings and chat completions.

Explore OpenSearch connector blueprints for pre and post processing and learn how to configure default and custom functions. See a batch ingestion demo that creates embeddings using OpenAI and SageMaker.

Implement guardrails to prevent offensive or harmful outputs from large language models using prompts and stop words. Demonstrate input-output filtering with a stop-words index and percolator queries in OpenSearch.

Explore how agentic frameworks in OpenSearch coordinate specialized tools and large language models through flow, conversational flow, and conversational agents, using retrieval augmented generation (rag) with vector embeddings.

Demonstrates retrieval augmented generation using a non-supported sentence transformer with OpenSearch, including embedding via a custom ingest pipeline, indexing data, and deploying a GPT-3.5 model.

Demonstrates rag with the interns dataset using OpenSearch, vector embeddings, and a recruiter agent to extract salaries from unstructured job content and answer salary questions with OpenAI.

Explore OpenSearch tools for ai/ml search, including ml model, agent, rerank, indexing, connector, rag, search alerts, anomaly detector, and vector db tools, noting evolving stability.

Register and deploy a cross-encoder model and use a rerank pipeline to order results by semantic similarity. Show that reranked results outperform search for queries like capital of United States.

Demonstrates a rag conversational flow agent with multiple knowledge bases in an OpenSearch cluster, using two vector db tools and retrieval augmented generation with OpenAI and sentence transformers.

Explore how a conversational agent dynamically selects an OpenSearch index from multiple knowledge bases, using BM25, neural, and a hybrid approach with a min-max normalizer to rank results.

Demonstrates a rag chatbot conversation agent that dynamically selects the appropriate knowledge base, tech news or population data, based on the question, using embeddings, OpenSearch, and OpenAI tooling.

Delve into OpenSearch dashboards, explore a range of query languages—DSL, DQL, query string language, SQL—and create visualizations, discover data, and explore observability features.

Learn end-to-end data exploration in OpenSearch dashboards by adding sample data, using discover and dashboards, applying filters and boolean queries, and creating visualizations with dxl and dark mode customization.

Analyze data with OpenSearch dashboards using discover to search, filter, view document details, and save searches as templates, while applying time range filters across sample flight and e-commerce data.

Create dashboards in OpenSearch, add panels powered by searches, and visualize e-commerce data using index pattern; customize visuals, adjust time filters, and save with markdown options.

Explore how to customize branding in OpenSearch dashboards by changing the logo, application title, and favorite icon via the OpenSearch dashboards YAML file, with live browser updates.

Explore the dashboards query language (DQL) to search terms and phrases, escape reserved characters, perform field-specific searches with wildcards and boolean operators, and handle ranges and nested fields.

Explore OpenSearch observability dashboards, integrations, notebooks, and reporting features, including sample data, dashboards for Nginx and S3, SQL querying, and alert-ready anomaly detection.

Explore data prepper as a server-side data collector for OpenSearch, detailing ETL pipelines for trace and log analytics, with sources, buffers, processors, and sinks.

Compare data prepper and logstash in the elk stack and OpenSearch ecosystem, examining pipelines with sources, buffers, processors, and sinks across cloud and on premise inputs.

Explore a live demo of ingesting Apache logs into OpenSearch via Data Prepper and Fluent Bit, using Docker Compose to read common Apache log format.

Demonstrate end-to-end log ingestion with OpenTelemetry logs into OpenSearch via Fluent Bit and Data Prep, using a Docker container and a generate hotel logs script.

Load data, create snapshots, and restore OpenSearch indices using docker, illustrated with the e-commerce and interns dataset. Build and apply vector embeddings for ai/ml workflows in search.

Demonstrate migrating from Elasticsearch to OpenSearch by loading data, creating a snapshot, and restoring it into OpenSearch, using docker, OpenSearch dashboards, and end-to-end verification of the e-commerce index.

Explore geospatial analysis with OpenSearch by indexing geo points and performing geo bounding box, distance, polygon, and shape queries using GeoJSON, with attention to precision and rounding.