LLM on OpenShift AI: Deployment Masterclass

In this session, you will get a practical introduction to Large Language Models (LLMs), LLM inference, model runtimes, and model selection while understanding how to build a private AI solution using Red Hat OpenShift AI. The session explains the architecture behind a ChatGPT-style application, including the user interface, API gateway, LLM model, inference process, and response flow.

You will also understand the difference between using cloud-based AI services and deploying your own private or sovereign AI infrastructure. The session introduces open-source models, Hugging Face, model registries, LLM runtimes such as vLLM, and the role of OpenShift in managing AI workloads.

This video provides the foundation required before deploying and serving an LLM on OpenShift AI.

Topics covered: LLM fundamentals, OpenShift AI, sovereign AI, private AI, model selection, Hugging Face, model registry, inference, vLLM, API gateway, and LLM architecture.

In this session, you will learn how to prepare the OpenShift infrastructure for AI and LLM workloads. The video demonstrates the OpenShift AI platform setup, installation of the required operator, and creation of a Data Science Cluster.

You will understand how OpenShift AI brings together important AI components such as KServe, Workbench, Dashboard, Model Registry, Pipelines, and related services. The session also explains how these components support MLOps and AIOps workflows on top of OpenShift.

You will learn how to monitor the installation process using OpenShift events and understand how the Data Science Cluster configuration controls the components required by the AI platform.

This session is especially useful for learners preparing an OpenShift cluster for LLM deployment, model serving, AI application development, and inference workloads.

Topics covered: OpenShift AI, OpenShift AI Operator, Data Science Cluster, KServe, Workbench, Dashboard, Model Registry, AI Pipelines, MLOps, AIOps, and cluster activation.

In this session, you will learn how LLM parameters, model size, precision, memory requirements, and quantization are connected. The video explains why large language models require significant storage and memory resources when they are loaded for inference.

You will understand the relationship between model parameters and storage requirements and learn how precision formats such as FP32, FP16, and lower-bit representations can significantly reduce the resources required to run an LLM.

The session also introduces LLM quantization as a technique for reducing model size and infrastructure requirements while attempting to maintain model quality and accuracy. You will see why quantization is particularly important when deploying LLMs on your own infrastructure and how it can help reduce memory, CPU/GPU, storage, and overall infrastructure requirements.

Topics covered: LLM parameters, model weights, FP32, FP16, INT8, INT4, memory requirements, model size, quantization, inference resources, infrastructure planning, and LLM optimization.

In this hands-on session, you will learn how to identify and resolve pending pods caused by insufficient cluster resources in an OpenShift AI environment.

The video demonstrates how to investigate pod scheduling problems, examine pod events, identify CPU and memory constraints, and determine when additional worker-node capacity is required. You will learn how MachineSets can be used to increase the number of worker nodes and provide additional CPU and memory resources to the OpenShift cluster.

The session also demonstrates how Kubernetes scheduling works when a workload requires more resources than are currently available on individual worker nodes. This is particularly important when deploying AI services and LLM workloads that require significant compute and memory capacity.

Topics covered: OpenShift worker nodes, MachineSets, pod scheduling, pending pods, CPU resources, memory resources, cluster scaling, Kubernetes scheduling, OpenShift AI infrastructure, and AI workload capacity planning.

In this hands-on session, you will begin deploying a Large Language Model on OpenShift AI. The video demonstrates how to create a project, explore available models through the AI Hub, select an appropriate LLM, and configure the deployment for inference.

You will learn how to select an LLM based on parameters, quantization, hardware requirements, and intended use case. The session demonstrates configuring the model to run with vLLM as the model runtime and KServe as the model-serving layer.

You will also learn how OpenShift AI hardware profiles control CPU and memory resources for the model pod, including the concepts of memory requests and limits. The video shows how these settings influence Kubernetes scheduling and model deployment.

Topics covered: OpenShift AI, AI Hub, LLM deployment, vLLM, KServe, model serving, quantized LLMs, hardware profiles, CPU, memory limits, memory requests, Kubernetes pods, and inference.

In this practical troubleshooting session, you will learn how to diagnose and resolve common issues that can occur while deploying LLM inference workloads on OpenShift AI.

The video demonstrates how to investigate a model that remains in a starting or pending state, inspect pods and events, review container logs, and identify the actual cause of a failed deployment. You will learn how CrashLoopBackOff and out-of-memory conditions can occur when the resources allocated to an LLM are insufficient.

The session also demonstrates how to create and modify a hardware profile with increased CPU and memory resources and apply it to the model deployment. This provides a practical understanding of resource troubleshooting for LLM workloads.

Topics covered: OpenShift AI troubleshooting, LLM deployment errors, pod failures, CrashLoopBackOff, out-of-memory errors, container logs, Kubernetes events, CPU resources, memory resources, hardware profiles, vLLM, and KServe.

In this session, you will complete the practical process of launching an LLM pod and serving the model for inference on OpenShift AI.

The video explains why model weight size alone is not enough when planning infrastructure and why additional memory and compute resources may be required during model startup and inference. You will see the deployed model running through KServe and vLLM and learn how the serving layer exposes APIs for interacting with the model.

The session introduces important inference endpoints, including health and model-related APIs, and explains how an LLM can be accessed through an inference endpoint. You will also learn how the OpenShift AI Playground can be used to test and interact with a deployed model.

Topics covered: LLM pods, model deployment, KServe, vLLM, model serving, inference endpoints, REST APIs, OpenShift Routes, model health, model APIs, OpenShift AI Playground, and LLM inference.

In this session, you will learn how to use a deployed LLM inference service to build a ChatGPT-style web application. The video explains the role of KServe in exposing inference APIs and demonstrates how an application can communicate with the deployed LLM through an API endpoint.

You will explore the model's inference APIs, including the chat completion workflow, HTTP methods, JSON request data, model identification, user prompts, system messages, temperature, and streaming responses.

The session then moves into application development using Python and Gradio to create a simple web interface for interacting with the LLM. You will learn how the application connects to the inference endpoint, passes prompts to the model, and displays streaming responses.

Topics covered: KServe, LLM inference API, REST API, chat completion, Python, Gradio, OpenShift AI, vLLM, LLM web application, ChatGPT-style UI, streaming responses, prompts, system messages, temperature, and model APIs.