AI Infrastructure Explained (GPUs, vLLM, and LLM-D)

Every AI product you use, from chat assistants to coding copilots, runs on a stack of GPUs, model servers, and orchestration systems working together behind the scenes. This beginner-friendly course gives you a clear mental model of that stack without requiring any machine learning background.

Instead of memorizing buzzwords, you will follow a single request from the moment a user types into a chat box all the way down to the hardware that answers it. Each lesson follows a problem, solution, new problem pattern, so you understand not just what each component does, but why it exists. Along the way, you will build up an end-to-end picture of how large language models are served to many concurrent users.

You will start with what a model actually is and what it takes to load one onto a GPU, then see how a model server like vLLM turns that model into a usable service. You will learn why models generate text one token at a time, why there is a pause before an answer starts streaming, and how the KV cache lets the system reuse work. From there, you will explore what makes serving hard at scale: batching multiple users on a single GPU, managing memory limits, balancing throughput against latency, and splitting models across multiple GPUs. Finally, you will meet llm-d, the open project backed by Red Hat, Google, IBM, and NVIDIA, and see how it routes LLM traffic intelligently across a Kubernetes cluster.

By the end of this course, you will be able to:

• Explain how GPUs, model servers, and supporting infrastructure work together to serve large language models

• Describe how vLLM serves models, including token generation, prompt processing, and KV cache reuse

• Analyze the trade-offs of batching, GPU memory limits, throughput, and latency when serving many users

• Evaluate why traditional load balancing struggles with LLM traffic and how llm-d addresses it on Kubernetes

This course is ideal for DevOps engineers, platform and infrastructure engineers, SREs, and anyone curious about the systems that power modern AI. No prior machine learning experience is required.