AI System Design: LLM Gateways, RAG & Agent Platforms

In this lecture, we start with three product teams who use the same model and find it fast, slow and too expensive. On Polaris simulator profiles, the large model's ninety-fifth percentile is eight point nine seconds, at about twenty-five point six times the small model's cost. You see the design response to each dimension that varies: a deadline, an evaluation gate, cost per outcome and a degraded mode. In the mini challenge, you decide whether a five-second target is achievable on the large profile.

In this lecture, we ask the first question of the AI Architecture Loop: does this need a model at all? You score use cases with five questions that point to rules, search, a classic model, a large language model, a hybrid or no model at runtime. In the workshop, you score ten use cases from Polaris Group's customer operations, then compare with the reference answer, where only four of the ten use a large language model.

In this lecture, we turn a requirement that says the assistant must be fast and accurate into numbers, using the Polaris requirements template. You see the four autonomy levels, A0 to A3, why quality is a distribution, and how the Wilson lower bound sizes the golden set: eighty-five out of one hundred proves only seventy-seven percent. You also write latency as percentiles, cost per outcome and data classes. In the mini challenge, you decide whether a golden set of fifty is big enough.

In this lecture, you build tools/estimate.py, the estimation tool Polaris uses for the rest of the course, where every input is an assumption with a reason. You see why the peak rate uses busy seconds and a peak factor, and how Little's law gives about three point two eight requests in flight. In the lab, you write six functions until the support assistant estimate matches the reference of about seven hundred and fourteen dollars a month and the tool rejects a missing assumption.

In this lecture, we take a model call apart into time to first token and generation time. On the simulator, the first token arrived in eight hundred and two milliseconds at the ninety-fifth percentile, against four point one seconds for the whole answer, so streaming changes what users feel. You see why long answers and parallel calls sit in the tail, and how to build a latency budget with deadlines. In the exercise, you compute the p99 budget for a streamed answer.

In this lecture, we turn a support copilot for about four hundred agents into a requirements sheet and an estimate. You see the reference answer: autonomy level A1, an eighty-five percent target on a golden set of three hundred tickets, and four hundred and one thousand seven hundred tokens per minute at peak. Cost, at about four thousand seven hundred and twenty-eight dollars a month, is not the binding constraint. In the lab, you fill the template, run estimate.py and compare with the reference.

In this lecture, we open the box labelled LLM and draw the full request path once, with numbered hops and an owner for every layer. You see the knowledge, tools and operations planes, and the integration estate that most AI diagrams leave out. We cover asynchronous enrichment through a broker, request-reply with a correlation id, and why every worker still calls the same AI gateway. In the mini challenge, you place the copilot's draft request on the Polaris map.

In this lecture, we compare three ways to get an AI platform: buy a managed one, assemble one around an open-source gateway, or build a thin layer of your own policy. You judge them on criteria rather than product names, using the course's trade-off framework, a two-week proof of concept and an estimate of the exit cost. You also see why you always keep your own gateway interface. In the mini challenge, you choose an option for Polaris Group.

In this lecture, we choose between a hosted API, cloud-hosted models and self-hosting with a decision table. You see why residency asks where data is processed, not just stored, and why hosted cost is a line while self-hosted cost is a staircase. For Polaris, self-hosting costs two point one eight times the hosted cost today, and ADR 0001 records the decision with a refresh trigger.

In this lecture, we make three decisions every AI feature needs: the interaction pattern, where session state lives, and the output contract. You see why the model call stays stateless, why any output a system acts on needs a schema, and why asynchronous workers must be idempotent. In the workshop, you decide pattern, state and contract for four features, from a customer support chat answer to a nightly extraction from claim documents, then compare with the reference answer.

In this lecture, we start with a Monday morning where the copilot's peak, about thirteen point three times the customer assistant's, causes rate-limit errors. You compare three tenancy models, shared pool, reserved share and isolated, and see why quotas are written in tokens and requests. We define the paved road a platform team offers and place Polaris on a seven-step maturity ladder.

In this lecture, you play the architecture board and review five designs, each with one fatal flaw. They include a browser that holds the API key, one shared key for three teams, a semantic cache over personalised answers, retrieval with no access control, and a single region for EU and Gulf customers. For each design, you pause and decide whether to approve it, then hear the flaw, its impact, the fix and how to spot it in production.

In this lecture, the reference architecture becomes data you can check. You map Polaris in docs/architecture/polaris_map.toml, with layers, owners, components and numbered flows, and write ADR 0001 on hosting with at least two rejected options. You see two genuine failing runs: a map with no response path, and an ADR with only one rejected option. In the lab, you work from s02-start until the map and ADR structure checks pass, then compare with the solved tag.

In this lecture, we start with four teams at Polaris Group calling three providers directly, with no cost attribution, one team throttling the rest, and EU data sent to a non-EU endpoint. You see the five jobs of an AI gateway: identity, budgets, routing, resilience, and policy with evidence, and what fails without each one. You also see why the extra hop costs milliseconds.

In this lecture, we fix Monday morning with token-bucket quotas in gateway/quota.py and config/quotas.toml. You see why each tenant needs two buckets, one in tokens and one in requests, why admission charges an estimate and refunds the rest, and why the quotas must add up to less than the provider's limit. In the lab, you run the burst-tenant-a profile until support loses none of its seven hundred and twenty requests and every refusal is a four-two-nine with a retry time.

In this lecture, we move routing out of scattered if-statements into one ordered policy table in config/routing.toml. You see why the residency rules sit at the top, why regions filter every fallback, and why EU personal data gets the degraded answer when the EU route is down. A live demonstration sends one request to a hosted provider and one to a local model server. In the lab, you complete the table until the routing cases pass with zero residency violations during an EU outage.

In this lecture, a provider slows down and naive retries multiply the bill. You see the gateway's four mechanisms: a deadline per request, attempt timeouts from the time that remains, a shared retry budget, and a rule for streams that stop halfway. In the lab, you write five functions under the retry-storm, server-errors and partial-stream faults until no request runs past its deadline, retries stay inside the budget, and no truncated answer is returned as complete.

In this lecture, we compare exact, semantic and prompt-prefix caching, and show when each one lies. You see how the threshold of zero point nine was chosen from measured pairs, and how two customers with different booking references scored zero point nine four, causing seven hundred and nine wrong answers when personal data was allowed. You learn to scope the semantic cache by tenant, route and data class. In the mini challenge, you pick the cache for four features and name its failure test.

In this lecture, we work through an incident where three services with default timeouts and three retries turn a slow provider into a bill about three point eight times the budget. On Polaris, the rejected design made three point seven eight calls per request, and only thirteen percent succeeded. You pause and decide what to check first, then follow the root cause, the mitigation and the permanent fix, with model calls per request as the detection signal.

In this lecture, we set Project One: one gateway for Polaris Group that fixes cost visibility, noisy tenants and residency without slowing the support assistant. Done means five fitness functions passing from a clean clone, covering quotas, EU residency, bounded retries, cache scoping and a ninety-fifth percentile within four seconds. You see the reference result, the rejected design and the rubric. In the project, you spend about four to six hours combining the three labs and scoping the semantic cache.

In this lecture, we start with Polaris Group's support assistant still quoting on Wednesday a refund policy that changed on Monday, although nobody touched the prompt or the model. You see retrieval as a data pipeline, from connectors to the answer cache, with at least five copies of every document, and a service level per stage for freshness, completeness, deletion, quality and cost. In the mini challenge, you assign an owner and a service level with a number to each stage.

In this lecture, we compare two chunkers on a retrieval set of three hundred and twenty questions about eighty invented Polaris policy documents. Fixed windows found the right section for twenty-three percent of questions and cut forty-six facts in half, while section chunks with a title and heading prefix found seventy-two percent and cut none. You also see the metadata later stages need, including access groups and a content hash. In lab s04-1, you build the chunkers, attach the metadata and record the decision.

In this lecture, we upgrade Polaris to a better embedding model and see re-embedding in place fail: eighty of eighty queries during the upgrade silently could not see part of the corpus. You learn why vectors from two models are not comparable, and how a dual index with a backfill, an alias and a cut-over removes that window. In lab s04-2, you build the four functions, reach zero failed queries out of one hundred and ten and see recall rise from seventy-two to eighty-nine percent.

In this lecture, we hold Polaris to a freshness SLO of ninety-five percent of changes visible within five minutes, and see the nightly rebuild make none of two hundred and forty changes visible within the run. You see change capture, tombstones and why the answer cache is invalidated when the replica applies a change. In lab s04-3, you wire the change feed, reach a ninety-fifth percentile lag of sixty seconds and prove all sixteen deleted documents are gone from every copy.

In this lecture, we replace the question of which vector database to use with five questions: filtering, hybrid search, where it runs, updates and deletes, and tenancy. You meet flat, inverted-file and graph indexes and quantisation, with products named only as dated examples. In lab s04-4, you complete tools/capacity.py and size the Polaris index at four million eight hundred thousand chunks: about eighty gigabytes with float32 across three copies, one hundred and sixty during an embedding upgrade, and about twenty-seven with int8.

In this lecture, we work through an incident in which the assistant quotes withdrawn policies while the source system and the primary index are both clean and the dashboard stays green. Reproduced with the stale-replica fault, twenty-eight answers quoted a deleted document, and the deletion audit finds all sixteen deleted documents still in the replica. You trace the root cause to dropped tombstones and specify the mitigation, permanent fix and detection signal. Before the answer, you pause and decide what you would check first.

In this lecture, we bring the earlier labs together in config/retrieval.toml, where every value is a decision backed by evidence. At the start tag, the configuration still describes the old Polaris, so the fitness functions fail even with correct code. You set the chunker, the dual-index upgrade and the change feed, then trace one change through every copy. In lab s04-5, you pass all ten Section Four fitness functions, break the pipeline with the stale-replica fault and tune the poll interval against the SLO.

In this lecture, we replace decisions made on feelings, a reranker added, then removed, on ten questions each, with numbers. You separate retrieval metrics such as recall at k, precision at k and mean reciprocal rank from answer metrics such as faithfulness, and build a judgement set of two hundred and forty questions in three kinds. Vector search finds eighty-nine percent overall but only sixty-eight percent of code questions. In the mini challenge, you match three failures to the metric that would catch each one.

In this lecture, we measure hybrid search, reranking and query rewriting against a recall at ten of ninety percent and a budget of three hundred milliseconds. Hybrid search with reciprocal rank fusion adds about five point eight points for eight point five milliseconds, and reranking lifts recall at three by about fifteen point four points. Query rewriting pushes the ninety-fifth percentile to five hundred and eighty-seven milliseconds and is rejected. In lab s05-1, you measure each step's gain and cost.

In this lecture, we serve three customer organisations, acme, globex and initech, from one service. You compare a shared index with a tenant filter, a partition per tenant and a separate index per tenant, and see how a bulkhead caps one tenant's share of the worker pool. At the start tag, two thousand three hundred and five chunks reach the wrong organisation. In lab s05-2, you pick an isolation model and pass the noisy-neighbour profile, keeping globex at twenty-six milliseconds and initech at fourteen.

In this lecture, we start with an HR policy passage appearing in a travel team employee's answer. You carry access groups on every chunk, resolve membership at query time and deny by default when a chunk has no groups. Filtering after ranking leaks nothing but leaves all one hundred and twenty narrow-principal queries short of k, while filtering before ranking gives full results. In lab s05-3, you move the filter before ranking and prove zero leaks across a generated attack set.

In this lecture, we spend the three-hundred-millisecond retrieval budget step by step and see that the reranker owns the tail. Then we compare long context, retrieval and fine-tuning. For one organisation's library of about three hundred thousand tokens and one hundred thousand questions a month, long context costs about one hundred and thirty-five point one times retrieval on one dated price row, and about two hundred and forty-seven point nine times on another with a long-context surcharge.

In this lecture, we review six retrieval designs. Two leak, by padding the list back to k and by keying an answer cache on question text alone. Two lag, through a nightly rebuild and through user lists frozen at indexing time. Two lie, by answering when nothing relevant was found and by serving two versions of a policy at once. For each design, you pause, find the flaw, state its impact and propose a fix before comparing.

In this lecture, we set Project Two: three customer organisations sharing one document-intelligence service, with access rules by group, a budget of three hundred milliseconds and a recall at ten of ninety percent. You compose retrieval/service.py in the order isolation, access, then quality, and configure config/rag.toml from your lab evidence. In the project, planned at six to eight hours, you pass five fitness functions and present the rejected shared, post-filtered design, which leaked five hundred and seventy-two cross-tenant chunks at k of three.

In this lecture, we start with five agents built by four teams, each with its own credentials and approval rules, and a refund nobody approved. You see an agent as a dependency with hands, and what the Polaris agent platform owns as shared services: a tool registry, delegated identity, a policy layer, an approval queue, durable execution and an audit trail. You also meet risk tiers for the seven registered tools. In the mini challenge, you assign ten responsibilities to the platform or the agent.

In this lecture, we start with a tool server connected on a Friday and used by three agents by Monday with no owner. You design a tool registry in config/tools.toml with owners, risk tiers, scopes and egress allow-lists, and an MCP gateway that enforces them on every call. You see how the July twenty twenty-six MCP revision, now stateless with server discovery, lets the gateway scale horizontally. In the mini challenge, you list what the registry stores and what the gateway enforces.

In this lecture, we remove the ambient service account, svc-support-agent, behind a refund that nobody could trace to a user. You learn OAuth 2.0 Token Exchange, RFC 8693, with the user as subject and the agent as actor, and Polaris's rule of one token per tool call, with one scope and a life of at most five minutes. In lab s06-1, you write the exchange and check functions, so all twenty-one tool calls carry delegated tokens and Sam's refund is refused at the exchange.

In this lecture, we start with a bulk task that a routine restart ran twice, sending four customers two closure notices each. You learn why durable execution needs both checkpoints, in a store that outlives the worker, and stable idempotency keys. You also give every approval a fifteen-minute timeout whose safe default is not acting, followed by escalation. In lab s06-2, you reproduce the worker crash, resume at step three with zero duplicate closures, and make sure an unapproved refund never happens.

In this lecture, we answer three design questions. You see how isolated model-written code must be, from containers through user-space kernels and microVMs to full virtual machines, with egress denied by default. You decide when a second agent pays for itself, and what the A2A protocol, version one point zero point one, standardises between agents and what it leaves to you. In the mini challenge, you decide one agent or several for three workloads.

In this lecture, we test whether the support agent obeys a harmless canary instruction, CANARY-7, planted in a retrieved triage guide. You meet the sixth mental model, text crosses trust boundaries, and build a policy layer that tracks the provenance of every argument and applies its rules in order, plus a hash-chained audit trail. In lab s06-3, you complete the policy and audit functions so zero injected calls run, high-risk calls wait for a named approver, and the chain of twenty-nine records verifies.

In this lecture, we start with a Friday prompt edit that, by Monday, had refund and cancellation questions answered wrongly about a third more often. You meet the fifth mental model, no eval, no release, and place offline, pre-release and online evaluation in the architecture. You see the shortened prompt, candidate b, fall about eight point nine points overall but about thirty-three point three on cancellations. In the mini challenge, you assign five failures to the stage that should catch them.

In this lecture, we build a golden set with at least twelve cases per intent and twenty-four for each high-risk intent, then calibrate two judges against three hundred and twenty human labels. Judge one agrees on zero point eight four but has a kappa of only zero point three, and passes every long wrong answer. Judge two, grading against the reference with a rubric, reaches a kappa of zero point seven. In lab s07-1, you measure judge agreement and find the length bias.

In this lecture, we start with an incident review that cannot say what was running, because only one of five moving parts was recorded. You learn to pin the prompt, model, index and policies in one release manifest, using content hashes for text, exact dated model versions instead of aliases, and snapshots for indexes. You also see how manifests make promotion between environments and rollback simple. In the mini challenge, you find the unpinned component in three manifests.

In this lecture, we replace a release check that read one average, and let the shortened prompt through, with a gate that reads slices. You learn Wilson confidence intervals, a margin and a floor agreed in advance, and promotion and rollback rules for shadow traffic and canaries. In lab s07-2, you pin the candidate manifests and build the gate, which blocks candidate b on cancellations at zero point six two against production's zero point nine six and promotes candidate c.

In this lecture, we trace retrieval, model and tool calls with OpenTelemetry's generative AI conventions, which are still in Development status. You keep the convention names behind one mapping module, ops/genai_semconv.py, keep prompts and answers off spans, and record what an exact replay needs. In lab s07-3, you build the span and replay functions so all sixty spans carry their required attributes and twenty of twenty model calls replay identically. A live demonstration then traces one real provider call.

In this lecture, we keep measuring quality after release. Polaris judges six hundred production answers a day, sampled by slice, adds customer feedback, and alerts per high-risk intent over a three-day window against a reference week; over a quiet simulated fortnight, it raised zero alerts. You also add a daily golden re-run, on-call ownership and mitigations ready in configuration. In the mini challenge, you write the first five checks of a runbook for a quality alert on refund answers.

In this lecture, we work through a model upgrade that passed the gate, shadow traffic and a canary, yet a week later gets refund and delay answers wrong with no manifest change. Reproduced with the wrong-answer-drift fault, per-intent online sampling alerted on day nine and the daily golden re-run on day eleven. You trace the root cause to an evaluation gap and specify the mitigation and permanent fix. Before the answer, you pause and decide what you would check first.

In this lecture, we return to the Friday prompt change and prove it is now blocked. At the start tag, config/evals.toml names the lenient judge one, a mean gate and no replay, and the pipeline promotes candidate b on forty-eight golden cases. You set the judge, the slice gate and replay records, then switch back to the biased judge to see that every part is needed. In lab s07-4, you pass all ten Section Seven fitness functions with candidate b blocked.

In this lecture, we start at a launch review for a support assistant in Europe, where the team has good answers about the network and the database and almost none about the text. You see why text crosses trust boundaries, mark where untrusted text enters Polaris and the three places it can become an instruction, and keep the threat model as a table in the repository. In the mini challenge, you mark every trust boundary on the Polaris request path and name its control.

In this lecture, we map the OWASP Top Ten for LLM Applications and the OWASP Top Ten for Agentic Applications to Polaris controls. You see why a mapping by name survives the next edition, how unbounded consumption maps to the gateway's quotas, retry budgets and spend caps, and why prompt injection is reduced, never removed. In the mini challenge, you name the Polaris control, and the section that built it, for five risks.

In this lecture, we replace the single filter in front of the model with guardrails at four layers: input, retrieval, output and tool. You see what each layer misses and what it costs, from about four hundred milliseconds at the ninety-fifth percentile for a model-based check to zero per query for the retrieval guard, and why their gaps should not line up. In the mini challenge, you assign three failures to the layer that should stop them.

In this lecture, we treat the gateway's request logs as a data store, using a test set of four hundred prompts with six hundred and thirty-four labelled personal values. You implement keyed tokenisation, measure the redactor's recall by type, and follow one rule: redact what you can enumerate, and do not log content you cannot. In the lab, you complete the logging policy in config/logging.toml until zero labelled values reach the logs, and report the redactor's misses honestly.

In this lecture, we design residency as a tenant property that covers every hop: processing, storage, keys and support access. You compare the dated rules for the EU, Saudi Arabia, the United Arab Emirates and India, and see how a regional cell keeps failover inside the region or degrades. In the exercise, you price one month of residency for the EU support tenant from a dated price row, with and without the regional uplift of one point one.

In this lecture, we cover the dated duties that affect architecture, without giving legal advice. You see which EU AI Act dates moved after the Digital Omnibus on AI and which did not, where the NIST AI RMF and ISO/IEC forty-two thousand and one fit, and why none substitutes for another. You also see why audit evidence should be a tamper-evident by-product of running the system. In the mini challenge, you list at least eight items of evidence Polaris already produces.

In this lecture, we turn three claims an EU data-protection officer needs into fitness functions, and add a fourth for the canary. You write the EU tenant's residency plan and a tenant routing rule, taking hops outside the EU from nine hundred of one thousand to zero, anchor the audit trail's head every ten records so all three tampering cases are detected, and keep the canary from agents with source policy. In the lab, you pass every Section Eight fitness function from the start tag.

In this lecture, we start with a spinner that turns for six seconds and ends in "Something went wrong", and meet the seventh mental model: design the degraded mode first. You learn to tell down, hung, rate-limited and wrong apart, build a ladder of responses per feature that keeps every constraint, and give Polaris traffic critical, standard and batch priority classes. In the mini challenge, you write the degraded mode for three Polaris features, including the EU tenant.

In this lecture, we take Polaris from one route per tenant to a failover chain in config/reliability.toml: provider A east, then provider A west, then provider B. You see why both another region and another provider are needed, treat rate limits as shared capacity with quota pools, and share them by priority class. In the lab, you implement admit in reliability/pools.py and take critical availability through the region outage from sixty-nine percent to one hundred percent, with zero EU attempts leaving the EU.

In this lecture, we see why a hung provider is worse than a failed one, and add two defences: a first-token timeout of one point five seconds and a circuit breaker per route that opens after five failures within thirty seconds. You also shed load by priority class, queueing batch work and refusing standard requests with a retry-after. In the lab, you add the breaker and the shedding policy, cutting attempts into the hung provider from one thousand five hundred to twenty-three.

In this lecture, we start with a refund issued twice because the payments response was lost and the workflow retried. You see why retries repeat side effects, how an idempotency key per workflow step, the same on every attempt, stops them, and how outboxes and compensation cover the rest, with the model deciding while the workflow acts. In the mini challenge, you find the duplicate side effect in a refund flow and fix both of its problems.

In this lecture, we define three service levels for the critical class in config/slos.toml: availability of ninety-nine point five percent, time to first token of two seconds at the ninety-fifth percentile, and quality of eighty-five percent, measured from a judged sample with a Wilson interval. You also see error budgets and alerts on the interval. In the lab, you implement the measurements and run chaos profiles for slow first tokens, wrong answers and empty retrieval.

In this lecture, we reconstruct a composite launch-day incident, where provider A's east region hangs just before a tenfold spike and critical availability over the half hour falls to twenty-six percent. You find two failures, the missing fallback and the cost spike from a hot fix running at forty-six dollars and two cents an hour against a cap of twenty-five dollars, plus a duplicate refund, and specify the permanent fix. At the pause, you decide as the on-call engineer what to check first.

In this lecture, we take Polaris through the launch-day profile, a hung region and a tenfold spike together, keeping critical availability at ninety-nine point five percent or more and spend under twenty-five dollars an hour. You tune the failover chain, shares and shedding thresholds, implement step_key, and compare with the reference result of sixteen dollars and fifty-two cents an hour. In the project, six to eight hours, you pass the four project fitness functions and write ADR four with the rejected design.