LLM Production Engineering: Ship AI Apps That Scale

A build that cannot be judged is a demo with users. This lecture wires Lecture 11's golden set into a gate that can block a deploy: hit@1, hit@3 and MRR for ranking quality, abstention accuracy on the three questions the corpus cannot answer, and a deterministic groundedness checker that verifies every quoted sentence exists verbatim in one of the chunks the answer cites. The baseline board is perfect — 1.000 across all five numbers, 17 quotes checked — and then the lecture breaks the system on purpose: a rushed 40-token-window chunker goes in, abstention collapses from 1.000 to 0.667, and the gate goes red. That red board is the product. Thresholds are set from the baseline — headroom where headroom is safe, held at the measured value where it is not — never from hope, and the gate exits 0 for ship, 1 for blocked. AI disclosure: this course's narration uses a cloned voice of the instructor, created with the instructor's consent; all content, code, and curriculum are authored and verified by the instructor team. Music: "Frozen Star" by Kevin MacLeod (incompetech.com) — CC BY 4.0

The three levers from Lecture 5, run in pipeline order on Lecture 14's measured fixture — each one's savings read off the board, none asserted. Compression kills the doubled verbose preamble: 213 tokens become 39, fixture cost falls 39.2%. The cache is the lecture's centrepiece: two earlier versions of it were wrong in instructive ways (a word-set similarity that ate the whole workload, a key built from parts every request shares), and the shipped version keys on each request's payload with its threshold examined on screen — true duplicates score 0.935 and above, but distinct requests reach 1.000: the ranges overlap, no gap exists, and the board prints what the 117 hits are actually made of (40 planted near-duplicates, 2 verbatim repeats, 75 fuzzy same-kind matches). Result: 117 hits among 189 cacheable requests, a 61.9% hit rate, cost down to $0.1187. The cascade then routes what remains — 114 of 123 served requests to the small tier — for a cumulative 94.6% reduction. Personalised drafts are never cached, on principle. Lecture 16 exists because none of this ships until quality is proven. AI disclosure: this course's narration uses a cloned voice of the instructor, created with the instructor's consent; all content, code, and curriculum are authored and verified by the instructor team. Music: "Frozen Star" by Kevin MacLeod (incompetech.com) — CC BY 4.0

Savings claims are cheap; signed boards are not. This lecture re-runs Lecture 15's compression + cache + cascade pipeline end-to-end and adds the quality guard the cascade was betting on: a deterministic keyword classifier stands in for the small tier, and its agreement with the recorded gold labels is measured on the 36 classification requests the cascade actually routes to it — the cache answers the other 31 classify rows before any tier sees them, and the rag and draft requests the length rule sends small are printed on the board as unmeasured, not silently covered by the agreement number. Low-confidence cases escalate to the flagship tier instead of guessing. The signed board: $479.66 a month becomes $25.91, a 94.6% reduction, with the quality gate at 95% passed — and the lecture states plainly what the board does not prove, including why agreement reads 1.000 on this fixture and what would make it fall on real traffic. The board is written to board.json, where Lecture 18's ShipGate will read it as the cost gate. AI disclosure: this course's narration uses a cloned voice of the instructor, created with the instructor's consent; all content, code, and curriculum are authored and verified by the instructor team. Music: "Frozen Star" by Kevin MacLeod (incompetech.com) — CC BY 4.0