AI Incident Response: LLM & Agent Failures in Production

What to log, how to build the detection stack, and the triage tree for the first five minutes. Lab 2 has you read the span tree Atlas already emits, then author detections and prove each one fires.

An agent run is a distributed trace, so reuse the tracing stack your platform team already runs. Guardrails must emit on allow as well as block, or you have no baseline to compare against.

The most-referenced lecture in the course: failure mode against observable signal, with what fires first, what confirms it, and where it hides. Single signals are ambiguous; combinations are fingerprints.

Alert fatigue is a detection control that degrades on its own. Static thresholds are the main source of noise, and the fix is to page on combinations and on irreversible actions.

Nobody thinks clearly at four in the morning, so the triage tree is written in daylight. Four questions in a fixed order, one action each — and 'is it still acting?' comes first.

Atlas already emits a full span tree. Read it, account for all thirteen attributes, then author four detections and prove each one fires.

Scoring severity on four axes, declaring an incident, and preserving evidence before you touch anything. Lab 3 gives you three deliberately ambiguous incidents to score and defend.

Roles around the bridge, and the seat most organisations have never filled: who actually owns the model. The commander is accountable and singular, and does not need to understand transformers.

The lecture most responders get wrong out of competence. Restarting a misbehaving agent is correct everywhere else and here it deletes the only copy of your evidence.

The model will not produce the same output twice, so one clean run proves nothing about a fix. You measure a rate, with its denominator and conditions, and that rate becomes your regression test.

Three deliberately ambiguous incidents. Score each on the four axes, declare or stand down, and defend the call to a peer who disagrees.

Five containment levers and what each one costs. Lab 4 is a timed drill scored on both speed and collateral damage.

An agent step is usually several tool calls, and a hard stop lands between them. Half-changed is worse than either state — a denial with no audit entry still gets a letter sent by tomorrow's reconciliation.

Take away the doing, not the thinking. Capability lives in four places at once, and revoking three of them revokes nothing.

Rollback is either ninety seconds or completely unavailable, and which one you get was decided months ago by whoever chose where the prompt lives.

Containment is itself an unreviewed production change made at speed by one tired person. 'Stop it' versus 'keep it running' is almost always a false choice.

A timed containment drill, scored on two things that pull against each other: seconds to stop the harm, and collateral damage.

Playbooks for injection, poisoned memory, leakage, a compromised tool server, and supplier compromise. Lab 5 runs the signature incident end to end.

The lecture that separates AI incident response from ordinary incident response. You contained it, cleaned the index and restarted — and it came back, because the payload was never in the index.

Leakage usually involves no breach at all. Two individually correct permissions and a helpful model are entirely sufficient, and notification is legal's call, not yours.

A supply-chain incident wearing an AI costume. Tool descriptions are instructions, so a compromised server changes what your model believes a tool does — without touching your code.

When the failure is upstream there is no host to isolate and no patch to apply. Your own telemetry will tell you before your vendor does, if you run a small hourly canary.

The signature incident end to end — detect it in the trace, contain it, scope the exposure, purge every persistent layer, and write the notification assessment.

Playbooks for tool abuse, agent loops, runaway cost, provider degradation, hallucination and insecure generated code. Lab 6 is the weekend a retry loop burned the budget.

The most boring failure in the course and the one most likely to happen to you. No attacker: a planner, a critic and no stopping condition are enough on their own.

Cost is the incident nobody wants to call an incident, which is exactly why it keeps happening. Watch per-unit metrics, because the daily total stays unremarkable while spend per run triples.

A hard outage stops. Partial degradation keeps deciding, on truncated context, with full autonomy — which is strictly worse.

The hardest lecture here: nothing was breached, no control failed, and a confident fabrication still reached a customer and shaped a real decision. Includes the Canadian tribunal ruling on chatbot liability.

This incident happened four months ago, was reviewed, approved and shipped. Fluent code reads as reviewed code, which is how it gets past competent people.

A retry loop burned a weekend of budget on one malformed claim. No attacker, and every individual step the system took was correct.

Reconstructing an incident you cannot re-run. Lab 7 hands you a sealed evidence bundle — with declared gaps and a clock skew — and asks for a defensible timeline.

The question DFIR practitioners ask in their first week and rarely get answered: what exactly do you seize when the weights belong to a vendor and the context window lives for milliseconds?

'The model did it' is a symptom written where a cause should be. Four categories — data, model, infrastructure, policy — and you keep asking why until you reach a decision a human made.

Attribution is a different question from root cause, and it decides who fixes it and who gets told. You eliminate by replay: previous model, previous prompt, no retrieval.

A sealed case file from an incident you did not witness, on a system you cannot re-run. Verify it, rebuild the timeline, and state the gaps.

Purging poisoned state, restoring safely, and gating recovery on evals. Lab 8 builds the incident into a permanent regression test behind a gate that refuses to open.

Restoration is where good responses go to die. Five stages — shadow, canary, gated, progressive, full — each with an exit criterion somebody signs.

The idea the whole course builds toward: the incident becomes a permanent test case. Not a wiki page — an executable test wired into the deployment gate.

Blameless review is a solved problem, but AI adds a new failure mode: blaming the model is seductive precisely because it never attends the meeting and never objects.

Build the incident into a permanent regression test, run shadow mode, and stand behind a gate that refuses to open until the fix is proven.

Standing up the programme, running a tabletop that is not theatre, the regulatory clock, and your first thirty days. Lab 9 is unscripted.

Most tabletops are theatre. Do not announce the scenario, use real tools on a real clock, and treat the findings list as the deliverable.

Awareness level only. Recognise that a clock has started and escalate — the interpretation and the filing belong to counsel, not to the engineer holding the terminal.

What to do on Monday. Five things in week one that need no budget, the five artefacts you already built, and a deliberately modest thirty-day plan, because modest plans get finished.

Unscripted. A page, a clock, and injects that arrive whether you are ready or not. The findings list is the real deliverable.