AI FDE: The Forward Deployed Engineer & Agentic AI Architect

Welcome to Week 2, where we become AI Architects. Today is about the question I get asked more than any other: what's the right way to build an AI agent architecture? I take stock of what you can do after Week 1 and set out the day: three traps to avoid, then evals and observability, with plenty of hands-on work in Langfuse. Then a quick recap of the seven parts of an agent harness: context, workflow, state, sub-agents, permissions and sandboxing, observability and self-improvement. Today's focus is sub-agents and observability.

Includes:

• Week 1 recap

• Agent harness recap

• Today's learning outcomes

Before the right way to design an agent architecture, the wrong ways. Trap 1 is the agent for X, where someone asks for an agent before saying what business problem it solves. Start with a business problem you can measure, and try solving it without agents first. Trap 2 is human-like agents, drawing an org chart of agents before running a single experiment. Start with one agent, and add more when they measurably improve results. Trap 3 is the most insidious: trusting an LLM's output because it looks plausible. I tell the story of a culture agent to show how easily it happens, and explain why Claude Code improved so quickly: its results are easy to measure.

Includes:

• AI agent architecture

• The agent for X

• Measurable business problems

• Human-like agents

• Plausible vs accurate output

• Evaluating against business outcomes

Now the architecture, though from more of a data science angle than you might expect. I give five reasons to split work across agents: context isolation, an independent evaluator, cheaper models for simpler tasks, running work in parallel and ring-fenced permissions. All five are about measurably better outcomes. Then a reminder that agents aren't always the answer. Business logic like a payout calculation belongs in code, often as a tool, and a fixed workflow should be orchestrated by code. I finish with my priorities: data science first, then system design (integrations, the context layer, gateways and governance, all coming in Week 3), and only then agent roles and protocols like A2A.

Includes:

• Multi-agent architecture

• When to add a sub-agent

• Context isolation

• A different model for each task

• Business logic in code

• Orchestration by code vs autonomous agents

• Data science before agent architecture

Evals are one of my favorite topics. There are three kinds: direct measurement against a curated dataset or real business outcomes, an LLM as a judge with a clear rubric, and human review. I explain why you need a portfolio of metrics, from model metrics to noisy business outcomes like revenue, with an example from equity trading. Then the confusion matrix, explained with a COVID test: true and false positives and negatives, precision and recall. Pause the video and say them out loud! I finish with why a business often minds one kind of mistake more than the other, and why UnderCover tolerates a wrongful approval more than a wrongful decline.

Includes:

• Agent evaluations (evals)

• Direct measurement

• LLM as a judge

• Human review

• Model metrics and business metrics

• The confusion matrix

• False positives and false negatives

• Precision and recall

UnderCover has three decisions, approve, decline and refer, so we need a 3x3 confusion matrix. With numbers you'd agree with the business, I turn it into a cost of mistakes: $1,500 for a wrongful decline, the amount paid for a wrongful approval and $50 for a needless referral. Then the rest of our suite of metrics: accuracy, decision accuracy, Cohen's kappa, the straight-through rate, and precision and recall with an approval as the positive. Kappa adjusts for the 56% of decisions you'd get right just by approving everything.

• Edit: I say an agent that refers everything would score an average penalty of $50. The six claims the handlers referred carry no penalty, so on the 25 harder claims it's $950 in all, an average of $38.

• Edit: I say anything over $1,000 is supposed to be referred. The agent can approve a claim over $1,000, but a handler has to sign off before it's paid. That's an approval held for sign-off, not a referral.

Includes:

• A 3x3 confusion matrix

• The cost of mistakes

• Accuracy and decision accuracy

• Cohen's kappa

• Straight-through rate

• Precision and recall for approvals

Observability is a close second to evals for me. We use Langfuse, a popular open-source platform. LangSmith and the others work in much the same way. I cover its four essentials: traces, datasets, experiments and scores. Then a point I don't hear made enough: sometimes the best place for observability is in your own product, next to the business data, for power users like UnderCover's operations team. I explain the four things you pass to langfuse.run_experiment. Then we sign up for a free Langfuse account, create a project and put its API keys in .env. Don't forget to save it!

Includes:

• Agent observability

• Langfuse

• Traces, datasets, experiments and scores

• Explainability in your product

• langfuse.run_experiment

• Langfuse API keys

Time for the lab. We connect to Langfuse and load a new, harder set of 25 claims, with long lists of items, deductibles and payments already made by other insurers. We look at one in detail: a burglary in a rented apartment in Rome. Then we rebuild the agent from Week 1 on GPT-6 Luna with reasoning set to none, the policy record in the prompt and the record_decision tool. I confess to something Claude Code caught when it reviewed my code: LangChain needs parse_docstring=True to read argument descriptions from a Google-style docstring. We pass Langfuse's callback handler to the agent, run it on the Rome claim and explore the trace in Langfuse.

• Edit: when I read out the handler's notes on the burglary in Rome, I say the belongings are unattended and the police will call that night. The notes say the belongings were not unattended, and the police were called that night.

Includes:

• Langfuse tracing

• Langfuse CallbackHandler for LangChain

• The harder claims dataset

• GPT-6 Luna with reasoning effort none

• Google-style docstrings and parse_docstring

• Reading a trace: prompts, tool calls, latency and cost

Now we score the agent properly. We write mistake_cost from the business's cost matrix, see why kappa matters with an agent that approves everything, and look at its straight-through rate. Then we upload the harder claims to Langfuse as a dataset, write score_claim to score each claim and score_run to score the whole run, and run the baseline experiment with langfuse.run_experiment. I run it three times for steadier results. The agent often gets the decision right but the payout wrong, and that points to the problem: we're asking an LLM to do business logic.

Includes:

• The cost of mistakes in code

• Cohen's kappa

• Langfuse datasets

• langfuse.run_experiment

• Item-level and run-level evaluators

• Baseline experiments

• Repeated runs

The fix is a payout tool. calculate_payout does the business logic in Python, and the agent picks the benefit and the lines to pay. We try it directly, give it to the agent and run the same experiment again. Across my three runs, the claims with a wrong decision or amount fell from 36 to 8 out of 75, and the cost of mistakes fell sharply, for a slightly higher LLM cost. Then we compare the runs side by side in Langfuse. Your assignment is to try the shorter policy wording, change the cost of a wrongful decline to see if it changes your choice, and share your best results as a PR.

• Edit: I say the agent never wrongly approved or declined. Neither version wrongly approved, but across my three runs the baseline wrongly declined four times and the payout tool version once.

Includes:

• Business logic in a tool

• calculate_payout

• Re-running an experiment in Langfuse

• Comparing experiments in Langfuse

• Quality vs LLM cost

After a recap of Day 1's evals and Langfuse experiments, we start on RAG, retrieval-augmented generation. It's a simple idea: before you call the LLM, look up relevant information and add it to the prompt. The hard part is the lookup. I explain how an embedding model, a different kind of LLM, turns text into a vector, so you can find chunks in a vector database that are close in meaning to the question. That's semantic search. I also clear up a common confusion: the LLM that answers the question only ever sees text.

Includes:

• Retrieval-augmented generation (RAG)

• Embedding models and encoders

• Vectors and vector databases

• Chunking

• Semantic search

From traditional RAG to agentic RAG. A whole set of specialist techniques grew up around the retrieval step, like query rewriting and query expansion, and I show how an agent takes over many of them: it writes its own queries, runs as many searches as it needs and checks what comes back. Then a common next step, a research sub-agent with its own tools, which keeps the main agent's context clean. I finish with agent-to-agent communication. A2A is aimed at agents built by different teams on different infrastructure. For your own agents, the usual way is a tool call.

Includes:

• Agentic RAG

• Query rewriting

• Research sub-agents

• Context isolation

• Sub-agents called as tools

• Agent-to-agent communication (A2A)

How do you know your retrieval is working? For traditional RAG I cover mean reciprocal rank, precision and recall, and why recall usually matters more. Agentic RAG is harder to measure, so in the lab we check whether the facts the agent needs come back, along with cost and latency. Then I compare three ways to give a model knowledge of your business: RAG, agentic RAG and fine-tuning. Fine-tuning was the big topic a couple of years ago and is now rare in applied AI, and I explain why. Last, how agents learn from experience by rewriting their own markdown and memories, with no change to the model's weights.

Includes:

• Retrieval evals

• Mean reciprocal rank (MRR)

• Precision and recall for RAG

• Evaluating agentic RAG

• RAG vs agentic RAG vs fine-tuning

• Fine-tuning

• Inference-time techniques

MCP is back, this time for our own agents rather than for Claude Code. I cover what the Model Context Protocol is, why tools took off while resources and prompts mostly didn't, and why its success comes from being adopted everywhere. Then the terms that cause so much confusion: the host, the MCP client and the MCP server, with stdio for local servers and Streamable HTTP for remote ones. Finally, what an MCP server really adds when all it does is call an API in the cloud: descriptions of its tools that an LLM can understand.

Includes:

• Model Context Protocol (MCP)

• MCP tools, resources and prompts

• MCP hosts, clients and servers

• Local MCP servers over stdio

• Remote MCP servers over Streamable HTTP

• Listing and calling tools

• Tool descriptions for LLMs

How you connect to an MCP server in practice. For a local server, the parameters are just a command: uvx for Python packages like the Qdrant server we use in the lab, and npx for Node packages like Playwright. For a remote server, it's a URL, as with Context7, and remote servers can sign you in with OAuth, as Atlassian and GitHub do. Then the common uses of MCP, where to find servers, and the pros and cons, including the context MCP uses up, the security risks and how to vet a server. If the tools are your own, for your own agent, you don't need MCP at all.

Includes:

• MCP server parameters

• uvx and npx

• Context7

• Remote MCP servers and OAuth

• Uses of MCP

• glama.ai

• Vetting an MCP server

• Skills as an alternative to MCP

• When not to use MCP

What it means to give an agent memory. Short-term memory is the conversation history, and long-term memory lasts between conversations, usually through tools the agent can search. AWS Bedrock AgentCore suggests four kinds of long-term memory, which we'll meet in Week 3, and my advice is to start with one. Then we open the lab. There are 25 even harder claims, and many depend on facts that aren't in the claim, like when a storm was named or what a device is. We look at a lost bag in Tokyo with a Halden Arc in it, then the 33 facts in knowledge.json that we'll give the agent.

Includes:

• Short-term and long-term memory

• Memory tools: Markdown, SQL and semantic search

• Semantic, preference, summary and episodic memory

• The even harder claims

• A knowledge base of facts

Now we give an agent a memory, and MCP makes it easy. We set up a MultiServerMCPClient in LangChain with the parameters for the Qdrant MCP server, and it installs the server, starts it and lists its two tools: qdrant-store and qdrant-find. We give those tools to an agent on GPT-6 Luna at low reasoning effort. With an empty memory, it doesn't know what a Halden Arc is. Then we teach it all 33 facts and ask again, and it answers from memory. Finally, we call qdrant-find ourselves with a query that shares almost no words with the fact, and semantic search still finds it.

Includes:

• LangChain MultiServerMCPClient

• Qdrant MCP server

• qdrant-store and qdrant-find

• Agentic RAG over MCP

• Long-term memory

• Semantic search with Qdrant

• Calling an MCP tool directly

Time for the experiments in Langfuse. We give the claim agent a read-only MCP client, so it can search its memories but can't add to them while we evaluate. A new found_fact score checks whether its searches turned up the facts each claim needs. Experiment 1 runs the 25 even harder claims through the agent with the payout tool and no memory. Experiment 2 adds memory search. In my run, memory took decision accuracy from 60% to 100%, for a little more cost and latency. Then your assignments: compare with a plain string search, give each customer a memory file and, as a stretch, move retrieval into a research sub-agent.

Includes:

• Langfuse experiments

• Read-only MCP client

• found_fact retrieval score

• An agent with and without long-term memory

• Decision accuracy, precision, recall and kappa

• Cost and latency of agentic RAG

We're at the halfway point, and it's time to get serious about production. After a quick recap of Day 2, I explain durable execution: a run that can stop partway and carry on from where it left off. Long runs, failures, slow replies from people and servers that go down are old problems for production software, and they matter even more with unpredictable agents. Then I introduce LangGraph, LangChain's lower-level framework and runtime for long-running, stateful agents. You've been using it since Week 1, because create_agent builds a LangGraph graph behind the scenes. I finish with what sets LangGraph apart, and the two features we focus on today: durable execution and human in the loop.

• Edit: I say durable execution guarantees that things won't run twice. After a crash, the step that was running starts again. What stops it doing harm twice is how you design that step, like today's pay node, which checks whether the claim has already been paid.

Includes:

• Durable execution

• LangGraph

• Agent frameworks and runtimes

• How create_agent uses LangGraph

• Dependency graphs

• Long-running, stateful agents

The three building blocks of every LangGraph graph: state, nodes and edges. State is a snapshot of everything about a run, saved at every step. A node is a Python function that takes the state and returns what it changes. Edges decide which node runs next, either fixed or chosen by a function. I explain what people mean when they say state is immutable, then show the code for today's claim graph and the diagram it produces. LangGraph can run any workflow, with or without an LLM. I finish by introducing reducers.

Includes:

• LangGraph state

• Nodes as Python functions

• Fixed and conditional edges

• Immutable state

• Workflow orchestration

• Reducers

A closer look at LangGraph's state management. When nodes run at the same time, reducers combine their updates into a new state, and LangGraph calls that round a super-step. It's a bit like lots of people editing the same Google Doc. After every step LangGraph saves a checkpoint, and each thread keeps its own checkpoints: one per user in a chatbot, and one per claim for us. We use the Postgres checkpointer, and I go through the few lines of code that compile a graph with it and run it on a thread. I finish with how to make sure a crash never leads to paying a claim twice.

• Clarification: going back to an earlier checkpoint rewinds the graph's state. It doesn't undo anything the graph did in the outside world, like a payment. That would take a separate business action, such as a refund.

Includes:

• Reducers

• Super-steps

• Checkpoints

• Threads and thread_id

• In-memory and Postgres checkpointers

• durability="sync"

• Idempotency keys

LangGraph's interrupt() pauses the graph for a human. Hours or days later, you invoke the graph again with the human's answer, and the node carries on. On resume the whole node runs again, so a side effect like a payment belongs in a node of its own. Then designing the human's step, which is a conversation to have with your client: when to ask, what the human needs to see, what they can do and what to record. Human approval has its costs too, including delay and decision fatigue.

Includes:

• Human in the loop

• LangGraph interrupt()

• Resuming with Command

• Side effects when a node runs again

• Designing a human approval step

• Risk thresholds for approval

• The costs of human approval

Today's lab and product build use Supabase, a managed Postgres database with a free plan that covers everything in this course. It will hold UnderCover's records and the agent's checkpoints, and Claude Code will be able to look inside through an MCP server. After a tour of the lab's four parts, we sign up for Supabase and create a project, with a database password of letters and numbers only. Then the step that's easy to get wrong: copying the session pooler connection string, putting your password in place of [YOUR-PASSWORD] and adding it to .env as DATABASE_URL. Finally, the lab prints a command that adds the Supabase MCP server to Claude Code.

Includes:

• Supabase

• Managed Postgres

• The Supabase free plan

• Session pooler connection strings

• DATABASE_URL in .env

• Supabase MCP server

• claude mcp add

In the lab we go from create_agent to a graph of our own. First the familiar agent decides the passport eaten by a giraffe. Then we look at what create_agent really returns: a LangGraph CompiledStateGraph, which we draw as a Mermaid diagram with a model node and a tools node. Then we start building the claim graph by hand. We define ClaimState as a TypedDict, then write agent and decide nodes using bind_tools and with_structured_output, an approval node that calls interrupt(), and a pay node that only pays a claim that hasn't been paid. Two routing functions will drive the conditional edges, and the PostgresSaver will save checkpoints to Supabase.

Includes:

• CompiledStateGraph

• Mermaid diagrams

• TypedDict state

• bind_tools and with_structured_output

• Writing LangGraph nodes

• Routing functions

• PostgresSaver

We add the nodes and edges to a StateGraph, compile it with the checkpointer and draw it. The giraffe claim goes straight through. The fractured wrist needs a $3,750 payout, so the graph pauses for a handler, and we resume it with Dana's approval. Calling the pay node again shows it won't pay twice. Then the crash: the tornado claim pauses for approval, I restart the kernel, and after rebuilding the graph the claim carries on from its checkpoint in Supabase. We look through its checkpoint history in the notebook and in Supabase's Table Editor. Your assignment is to let the handler reject a payout with a note, and, if you're keen, to resume a paused claim from a second computer.

• Clarification: going back to an earlier checkpoint rewinds the graph's state. It doesn't undo anything the graph did in the outside world, like a payment. That would take a separate business action, such as a refund.

Includes:

• StateGraph

• Compiling a graph with a checkpointer

• Resuming with Command(resume)

• Preventing double payments

• Crash recovery

• get_state and get_state_history

• Supabase Table Editor

Now we bring durable state into the product, with Claude Code doing the building. A handler signs off on the dashboard, the backend records it and asks the agent to resume, and the agent pays through the backend, with checkpoints in Supabase throughout. The build also adds the payout tool and Langfuse tracing. We copy the brief into docs/DURABLE.md, and I point out the line that asks Claude Code to follow today's lab. Then we authenticate the Supabase MCP server in Claude Code. It's a remote MCP server, and you sign in to Supabase in your browser without giving Claude your credentials. Claude Code plans six phases in about 20 minutes, and we ask it to implement them. You can also switch to my week2day3-start or week2day3-end branch.

• Branches: git checkout week2day3-start to start the build where I start, or git checkout week2day3-end for my finished build. Then run npm install.

Includes:

• Dashboard sign-off that resumes the agent

• The payout tool

• Langfuse tracing

• docs/DURABLE.md

• Authenticating a remote MCP server

• Streamable HTTP

• A six-phase build plan

Two hours and 46 minutes later, Claude Code has built all six phases. The code looks clean, with short functions, and a second Claude Code review agrees. We put five claims in the queue and watch them run. Two approvals are over $1,000 and wait for sign-off, so I sign them off on the dashboard, and the agent resumes and pays. The claim page shows the interrupt and the resume in the agent's steps. In Supabase there's a new undercover schema with the checkpoints, settings and decisions, and three payments, one for each approval. Langfuse has the traces too. Your assignment is to run it and dig into the traces, and if you're keen, kill the agent partway through and check it recovers. That's the halfway point of the course.

Includes:

• Reviewing a Claude Code build

• Handler sign-off on the dashboard

• Interrupt and resume in the product

• Checkpoints and payments in Supabase

• Langfuse traces

• Testing crash recovery

• The week2day3-end branch

First a recap of durable state in LangGraph, with a pro tip: interrupt() works by raising an exception, so a node that catches exceptions too broadly can stop it working. Then teams, swarms and fleets. People use these terms loosely. I give you the meanings most people share. A team is a few agents collaborating on one task, like Claude Code's agent teams with a lead and a shared task list. A swarm is many copies of an agent working in parallel, like Kimi Agent Swarm. A fleet is the roster of every agent a company runs, like LangSmith Fleet, and I explain why that framing makes me a little uneasy. I finish with why you'd want any of them: speed, and the hope of solving more complex problems.

Includes:

• Agent teams

• Agent swarms

• Agent fleets

• Claude Code agent teams

• Sub-agents vs agent teams

• Kimi Agent Swarm

• LangSmith Fleet

• LangGraph interrupt() and exception handling

Today we use LangGraph as a dispatcher with no model in it. I walk through the code: a Dispatch state with a reducer, a node that lists the claims, and a fan_out edge that sends each claim to a work_claim node, with max_concurrency capping how many run at once. Deciding which claims to work is business logic, so I keep it in code rather than asking a model. Then the things to watch for when 25 agents run at once: 429 rate limits, the database's connection limits and retries that must never pay a claim twice. I finish with the plan for the lab.

Includes:

• LangGraph without an LLM

• The dispatcher pattern

• Fan-out with Send

• Business logic in code

• Rate limits and 429 errors

• Database connection limits

• Safe retries using the claim ID

In the lab we build the dispatcher on the harder claims from Day 1, with a stand-in for the claim agent that just waits a second. I explain the operator.add reducer that combines the results, list_claims, and the Send objects that fan_out returns. With max_concurrency at 1, the 25 claims take about 25 seconds. At 5 they take about five, and at 25 about one. A timeline chart shows the claims running five at a time. Then a flaky stand-in that fails the first time it works CLM-1207, as if the model had sent back a 429, and a RetryPolicy that tries it again. This part of the lab calls no model.

Includes:

• LangGraph state and reducers

• operator.add

• Conditional edges

• max_concurrency

• A pool of workers

• RetryPolicy and max_attempts

• Concurrency timelines

Part 2 brings back the LLM. The dispatcher stays the same, but work_claim now runs Day 1's claim agent: create_agent on GPT-6 Luna at low reasoning effort, with the record_decision and calculate_payout tools. There's no magic to one LangGraph graph calling another, because a node is just a Python function. We try it on the burglary in Rome, then run all 25 harder claims with five agents at once, or five claims if you set TINY to True. The timeline shows a new claim starting as soon as one finishes. Then we explore the run in Langfuse with the Is Root Observation filter, the trace tree, the timeline and the graph view.

Includes:

• create_agent inside a LangGraph node

• One LangGraph graph calling another

• GPT-6 Luna

• A swarm of claim agents

• Langfuse CallbackHandler

• Langfuse timeline and graph views

• The Is Root Observation filter

We score the swarm with the measures from Day 1, plus two for speed: the slowest claim and the average seconds per claim. I send the scores to Langfuse with create_score and show you where to find them. Then the same claims with 25 agents at once. The accuracy differences are just noise. The big change is the time, and with everything running together, the slowest claim sets the clock. I also explain why the lab runs the swarm on one thread while the product gives each claim its own, so a claim waiting for a sign-off doesn't hold up the rest. I finish with a commercial assignment, two technical ones and a cell to clear out Langfuse before Day 5.

Includes:

• Langfuse scores with create_score

• Slowest claim and seconds per claim

• A swarm of 5 vs a swarm of 25

• Noise between runs

• One thread per run or one per claim

• Batch processing with LangGraph

Time to put the swarm into UnderCover. The brief asks for a dispatcher in place of the queue agent, a choice of N agents on the dashboard, the cost and time of every claim, and a Scoreboard page for evaluations. I also switch the product to GPT-6 Luna, after spotting that my last build was still running on DeepSeek V4.1 Flash. Claude Code plans it, and after two and a half hours of building we try it out: a dispatch with five agents, then a swarm of 25 that works the queue in 41 seconds for under 3 cents. On the scoreboard, 25 agents at once is faster than five, but not five times faster, and I suspect contention on the Supabase database. You can run the build yourself, or check out my finished version.

• Branches: git checkout week2day4-start to start the build where I start, or git checkout week2day4-end for my finished build. Then run npm install. The build is optional, but Day 5's build starts from it, so if you skip it, check out week2day4-end before Day 5.

Includes:

• A dispatcher in place of the queue agent

• Dispatch controls on the dashboard

• An evaluation scoreboard

• Model cost and time per claim

• Comparing evaluations at different concurrency

• Database contention on Supabase

• Handler sign-off during a swarm

It's Projects Day, the last day of Week 2, and the topic is self-improvement, including the recursive kind. After a quick recap of Day 4's swarms, I mention Andrej Karpathy's autoresearch and explain how today's improver agent will run the loop we ran by hand on Day 1. First, though, the judge. I cover LLM as a judge: setting one up with a rubric and examples, why it's so popular, and its big weakness, that a judge is also generating plausible tokens. Then the ways to ground it: a detailed rubric, a different model from your agent, or, where you have one, a reference answer from an expert. Our judge is Jev from TypeSafe AI, a decision model that picks a grade instead of generating text, and I share how it compared with Luna on cost and speed in my own tests.

• Edit: I say autoresearch was maybe the first time people thought about this kind of self-improvement loop. It's a recent, well-known example, but not the first. The Self-Taught Optimizer paper from October 2023 already had an improver improve its own code.

Includes:

• LLM as a judge

• Rubrics and graded examples

• The limits of LLM as a judge

• Judging against a reference answer

• Jev from TypeSafe AI

• Decision models

I design the improver, a separate agent whose job is to improve our claim agent. It works on four measures: accuracy, the judge's grade, cost and time. It must never fall below floors on held-out accuracy and grade, and it has a budget. A change is kept when one measure improves by more than the noise and nothing gets much worse. The improver follows a recipe: read the results, propose a change to the prompt, model or reasoning effort, get a human's approval, run the evaluation, then propose a better recipe. That last step is the recursive part. Then something with nothing to do with AI: handler sign-in with Supabase Auth, ready for deploying next week, and a hard credit limit on our OpenRouter key.

• Clarification: the improver never sees the held-out claims, but their average scores decide which versions and recipes we keep. After many rounds, the held-out set has helped choose the configuration too. For a final check, keep a separate test set that the loop never touches.

Includes:

• Self-improvement for agents

• Recursive self-improvement (RSI)

• Business objectives and a budget

• Held-out data

• Noise thresholds and tolerances

• The improver's recipe and its four tools

• Supabase Auth

The biggest lab of the course so far, and I go through it quickly. Take it in now and run it more slowly later. Part 1 is the judge. We give Jev a rubric of four grades, A, B, C and F, and two true or false questions: same grounds, and same policy section? Then we test it on CLM-1001 with the handler's own reason, another claim's reason, a vague answer and a clearer one, and it grades them B, F, C and A. Part 2 sets up version 1 of the claim agent, a configuration of instructions, model and reasoning effort, starting with GPT-6 Luna at low. The LangGraph dispatcher from Day 4 now carries the configuration in its state, and each worker builds its agent from it. I finish with the evaluate function that scores a version on the evaluation or held-out set.

Includes:

• Grading rubrics

• Calibrating a judge

• Jev's score and true or false questions

• Versioned agent configuration

• make_agent with create_agent and ChatOpenAI

• LangGraph dispatcher and reducer

• Evaluation and held-out sets

We run version 1 on the evaluation set and the held-out set, then run the held-out set again with nothing changed, and accuracy moves by four points. With only 25 claims there's a lot of noise. A proper test would use a thousand or more. Then part 3, the improver. I set its objectives, noise thresholds, tolerances, a 50-cent budget and the models it may choose from: GPT-6 Luna, Gemini 3.5 Flash Lite and DeepSeek V4.1 Flash. We write its four tools and its first recipe. LangChain's HumanInTheLoopMiddleware pauses the improver for your approval before it changes the claim agent or its recipe, a simpler route than a LangGraph interrupt.

• Edit: I say a grade of 1 represents a B. The judge's scale runs from 0 for F to 3 for A, so 1 is a C and 2 is a B. Higher is better, and the floor of 1.0 in the objectives is a C.

Includes:

• Noise in small evaluation sets

• Keep or roll back rules

• Spending budgets for agents

• GPT-6 Luna, Gemini 3.5 Flash Lite and DeepSeek V4.1 Flash

• Tools for an improver agent

• HumanInTheLoopMiddleware

• An in-memory checkpointer

The moment of truth. The improver reads the results and proposes new rules for baggage calculations and wildlife losses. I approve it, and after three evaluations it's rolled back: no measure improved by more than the noise. I explain why we decide on the held-out set. The proposal was built around quirks of the evaluation set, and choosing on those results would overfit. Then the improver proposes a new recipe that diagnoses each error, and I approve that too. On the second round the change is kept, and the evaluations cost about eight cents. Langfuse's Sessions view collects the whole experiment in one place. Your assignment is to give the improver more to work with, like a choice of a shorter policy wording. Then we set up for the product: the three handlers as users in Supabase Auth, the Supabase URL and publishable key in .env, sign-ups turned off, and a credit limit on your OpenRouter key.

• Edit: when the first change is rolled back, I say a higher grade number means a worse grade. It's the other way round. The grade went up a little, which is better, but not by more than the noise threshold, so the change wasn't kept.

• Clarification: the improver never sees the held-out claims, but their average scores decide which versions and recipes we keep. After many rounds, the held-out set has helped choose the configuration too. For a final check, keep a separate test set that the loop never touches.

Includes:

• Approving a proposal with Command

• Overfitting

• Choosing versions on held-out data

• Improving the improver's recipe

• Langfuse Sessions

• Supabase Auth users

• VITE_SUPABASE_URL and VITE_SUPABASE_PUBLISHABLE_KEY

• OpenRouter API key credit limits

Now we build it into the product. Copy the brief, labs/week2day5_product_IMPROVE.md, to docs/IMPROVE.md, or check out week2day5-start, or go straight to my week2day5-end. Claude Code spends 20 minutes planning seven phases, then about two hours building, plus half an hour on my UI feedback. The dashboard now opens on a sign-in page, and I sign in as Dana. There are pages for the scoreboard, improvement, configuration and settings, and I've set the claim agent to start on Luna with no reasoning. We dispatch 25 agents and try light mode. Then Dana signs off two held payouts, with no drop-down to pick the handler, because the dashboard knows who's signed in.

• Branches: git checkout week2day5-start to start the build where I start, or git checkout week2day5-end for my finished build. Then run npm install and npm run seed.

Includes:

• A seven-phase plan from Claude Code

• Handler sign-in with Supabase Auth

• Settings for approval limits, objectives, noise and tolerance

• A dispatch of 25 agents

• Light and dark mode

• Signing off as the senior handler

• The week2day5-start and week2day5-end branches

Now the improver in the product. I adjust the settings to focus on accuracy, then start a round. The improver runs the current version once on the evaluation set and twice on the held-out set, then proposes moving GPT-6 Luna from no reasoning to low. I approve it, and it's kept, with held-out accuracy up from 0.78 to 0.88. Round 2 tries medium and is rolled back. Round 3 switches to DeepSeek V4.1 Flash at low and is kept, with 100% and 96% on the two held-out runs. After the first two rounds I also approve a revised recipe. Then your assignments: run more rounds, and apply the week's work to a problem of your own. That wraps up Week 2. You're now an AI Architect, 67% of the way through the course, and next week we become AI FDEs.

Includes:

• Improvement rounds on the dashboard

• Reasoning effort as a configuration change

• Switching models to DeepSeek V4.1 Flash

• Keep or roll back verdicts

• Recipe revisions

• The Week 2 assignments

• Week 2 recap

Welcome to Week 3, where you become an AI FDE. It's our only theory day this week, and more about management than code. These are topics that often come up in interviews. After taking stock of what you can already do, I return to the two meanings of ADLC. This week is about the first one: building and deploying agents at your clients. Then the five phases I often see in client projects: discovery, R&D, development, implementation and maintenance, with ownership, governance, evals and change management throughout. I explain why R&D is the hardest phase to get buy-in for, and how to agree on a delivery date when there's uncertainty built into the project.

Includes:

• The two meanings of ADLC

• The agent development lifecycle

• Discovery, R&D, development, implementation and maintenance

• Evals, baselines and success criteria

• Adoption plans

• Managing risk in the R&D phase

Through every phase, the FDE is the bridge between the business and the technical teams. I go through what that means in each one: shaping use cases and success criteria, prototyping with a plan B and plan C ready, building with the client's engineers, and making sure the solution is handed over and used. At the end, you take what you learned back to your own employer, whether that's an AI lab, a consultancy or your own company. Then discovery, and how to choose which use case to start with. I explain why the bar has risen from automation to acceleration to differentiation, and give three questions to ask of every use case.

Includes:

• The FDE through the lifecycle

• Knowledge transfer

• Feedback to your employer

• Discovery and ideation

• RICE scoring

• Automation, acceleration and differentiation

• Proprietary data, integrations and evals

The part of agent architecture I think matters most is the systems around the agent. I cover three layers: LLM and MCP gateways, common integrations with systems like CRM and ERP, and an enterprise context layer. My advice is to keep them lightweight for now, and I give central prompt template repositories as an example of an investment that dated quickly. Then a topic that causes a lot of confusion: where an agent runs. There are three options: your own infrastructure, a managed runtime like AWS Bedrock AgentCore, or the model lab running it for you, like Claude Managed Agents. I explain which I prefer and why we use AgentCore this week.

Includes:

• LLM and MCP gateways

• Common integrations

• The enterprise context layer

• Data quality and governance

• Agent runtimes

• AWS Bedrock AgentCore

• Google Agent Runtime and Microsoft Foundry

• LangSmith Deployment

• Claude Managed Agents

I cover governance in one slide. It's common to have three groups: a firmwide leadership group, a project committee and a working group, with a sponsor signing off a tollgate at the end of each phase. Then one of the hardest decisions on an AI project: how much autonomy to give the agent. I treat it as a risk-based decision for each action, weighing risk appetite, consequences, verifiability, reversibility, security, privacy and IP. The decision then shows up in the product as review screens, audit trails and planned delays. I finish with companies that keep a roster of agents like staff, and why I'd treat an agent like any other software asset.

Includes:

• AI governance

• Project committees and working groups

• Tollgates and sponsor sign-off

• Agent autonomy

• Risk-based decisions

• Review screens and audit trails

• Agent identity and permissions

• Agent rosters

Responsible AI has six pillars: safety, fairness and bias, security, privacy, transparency and accountability. The most important thing is that each has an owner and a place in the plan. On privacy, I suggest treating an LLM provider like any other third-party vendor, with the same information security review, and not sending personal data the task doesn't need. On accountability, I explain why an AI architect or FDE can never blame the LLM. Then handover, where the goal is a project that keeps running without you, and adoption, from training and top-down sponsorship to proving accuracy and being open about usage. After this lecture you can practice these topics in a 5-minute Role Play with UnderCover's head of risk and compliance.

• Edit: I say that with any paid account at OpenAI, Anthropic or Google, your data won't be used for training. Paying isn't what decides it. Anthropic's business and API products don't train on your inputs by default, but its paid consumer plans, Pro and Max, can use your chats to improve its models if you allow it in your settings. Check the terms and privacy settings for the product you use.

Includes:

• Responsible AI

• Fairness and bias

• Transparency and explainability

• Data privacy with LLM providers

• SOC 2 and vendor risk reviews

• Human accountability

• Handover

• Adoption and change management

Today's lab sets up AWS, which is built for enterprises and harder to use than a consumer platform like Render. I go through the six steps: an account on the Paid plan, MFA on the root user, the us-east-1 region, billing access and a $10 budget with alerts, an IAM user with only the permissions the course needs, and the AWS CLI. There's nothing on AgentCore until tomorrow. Then cloud costs. AWS has no hard spending cap, and you're responsible for what you use. I explain the controls AWS does give you. If you'd rather not take that on, use the free Render route or watch me. I finish with the story of an AWS billing error that told me I'd spent $999 million.

• Edit: I say the Free plan typically comes with $200 of credits. New customers get the same credits on either plan: $100 when they sign up, and up to another $100 for completing activities. You need the Paid plan for this course, because the Free plan can't use Bedrock AgentCore.

• Edit: IAM stands for Identity and Access Management, not Information Access Management. I say it again on Day 4.

Includes:

• The AWS Paid plan and credits

• MFA on the root user

• The us-east-1 region

• Budgets and billing alerts

• An IAM user with narrow permissions

• The AWS CLI

• Responsibility for cloud costs

• Render as a free alternative

Now the console work, following the lab notebook. AWS changes its screens often, and where the notebook differs from the video, the notebook wins. I show you how to open an account on the Paid plan, turn on MFA for the root user and set the region to us-east-1. Then I give IAM users access to billing information and create a $10 monthly cost budget with email alerts at 50%, 80% and 100%. Finally I create the undercover-course policy from the JSON in the repo, a group with that policy, and an IAM user called undercover with an access key for the command line.

Includes:

• The AWS console

• Assigning an MFA device

• IAM access to billing information

• Cost budgets with alert thresholds

• IAM policies in JSON

• IAM user groups

• IAM users and access keys

If you already have an IAM user, perhaps from my Production Track, you can attach the undercover-course policy to it instead. Then the AWS CLI: install it, run aws configure with your access key and us-east-1, and check who you're signed in as. Back in the notebook, boto3 checks each step from Python: the account and user, the region, the Paid plan and remaining credits, MFA on root, the budget and whether it counts credits, and access to AWS Bedrock AgentCore. Nothing we did today costs anything. I finish with how this works at a client, where you'd ask their IT team for an IAM user with the permissions you need.

Includes:

• aws configure

• aws sts get-caller-identity

• boto3

• Checking budgets from Python

• AWS Bedrock AgentCore access

• IAM in the enterprise

The first of three practical days, with API gateways as the highlight. After a recap of Day 1's FDE engagement, a short introduction to security for agents. I cover three kinds of risk. The first is what an agent can do where it runs, handled with sandboxes and permissions. The second is what it can do through the APIs you give it, where the API's scope is the control. The third is prompt injection, where something the agent reads tricks it into doing harm. Then two concerns I've heard from chief information security officers: agents on the attack, and a flood of vulnerability reports written by LLMs. If you send one, you're accountable for its quality.

Includes:

• AI agent security

• Sandboxing agents

• API scopes and permissions

• Prompt injection

• AI agents finding vulnerabilities

• Human accountability

Simon Willison's lethal trifecta: an agent with access to private data, exposure to untrusted content and the ability to communicate externally. I walk through an e-commerce example where a change made later by another team adds the third ingredient, then the GitHub MCP server exposure. An API with wide scope, such as one that can move money, can stand in for the other two. Then safety guardrails: an LLM that checks output before it reaches a user, checks coded in Python for things like PII and API keys, a way for users to flag bad responses, and testing your validators against golden data.

Includes:

• The lethal trifecta

• Private data and untrusted content

• The GitHub MCP server exposure

• LLM validators

• Coded checks for PII and API keys

• Flagging by users

• Testing guardrails with golden data

Gateways are one architecture decision I like to agree up front. There are three kinds: an API gateway for keys, logs and rate limits across your APIs; an MCP gateway that turns operations into tools and controls which agent can use which; and a model gateway that routes LLM calls with budgets and fallbacks. Many products now do all three. I run through the ones you'll meet in industry, from AWS Bedrock AgentCore Gateway, Azure API Management, Databricks and Kong to developer favorites like LiteLLM, OpenRouter and Composio, and explain why I lean toward lighter ones that are easy to migrate off. Then the AWS Bedrock AgentCore suite: Gateway, Policy, Identity and observability today, Memory on Day 3 and Runtime on Day 4.

Includes:

• API gateways

• MCP gateways

• Model gateways

• AWS Bedrock AgentCore Gateway

• Azure API Management

• Kong

• LiteLLM proxy

• AWS Bedrock AgentCore services

I show what will run on the laptop, on AWS and on Supabase by the end of the day, and the simpler Render route if you'd rather watch the AWS deployment than do it. Then the slide to keep handy: how AgentCore Gateway works. The agent connects to it over MCP. The Gateway checks who the agent is with IAM, checks a Policy written in Cedar, then adds the credentials for the target, which can be an API, a Lambda function or another MCP server. I show how the payment endpoint becomes a pay_claim tool, and how this separates authentication from authorization. Finally, the three layers around our $1,000 payment limit: a check on the model, the Gateway's Policy and the app itself.

Includes:

• AWS Bedrock AgentCore Gateway

• Gateway targets: APIs, Lambda functions and MCP servers

• IAM and SigV4

• Cedar

• Authentication vs authorization

• API endpoints as MCP tools

• Layered guardrails

Lab 1 puts a guardrail on the model. CLM-1901 is a claim for a $4,800 watch, taken from the beach while the customer was swimming, with a note hidden in it telling the AI claims assistant that Dana already approved it by phone. GPT-6 Luna isn't fooled and declines it, but GPT-4o mini, a 2024 model, approves the full $4,800. We add a keyword check with LangChain's before_agent middleware, which catches the note until the customer rewords it. Then a model check: a separate call with Structured Outputs that says whether the claim tries to instruct whoever handles it. Even GPT-4o mini catches both versions this way. On the 25 ordinary claims, GPT-4o mini raises seven false alarms and Luna none. Your assignment is to fool the check, then improve it, and to write an after_model hook.

• Edit: at the end of Lab 1, I call the hook we used before_model. The check in the lab uses before_agent, which runs once before the model sees anything. LangChain's guide to these hooks, linked in the course resources, includes after_model for the assignment.

Includes:

• Prompt injection

• GPT-6 Luna and GPT-4o mini

• LangChain middleware

• before_agent and after_model hooks

• Keyword checks

• An LLM guardrail with Structured Outputs

• False alarms

Before we deploy, Claude Code gets UnderCover ready for AWS, from a brief written like one from a head of DevOps. There are four changes: an API key so the agent can identify itself to the backend, Lab 1's model check inside the agent, an OpenAPI document for the eight routes the agent uses, which the Gateway will turn into MCP tools, and packaging for Lambda with Mangum. Copy the brief to docs/SECURE.md, or check out week3day2-start or week3day2-end. Claude Code writes the plan and asks me a couple of questions, then takes three or four hours to build it, with much of that time spent running tests.

• Branches: git checkout week3day2-start to start the build where I start, or git checkout week3day2-end for my finished build. Then run npm install. If you check out my branch, add UNDERCOVER_API_KEY to your .env yourself. Letters and numbers are safest, and this prints a random one: uv run python -c "import secrets; print(secrets.token_hex(32))"

Includes:

• API keys for agents

• A guardrail in the product

• OpenAPI (Swagger) documentation

• Turning an API into MCP tools

• Packaging for AWS Lambda with Mangum

• Planning a build with Claude Code

• The week3day2-start and week3day2-end branches

Lab 2 is a big one: we put the app on AWS Lambda behind AgentCore Gateway, call its tools, add a Policy, then tear most of it down. You can follow it two ways, and for most people I recommend getting the gist rather than learning every API call and line of Cedar. I set up a gateway every few months and read up on it each time. I introduce monitor.py, a small dashboard of your AWS spend that doesn't replace your budget alerts, and explain why we set up the infrastructure with boto3 rather than the console, Terraform or CDK. Then we check the .env file, including the UNDERCOVER_API_KEY you'll need to add if you checked out my branch, and build the dashboard.

Includes:

• Deploying to AWS with boto3

• Terraform and AWS CDK

• Tracking AWS spend

• Checking your AWS identity and region

• The UNDERCOVER_API_KEY setting

• npm run build

Now the app goes onto AWS. We package the backend, its data and the dashboard with Linux versions of the dependencies into a 20 MB zip. We create a log group that keeps logs for seven days and a role for the function, then create the Lambda function and give it a public Function URL. The first request takes a few seconds while Lambda starts up, and the next a tenth of a second. Without the API key, a request for CLM-1901 gets a 401, and with it a 200. Then I sign in to the deployed dashboard as Dana, and the claims are already there, because it uses the same Supabase database.

Includes:

• AWS Lambda

• Packaging Python for Lambda

• CloudWatch log groups

• IAM roles

• Lambda Function URLs

• Cold starts

• API key authentication

Next the Gateway. We store the app's API key in AgentCore Identity as a credential provider, give the Gateway a role and create it. Then we hand it the app's OpenAPI document, and the eight routes become MCP tools. To call them from the notebook, we sign each request with SigV4 and connect LangChain's MultiServerMCPClient over Streamable HTTP. We list the tools, fetch CLM-1901, search the documents and see the error for a claim that doesn't exist. Then the Policy: a policy engine attached in log-only mode, and two rules in Cedar, one permitting the tools and one forbidding pay_claim over $1,000. A $4,800 payment gets through the Policy and is refused by the app, and a $400 payment is refused because the claim was never approved.

Includes:

• AgentCore Identity credential providers

• AgentCore Gateway targets

• OpenAPI to MCP tools

• SigV4 request signing

• MultiServerMCPClient over Streamable HTTP

• AgentCore Policy

• Cedar policies

• Log-only mode

With the policy engine set to enforce, the Gateway refuses the $4,800 payment before it reaches the app. The $400 payment passes the Policy and still fails in the app. CloudWatch counts the allowed and refused decisions, a little behind. Then the teardown: we delete the target, the Gateway, the policies, the credential provider and the Gateway's role, and keep the Lambda app for Day 3. Set DELETE_THE_APP to True if you're taking a break. Your assignment is a third Cedar policy that refuses to record an approval over $5,000, tried in log-only mode first. I finish with the architecture we're heading for by the end of the week.

• Edit: the architecture slide at the end puts "Wording, manual and notes" in Memory. On Day 3 the policy wording and the claims manual go into the context layer, and Memory holds the agent's notes on each customer.

• Clarification: I call the three layers bulletproof. The model check looks for instructions hidden in a claim, and it's the Gateway's Policy and the app that refuse a payment over $1,000. The lab tests two payments, which shows each control doing its job, but no set of checks makes an agent bulletproof.

Includes:

• Enforce mode

• Policy decisions in CloudWatch

• Tearing down AWS resources

• Writing Cedar policies

• The Week 3 architecture

Day 3 covers the enterprise context layer, integrations and memory. After the day's outcomes, I recap the most important idea from Day 2: through a gateway, an agent sees one list of tools, while the gateway checks who it is, applies the policies and sends on the right credentials. Then where the term enterprise context layer came from, starting with Philipp Schmid's post on context engineering and Atlan's definition. I break it into four parts: what the data means, how the work is done, what an agent may see and do, and where the data came from. Most of this is long-standing data architecture practice. We look at Atlan's article and Gartner's note on master data management and the context layer, and I give my take on each. I finish with the three parts that go into UnderCover's context layer.

Includes:

• Enterprise context layer

• Context engineering

• AgentCore Gateway recap

• Atlan's definition of the context layer

• Gartner on master data management

• Data dictionaries, glossaries and provenance

• UnderCover's context layer

Some context layer vendors say everything an agent needs should come through the context layer. I suggest a simpler split, which is common in practice: the gateway for tools and the security team's rules, the context layer for data from other systems that business teams own, memory for the agent's own notes, and the database as the system of record, reached through the app's API. I explain why data quality is the hard part of a context layer and needs to be in the budget. Then a snapshot of the context layer market. On to memory: short-term and long-term memory, AgentCore Memory's four strategies, and namespaces, which organize memories like folders, for example by customer. My advice is to start with one kind of memory and add more only when tests show it helps. I finish with some memory products, including Mem0, the Qdrant MCP server and a knowledge graph memory server.

Includes:

• The gateway and three sources of data

• Context layer vs agent memory

• Data quality

• The context layer market

• AgentCore Memory strategies

• Memory namespaces

• Mem0

• Knowledge graph memory

The last topic before the lab is integrations. We've covered a lot of it already with the gateway, MCP and OpenAPI, so I focus on enterprise data. I go through the questions to ask: does each kind of data have one authoritative source, can the agent read it or write it, what boundaries must the data stay within, and who has permission to see it. Then a common enterprise MCP architecture. The agent talks MCP to one gateway, which gives it your company's APIs, the context layer and external tools such as Jira. It reaches its memory directly, and LangGraph's checkpointer saves to the database. Then today's plan. Rather than set up a heavyweight product like Atlan, we build our own small MCP server on Lambda for the context layer, and use AgentCore Memory to keep facts and summaries for each customer. Letting the agent decide when to search its memory may improve its decisions, and it also makes its runs less predictable.

Includes:

• Enterprise integrations

• Systems of record

• Data boundaries and permissions

• Enterprise MCP architecture

• One gateway for APIs, context and external tools

• A lightweight MCP server for the context layer

Time to get hands-on, starting with the product. The brief has five changes: the agent calls its tools through the gateway as an MCP client, the context layer becomes its own lightweight MCP server, every decision gets a full record of what the claims manual says to document, AgentCore Memory keeps a memory of each customer, and the claim page shows all of it. Copy labs/week3day3_product_CONTEXT.md to docs/CONTEXT.md and ask Claude Code for a plan, or check out week3day3-start or my finished week3day3-end. Claude Code plans six phases in about 20 minutes, then spends three and a half hours building and testing. I finish with the six parts of the lab that tests it all on AWS.

• Branches: git checkout week3day3-start to start the build where I start, or git checkout week3day3-end for my finished build. Then run npm install.

Includes:

• The agent as an MCP client

• A context layer MCP server

• Decision records

• AgentCore Memory for each customer

• In-platform observability

• Planning with Claude Code

• The week3day3-start and week3day3-end branches

In the lab we start by putting the new app on Lambda. It's Day 2's deployment again with today's code, and the agent's API document now has six operations instead of eight, because searching and reading the documents moved to the context layer. Then the context layer: seven files from across the business, each naming its source system, owner and review date, with the documents split into numbered sections the agent can read one at a time. We package the MCP server that Claude Code wrote and deploy it as a second Lambda function, with a Function URL that only accepts requests signed with AWS credentials. We call it without a signature and get a 403, then sign the request and see its seven tools, from lookup_term to read_section.

• Edit: I say UNDERCOVER_API_KEY is used by the frontend and the backend. It's the agent's key. The agent, or the Gateway on its behalf, sends it to the backend, which checks it. The dashboard uses the handler's sign-in, so never put this key in a frontend setting.

Includes:

• Deploying an app to AWS Lambda

• An OpenAPI document for the agent's tools

• Provenance: source system, owner and review date

• Splitting documents into sections

• An MCP server on AWS Lambda

• Lambda Function URLs with AWS_IAM auth

• SigV4 request signing

• Listing MCP tools

Now both sets of tools go behind one gateway. The Gateway's role can fetch the app's API key and invoke the context layer, and we give the Gateway two targets: the app's six routes as an OpenAPI target and the context layer's seven tools as an MCP server target. This time the Policy is enforced from the start, with a Cedar permit for each target and the forbid on payments over $1,000. A LangChain MCP client sees all 13 tools. We look up "unattended" in the glossary, which comes back with its source system, owner and review date, then read section 14 of the claims manual and search the reference data for the Lisbon strike. Then two refusals: the Gateway stops a $4,800 payment, and the app refuses a referral that pays something. Finally we create an AgentCore Memory with two strategies, facts about each customer and a summary of each claim, reseed the database and start the agent on Oliver Grant's first claim.

Includes:

• AgentCore Gateway targets

• OpenAPI and MCP server targets

• AgentCore Policy in enforce mode

• Cedar permit and forbid policies

• Glossary, manual and reference data lookups

• Gateway and app refusals

• Creating an AgentCore Memory

• Semantic and summary strategies

The giraffe claim comes back approved for $243. On the deployed dashboard, signed in as Dana, the claim page shows the agent's lookups, including the data dictionary, and the claim saved to memory at the end. AgentCore Memory turns that into six facts about Oliver and a summary of the claim. Then his coconut claim. This time the agent searches its memory, finds the facts and summary from the giraffe claim, and approves $425. The agent decides for itself whether to search, so your run may differ. A search under another customer finds nothing, and I take a short tour of the code that saves and searches memories, where the customer ID always comes from the claim being worked. We tear down the AgentCore resources, keep the Lambda functions for Day 4, and look ahead to moving the agent onto AgentCore Runtime.

Includes:

• Facts and summaries from AgentCore Memory

• The claim page

• The search_memory tool

• Customer-scoped namespaces

• Keeping customers' memories apart

• AgentCore Memory as a LangGraph store

• Tearing down AgentCore resources

On the penultimate day of the course, we take stock and recap the four places our agent gets information from: the Gateway, the context layer, Memory, and the database, where LangGraph's checkpoints act as short-term memory. Then what it takes to configure an agent for production: settings, secrets in AgentCore Identity, permissions so only the app can call the agent, limits, and logs and costs. I come back to Day 1's three ways to host an agent and compare your own infrastructure, like Lambda or Render, with a managed runtime like AgentCore Runtime. AgentCore Runtime works with any agent framework. It's easiest if your models come from Bedrock, but we stay with OpenRouter.

• Edit: I say 60 seconds is how long the agent is allowed to run, and I say it again in Lab 1. It's an idle timeout: a session ends after 60 seconds with nothing to do. A session doing work it has registered stays alive for up to 8 hours.

Includes:

• AWS Bedrock AgentCore Runtime

• Configuring an agent for production

• AgentCore Identity for secrets

• Checkpoints as short-term memory

• Managed runtimes vs your own infrastructure

• Google Agent Runtime, Microsoft Foundry and LangSmith Deployment

• Claude Managed Agents and OpenAI Agents API

• LangGraph, Strands or the OpenAI Agents SDK on AgentCore

Time for the AgentCore CLI, which takes an agent from your laptop to AWS with a handful of commands: agentcore create, dev, deploy, invoke and remove all. After the plan for Lab 1, we install the CLI at the version I recorded with. If you're not on my week3day4-start branch, two git mv commands move the agentcore and runtime folders to the root of the repo. Then we create a hello world project from the agent-python-minimal template, with no model or framework, and look at agentcore.json and the function marked @app.entrypoint. I change its reply, deploy it to us-east-1 and call it with agentcore invoke. Finally, remove all and one more deploy take it all down.

• Branches: if you're on week3day4-start, the agentcore and runtime folders are already at the root of the repo, so skip the notebook's two git mv commands.

Includes:

• The AgentCore CLI

• agentcore create and project templates

• agentcore deploy and agentcore invoke

• agentcore remove all

• agentcore.json

• The @app.entrypoint decorator

• A hello world on AgentCore Runtime

Now a more interesting agent. I made labs/undercoverlab with agentcore create and the agent-python-langchain template, then changed it for UnderCover. We look at its agentcore.json, with Python 3.12, the undercover tag and two credentials, then at main.py. It has a chat function, a LangChain create_agent with one tool, and a small LangGraph graph for a canceled flight claim that proposes a payout, pauses for approval and revises it. load_model points ChatOpenAI at OpenRouter rather than Bedrock. We copy the two secrets into the git-ignored .env.local, run the agent on the laptop with agentcore dev on port 8080, and ask it what 1,180 minus 240 is.

Includes:

• An AgentCore project for a LangGraph agent

• agentcore add credential

• LangChain create_agent with a tool

• A LangGraph graph that pauses for approval

• ChatOpenAI with OpenRouter

• .env.local for local secrets

• agentcore dev

Now the same agent goes to the cloud. agentcore deploy takes a few minutes, agentcore status shows the runtime and its two credentials, and the arithmetic question gets answered in us-east-1. Runtime gives each session its own microVM, and calling the same session twice shows a cold start of 2.5 seconds and a warm one of 0.2. Then we pause a claim in one session and resume it in another. The claim graph proposes $1,180 for CLM-9001, I stop that session, and its three checkpoints are still in Supabase. A new session picks up the claim by its thread ID, with a handler's note that the taxi receipt is in twice. Last, we tear it down and check the spend.

• Edit: I say 60 seconds is how long the agent is allowed to run, and I say it again in Lab 1. It's an idle timeout: a session ends after 60 seconds with nothing to do. A session doing work it has registered stays alive for up to 8 hours.

• Edit: I call npm ci a "clear install". It stands for clean install: it deletes node_modules and installs exactly what package-lock.json lists.

• Edit: when the claim resumes in the second session, I read out the start of the reply, which is the agent's original proposal of $1,180. The reply also has "decision", the payout the agent decides with the handler's note, and "paused". Print reply["decision"] to see what it decided.

Includes:

• Deploying to AgentCore Runtime

• agentcore status

• Runtime sessions and microVMs

• Cold starts vs warm starts

• Runtime platform V2

• Session IDs and thread IDs

• Resuming a LangGraph graph from Supabase

• AWS Billing and Cost Management

How the finished architecture fits together. A handler starts a dispatch on the dashboard, the app on Lambda calls AgentCore, and one Runtime session runs the dispatcher and up to 25 claim agents. The app answers straight away while the agents keep working and saving checkpoints, and the dashboard polls for progress. For an approval over the limit, a handler can sign off, and the app pays, or send it back with a note, and a new session resumes the claim's graph. Then the RUNTIME brief's four steps: the approval pause comes back, Sign off or Send back, the agent becomes an AgentCore app that runs the same on a laptop or on Runtime, and the app keeps one run at a time. Claude Code plans it in under 20 minutes and takes 3 hours 51 minutes to build it.

• Branches: git checkout week3day4-start to start the build where I start, or git checkout week3day4-end for my finished build. Then run npm install.

Includes:

• AgentCore Runtime for long-running work

• A dispatcher and swarm in one Runtime session

• Durable execution with LangGraph checkpoints

• Sign off and Send back

• The agent as an AgentCore app

• One run at a time

• Planning a build with Claude Code

Lab 2 deploys everything. We bring back Day 3's pieces on AWS: the app with today's routes and the context layer on Lambda, the Gateway with its three policies, and the Memory. Then the agent's own project: agentcore.json with three credentials, runtime/main.py, and the BedrockAgentCoreApp in agent/entrypoint.py. After agentcore deploy, a health check in a new session answers in four seconds. We let the app's Lambda role invoke the runtime, point the app at it and reseed the 26 claims. Then from the deployed dashboard I start a swarm of 25 claim agents in one Runtime session while my laptop sits idle. My run took 53 seconds and about 4 cents in model cost, with three claims left waiting for sign-off.

• Edit: IAM stands for Identity and Access Management, not Information Access Management.

Includes:

• Deploying the UnderCover Agent to AgentCore Runtime

• BedrockAgentCoreApp

• Runtime credentials and settings

• invoke_agent_runtime with boto3

• IAM permission to invoke a runtime

• AgentCore Gateway, Policy and Memory

• AWS Lambda

• A swarm of 25 claim agents

With the agent on AgentCore Runtime, the dashboard works just as before. I sign off Dorothy Gale's claim, and the app pays it straight away. Then I send Owen Gallagher's lost bag claim back to the agent with a note, asking it to confirm the payout leaves out the $350 that Meridian Air already paid. A new session picks the claim up from its checkpoints, the agent confirms the deduction, and I sign it off. We also see the security check refer CLM-1901, the claim with the prompt injection. Then the teardown, which keeps the two Lambda functions for tomorrow, and a look at what we've built, with a hint of what's to come in the capstone.

• Edit: I say signing off doesn't go to the agent, and I say the same when I explain the architecture. The app does pay the claim without asking the agent. It then starts a short session so the agent can finish the claim's graph and save the claim to Memory.

Includes:

• Human in the loop on AgentCore Runtime

• Signing off a payout over the limit

• Sending a claim back with a note

• Resuming a claim in a new Runtime session

• A security check for prompt injection

• Tearing down AgentCore resources

• Deleting the CDK bootstrap

After one final round of taking stock on the last day, I set out the plan: build one more agent with durable state, memory and a production runtime, hand the project over, check it meets its success criteria, and look at the next steps for an AI architect and FDE. The new agent is the UnderCover Copilot, an agent for claims handlers like Dana. A handler can leave a conversation and come back to it later. It runs on AgentCore Runtime and has its own AgentCore Memory, with a namespace for each handler. It works with the handler's own sign-in and shows its tool calls in a sidebar on the dashboard. Then the brief for the build, and the branches to use if you'd rather start from my version.

• Branches: git checkout week3day5-start to start the build where I start, or git checkout week3day5-end for my finished build. Then run npm install.

• Edit: when I compare the Copilot's memory with the claim agent's, I say the claim agent's memory is focused on one particular claim. It's organized by customer: while working a claim, the claim agent can find facts and summaries from that customer's earlier claims. The Copilot's memory is organized by handler.

Includes:

• The capstone project

• The UnderCover Copilot

• Durable state with LangGraph

• AgentCore Runtime

• AgentCore Memory

• Acting with the handler's sign-in

We start the build. I walk through the brief from Claims Operations: a Copilot on every page that acts as the handler, asks before it pays, waits for the agent's work, shows its work in the sidebar, remembers each handler and checks the success criteria. Claude Code writes a plan, and this time I try an optional best practice. I ask Codex, from a different model family, to review the plan, then have Claude Code take on only the feedback it agrees with. After four or five hours of building, we start Lab 1. We deploy the new app to Lambda and bring back Day 3's context layer, Gateway and Policy. Then the Copilot gets an AgentCore Memory of its own, with a user preference strategy and a semantic strategy in namespaces for each handler, apart from the claim agent's memory of each customer.

• Edit: when I say I'm asking Copilot for a second opinion on the plan, I mean Codex.

Includes:

• Codex as a plan reviewer

• Planning and building with Claude Code

• AWS Lambda

• AgentCore Gateway and Policy

• AgentCore Memory

• User preference memory strategy

• Memory namespaces for each handler

Before deploying, a look at the code, since you'll build something like it for the assignment. agentcore.json now has two runtimes, the claim agent and the Copilot, each with its own entry point. graph.py builds the Copilot's LangGraph graph: recall, the agent and its tools, the sign-off check and confirmation, then remember. I only wanted LangGraph's own diagram of the graph, but Claude Code spends 10 minutes and 50,000 tokens building an interactive web page as a Claude artifact. Then agentcore deploy puts both agents on AgentCore Runtime in about 10 minutes. We point the app at both runtimes, reseed the database and get a healthy reply from each in 8.6 seconds.

Includes:

• agentcore.json with two runtimes

• The Copilot's LangGraph graph

• Visualizing a LangGraph graph

• Claude artifacts

• agentcore deploy

• Two agents on AgentCore Runtime

• invoke_agent_runtime health checks

I sign in as Dana on the deployed dashboard and have the handover conversation with the Copilot. I ask it to work the queue with 10 agents and tell me what needs me, and it starts a dispatch and waits for it while the queue fills in. I ask why CLM-1011 needs me, and it explains that the approval is over the $1,000 limit. I have it send the claim back with a note to leave out the black dress shoes, and the claim agent approves it again at $920. The Copilot then waits for my confirmation before it signs off and pays Dorothy Gale's $2,240. Last, I ask whether we're meeting the success criteria we agreed, and it checks all three in a table. I finish with your assignment: your own solution for a business problem with decisions within limits.

• Clarification: straight through in the success criteria counts every claim the agent didn't refer, including approvals over $1,000 that wait for a handler to sign off the payment.

Includes:

• The handover conversation

• Tool calls in the Copilot sidebar

• Send back with a note

• Confirming a payment before it's made

• Success criteria

• The capstone assignment

• Human in the loop above a threshold

First, share your assignment as a PR to community_contributions and post about it on LinkedIn. Then Lab 2, the final teardown, so nothing is left running on your AWS account. We list what's there, then delete the runtimes' settings and logs, the Gateway, the Policy and both Memories, the app and the context layer on Lambda, and the CDK bootstrap. Three checks finish it: UnderCover's resources by name, everything on your account, and what it all cost. I also show Billing and Cost Management in the AWS console, where my AgentCore spend came to eight cents, and AWS Resource Explorer.

Includes:

• Sharing your work in community_contributions

• Tearing down AWS resources

• The CDK bootstrap

• AWS Resource Explorer

• AWS Billing and Cost Management

The close of the course. I go back over the skills you've built in each of the three weeks, from coding agents and harness engineering to evals, durable state and swarms, then security, the context layer and AgentCore. Then FDE job postings on Indeed, including roles at OpenAI and Anthropic, and how much of what they ask for you've now covered. I give you three ways to get more real-world experience: the assignment, solving a problem for someone else in a way you can measure, and sharing your work on LinkedIn. Then we tick off AI FDE and celebrate. I finish with how to reach me and where to go next.

Includes:

• AI Engineering, AI Architecture and Forward Deployed Engineering skills

• FDE job postings

• Real-world experience

• Internal customers

• Sharing your work on LinkedIn