A Systems Build Story
╔══════════════════════════════════════════════════════════════════════════════╗
║ ║
║ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────────┐ ║
║ │ Image │ ───▶ │ Retrieve │ ───▶ │ Decide │ ───▶ │ Tutor/Answer │ ║
║ │ + │ │ (text │ │ (tools, │ │ (probes, │ ║
║ │Question │ │ +image) │ │ actions)│ │hints,answer) │ ║
║ └─────────┘ └──────────┘ └────┬────┘ └──────────────┘ ║
║ │ ║
║ ▼ ║
║ ┌────────────┐ ║
║ │ Tool Exec │ ║
║ │(zoom,seg, │ ║
║ │ retrieve…) │ ║
║ └────────────┘ ║
║ ║
║ A M U L T I M O D A L M E D I C A L T U T O R. ║
║ ║
╚══════════════════════════════════════════════════════════════════════════════╝TL;DR
I built a multimodal medical tutoring system that reads medical images, retrieves supporting knowledge from text and image databases, uses visual tools to gather evidence, teaches Socratically with calibrated probes and hints, and produces a final answer only when the student is ready.
The system is stochastic by design. At each decision point, the tutor samples actions under uncertainty—which tool to use, which question to ask, whether to reveal the answer. The stochasticity lives in the controller's sampling; the surrounding runtime is deterministic. Those decisions are improved through three complementary mechanisms: SFT for reliable action formatting, RL for tool-use behavior, and ACE-style playbook optimization for fast iteration on recurring mistakes.
This is a long-form technical build story. I wrote it for engineers and recruiters who want to understand not just what the system does, but how it was designed and why.
The problem I actually solved
"Medical VQA" and "medical tutoring" look similar on the surface. Both start with an image and a question. But tutoring adds requirements that most VQA systems never address.
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ MEDICAL VQA MEDICAL TUTORING │
│ ─────────── ──────────────── │
│ │
│ Image + Question Image + Question │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌──────────────┐ │
│ │ Model │ │ Ask student │◀──┐ │
│ └────┬─────┘ └──────┬───────┘ │ │
│ │ │ │ │
│ ▼ ▼ │ │
│ ┌──────────┐ ┌──────────────┐ │ │
│ │ Answer │ │ Assess │ │ (loop until │
│ └──────────┘ │ response │ │ ready) │
│ └──────┬───────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌──────────────┐ │ │
│ │ Hint / Probe │───┘ │
│ │ / Reveal │ │
│ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘A tutor has to ask what the student thinks before explaining. It has to escalate hints based on performance rather than revealing immediately. It has to use tools when the image is ambiguous instead of guessing. It has to use retrieval when medical knowledge is needed, and it has to separate observation from diagnosis. Most importantly, every decision the system makes should be inspectable—which tool was called, what it returned, and how it changed the next step.
I built a system that treats tutoring as a control problem, not a single model call. The goal wasn't just accuracy. I wanted a system that is consistent in how it reasons, inspectable when it fails, and trainable when I want to change behavior. This is why the final system looks more like a small operating system than a prompt-and-model demo.
The system at a glance
The system has two interacting loops. At runtime, it behaves like a state machine that iterates until it produces an instructional response—a probe, a hint, a micro-lesson—or a final answer. Offline, multiple improvement channels operate on the same underlying runtime traces: SFT trains the controller to emit structured tutoring actions reliably, RL trains tool-use behavior with intermediate rewards, and ACE evolves a playbook of behavioral rules to reduce failure modes without retraining.
╔═══════════════════════════════════════════════════════════════════════════════╗
║ ONLINE RUNTIME ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║ ║
║ ┌───────────────┐ ┌────────────┐ ┌───────────┐ ┌────────────────┐ ║
║ │ Pre-Retrieve │───▶│ Decide │───▶│ Tool Exec │───▶│ Answer/Respond │ ║
║ │ (gather ctx) │ │ (policy) │ │ (1 tool) │ │ (terminal) │ ║
║ └───────────────┘ └─────┬──────┘ └─────┬─────┘ └────────────────┘ ║
║ │ │ ║
║ │◀────────────────┘ ║
║ │ (loop back) ║
║ ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║ ▲ ▲ ║
║ │ traces + state snapshots │ playbook rules ║
║ │ │ ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║ OFFLINE IMPROVEMENT ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║ ║
║ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────────┐ ║
║ │ SFT │ │ RL │ │ ACE │ ║
║ │ (action format, │ │ (tool-use │ │ (playbook rules, │ ║
║ │ reliability) │ │ rewards) │ │ fast iteration) │ ║
║ └─────────────────┘ └─────────────────┘ └─────────────────────────────┘ ║
║ ║
╚═══════════════════════════════════════════════════════════════════════════════╝This separation matters because it keeps the runtime stable, keeps training focused, and makes improvements measurable. The runtime never changes during training—only the policies that run inside it.
When I say "stochastic," I mean something specific. At decision points, the controller can sample different actions under uncertainty: tool versus probe versus hint versus answer. But the runtime around it is deterministic and constrained—schemas, gates, routing, budgets. This combination gives you both creativity when the model needs it and predictability when the system needs it.
Every design decision in this system is downstream of a few constraints I couldn't ignore. Raw retrieval and raw tool outputs blow up your prompt, so I needed a context budget. A tutor can't take two minutes per turn and still feel interactive, so I needed a latency budget. Tool-using systems fail in dozens of new ways—parsing errors, missing dependencies, wrong modalities, empty retrieval, repeated tools—so I needed explicit failure handling. If I can't replay a run, I can't improve it reliably, so I needed reproducibility. And I needed hard constraints that prevent "answer dumping" even when the policy is stochastic.
The backbone: one state object, one traceable story
The most important decision I made was that everything flows through one explicit state object.
┌─────────────────────────────────────────────────────────────────────────────┐
│ AgentState │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ messages: List[Message] ◀── conversation history (multimodal) │
│ image: ImageRef | None ◀── current image reference │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ RETRIEVAL │ │
│ │ retrieval_hits: List[RetrievalItem] ◀── full structured results │ │
│ │ retrieval_summary: str ◀── compact for prompts │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ TOOLS │ │
│ │ tool_calls: List[ToolCall] ◀── what was requested │ │
│ │ tool_results: List[ToolResult] ◀── what came back │ │
│ │ artifact_refs: List[ArtifactRef] ◀── persisted outputs │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ TUTORING │ │
│ │ student_profile: StudentProfile ◀── novice/medium/expert │ │
│ │ session_state: SessionState ◀── attempts, performance │ │
│ │ hint_level: int ◀── escalation tracking │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ errors: List[str] ◀── captured, not hidden │
│ route: str ◀── deterministic control flow │
│ │
└─────────────────────────────────────────────────────────────────────────────┘If you want a tool-using system to be reliable, you need to be able to answer questions like: What did the model see at this step? Why did it call this tool? What did the tool return? What changed after that? So the runtime carries a single state object with conversation messages, the image reference, retrieval hits and a compact retrieval summary, tool calls and tool results, artifact references for persisted outputs, errors captured rather than hidden, tutoring fields for the student model and session progress, and a routing field that makes control flow deterministic.
This design has two practical benefits. First, I can debug failures as data rather than intuition. Second, I can run offline loops—SFT, RL, ACE—that learn from the same state and trace structure.
In early iterations, I tried the "just append tool results to the prompt" approach. It breaks in predictable ways. Retrieval results are verbose, so the prompt bloats. Tool outputs are heterogeneous, so formatting becomes inconsistent. The model begins to lose the thread of the task—a phenomenon often called context rot. Making state explicit and structured was the turning point that made the system scalable.
The runtime loop
The runtime is intentionally simple: pre-retrieve to gather context, decide to choose the next action, execute exactly one tool if requested and persist outputs, then either teach (probe, hint, micro-lesson) or produce a final answer.
state = init_state(user_input, image)
state.retrieval = pre_retrieve(state)
while True:
action = decide(state) # stochastic: model-driven policy
if action.type == "TOOL":
result = tool_exec(action, state) # deterministic tool handler
state = state.with_tool_result(action, result)
continue
if action.type == "TUTOR":
state = state.with_tutor_message(apply_gates(action, state))
break
if action.type == "ANSWER":
state = state.with_answer(run_solver(state))
break
return stateWhere the "agent" feeling comes from is the decide step—model-driven policy—but the system remains stable because routing is deterministic. Two details matter in practice: one tool per turn, which prevents tool spam, keeps traces readable, and makes the policy learnable; and explicit termination, meaning every run ends with a coherent "what happens next."
Without hard boundaries, a stochastic controller can get into loops. Zoom, zoom, zoom with slightly different bounding boxes each time. Retrieve, retrieve with the same query returning the same hits. Web search after web search because it "feels" like progress. The runtime constraints are not just for neatness—they prevent the most common agent failure: doing busy work until the budget runs out.
Why I split "controller" from "solver"
I treat "what to do next" as a fundamentally different problem from "produce the best medical answer."
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ CONTROLLER(S) │ │
│ │ │ │
│ │ Input: state (image, history, retrieval, student signals) │ │
│ │ │ │
│ │ Output: next action │ │
│ │ • TOOL: { name, args } │ │
│ │ • TUTOR: { type: probe | hint | microlesson | ... } │ │
│ │ • ANSWER: trigger solver │ │
│ │ │ │
│ │ Trained via: SFT (format), RL (behavior), ACE (rules) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ "ready to answer" │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ SOLVER │ │
│ │ │ │
│ │ Input: state (image, retrieval, tool artifacts, question) │ │
│ │ │ │
│ │ Output: final medical answer │ │
│ │ • answer text │ │
│ │ • confidence │ │
│ │ • OR: "insufficient evidence" (triggers retry path) │ │
│ │ │ │
│ │ Model: MedGemma (vision-language) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘The controller decides what to do next—tool call versus tutor action versus answer. The solver produces the final medical answer once enough evidence exists. This split buys me faster iteration on policy behavior, cleaner debugging since I can distinguish policy errors from solver errors, and the ability to improve tool-use without destabilizing answer generation.
Tutoring systems have an uncomfortable truth: sometimes the right output is "I don't have enough evidence." I explicitly preserve that as a valid outcome and build a retry/repair path around it. When a solver is forced to always answer, it will hallucinate. When it can refuse, the rest of the system can respond by collecting more evidence.
Tools: evidence gathering with artifacts
In tool-using systems, the failure mode is almost always the same. Tools become just strings in a prompt. Results get inlined as huge blobs. Context collapses. The "agent" starts hallucinating tool outputs.
┌─────────────────────────────────────────────────────────────────────────────┐
│ TOOL REGISTRY │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ zoom │ │ enhance │ │ segment │ │ image_findings │ │
│ │ ─────────── │ │ ─────────── │ │ ─────────── │ │ ─────────────────── │ │
│ │ bbox crop │ │ upscale │ │ MedSAM2 │ │ describe features │ │
│ │ + padding │ │ + sharpen │ │ mask gen │ │ in region │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────────┘ │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌───────────────────────────────────────┐│
│ │ retrieve │ │ web_browser │ │ ocr ││
│ │ ─────────── │ │ ─────────── │ │ ───────────────────────────────────── ││
│ │ query local │ │ fallback │ │ extract text from image regions ││
│ │ database │ │ web search │ │ ││
│ └─────────────┘ └─────────────┘ └───────────────────────────────────────┘│
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ TOOL EXECUTION FLOW │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ToolCall ToolResult │
│ ┌──────────────┐ ┌──────────────────────────────────────┐ │
│ │ name: "zoom" │ │ ok: true │ │
│ │ args: { │ ──────▶ │ summary: "Zoomed into upper-left…" │ │
│ │ bbox_2d, │ (execute) │ artifact_refs: [ │ │
│ │ padding │ │ "artifact://zoom_abc123.png" │ │
│ │ } │ │ ] │ │
│ └──────────────┘ └──────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ ARTIFACT STORE │ │
│ │ (images, masks, │ │
│ │ text files) │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘I built tools as first-class operations. Tools have schemas. They return structured outputs and a human-readable summary. Tool outputs are persisted as artifacts—images, masks, text—referenced by ID or path instead of being dumped into the prompt.
The design principle is simple: summaries are for prompts, artifacts are for truth. The summary goes into the controller's context; the artifact gets persisted for humans and offline evaluators. This avoids a classic failure mode where a system becomes "prompt-native" and loses its grounding in actual data.
Tools fail for boring reasons: no image present, missing optional dependency, malformed bounding box, empty crop after clamping, retrieval index unavailable, network keys missing for web search. These boring failures destroy a demo if you don't capture them cleanly. I made tool failure a first-class output so the system can log it, surface it, and recover without crashing the run.
Retrieval: multimodal fusion under a context budget
Retrieval in this system has two jobs: return evidence in the form of text snippets and image provenance, and produce a compact, controller-friendly representation.
┌─────────────────────────────────────────────────────────────────────────────┐
│ MULTIMODAL RETRIEVAL PIPELINE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────┐ │
│ │ Query │ │
│ │ (text+image) │ │
│ └───────┬───────┘ │
│ │ │
│ ┌──────────────┼──────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ BM25 │ │ Text │ │ Image │ │
│ │ (sparse) │ │ Vectors │ │ Vectors │ │
│ │ │ │ (dense) │ │ (dense) │ │
│ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
│ │ │ │ │
│ │ ranked │ ranked │ ranked │
│ │ results │ results │ results │
│ │ │ │ │
│ └──────────────┼──────────────┘ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ RRF Fusion │ │
│ │ (rank-based │ │
│ │ aggregation) │ │
│ └────────┬────────┘ │
│ │ │
│ ┌────────────┴────────────┐ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────────────┐ │
│ │ retrieval_hits │ │ retrieval_summary │ │
│ │ (full structs) │ │ (compact for prompt) │ │
│ └─────────────────┘ └─────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘I use multimodal retrieval: text retrieval with BM25 plus dense vectors, image retrieval with dense vectors, fused via a rank-based aggregation strategy so one modality doesn't steamroll the other. Crucially, the controller does not get everything. It gets top hits as structured data and a short summary string that fits comfortably in a prompt.
This is a real engineering constraint. If you dump raw retrieved documents into the prompt, you will destroy the agent's ability to think.
I'm strict about retrieval order: prefer the local database before web, use web only when the database is thin or missing key facts. This policy shows up in three places: in the prompt policy telling the model what to do, in runtime constraints defining what tools exist and when they're appropriate, and in RL shaping that penalizes web calls when database confidence is high. The same principle is reinforced by multiple layers rather than relying on a single prompt instruction.
Naively merging text and image retrieval can backfire. Dense similarity can return superficially similar but clinically irrelevant items. BM25 can overweight short keyword overlaps. Image retrieval can "look right" but be wrong in context. Fusion helped, but the bigger win was controlling what the controller sees: a small number of hits plus a compact summary that doesn't drown out the question.
Student modeling: personalization without pretending I solved cognition
I intentionally avoided unsupported "we solved knowledge tracing" claims.
┌─────────────────────────────────────────────────────────────────────────────┐
│ STUDENT MODELING │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ StudentProfile │ │
│ │ level: novice | medium | expert │ │
│ │ (set at session start, influences scaffolding depth) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ SessionState │ │
│ │ attempt_count: int ◀── how many tries so far │ │
│ │ performance: correct | partial | wrong | none │ │
│ │ misconceptions: List[str] ◀── detected errors in reasoning │ │
│ │ consecutive_wrong: int ◀── escalation trigger │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Tutoring State Machine │ │
│ │ │ │
│ │ hint_level: 0 ──▶ 1 ──▶ 2 ──▶ 3 │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ (reveal allowed) │ │
│ │ │ │
│ │ reveal_gate: closed until attempt_count > 0 AND │ │
│ │ (performance == correct OR hint_level == max) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘My student modeling is built around what a system can actually observe during a session: a student profile indicating novice, medium, or expert level; attempt count; performance signals like correct, partial, or wrong; misconceptions when detectable; and a tutoring state machine that tracks hint level escalation and reveal gating.
This delivers visible behavior differences immediately. Novices get more probing and scaffolding. Experts get faster convergence and higher-level prompts. The system refuses to dump the answer on turn zero. That last point is not a prompt trick—it's enforced by gates.
For a tutoring system, you can either build a long-horizon mastery estimator early or build a robust session-level tutor policy first. I chose the second path because it produces visible, demoable behavior quickly, it's easier to verify, and it creates the scaffolding you need for knowledge tracing later: structured student signals, explicit state, traceable decisions.
The Socratic tutor policy: actions, not essays
Instead of generating free-form tutoring transcripts, I modeled tutoring decisions as one action per step: ASK_PROBE, HINT at levels 1 through 3, MICROLESSON, QUIZ, REQUEST_TOOL, REVEAL_ANSWER, or SAFETY_REFUSE.
┌─────────────────────────────────────────────────────────────────────────────┐
│ TUTOR ACTION SPACE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ASK_PROBE "What do you observe in the image?" │
│ HINT (1-3) Progressively more specific guidance │
│ MICROLESSON Brief teaching moment on a concept the student is missing │
│ QUIZ Follow-up question to check understanding │
│ REQUEST_TOOL "I need to examine this region closer" │
│ REVEAL_ANSWER Give the answer (only after gates allow it) │
│ SAFETY_REFUSE "This requires clinical judgment—consult a physician" │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ GATING LAYER │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Controller ──▶ [proposed action] ──▶ GATE CHECK ──▶ [actual action] │
│ │
│ REVEAL_GATE: Block reveal if attempt_count == 0 │
│ ASSESSMENT_GATE: Convert reveal to hint if performance != correct │
│ │
└─────────────────────────────────────────────────────────────────────────────┘That's a design choice that pays off everywhere. It's easy to validate, easy to log, easy to train with SFT, and easy to optimize with RL and ACE.
Even if a controller tries to reveal too early, runtime gates can override. No reveal until an attempt exists. Wrong or partial performance converts "reveal" into calibrated hints. Repeated tool calls within a single step are discouraged. This is how I guarantee the system behaves like a tutor even under stochastic policies.
If you're tutoring from a medical image, it's dangerously easy for a system to skip straight to a diagnosis or answer choice. I explicitly keep a pathway that produces objective findings without committing to a diagnosis. This supports tutoring behaviors like "Describe what you see first" or "Which finding most narrows the differential?" or "What would you expect next?" This observation-versus-diagnosis distinction is one of the most important tutor-versus-solver differences in the entire system.
SFT: training the controller to output valid tutoring actions
SFT is about reliability: consistent action formatting, fewer invalid outputs, better alignment to the action schema. The training target is not a perfect tutor transcript. The target is the next action.
┌─────────────────────────────────────────────────────────────────────────────┐
│ SFT: ACTION-ONLY CONTROLLER TRAINING │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ INPUT: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ student_profile: { level: "novice" } │ │
│ │ kc_tags: ["cardiac", "imaging", "tamponade"] │ │
│ │ history: [ │ │
│ │ { role: "user", content: "What's wrong with this chest X-ray?" }, │ │
│ │ { role: "assistant", content: "What features do you observe?" }, │ │
│ │ { role: "user", content: "The heart looks big?" } │ │
│ │ ] │ │
│ │ tool_summaries: [ "Zoomed into cardiac silhouette region" ] │ │
│ │ assessment: { performance: "partial", misconceptions: [] } │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ TARGET OUTPUT: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ { │ │
│ │ "action": "HINT", │ │
│ │ "level": 1, │ │
│ │ "content": "Good observation. Can you describe the shape of │ │
│ │ the cardiac silhouette more specifically?" │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Training Method: LoRA (lightweight adaptation) │
│ │
└─────────────────────────────────────────────────────────────────────────────┘If the runtime expects a structured action and the model outputs a 2,000-token lecture instead, you can't gate or route safely. So I trained the controller on examples that include student profile, coarse knowledge-component tags, conversation history, tool summaries, student assessment signals, and the desired next tutoring action. This is the same structure used at runtime—the training distribution matches the control interface.
The goal was to make the controller reliably output valid structured actions that the runtime can gate and route. The format is action-only, JSON-shaped outputs with one decision per step. Training used lightweight adaptation via LoRA so I could iterate quickly.
What changed after SFT: fewer invalid outputs, more consistent hint calibration across levels, and less style drift into answer-dumping. The runtime became a stable interface, and the controller became something I could upgrade without rewriting the system.
RL: training tool-use as sequential decision making
Tool use is not a single decision. It's a sequence: decide to zoom, observe, decide to retrieve, observe, decide to answer. If you only reward the final answer, tool learning becomes painfully sparse.
┌─────────────────────────────────────────────────────────────────────────────┐
│ RL: TOOL-USE AS SEQUENTIAL DECISION MAKING │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Episode: One medical case/question │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ State₀ │──▶──│ Action₀ │──▶──│ State₁ │──▶──│ Action₁ │──▶ ... │
│ │ (init) │ │ (zoom) │ │ (after │ │(retrieve)│ │
│ └─────────┘ └────┬────┘ │ zoom) │ └────┬────┘ │
│ │ └─────────┘ │ │
│ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ │
│ │Reward₀ │ │Reward₁ │ │
│ │+0.1 │ │+0.2 │ │
│ │(curious)│ │(relevant)│ │
│ └─────────┘ └─────────┘ │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ REWARD SHAPING │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ CURIOSITY REWARD: +reward when image tool produces meaningfully new view │
│ ANTI-REPEAT PENALTY: -penalty when same tool called with same/similar args│
│ RETRIEVAL QUALITY: +reward when retrieval returns relevant evidence │
│ SAFE FALLBACK: -penalty when web called AND local DB has high-conf hits │
│ TERMINAL REWARD: +reward for correct answer,-penalty for incorrect/timeout│
│ │
└─────────────────────────────────────────────────────────────────────────────┘So I added intermediate rewards that make tool-use learnable. Curiosity reward gives positive signal when an image tool produces a meaningfully new view. Anti-repeat penalty discourages repeated identical tool calls. Retrieval shaping rewards queries that return relevant evidence. Safe fallback shaping discourages web calls when the local database has high-confidence hits.
This turns tool-use learning from "pray the gradient finds it" into a tractable training signal.
I treated RL as an engineering problem rather than a label. One episode is one case/question. Actions are tool calls or terminal answers. Observations include the current view of the image via artifacts plus compact textual context. Termination happens when an answer is produced, the tool budget is exhausted, or there's an unrecoverable parse failure.
What changed after RL: the planner learned to "look first" instead of guessing, retrieval became the default knowledge source, and tool sequences became shorter and more purposeful. Tool use stopped being a brittle prompt behavior and became a policy that can improve with data and rewards.
ACE: playbooks that evolve without retraining
Not every failure requires gradient updates. Many failures are policy mistakes that can be captured as explicit rules: use retrieval before web, if the image is ambiguous zoom first, don't exceed one tool per turn, if you failed to parse a tool call simplify output formatting.
┌─────────────────────────────────────────────────────────────────────────────┐
│ ACE: AGENTIC CONTEXT ENGINEERING │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ ACE LOOP │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Run │────▶│ Analyze │────▶│ Curate │────▶│ Update │ │ │
│ │ │ samples │ │ failures │ │ ops │ │ playbook │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ └────┬─────┘ │ │
│ │ ▲ │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ │ (next iteration) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ PLAYBOOK EXAMPLE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ # Tool Policy Rules │
│ 1. Always use `retrieve` before `web_browser` │
│ 2. If the image region is ambiguous, use `zoom` first │
│ 3. One tool per turn maximum │
│ 4. If tool parsing fails, simplify output format │
│ 5. When evidence is insufficient, request a tool—do not guess │
│ │
└─────────────────────────────────────────────────────────────────────────────┘ACE gives me a clean loop: run the system on samples, analyze failures including tool failures, update a playbook, inject the playbook into controller prompts on the next run.
This is powerful because it is fast to iterate, transparent since the playbook is readable, and complementary to SFT and RL since it patches behavior while training catches up.
ACE is my "fast patch lane." Instead of waiting for a training cycle, I can run the system on a batch, detect systematic failure modes like tool parsing errors, tool ordering mistakes, and retrieval misuse, and evolve a playbook that steers the controller away from repeating those mistakes.
What changed after ACE iterations: fewer tool-policy violations, better ordering with retrieve before web, and better behavior under uncertainty where the system requests a tool when evidence is unclear. It turns repeated mistakes into explicit, reviewable rules and gives the system a path to improve even when training is expensive.
Observability: if you can't debug it, it's not done
Every time I've seen an "agent" fail in the wild, the root cause is the same: nobody can reconstruct what happened.
┌─────────────────────────────────────────────────────────────────────────────┐
│ OBSERVABILITY STACK │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Runtime Execution │
│ │ │
│ ┌───────────────────────┼───────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Trace Events │ │ Tool Cards │ │ Artifacts │ │
│ │ • step_id │ │ • tool_name │ │ • artifact_id │ │
│ │ • state_before │ │ • arguments │ │ • type │ │
│ │ • action │ │ • summary │ │ • path │ │
│ │ • observation │ │ • ok/error │ │ • metadata │ │
│ │ • state_after │ │ • duration_ms │ │ │ │
│ │ • timestamp │ │ │ │ │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ └──────────────────────┼──────────────────────┘ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Trace Store │ │
│ └──────────┬──────────┘ │
│ │ │
│ ┌──────────────────────┼──────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────────┐ │
│ │ Deterministic │ │ Audit │ │ Targeted Improvement │ │
│ │ Replay │ │ (provenance) │ │ (feed into SFT/RL/ACE) │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘So I built the system to emit per-step trace events with state before and after, action, and observation. It emits tool cards with arguments, summary, and errors. It emits retrieval hits. And it persists artifacts like images, masks, and text outputs.
This makes it possible to replay failures deterministically, audit whether a claim is evidence-backed, and do targeted improvements instead of random prompt tweaks.
The most important debugging feature is replayability. When an agent fails, there are always two questions: what happened, and what should change. Replayability answers the first question with certainty. And once you can replay, you can build the second half of the system: targeted improvements through SFT, RL shaping, or ACE playbook updates.
The decisions that shaped the system
One tool per turn. It's a constraint that makes everything else possible. It bounds latency. It bounds context. It makes policy learning easier. And it keeps traces readable.
Controller is not solver. This is what lets the system be both a stable medical answerer and a continuously improving tutor. I can iterate on policy behavior without destabilizing answer generation.
Student modeling is session-grounded. I'm modeling what I can observe reliably in-session: attempts, correctness, misconceptions when detectable, plus profile-based priors. It's enough to produce observable personalization without over-claiming.
Optimize with multiple knobs. I didn't bet everything on one method. SFT makes outputs reliable. RL makes tool-use behavior learnable. ACE patches recurring mistakes quickly. Together, they form a robust improvement stack.
The training recipe
┌─────────────────────────────────────────────────────────────────────────────┐
│ THE BUILD RECIPE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Step A Step B Step C Step D │
│ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ │
│ │Baseline│──────▶│ State │──────▶│Retriev│──────▶│ Tools │ │
│ │ │ │+Traces│ │ al │ │+Artif │ │
│ └───────┘ └───────┘ └───────┘ └───────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ "Can we "Can we "Can we "Can we │
│ measure?" debug?" ground?" inspect?" │
│ │
│ Step E Step F Step G Step H │
│ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ │
│ │Student│──────▶│ SFT │──────▶│ RL │──────▶│ ACE │ │
│ │Model+ │ │ │ │ │ │ │ │
│ │ Gates │ │ │ │ │ │ │ │
│ └───────┘ └───────┘ └───────┘ └───────┘ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ "Can we "Is output "Can policy "Can we fix │
│ tutor?" reliable?" improve?" fast?" │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ Core Principle: Make the runtime a stable interface first—then │
│ optimize behavior against that interface. │
└─────────────────────────────────────────────────────────────────────────────┘If I had to compress the build into one principle, it would be this: make the runtime a stable interface first, then optimize behavior against that interface.
Most agent projects invert this. They start by tuning prompts and policies, then add tooling later. That works for a demo, but it collapses once you add multiple tools, retrieval, tutoring behaviors, and long multi-turn traces. I ended up with a recipe that deliberately moves in the opposite direction.
Step A: establish a measurable baseline. Before adding tools or retrieval, I started with the simplest solvable loop: image plus question to answer. This isn't because it's the end goal. It's because you need a control group. Once you have a baseline, you can answer questions like: did retrieval actually help, or just add latency? Did tools improve evidence gathering, or just increase failure surface? Did tutoring gates improve pedagogy, or just prevent answers?
Step B: define the system boundary as explicit state plus traces. The next step is not "add more capability." It's "make behavior observable." I converted the runtime into an explicit state machine where every step has a before-state, an action, an observation, and an after-state. This is the point where the project stops being prompt engineering and becomes systems engineering. Once traces exist, failures become data.
Step C: introduce retrieval as a first-class dependency. Retrieval adds value, but it also adds a context budget problem. The key pattern that made retrieval usable was storing full hits as structured evidence but injecting only a compact summary into controller prompts. This keeps the controller thinking on the actual question rather than being drowned in text.
Step D: add tools with artifact discipline. Tools are only useful if they're trustworthy. The design that worked was tools returning a short summary for prompts and persisting outputs as artifacts. Prompts stay small. Humans can audit what happened. Offline loops can score behavior without re-running everything.
Step E: turn the solver into a tutor. At this point, the system can answer. But tutoring requires control. I treated student modeling as session-grounded signals: student profile, attempt count, performance grade, misconceptions when detectable, and tutoring state for hint escalation. Then I enforced pedagogy with hard gates: don't reveal on turn zero, wrong or partial turns become hints and probes, and objective findings can be separated from diagnosis.
Step F: SFT for reliability. Once the runtime interface is stable, SFT becomes straightforward. The model is trained to output the next structured action, conditioned on the same information the runtime uses. The effect is not "the model got smarter." The effect is fewer invalid actions, less formatting drift, and more predictable control flow.
Step G: RL for behavior. Tool use is sequential, and terminal-only rewards are too sparse. The recipe that worked was adding step shaping rewards that encode the behaviors the system needs: curiosity reward when visual tools create a new informative view, penalties for repeats and tool spam, reward retrieval when it produces relevant evidence, penalize web usage when local retrieval is clearly sufficient.
Step H: ACE for fast iteration. Even with training, some failures are not model skill problems—they're policy mistakes that recur. Ordering mistakes like web before retrieval. Repeated formatting failures. Avoidable tool misuse. ACE turns these into explicit, reviewable rules that get injected into the controller context.
Failures that forced the architecture
Context collapse. The model forgets the question after large retrieval or tool outputs, or starts repeating generic answers. The fix: compact summaries for prompts, artifacts for everything else, and strict budgets like one tool per turn.
Tool spam and busy work. Multiple tools called without changing the outcome. Repeating the same action with minor variations. The fix: bounded tool calls, repeat penalties in RL, and routing that forces progress.
Brittle formatting. Malformed tool calls. Tool outputs that can't be parsed. Controllers drifting into unstructured text. The fix: action schemas, action-only training targets, and system-level gates that sanitize unsafe decisions.
Answer dumping. The system gives the correct option immediately. The fix: reveal gates and assessment gates, tutor actions as the primary interface, and a separation between objective findings and diagnosis.
Closing
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ WHAT THIS ARCHITECTURE ENABLES │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Multi-turn │ │ Under │ │ With tools │ │ Debuggable │ │
│ │ tutoring │ │ uncertainty │ │& retrieval │ │& improvable │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │ │
│ └─────────────────┼─────────────────┼─────────────────┘ │
│ ▼ ▼ │
│ ┌───────────────────────────────────────┐ │
│ │ Deterministic Runtime Interface │ │
│ │ (state, tools, traces, gates) │ │
│ └───────────────────┬───────────────────┘ │
│ │ │
│ ┌─────────────────┼─────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ SFT │ │ RL │ │ ACE │ │
│ │ (format) │ │ (behavior)│ │ (rules) │ │
│ └───────────┘ └───────────┘ └───────────┘ │
│ │
│ ═══════════════════════ │
│ A TUTOR, │
│ NOT A CHATBOT │
│ ═══════════════════════ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘This architecture is designed to make a difficult problem tractable: multi-turn tutoring, under uncertainty, with tools and retrieval, while staying debuggable and improvable.
The core idea is simple. Treat the runtime as a stable interface—state, tools, traces, gates—then improve behavior through multiple complementary mechanisms: SFT for reliable formatting, RL for learned tool-use policies, and ACE playbooks for fast behavioral fixes. That structure is what makes the system behave like a tutor rather than a single-shot chatbot.