Agent Prompt Caching Guide

How the Loop agent lays out each request so that provider prompt caches survive from one iteration to the next, how that layout adapts per provider, and the run-scoped action circuit breaker that the same work exposed and fixed.

Audience: developers changing BaseAgent, the Loop system prompt template, a provider driver, or the model catalog; anyone reading a prompt-run cost report and wondering why the cached share is what it is.

Ships with: @memberjunction/ai-agents, @memberjunction/ai, @memberjunction/ai-core-plus, @memberjunction/ai-prompts, @memberjunction/ai-anthropic (PR #4508). Before-and-after numbers: packages/AI/Agents/docs/PROMPT_CACHE_PERFORMANCE_BRIEFING.md.

---

1. The problem

Every provider’s prompt cache is a prefix match over tools → system → messages. Anything that changes inside the system prompt therefore invalidates everything after it, including the entire (and growing) conversation history.

Until 6.2 the Loop agent’s system prompt template rendered three blocks at its tail on every iteration: the current date/time (changes per minute), the Scratchpad (changes per turn) and the Payload (changes per turn). Measured on Sage on 2026-09-14: cache reads plateaued at ~21.5K tokens (the static part of the system prompt) while uncached input grew to 73K per call. Anthropic and OpenAI showed 0% cached on later iterations, because their caches need the prefix to match from token 0.

2. The layout

The template no longer renders those blocks. BaseAgent.preparePromptParams builds the same content into a fragment and appends it as the final user message of each request:

<mj-runtime-state>
## Current Date/Time
- **Date**: 2026-09-27 (Sunday)
- **Time**: 4:37 PM CDT

## Scratchpad State
...

## Current State
**Payload:** ...
```json
{...}

```

The system prompt keeps a static ## Runtime State pointer telling the model where the state lives. The system prompt is therefore byte-identical across iterations, and the history becomes an incrementally cacheable prefix.

Rules the layout keeps:

• State only, never rules. Instructions stay in the system prompt. The fragment is authoritative framework state and the pointer says so.
• Never persisted. The fragment is appended to a copy of the history for one request (assembleOutgoingMessages). The stored history stays byte-stable. The one exception is append-only mode, below, where prior fragments are deliberately retained because the provider needs them.
• Tag literals are escaped at send time. <mj-runtime-state> and <mj-agent-specialization> in any non-system history message (user turns, action results, sub-agent results, tool-call arguments) are rewritten to <...> in the outgoing copy, so nothing can pose as framework state. Escaping is deterministic, so it does not disturb the cached prefix.
• Delivery gate. The fragment is only sent when the system prompt template carries the pointer. A template that still embeds the old blocks (a database whose metadata/prompts has not synced) gets no fragment, so the model never sees the state twice. A template with neither (Flow agents, custom prompts) gets none either. The markers that identify the old layout live in VOLATILE_TEMPLATE_MARKERS (constants.ts) and are overridable through BaseAgent.volatileTemplateMarkers.

All string literals involved (placeholder names, tag literals, block headings, markers) live in one place: packages/AI/Agents/src/constants.ts.

3. Replace-in-place vs append-only: PrefixPromptCache

Two kinds of cache exist:

PrefixPromptCache	Providers	Behaviour of the cache	What the agent does with the previous iteration’s fragment
absent / false (block cache)	Anthropic (explicit breakpoints), Gemini (implicit), Cerebras (sliding)	Matches on block or segment boundaries inside the prompt	Replaces it. Only the latest fragment is attached; history stays compact.
true (byte-prefix cache)	OpenAI automatic cache, xAI	Reuses a prior request only when that request’s entire prompt is a byte prefix of the new one	Retains it and appends the new one, so each request is an exact prefix extension of the last. Costs a few hundred stale (cached) tokens per iteration.

Replacing the fragment on a prefix cache breaks the prefix right after the system prompt and caps the cached share there (measured: GPT 5.6 flat at 21K cached every iteration). Appending recovers the history (~93% cached).

Which provider is which is metadata, not code. PrefixPromptCache is a boolean in LLMConfigurationSettings in the model catalog’s ModelConfiguration cascade, which since 6.2 has a vendor layer. AI Vendors carries its own Configuration bag (IAIVendorConfiguration), and its ModelDefaults key is a model-configuration bag that forms the default for every model the vendor serves:

MJ: AI Model Types . ModelConfiguration      (type-wide default)
  < MJ: AI Models . ModelConfiguration       (per model)
    < MJ: AI Vendors . Configuration.ModelDefaults   (host-wide default for every model this vendor serves)
      < MJ: AI Model Vendors . ModelConfiguration   (the inference provider's row — the tie-breaker, wins)

The vendor default sits above the model’s own bag on purpose: a host’s statement about how it serves models beats the model’s generic description, and the model-vendor row is where a host diverges for one model. The merge is per key, so a vendor default only touches the keys it actually sets.

BaseAgent.resolvePrefixPromptCache(model, vendor) finds the vendor’s inference-provider model-vendor row and reads the effective configuration through AIEngine.GetEffectiveModelConfiguration, which folds in the ModelDefaults of the vendor named by that model-vendor row. true ships under Configuration.ModelDefaults on the OpenAI and x.ai vendor rows in metadata/ai-vendors, so every model they serve inherits it and a new OpenAI model needs no seed of its own. A host that caches differently for one model sets the flag on that model-vendor row in metadata/ai-models. Nothing in code knows a provider’s name.

The mode is decided once per run: from a runtime model override, else from the first iteration’s model selection, then frozen so a failover cannot flip the layout mid-run. On turn 1, before any selection is known, the layout is replace-in-place; if turn 2 resolves to append-only, turn 1’s fragment is restored at the turn-1 boundary. The prompt’s bound models are deliberately not consulted, because prompts bind several vendors for failover.

Per-agent override, on the Loop type’s prompt params (trailingStateMode): auto (default), appendOnly (a serving path whose rows carry no strategy yet), replace (context growth matters more than cache hits).

4. Anthropic’s breakpoint

Anthropic caches only up to an explicit cache_control breakpoint and read-hits require the new request to match a cached prefix at a breakpoint. If the breakpoint sat on the last message, it would sit on the fragment and miss every iteration.

BaseLLM owns the framework-generic half: isVolatileStateMessage (the volatileState metadata flag the agent layer sets on the fragment), trailingVolatileStateIndex and splitTrailingVolatileState. AnthropicLLM uses the split to place its breakpoint on the last real history message and sends the fragment uncached, inserting an OK assistant turn when two user turns would otherwise touch. Any other provider that needs a pre-fragment breakpoint reuses the same seam.

5. Specialization placement

The OS prompt cannot control what an agent designer puts in a child prompt. A child template that references a volatile placeholder (_CURRENT_DATE*, _CURRENT_TIME*, _CURRENT_PAYLOAD, _SCRATCHPAD_*) mutates the system prompt every iteration from a position ahead of the catalogs and the history, so relocating the runtime-state tail recovers nothing for it. Six active Loop agents in the shipped catalog do this (Agent Manager, Data Scout, Database Designer, Experiment Designer, Goal Analyst, Model Development Agent).

specializationPlacement on the Loop prompt params: auto (default) relocates the rendered child prompt into the fragment, inside <mj-agent-specialization>, only when its unrendered template text references a volatile placeholder; systemPrompt never; trailingMessage always. Decided once per run from the template text, read by TemplateID through the template engine, and it fails closed (no text → stays in the system prompt). When relocated, the runner receives the already-rendered child through AIPromptParams.PreRenderedChildTemplates so the child is not rendered twice.

6. The action-failure circuit breaker

Benchmarking the layout on the Research Agent exposed a pre-existing bug: executeActionsStep reported a failed action (ActionResult.Success === false) as success: true, so the model never learned the tool had failed and retried it until the two-hour benchmark timeout. The step result now carries the real outcome, and a run-scoped breaker in BaseAgent.ExecuteSingleAction bounds retries with three rules, checked fatal → identical-arguments → budget:

Rule	Trigger	Effect	Directive the model sees
Fatal lockout	One failure whose message is a credential or service-configuration error (isFatalActionError: API key missing/invalid, credentials not found, authentication failed, <service> not configured). Parameter-level “not configured” messages are deliberately non-fatal.	Action disabled for the rest of the run, any arguments.	[CRITICAL/ACTION_UNAVAILABLE]
Identical arguments	IDENTICAL_FAILURE_THRESHOLD (2) failures with the same normalized arguments	Further calls with those arguments blocked; different arguments still dispatch.	[CRITICAL/REPEATED_IDENTICAL_CALL]
Attempt budget	ACTION_FAILURE_BUDGET (5) consecutive failures across any arguments	Action disabled for the run.	[CRITICAL/ATTEMPTS_EXHAUSTED]

A success clears both counters. A blocked call returns a CircuitBreakerActionResult (a failed ActionResult carrying the Reason) in ~0 ms and never reaches ActionEngine.RunAction, so it writes no execution-log row. Other failures get [WARNING/ACTION_FAILURE] with an “attempt N of 5” counter. Directives are appended to the history as a user message, never injected into the system prompt, so the cacheable prefix stays intact.

Argument identity is normalizeActionParams: keys sorted at every depth, arrays kept in order, three protected layers (normalizeActionParams → normalizeActionParamEntry → normalizeActionParamValue) so a subclass can ignore a key or canonicalize values.

Exemption. Callers that do their own per-element failure accounting and have no model in their loop pass { skipCircuitBreaker: true }: the pipeline registry, and the ForEach and While operators. Those calls bypass all three rules and leave the breaker’s history untouched in both directions.

Known gaps: the thresholds are constants rather than metadata; a blocked call does not write an ActionExecutionLog row (the run step is still created and marked failed).

7. Verifying it

• Unit: base-agent-volatile-state.test.ts, base-agent-action-failure.test.ts, runtime-state-fragment.test.ts, volatile-child-prompt.test.ts, loop-agent-system-prompt-snapshot.test.ts (Agents); baseLLM.test.ts, modelConfiguration.test.ts (Core); anthropic.test.ts.
• Integration, deterministic tier: IT95 – Trailing Runtime State (synced template, live placeholders, catalog-driven mode, by-ID child template resolution, copy-on-assemble, Anthropic breakpoint), IT46 ALS7–ALS11 (breaker permutations against the real action engine), IT39 AP6 (the structured-failure contract the breaker consumes).
• Live: a prompt run’s CacheReadTokens / PromptTokens on iteration 3 and later. On a prefix-cache provider expect the cached figure to grow with the history; flat at the system-prompt size means the mode resolved to replace.

8. Checklist when you touch this

• Changing the Loop template? Keep the ## Runtime State pointer and do not reintroduce _CURRENT_DATE, _CURRENT_PAYLOAD or the three block headings; loop-agent-system-prompt-snapshot.test.ts pins this.
• Adding a provider? Decide whether its cache is a byte-prefix match and, if so, set PrefixPromptCache: true under the vendor row’s Configuration.ModelDefaults; every model it serves inherits it. Override per model-vendor row only where a host diverges. Nothing in code needs to know its name.
• Adding a placeholder that changes per iteration? Add it to constants.ts so IsVolatileChildPrompt and the template markers see it.
• Writing a child prompt? Referencing the date or payload is fine; auto placement relocates it. Prefer not to, since relocation costs ~3K uncached tokens per call.
• Adding an action that fails per resource (one URL, one record)? Make the failure message not look like a credential error, or the breaker’s fatal rule will lock the whole tool out. IT39 AP6 shows the contract.