How the trailing runtime-state message is carried across loop iterations.
Background: the fragment described under SpecializationPlacement is rebuilt every
iteration. Providers with block-level or sliding prefix caches (Anthropic, Gemini, Cerebras) are
happiest when the previous iteration's fragment is REPLACED, so the history stays compact.
OpenAI's automatic cache is different: it reuses a prior request only when that request's
entire prompt is a byte prefix of the new one, so replacing the fragment breaks the prefix
right after the system prompt and caps the cached share at the system prompt (~22% measured).
Retaining prior fragments and APPENDING the new one makes each request an exact prefix
extension of the last (~93% measured).
'auto' (default): append-only when the model catalog says the serving path's cache is a
byte-prefix cache — the PrefixPromptCache flag in ModelConfiguration.LLM, resolved
through the catalog cascade (Model Types < Models < Vendors' Configuration.ModelDefaults <
Model Vendors: the vendor's defaults beat the model's own bag and the inference provider's
model-vendor row is the tie-breaker) is true — otherwise replace-in-place. Nothing about a
provider is hard-coded: a new host, or one model on a host that caches differently from the
rest, is a metadata change. A runtime model override that names a vendor, or whose catalog says
yes, answers immediately; otherwise the FIRST iteration's model selection decides. The answer is
then frozen for the rest of the run so a failover cannot flip the layout mid-run. On turn 1, before any selection is known, the layout is replace-in-place;
if turn 2 resolves to append-only, turn 1's fragment is restored at the turn-1 boundary, so
deferring loses nothing. The prompt's bound models are deliberately not consulted: prompts
commonly bind several vendors for failover. One case is decided by the driver, not the catalog:
a run that declares native tools to a driver whose replayed reasoning needs the history before
it unchanged (BaseLLM.ReplayedReasoningRequiresStablePrefix, true for Anthropic) is
append-only too, so a thinking block is never replayed behind a dropped fragment.
'appendOnly': always retain prior fragments. Use this for a serving path whose catalog rows
carry no flag yet — an OpenAI-compatible gateway, say — until its metadata is filled in.
'replace': always replace. Use this to keep context compact on a run whose catalog rows say
true but where context growth matters more than cache hits.
Resolved by BaseAgent.shouldUseAppendOnlyTrailingState via BaseAgent.ResolveAppendOnlyState
(the catalog through resolvePrefixPromptCache, the driver through DriverRequiresStableReasoningPrefix).
How the trailing runtime-state message is carried across loop iterations.
Background: the fragment described under SpecializationPlacement is rebuilt every iteration. Providers with block-level or sliding prefix caches (Anthropic, Gemini, Cerebras) are happiest when the previous iteration's fragment is REPLACED, so the history stays compact. OpenAI's automatic cache is different: it reuses a prior request only when that request's entire prompt is a byte prefix of the new one, so replacing the fragment breaks the prefix right after the system prompt and caps the cached share at the system prompt (~22% measured). Retaining prior fragments and APPENDING the new one makes each request an exact prefix extension of the last (~93% measured).
'auto'(default): append-only when the model catalog says the serving path's cache is a byte-prefix cache — thePrefixPromptCacheflag inModelConfiguration.LLM, resolved through the catalog cascade (Model Types < Models < Vendors'Configuration.ModelDefaults< Model Vendors: the vendor's defaults beat the model's own bag and the inference provider's model-vendor row is the tie-breaker) istrue— otherwise replace-in-place. Nothing about a provider is hard-coded: a new host, or one model on a host that caches differently from the rest, is a metadata change. A runtime model override that names a vendor, or whose catalog says yes, answers immediately; otherwise the FIRST iteration's model selection decides. The answer is then frozen for the rest of the run so a failover cannot flip the layout mid-run. On turn 1, before any selection is known, the layout is replace-in-place; if turn 2 resolves to append-only, turn 1's fragment is restored at the turn-1 boundary, so deferring loses nothing. The prompt's bound models are deliberately not consulted: prompts commonly bind several vendors for failover. One case is decided by the driver, not the catalog: a run that declares native tools to a driver whose replayed reasoning needs the history before it unchanged (BaseLLM.ReplayedReasoningRequiresStablePrefix, true for Anthropic) is append-only too, so a thinking block is never replayed behind a dropped fragment.'appendOnly': always retain prior fragments. Use this for a serving path whose catalog rows carry no flag yet — an OpenAI-compatible gateway, say — until its metadata is filled in.'replace': always replace. Use this to keep context compact on a run whose catalog rows saytruebut where context growth matters more than cache hits.Resolved by
BaseAgent.shouldUseAppendOnlyTrailingStateviaBaseAgent.ResolveAppendOnlyState(the catalog throughresolvePrefixPromptCache, the driver throughDriverRequiresStableReasoningPrefix).