How the trailing runtime-state message is carried across loop iterations.
Background: the fragment described under SpecializationPlacement is rebuilt every
iteration. Providers with block-level or sliding prefix caches (Anthropic, Gemini, Cerebras) are
happiest when the previous iteration's fragment is REPLACED, so the history stays compact.
OpenAI's automatic cache is different: it reuses a prior request only when that request's
entire prompt is a byte prefix of the new one, so replacing the fragment breaks the prefix
right after the system prompt and caps the cached share at the system prompt (~22% measured).
Retaining prior fragments and APPENDING the new one makes each request an exact prefix
extension of the last (~93% measured).
'auto' (default): append-only when the model catalog says the serving path's cache is a
byte-prefix cache — the PrefixPromptCache flag in ModelConfiguration.LLM, resolved
through the catalog cascade (Model Types < Models < Vendors' Configuration.ModelDefaults <
Model Vendors: the vendor's defaults beat the model's own bag and the inference provider's
model-vendor row is the tie-breaker) is true — otherwise replace-in-place. Nothing about a
provider is hard-coded: a new host, or one model on a host that caches differently from the
rest, is a metadata change. The answer is taken from a runtime model override, else the FIRST
iteration's model selection, and then frozen for the rest of the run so a failover cannot flip
the layout mid-run. On turn 1, before any selection is known, the layout is replace-in-place;
if turn 2 resolves to append-only, turn 1's fragment is restored at the turn-1 boundary, so
deferring loses nothing. The prompt's bound models are deliberately not consulted: prompts
commonly bind several vendors for failover.
'appendOnly': always retain prior fragments. Use this for a serving path whose catalog rows
carry no flag yet — an OpenAI-compatible gateway, say — until its metadata is filled in.
'replace': always replace. Use this to keep context compact on a run whose catalog rows say
true but where context growth matters more than cache hits.
Resolved by BaseAgent.shouldUseAppendOnlyTrailingState via BaseAgent.resolvePrefixPromptCache.
How the trailing runtime-state message is carried across loop iterations.
Background: the fragment described under SpecializationPlacement is rebuilt every iteration. Providers with block-level or sliding prefix caches (Anthropic, Gemini, Cerebras) are happiest when the previous iteration's fragment is REPLACED, so the history stays compact. OpenAI's automatic cache is different: it reuses a prior request only when that request's entire prompt is a byte prefix of the new one, so replacing the fragment breaks the prefix right after the system prompt and caps the cached share at the system prompt (~22% measured). Retaining prior fragments and APPENDING the new one makes each request an exact prefix extension of the last (~93% measured).
'auto'(default): append-only when the model catalog says the serving path's cache is a byte-prefix cache — thePrefixPromptCacheflag inModelConfiguration.LLM, resolved through the catalog cascade (Model Types < Models < Vendors'Configuration.ModelDefaults< Model Vendors: the vendor's defaults beat the model's own bag and the inference provider's model-vendor row is the tie-breaker) istrue— otherwise replace-in-place. Nothing about a provider is hard-coded: a new host, or one model on a host that caches differently from the rest, is a metadata change. The answer is taken from a runtime model override, else the FIRST iteration's model selection, and then frozen for the rest of the run so a failover cannot flip the layout mid-run. On turn 1, before any selection is known, the layout is replace-in-place; if turn 2 resolves to append-only, turn 1's fragment is restored at the turn-1 boundary, so deferring loses nothing. The prompt's bound models are deliberately not consulted: prompts commonly bind several vendors for failover.'appendOnly': always retain prior fragments. Use this for a serving path whose catalog rows carry no flag yet — an OpenAI-compatible gateway, say — until its metadata is filled in.'replace': always replace. Use this to keep context compact on a run whose catalog rows saytruebut where context growth matters more than cache hits.Resolved by
BaseAgent.shouldUseAppendOnlyTrailingStateviaBaseAgent.resolvePrefixPromptCache.