Platt calibration of the thread Likelihood (continues), per decision model, each tied to the
exact model it was fitted on (see FindDecisionCalibration): the MJ decision model that answered
(ModelName) and the model the driver reports behind it (ResolvedModel). Any other model,
including Jev at another version or LLM Decision answered by another chat model, is treated as
unsure, so routing keeps continuity.
Calibrated at the corpus's mix, not a chat's. Platt's B absorbs the base rate of the data it
was fitted on: 127 continuations to 174 switches, so 42% of messages continue. A calibrated value
is the chance that the message continues when 42% of messages do. Both models' raw answers lean
toward "continues": at the corpus's mix, Jev's raw 0.5 is a calibrated 0.03. At a 90%-continue
prior, closer to a chat, the calibrated logit gains ln(9 × 174/127) ≈ 2.51. Jev's raw 0.5 is then
about 0.30, and the leave bar of a calibrated 0.30 (DECISION_ROUTING_MIN_CONFIDENCE) is about
an 84% chance that the message continues. B is kept as fitted; that constant says what the bar
does at a chat's mix.
Where the fits come from. The Phase 2 Decision Eval (plan Task 2.4, 2026-09-29), cell
production (the production state layout and question), on the labelled continuity corpus with
its construction labels. The corpus has 374 points. 301 are scored (174 switch, 127 continue); the
rest are ambiguous or never got a usable probability. Each point is asked five times and scored on
its mean. A and B are fitted on all 301 points; the figures below are out of fold (5 folds). The
corpus is private and not in the repo. When the fits were made, corpus.jsonl hashed (SHA-256) to
08dd187af4318a93… and labels.jsonl to 625d1df9b1428809…, so a refit can be compared with this one.
Jev at its pinned APIName, typesafe/jev-1.13-20260917, which OpenRouter reports back as
the resolved model. Suite "Decision Eval — Conversation Routing (Jev rerun)", runs since
2026-09-29T21:36:27Z: 1,825 usable runs (45 more had no probability). Balanced accuracy 0.800
raw → 0.928 calibrated [0.897, 0.956], ECE 0.226 → 0.041.
LLM Decision when its chat model is GPT-OSS-120B (its prompt's first choice, via Cerebras
in the eval). Suite "Decision Eval — Conversation Routing", runs since 2026-09-29T20:29:56Z:
1,825 usable runs. 0.796 → 0.861 [0.826, 0.897], ECE 0.195 → 0.081.
Assumptions in the keying.
LLM Decision reports its chat model's MJ name, GPT-OSS-120B, with no vendor. So its fit also
covers answers served by Groq, which the LLM Decision prompt lists second, though the fit was
measured via Cerebras. This assumes both vendors serve the same model.
After a failover inside the LLM Decision prompt, the resolved model is the one AIPromptRunner
first selected, not the one that answered: its modelInfo comes from the selected model, and
failover moves only promptRun.ModelID. So if both GPT-OSS-120B vendors fail and another chat
model answers, the GPT-OSS-120B fit is applied to that answer. Jev would have to fail first, and
then both vendors, within DECISION_ROUTING_TIMEOUT_MS, so this is unlikely. The fix
belongs in AIPromptRunner: build modelInfo from the candidate that answered.
Refit, and add the new pair, whenever a model, its version, the question or the state layout
changes.
Platt calibration of the thread Likelihood (
continues), per decision model, each tied to the exact model it was fitted on (seeFindDecisionCalibration): the MJ decision model that answered (ModelName) and the model the driver reports behind it (ResolvedModel). Any other model, including Jev at another version orLLM Decisionanswered by another chat model, is treated as unsure, so routing keeps continuity.Calibrated at the corpus's mix, not a chat's. Platt's
Babsorbs the base rate of the data it was fitted on: 127 continuations to 174 switches, so 42% of messages continue. A calibrated value is the chance that the message continues when 42% of messages do. Both models' raw answers lean toward "continues": at the corpus's mix, Jev's raw 0.5 is a calibrated 0.03. At a 90%-continue prior, closer to a chat, the calibrated logit gains ln(9 × 174/127) ≈ 2.51. Jev's raw 0.5 is then about 0.30, and the leave bar of a calibrated 0.30 (DECISION_ROUTING_MIN_CONFIDENCE) is about an 84% chance that the message continues.Bis kept as fitted; that constant says what the bar does at a chat's mix.Where the fits come from. The Phase 2 Decision Eval (plan Task 2.4, 2026-09-29), cell
production(the production state layout and question), on the labelled continuity corpus with its construction labels. The corpus has 374 points. 301 are scored (174 switch, 127 continue); the rest are ambiguous or never got a usable probability. Each point is asked five times and scored on its mean.AandBare fitted on all 301 points; the figures below are out of fold (5 folds). The corpus is private and not in the repo. When the fits were made,corpus.jsonlhashed (SHA-256) to08dd187af4318a93…andlabels.jsonlto625d1df9b1428809…, so a refit can be compared with this one.APIName,typesafe/jev-1.13-20260917, which OpenRouter reports back as the resolved model. Suite "Decision Eval — Conversation Routing (Jev rerun)", runs since 2026-09-29T21:36:27Z: 1,825 usable runs (45 more had no probability). Balanced accuracy 0.800 raw → 0.928 calibrated [0.897, 0.956], ECE 0.226 → 0.041.Assumptions in the keying.
GPT-OSS-120B, with no vendor. So its fit also covers answers served by Groq, which the LLM Decision prompt lists second, though the fit was measured via Cerebras. This assumes both vendors serve the same model.AIPromptRunnerfirst selected, not the one that answered: itsmodelInfocomes from the selected model, and failover moves onlypromptRun.ModelID. So if both GPT-OSS-120B vendors fail and another chat model answers, the GPT-OSS-120B fit is applied to that answer. Jev would have to fail first, and then both vendors, within DECISION_ROUTING_TIMEOUT_MS, so this is unlikely. The fix belongs inAIPromptRunner: buildmodelInfofrom the candidate that answered.Refit, and add the new pair, whenever a model, its version, the question or the state layout changes.