Member Junction
    Preparing search index...

    Function BuildDecisionEvalActualOutput

    • What a run records as its ActualOutput: the summarized answers, the continues probability, the agent Choice, production's verdict and the policy it was reached under, the model record, the sampling record, the latency, the prompt run and its cost, and the error when there are no answers.

      The verdict is production's: the answers are read by InterpretRoutingAnswers with the model that answered and the model behind it, as the chat passes them from RunDecision, so the thread Likelihood is calibrated exactly when production would calibrate it.

      Parameters

      • config: {
            decision: "conversation-routing";
            failover?: boolean | null;
            modelId?: string | null;
            oracles: {
                config?: Record<string, unknown>;
                type: string;
                weight?: number;
            }[];
            promptName?: string
            | null;
            seed?: number | null;
            stateLayout: "production" | "structured";
            temperature?: number | null;
            vendorId?: string | null;
        }

        The cell's configuration.

        • decision: "conversation-routing"

          Which decision builder. A discovery test has its own schema (DiscoveryEvalConfigSchema).

        • Optionalfailover?: boolean | null

          Whether the runner may fail over. Defaults to false when the cell pins a model or vendor.

        • OptionalmodelId?: string | null

          Pins the MJ: AI Models row that must answer.

        • oracles: { config?: Record<string, unknown>; type: string; weight?: number }[]

          The oracles to run.

        • OptionalpromptName?: string | null

          The decision prompt. Defaults to DEFAULT_DECISION_PROMPT_NAME.

        • Optionalseed?: number | null

          Requested sampling seed. Recorded; see the driver for whether it reaches the model.

        • stateLayout: "production" | "structured"

          production sends BuildRoutingState; structured sends BuildRoutingStateStructured.

        • Optionaltemperature?: number | null

          Requested sampling temperature. Recorded; see the driver for whether it reaches the model.

        • OptionalvendorId?: string | null

          Pins the vendor.

      • promptName: string

        The decision prompt's name.

      • call: DecisionEvalCall

        The call and its result.

      Returns {
          Answers: Record<
              string,
              {
                  Confidence?: number;
                  Kind: "Likelihood"
                  | "Choice"
                  | "Score";
                  Probabilities?: Record<string, number>;
                  Probability?: number;
                  Value?: string | number;
              },
          >;
          ContinuesProbability: number
          | null;
          CostUSD: number | null;
          Decision: "conversation-routing" | "agent-discovery";
          Error: string | null;
          LatencyMs: number | null;
          Model: {
              AnsweredByPinned: boolean | null;
              AnsweredModelId: string | null;
              AnsweredModelName: string | null;
              AnsweredVendorId: string | null;
              AnsweredVendorName: string | null;
              DriverClass: string | null;
              FailedOver: boolean;
              FailoverAllowed: boolean;
              PinnedModelId: string | null;
              PinnedVendorId: string | null;
              ResolvedModel: string | null;
          };
          PromptName: string;
          PromptRunId: string
          | null;
          Route: { Confidence: number; Value: string } | null;
          RoutingPolicy?:
              | {
                  Calibration: { A: number; B: number }
                  | null;
                  MinConfidence: number;
                  TimeoutMs: number;
              }
              | null;
          RoutingVerdict: "Routed"
          | "SomeoneElse"
          | "KeptContinuity"
          | null;
          Sampling: {
              Applied: boolean;
              Note: string | null;
              RequestedSeed: number | null;
              RequestedTemperature: number | null;
          };
          StateLayout: "production"
          | "structured";
      }

      • Answers: Record<
            string,
            {
                Confidence?: number;
                Kind: "Likelihood"
                | "Choice"
                | "Score";
                Probabilities?: Record<string, number>;
                Probability?: number;
                Value?: string | number;
            },
        >

        Every answer, summarized. A Choice keeps its full distribution.

      • ContinuesProbability: number | null

        The continues Likelihood's raw probability, or null when there is none.

      • CostUSD: number | null

        The run's cost in USD, from the prompt run when it has one.

      • Decision: "conversation-routing" | "agent-discovery"
      • Error: string | null

        Why the run produced no answers, or null.

      • LatencyMs: number | null

        Wall-clock time of the one ExecuteDecision call, in milliseconds.

      • Model: {
            AnsweredByPinned: boolean | null;
            AnsweredModelId: string | null;
            AnsweredModelName: string | null;
            AnsweredVendorId: string | null;
            AnsweredVendorName: string | null;
            DriverClass: string | null;
            FailedOver: boolean;
            FailoverAllowed: boolean;
            PinnedModelId: string | null;
            PinnedVendorId: string | null;
            ResolvedModel: string | null;
        }
        • AnsweredByPinned: boolean | null

          Null when nothing was pinned.

        • AnsweredModelId: string | null
        • AnsweredModelName: string | null
        • AnsweredVendorId: string | null
        • AnsweredVendorName: string | null
        • DriverClass: string | null
        • FailedOver: boolean

          True when a pinned run was answered by another model or vendor.

        • FailoverAllowed: boolean
        • PinnedModelId: string | null
        • PinnedVendorId: string | null
        • ResolvedModel: string | null

          The vendor's resolved version, when the driver reports one. For LLMDecision, the chat model that answered.

      • PromptName: string
      • PromptRunId: string | null

        The MJ: AI Prompt Runs row the runner wrote.

      • Route: { Confidence: number; Value: string } | null

        The agent Choice's value and confidence, or null.

      • OptionalRoutingPolicy?:
            | {
                Calibration: { A: number; B: number }
                | null;
                MinConfidence: number;
                TimeoutMs: number;
            }
            | null

        The routing policy that verdict was reached under, or null without answers. Absent from runs recorded before routing was calibrated: their verdicts used raw thresholds, so the scorecard does not score them as production's.

      • RoutingVerdict: "Routed" | "SomeoneElse" | "KeptContinuity" | null

        What production would do with these answers (InterpretRoutingAnswers), or null without answers.

      • Sampling: {
            Applied: boolean;
            Note: string | null;
            RequestedSeed: number | null;
            RequestedTemperature: number | null;
        }
      • StateLayout: "production" | "structured"

    Settings

    Member Visibility