Member Junction
    Preparing search index...

    Function BuildBaselineEvalActualOutput

    • What a semantic-search baseline run records as its ActualOutput: the ranking, the action's first row as the chosen agent, and whether it lists anything at all as WouldInject. It has no answers, no model and no cost.

      Parameters

      • config: {
            baseline?: "semantic-search" | null;
            decision: "agent-discovery";
            failover?: boolean | null;
            modelId?: string | null;
            oracles: {
                config?: Record<string, unknown>;
                type: string;
                weight?: number;
            }[];
            promptName?: string
            | null;
            seed?: number | null;
            temperature?: number | null;
            vendorId?: string | null;
        }

        The cell's configuration.

        • Optionalbaseline?: "semantic-search" | null

          Runs this baseline instead of the decision: no decision call is made.

        • decision: "agent-discovery"
        • Optionalfailover?: boolean | null

          Whether the runner may fail over. Defaults to false when the cell pins a model or vendor.

        • OptionalmodelId?: string | null

          Pins the MJ: AI Models row that must answer.

        • oracles: { config?: Record<string, unknown>; type: string; weight?: number }[]

          The oracles to run.

        • OptionalpromptName?: string | null

          The decision prompt. Defaults to Default Decision, the prompt production's discovery asks.

        • Optionalseed?: number | null

          Requested sampling seed. Recorded, not applied: see the driver.

        • Optionaltemperature?: number | null

          Requested sampling temperature. Recorded, not applied: see the driver.

        • OptionalvendorId?: string | null

          Pins the vendor.

      • baseline: {
            Floor: number;
            Results: number;
            TopK: number;
            TopMatch:
                | {
                    AgentId: string;
                    AgentName: string
                    | null;
                    Lexical: number | null;
                    PassesFloor: boolean;
                    Rank: number;
                    Score: number;
                    Semantic: number | null;
                }
                | null;
            TopRanked: | {
                AgentId: string;
                AgentName: string
                | null;
                Lexical: number | null;
                PassesFloor: boolean;
                Rank: number;
                Score: number;
                Semantic: number | null;
            }
            | null;
        }

        The ranking (RankSemanticSearchBaseline).

        • Floor: number

          The action's similarity floor.

        • Results: number

          How many it returned.

        • TopK: number

          How many results the search was asked for.

        • TopMatch:
              | {
                  AgentId: string;
                  AgentName: string
                  | null;
                  Lexical: number | null;
                  PassesFloor: boolean;
                  Rank: number;
                  Score: number;
                  Semantic: number | null;
              }
              | null

          The best-ranked candidate that passes the floor: the first row the action returns, or null when it returns none.

        • TopRanked:
              | {
                  AgentId: string;
                  AgentName: string
                  | null;
                  Lexical: number | null;
                  PassesFloor: boolean;
                  Rank: number;
                  Score: number;
                  Semantic: number | null;
              }
              | null

          The best-ranked candidate, floor or not, or null when the search returned none.

      • latencyMs: number

        Wall-clock time of the search.

      • expected:
            | { agentId: string; label: "agent"; labelSource: string }
            | {
                kind: "chat" | "direct" | "workflow";
                label: "none";
                labelSource: string;
            }

        The request's label.

      • candidates: readonly { ID: string }[]

        The agents the action may list.

      Returns {
          Answers: Record<
              string,
              {
                  Confidence?: number;
                  Kind: "Likelihood"
                  | "Choice"
                  | "Score";
                  Probabilities?: Record<string, number>;
                  Probability?: number;
                  Value?: string | number;
              },
          >;
          AnyApplies: number
          | null;
          Arm: "decision" | "semantic-search";
          Baseline:
              | {
                  Floor: number;
                  Results: number;
                  TopK: number;
                  TopMatch: | {
                      AgentId: string;
                      AgentName: string
                      | null;
                      Lexical: number | null;
                      PassesFloor: boolean;
                      Rank: number;
                      Score: number;
                      Semantic: number | null;
                  }
                  | null;
                  TopRanked: | {
                      AgentId: string;
                      AgentName: string
                      | null;
                      Lexical: number | null;
                      PassesFloor: boolean;
                      Rank: number;
                      Score: number;
                      Semantic: number | null;
                  }
                  | null;
              }
              | null;
          ChosenAgentId: string
          | null;
          ChosenAgentName: string | null;
          Confidence: number | null;
          CostUSD: number | null;
          Decision: "agent-discovery";
          DiscoveryLatencyMs?: number | null;
          Error: string | null;
          LabelledAgentOffered: boolean | null;
          LatencyMs: number | null;
          MinConfidence: number | null;
          Model:
              | {
                  AnsweredByPinned: boolean
                  | null;
                  AnsweredModelId: string | null;
                  AnsweredModelName: string | null;
                  AnsweredVendorId: string | null;
                  AnsweredVendorName: string | null;
                  DriverClass: string | null;
                  FailedOver: boolean;
                  FailoverAllowed: boolean;
                  PinnedModelId: string | null;
                  PinnedVendorId: string | null;
                  ResolvedModel: string | null;
              }
              | null;
          Options: | {
              CatalogSize: number;
              Count: number;
              DeclaredCap: number
              | null;
              Limit: number;
              NarrowedFrom: number | null;
              WithoutDescription: number;
          }
          | null;
          PromptName: string
          | null;
          PromptRunId: string | null;
          Sampling: {
              Applied: boolean;
              Note: string | null;
              RequestedSeed: number | null;
              RequestedTemperature: number | null;
          };
          VerdictReason: string
          | null;
          WithinProductionTimeout: boolean | null;
          WouldInject: boolean | null;
      }

      • Answers: Record<
            string,
            {
                Confidence?: number;
                Kind: "Likelihood"
                | "Choice"
                | "Score";
                Probabilities?: Record<string, number>;
                Probability?: number;
                Value?: string | number;
            },
        >

        Every answer, summarized. The Choice keeps its full distribution, by agent ID.

      • AnyApplies: number | null

        The Likelihood that a specialist agent should handle the request, or null.

      • Arm: "decision" | "semantic-search"
      • Baseline:
            | {
                Floor: number;
                Results: number;
                TopK: number;
                TopMatch: | {
                    AgentId: string;
                    AgentName: string
                    | null;
                    Lexical: number | null;
                    PassesFloor: boolean;
                    Rank: number;
                    Score: number;
                    Semantic: number | null;
                }
                | null;
                TopRanked: | {
                    AgentId: string;
                    AgentName: string
                    | null;
                    Lexical: number | null;
                    PassesFloor: boolean;
                    Rank: number;
                    Score: number;
                    Semantic: number | null;
                }
                | null;
            }
            | null

        The baseline's ranking, or null for the decision arm.

      • ChosenAgentId: string | null
      • ChosenAgentName: string | null
      • Confidence: number | null

        The Choice's confidence, or null.

      • CostUSD: number | null
      • Decision: "agent-discovery"
      • OptionalDiscoveryLatencyMs?: number | null

        Decision arm: wall-clock time from building the options, the semantic search included, through the decision call, in milliseconds. That is the span production's discovery timeout bounds. Null for the baseline. Absent from runs recorded before it was measured.

      • Error: string | null

        Why the run produced no answer, or null.

      • LabelledAgentOffered: boolean | null

        For an agent label: whether the labelled agent was among the options (decision) or the catalog (baseline). Null otherwise.

      • LatencyMs: number | null

        Wall-clock time of the decision call alone, or of the search for the baseline, in milliseconds.

      • MinConfidence: number | null

        The calibrated threshold WouldInject was judged at (decision arm), or null.

      • Model:
            | {
                AnsweredByPinned: boolean
                | null;
                AnsweredModelId: string | null;
                AnsweredModelName: string | null;
                AnsweredVendorId: string | null;
                AnsweredVendorName: string | null;
                DriverClass: string | null;
                FailedOver: boolean;
                FailoverAllowed: boolean;
                PinnedModelId: string | null;
                PinnedVendorId: string | null;
                ResolvedModel: string | null;
            }
            | null

        The model record (decision arm), or null.

      • Options:
            | {
                CatalogSize: number;
                Count: number;
                DeclaredCap: number
                | null;
                Limit: number;
                NarrowedFrom: number | null;
                WithoutDescription: number;
            }
            | null

        The decision arm's options, or null.

      • PromptName: string | null

        The decision prompt, or null for the baseline.

      • PromptRunId: string | null
      • Sampling: {
            Applied: boolean;
            Note: string | null;
            RequestedSeed: number | null;
            RequestedTemperature: number | null;
        }
      • VerdictReason: string | null

        Why nothing would be injected, or null.

      • WithinProductionTimeout: boolean | null

        Whether the discovery (DiscoveryLatencyMs) finished within production's timeout, or null without an answer. Production gives up at the timeout and injects nothing, so a late answer is never WouldInject. In runs recorded before DiscoveryLatencyMs, it timed the call alone.

      • WouldInject: boolean | null

        Whether production would suggest the chosen agent, or null without a usable answer.

    Settings

    Member Visibility