Member Junction
    Preparing search index...

    Class DecisionEvalDriver

    Runs ONE typed decision on a corpus point and scores it against the point's label.

    The point is mapped to the routing decision's input, and the questions and state are built, with the same builders the chat uses (@memberjunction/ai-core-plus), so the eval sends exactly what production sends. One AIDecisionRunner.ExecuteDecision answers it, and the oracles score the answer. The prompt run is linked as the test run's target, so cost comes from it.

    Pinning. A cell that names modelId and/or vendorId pins them through the runner's own mechanism, AIDecisionParams.override, and runs with failover off (PinnedDecisionRunner) unless the cell sets failover: true. The model that answered is recorded; a pinned run answered by another model or vendor is flagged, and when failover was off it is an Error, not a data point.

    Sampling. temperature and seed are recorded but do not reach the model: the decision path has no parameter for them (see SAMPLING_NOT_APPLIED).

    Agent discovery. A test whose Configuration.decision is agent-discovery asks production's discovery decision instead (plan Task 3.1): the options come from BuildDecisionDiscoveryOptionSet over the agents the run's user may run, with the real semantic search, and the questions from BuildDecisionDiscoveryQuestions, both from @memberjunction/ai-agents, the code BaseAgent runs. Production gives up after DECISION_DISCOVERY_TIMEOUT_MS, counted from building the options, so the driver times the same span and records an answer that came later as not injected. The call itself runs to the end, so the late answer is still recorded. A semantic-search baseline cell makes no decision call: it records what Find Candidate Agents would have listed first.

    Never throws. A bad test, a missing prompt and a failed decision come back as an Error run with the reason. A point or request with nothing to decide is Skipped: production makes no call there.

    Hierarchy (View Summary)

    Index

    Constructors

    constructor

    Properties

    _provider

    Accessors

    _metadata Provider

    Methods

    calculateScore createLogMessage CreateRunner determineStatus Execute getEffectiveTimeout LoadDiscoveryEnvironment LoadEnvironment log logError logToTestRun parseConfig parseExpectedOutcomes parseInputDefinition SetProvider SetupSuite supportsCancellation TeardownSuite Validate

    Constructors

    Properties

    Protected_provider

    _provider: IMetadataProvider | null = null

    Metadata provider used by the driver for entity access. Set explicitly via the Provider setter; falls back to the global Metadata.Provider when not set. The engine (or test harness) should set this to thread a transaction-scoped provider.

    Accessors

    Protected_metadata

    Methods

    • Protected

      Get the effective timeout for a test.

      Priority (highest to lowest):

      1. Configuration JSON maxExecutionTime field (backward compatibility)
      2. Test.MaxExecutionTimeMS column
      3. DEFAULT_TEST_TIMEOUT_MS constant (5 minutes)

      Parameters

      • test: MJTestEntity

        The test entity

      • Optionalconfig: { maxExecutionTime?: number }

        Parsed configuration object (optional)

      Returns number

      Timeout in milliseconds

    • Loads what one discovery run needs, the way production's discovery loads it: AIEngine, the decision prompt by name (not for a baseline cell, which makes no call), the conversation manager, the agents the run's user may run, the prompt's option cap, and the semantic search when the provider can run it. Throws when the prompt or the conversation manager is missing. Overridable, so a test can run the driver without an engine or a provider.

      Parameters

      • config: {
            baseline?: "semantic-search" | null;
            decision: "agent-discovery";
            failover?: boolean | null;
            modelId?: string | null;
            oracles: {
                config?: Record<string, unknown>;
                type: string;
                weight?: number;
            }[];
            promptName?: string
            | null;
            seed?: number | null;
            temperature?: number | null;
            vendorId?: string | null;
        }
        • Optionalbaseline?: "semantic-search" | null

          Runs this baseline instead of the decision: no decision call is made.

        • decision: "agent-discovery"
        • Optionalfailover?: boolean | null

          Whether the runner may fail over. Defaults to false when the cell pins a model or vendor.

        • OptionalmodelId?: string | null

          Pins the MJ: AI Models row that must answer.

        • oracles: { config?: Record<string, unknown>; type: string; weight?: number }[]

          The oracles to run.

        • OptionalpromptName?: string | null

          The decision prompt. Defaults to Default Decision, the prompt production's discovery asks.

        • Optionalseed?: number | null

          Requested sampling seed. Recorded, not applied: see the driver.

        • Optionaltemperature?: number | null

          Requested sampling temperature. Recorded, not applied: see the driver.

        • OptionalvendorId?: string | null

          Pins the vendor.

      • context: DriverExecutionContext

      Returns Promise<DiscoveryEvalEnvironment>

    • Loads what one run needs: AIEngine, the decision prompt by name, the agent catalog and the conversation manager. Throws when the prompt is missing. Overridable, so a test can run the driver without an engine.

      Parameters

      • config: {
            decision: "conversation-routing";
            failover?: boolean | null;
            modelId?: string | null;
            oracles: {
                config?: Record<string, unknown>;
                type: string;
                weight?: number;
            }[];
            promptName?: string
            | null;
            seed?: number | null;
            stateLayout: "production" | "structured";
            temperature?: number | null;
            vendorId?: string | null;
        }
        • decision: "conversation-routing"

          Which decision builder. A discovery test has its own schema (DiscoveryEvalConfigSchema).

        • Optionalfailover?: boolean | null

          Whether the runner may fail over. Defaults to false when the cell pins a model or vendor.

        • OptionalmodelId?: string | null

          Pins the MJ: AI Models row that must answer.

        • oracles: { config?: Record<string, unknown>; type: string; weight?: number }[]

          The oracles to run.

        • OptionalpromptName?: string | null

          The decision prompt. Defaults to DEFAULT_DECISION_PROMPT_NAME.

        • Optionalseed?: number | null

          Requested sampling seed. Recorded; see the driver for whether it reaches the model.

        • stateLayout: "production" | "structured"

          production sends BuildRoutingState; structured sends BuildRoutingStateStructured.

        • Optionaltemperature?: number | null

          Requested sampling temperature. Recorded; see the driver for whether it reaches the model.

        • OptionalvendorId?: string | null

          Pins the vendor.

      • context: DriverExecutionContext

      Returns Promise<DecisionEvalEnvironment>

    • Suite-scoped setup. Runs ONCE after the suite run is created and before the first test's Execute, when the test runs inside a suite (mj test suite). Provision suite-shared fixtures here — discover/create users, create Query/Category rows, refresh engine caches — and stash them on context.Data / context.CreatedRecords so every Execute of the suite (which receives this same context via DriverExecutionContext.fixtures) can read them, and TeardownSuite can clean them up.

      Default is a no-op, so existing drivers (AgentEval, Computer Use) are unaffected. Does NOT fire for the standalone mj test run path (no suite).

      Parameters

      Returns Promise<void>

    • Whether this driver supports cancellation via AbortSignal.

      Drivers should override this to return true if they properly handle cancellation tokens. When a driver doesn't support cancellation, timeout will still mark the test as failed but the underlying execution may continue in the background.

      Returns boolean

      true if driver supports cancellation, false otherwise

    • Suite-scoped teardown. GUARANTEED by TestEngine.RunSuite via a finally, so it runs on pass, fail, a thrown Execute, and after a 'Timeout' result. MUST be best-effort: never throw (the engine logs and ignores any throw), only clean up. Delete what SetupSuite created (the context.CreatedRecords list and any driver-specific payload under context.Data).

      Default is a no-op. Does NOT fire for the standalone mj test run path.

      Parameters

      Returns Promise<void>

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Accessors
    Methods