Member Junction
    Preparing search index...

    Class AgentEvalDriver

    Test driver for AI Agent evaluation.

    Executes an AI agent with test input, runs configured oracles, and creates bidirectional link between TestRun and AgentRun.

    // Configuration JSON in Test entity
    {
      "agentId": "agent-123",
      "oracles": [
        { "type": "trace-no-errors", "weight": 0.2 },
        { "type": "llm-judge", "weight": 0.5, "config": { "criteria": [...] } },
        { "type": "schema-validate", "weight": 0.3, "config": { "schema": {...} } }
      ],
      "scoringWeights": { "trace-no-errors": 0.2, "llm-judge": 0.5, "schema-validate": 0.3 }
    }

    // InputDefinition JSON in Test entity
    {
      "userMessage": "Create a report showing sales by region",
      "conversationContext": null,
      "executionParams": { "temperatureOverride": 0.3 }
    }

    // ExpectedOutcomes JSON in Test entity
    {
      "responsePatterns": ["sales.*region", "chart|graph"],
      "expectedEntities": ["Report", "Dashboard"],
      "responseSchema": { "type": "object", "properties": {...} },
      "judgeValidationCriteria": [
        "Response accurately answers the user's question",
        "Report includes proper data visualization",
        "Response is professional and clear"
      ]
    }

    Hierarchy (View Summary)

    Index

    Constructors

    constructor

    Properties

    _provider

    Accessors

    _metadata Provider

    Methods

    calculateScore createLogMessage determineStatus Execute getEffectiveTimeout log logError logToTestRun parseConfig parseExpectedOutcomes parseInputDefinition SetProvider SetupSuite supportsCancellation TeardownSuite Validate

    Constructors

    Properties

    Protected_provider

    _provider: IMetadataProvider | null = null

    Metadata provider used by the driver for entity access. Set explicitly via the Provider setter; falls back to the global Metadata.Provider when not set. The engine (or test harness) should set this to thread a transaction-scoped provider.

    Accessors

    Protected_metadata

    Methods

    • Protected

      Get the effective timeout for a test.

      Priority (highest to lowest):

      1. Configuration JSON maxExecutionTime field (backward compatibility)
      2. Test.MaxExecutionTimeMS column
      3. DEFAULT_TEST_TIMEOUT_MS constant (5 minutes)

      Parameters

      • test: MJTestEntity

        The test entity

      • Optionalconfig: { maxExecutionTime?: number }

        Parsed configuration object (optional)

      Returns number

      Timeout in milliseconds

    • Suite-scoped setup. Runs ONCE after the suite run is created and before the first test's Execute, when the test runs inside a suite (mj test suite). Provision suite-shared fixtures here — discover/create users, create Query/Category rows, refresh engine caches — and stash them on context.Data / context.CreatedRecords so every Execute of the suite (which receives this same context via DriverExecutionContext.fixtures) can read them, and TeardownSuite can clean them up.

      Default is a no-op, so existing drivers (AgentEval, Computer Use) are unaffected. Does NOT fire for the standalone mj test run path (no suite).

      Parameters

      Returns Promise<void>

    • Suite-scoped teardown. GUARANTEED by TestEngine.RunSuite via a finally, so it runs on pass, fail, a thrown Execute, and after a 'Timeout' result. MUST be best-effort: never throw (the engine logs and ignores any throw), only clean up. Delete what SetupSuite created (the context.CreatedRecords list and any driver-specific payload under context.Data).

      Default is a no-op. Does NOT fire for the standalone mj test run path.

      Parameters

      Returns Promise<void>

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Accessors
    Methods