Member Junction
    Preparing search index...

    Class BaseTestDriverAbstract

    Abstract base class for test driver implementations.

    Each TestType in the database has a corresponding DriverClass that extends this base. The driver is responsible for:

    • Parsing test-specific configuration from Configuration JSON
    • Executing the test with appropriate logic
    • Running oracles to evaluate results
    • Calculating scores and determining pass/fail status
    • Returning structured results

    BaseTestDriver handles common functionality:

    • Configuration parsing
    • Score calculation
    • Status determination
    • Logging
    • Error handling

    Follows pattern from BaseScheduledJob and BaseAgent.

    @RegisterClass(BaseTestDriver, 'AgentEvalDriver')
    export class AgentEvalDriver extends BaseTestDriver {
        async Execute(context: DriverExecutionContext): Promise<DriverExecutionResult> {
            const config = this.parseConfig<AgentEvalConfig>(context.test);
            // Execute test logic
            return result;
        }
    }

    Hierarchy (View Summary)

    Index

    Constructors

    constructor

    Properties

    _provider

    Accessors

    _metadata Provider

    Methods

    calculateScore createLogMessage determineStatus Execute getEffectiveTimeout log logError logToTestRun parseConfig parseExpectedOutcomes parseInputDefinition SetProvider SetupSuite supportsCancellation TeardownSuite Validate

    Constructors

    Properties

    Protected_provider

    _provider: IMetadataProvider | null = null

    Metadata provider used by the driver for entity access. Set explicitly via the Provider setter; falls back to the global Metadata.Provider when not set. The engine (or test harness) should set this to thread a transaction-scoped provider.

    Accessors

    Protected_metadata

    Methods

    • Protected

      Create a log message for the test execution log.

      Parameters

      • level: "info" | "warn" | "error" | "debug"

        Log level

      • message: string

        Log message

      • Optionalmetadata: Record<string, unknown>

        Optional metadata

      Returns TestLogMessage

      TestLogMessage object

    • Execute the test.

      This is the main entry point for test execution. The driver should:

      1. Parse Configuration, InputDefinition, ExpectedOutcomes from test entity
      2. Perform test-specific execution (e.g., run agent, execute workflow)
      3. Run oracles to evaluate results
      4. Calculate score and determine status
      5. Return structured DriverExecutionResult

      The base engine will handle:

      • Creating/updating TestRun entity
      • Logging to database
      • Error handling
      • Timing and cost tracking

      Parameters

      Returns Promise<DriverExecutionResult>

      Promise resolving to execution result

    • Protected

      Get the effective timeout for a test.

      Priority (highest to lowest):

      1. Configuration JSON maxExecutionTime field (backward compatibility)
      2. Test.MaxExecutionTimeMS column
      3. DEFAULT_TEST_TIMEOUT_MS constant (5 minutes)

      Parameters

      • test: MJTestEntity

        The test entity

      • Optionalconfig: { maxExecutionTime?: number }

        Parsed configuration object (optional)

      Returns number

      Timeout in milliseconds

    • Protected

      Log a message to both the console (if verbose) and accumulate for test run log.

      Parameters

      • context: DriverExecutionContext

        Driver execution context

      • level: "info" | "warn" | "error" | "debug"

        Log level

      • message: string

        Log message

      • Optionalmetadata: Record<string, unknown>

        Optional metadata

      Returns void

    • Suite-scoped setup. Runs ONCE after the suite run is created and before the first test's Execute, when the test runs inside a suite (mj test suite). Provision suite-shared fixtures here — discover/create users, create Query/Category rows, refresh engine caches — and stash them on context.Data / context.CreatedRecords so every Execute of the suite (which receives this same context via DriverExecutionContext.fixtures) can read them, and TeardownSuite can clean them up.

      Default is a no-op, so existing drivers (AgentEval, Computer Use) are unaffected. Does NOT fire for the standalone mj test run path (no suite).

      Parameters

      Returns Promise<void>

    • Whether this driver supports cancellation via AbortSignal.

      Drivers should override this to return true if they properly handle cancellation tokens. When a driver doesn't support cancellation, timeout will still mark the test as failed but the underlying execution may continue in the background.

      Returns boolean

      true if driver supports cancellation, false otherwise

    • Suite-scoped teardown. GUARANTEED by TestEngine.RunSuite via a finally, so it runs on pass, fail, a thrown Execute, and after a 'Timeout' result. MUST be best-effort: never throw (the engine logs and ignores any throw), only clean up. Delete what SetupSuite created (the context.CreatedRecords list and any driver-specific payload under context.Data).

      Default is a no-op. Does NOT fire for the standalone mj test run path.

      Parameters

      Returns Promise<void>

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Accessors
    Methods