Protected_Metadata provider used by the driver for entity access. Set explicitly via the
Provider setter; falls back to the global Metadata.Provider when not set.
The engine (or test harness) should set this to thread a transaction-scoped provider.
Protected_The metadata provider this driver uses. Falls back to the global Metadata.Provider
when the engine hasn't injected an explicit provider — multi-tenant servers should
always set this so each driver run binds to the right database connection.
ProtectedcalculateProtectedCalculate overall score from oracle results.
If weights are provided, calculates weighted average. Otherwise, calculates simple average.
Results from oracle evaluations
Optionalweights: ScoringWeightsOptional scoring weights by oracle type
Overall score from 0.0 to 1.0
ProtectedcreateProtectedCreate a log message for the test execution log.
Log level
Log message
Optionalmetadata: Record<string, unknown>Optional metadata
TestLogMessage object
ProtecteddetermineProtectedDetermine overall test status from oracle results.
Test passes only if ALL oracles pass.
Results from oracle evaluations
'Passed' if all oracles passed, 'Failed' otherwise
Execute the test.
This is the main entry point for test execution. The driver should:
The base engine will handle:
Execution context including test, run, user, options
Promise resolving to execution result
ProtectedgetProtectedGet the effective timeout for a test.
Priority (highest to lowest):
The test entity
Optionalconfig: { maxExecutionTime?: number }Parsed configuration object (optional)
Timeout in milliseconds
ProtectedlogProtectedLog execution progress.
Log message
Whether to only log in verbose mode (default: false)
ProtectedlogProtectedLog errors.
Error message
Optionalerror: ErrorOptional error object
ProtectedlogProtectedLog a message to both the console (if verbose) and accumulate for test run log.
Driver execution context
Log level
Log message
Optionalmetadata: Record<string, unknown>Optional metadata
ProtectedparseProtectedParse and validate Configuration JSON.
Helper method for drivers to parse their configuration with type safety. Throws if configuration is invalid.
The configuration type
The test containing the configuration
Parsed configuration
ProtectedparseProtectedParse and validate ExpectedOutcomes JSON.
The expected outcomes type
The test containing the expected outcomes
Parsed expected outcomes
ProtectedparseProtectedParse and validate InputDefinition JSON.
The input definition type
The test containing the input definition
Parsed input definition
Suite-scoped setup. Runs ONCE after the suite run is created and before the
first test's Execute, when the test runs inside a suite (mj test suite).
Provision suite-shared fixtures here — discover/create users, create
Query/Category rows, refresh engine caches — and stash them on
context.Data / context.CreatedRecords so every Execute of the suite
(which receives this same context via DriverExecutionContext.fixtures) can
read them, and TeardownSuite can clean them up.
Default is a no-op, so existing drivers (AgentEval, Computer Use) are
unaffected. Does NOT fire for the standalone mj test run path (no suite).
Per-suite-run fixture bag, keyed by SuiteRunID
User context for data access
Whether this driver supports cancellation via AbortSignal.
Drivers should override this to return true if they properly handle cancellation tokens. When a driver doesn't support cancellation, timeout will still mark the test as failed but the underlying execution may continue in the background.
true if driver supports cancellation, false otherwise
Suite-scoped teardown. GUARANTEED by TestEngine.RunSuite via a finally, so
it runs on pass, fail, a thrown Execute, and after a 'Timeout' result.
MUST be best-effort: never throw (the engine logs and ignores any throw),
only clean up. Delete what SetupSuite created (the context.CreatedRecords
list and any driver-specific payload under context.Data).
Default is a no-op. Does NOT fire for the standalone mj test run path.
The same per-suite-run fixture bag passed to SetupSuite
User context for data access
Validate test configuration.
Called when creating or updating a test to ensure the configuration is valid for this test type. Override to add type-specific validation.
The test being validated
Validation result with errors and warnings
Runs ONE prompt and evaluates the single decision it produced.
This is the corpus workhorse (test plan §3.1). Given frozen mid-loop state — a payload, a conversation, a system prompt — it asks the model for exactly one decision and hands the raw turn to the decision oracles. No action ever executes.
AIPromptRunner.ExecutePromptreturns the model's reply and nothing is dispatched, which is what makes the evaluation a plain data assertion rather than a side-effecting integration test.Why a separate driver rather than reusing
AgentEvalDriver: an agent run is a loop, and a loop confounds the measurement. If the corpus is asking "does this model, in this state, choose action B", then a run that recovered on iteration three answers a different question. Loop-level cases still go throughAgentEvalDriver; this driver isolates the single decision.Three execution details are load-bearing and easy to get wrong:
contextUser— the CLI provider'sCurrentUseris null (#3251), so an omitted context user fails deep inside template rendering with an unrelated-looking error;AIEngine.Config()before the first run — otherwise prompt/model metadata is empty;WaitForPendingPromptRunSaves()before oracles read back — prompt-run persistence is fire-and-forget, so an oracle asserting onAIPromptRunwould race it.