Protected_Metadata provider used by the driver for entity access. Set explicitly via the
Provider setter; falls back to the global Metadata.Provider when not set.
The engine (or test harness) should set this to thread a transaction-scoped provider.
Protected_The metadata provider this driver uses. Falls back to the global Metadata.Provider
when the engine hasn't injected an explicit provider — multi-tenant servers should
always set this so each driver run binds to the right database connection.
ProtectedcalculateProtectedCalculate overall score from oracle results.
If weights are provided, calculates weighted average. Otherwise, calculates simple average.
Results from oracle evaluations
Optionalweights: ScoringWeightsOptional scoring weights by oracle type
Overall score from 0.0 to 1.0
ProtectedcreateProtectedCreate a log message for the test execution log.
Log level
Log message
Optionalmetadata: Record<string, unknown>Optional metadata
TestLogMessage object
ProtecteddetermineProtectedDetermine overall test status from oracle results.
Test passes only if ALL oracles pass.
Results from oracle evaluations
'Passed' if all oracles passed, 'Failed' otherwise
Execute the test.
This is the main entry point for test execution. The driver should:
The base engine will handle:
Execution context including test, run, user, options
Promise resolving to execution result
ProtectedgetProtectedGet the effective timeout for a test.
Priority (highest to lowest):
The test entity
Optionalconfig: { maxExecutionTime?: number }Parsed configuration object (optional)
Timeout in milliseconds
ProtectedLoadHuman and AI scores for the gold subjects, paired on the same major and criterion.
ProtectedlogProtectedLog execution progress.
Log message
Whether to only log in verbose mode (default: false)
ProtectedlogProtectedLog errors.
Error message
Optionalerror: ErrorOptional error object
ProtectedlogProtectedLog a message to both the console (if verbose) and accumulate for test run log.
Driver execution context
Log level
Log message
Optionalmetadata: Record<string, unknown>Optional metadata
ProtectedparseProtectedParse and validate Configuration JSON.
Helper method for drivers to parse their configuration with type safety. Throws if configuration is invalid.
The configuration type
The test containing the configuration
Parsed configuration
ProtectedparseProtectedParse and validate ExpectedOutcomes JSON.
The expected outcomes type
The test containing the expected outcomes
Parsed expected outcomes
ProtectedparseProtectedParse and validate InputDefinition JSON.
The input definition type
The test containing the input definition
Parsed input definition
ProtectedScoreRuns the named evaluator for one gold subject, in this test's context. A subclass can replace this in a test.
Suite-scoped setup. Runs ONCE after the suite run is created and before the
first test's Execute, when the test runs inside a suite (mj test suite).
Provision suite-shared fixtures here — discover/create users, create
Query/Category rows, refresh engine caches — and stash them on
context.Data / context.CreatedRecords so every Execute of the suite
(which receives this same context via DriverExecutionContext.fixtures) can
read them, and TeardownSuite can clean them up.
Default is a no-op, so existing drivers (AgentEval, Computer Use) are
unaffected. Does NOT fire for the standalone mj test run path (no suite).
Per-suite-run fixture bag, keyed by SuiteRunID
User context for data access
Whether this driver supports cancellation via AbortSignal.
Drivers should override this to return true if they properly handle cancellation tokens. When a driver doesn't support cancellation, timeout will still mark the test as failed but the underlying execution may continue in the background.
true if driver supports cancellation, false otherwise
Suite-scoped teardown. GUARANTEED by TestEngine.RunSuite via a finally, so
it runs on pass, fail, a thrown Execute, and after a 'Timeout' result.
MUST be best-effort: never throw (the engine logs and ignores any throw),
only clean up. Delete what SetupSuite created (the context.CreatedRecords
list and any driver-specific payload under context.Data).
Default is a no-op. Does NOT fire for the standalone mj test run path.
The same per-suite-run fixture bag passed to SetupSuite
User context for data access
Validate test configuration.
Called when creating or updating a test to ensure the configuration is valid for this test type. Override to add type-specific validation.
The test being validated
Validation result with errors and warnings
Scores a gold set with the AI evaluator and checks agreement with the submitted human scores on the same rubric major. The test score is the overall quadratic-weighted kappa, clamped to 0..1.