PrimaryValue(
output: string,
value: unknown,
resolved: ResolvedConstraint,
orderBy: string,
): string
Protected_Metadata provider used by the driver for entity access. Set explicitly via the
Provider setter; falls back to the global Metadata.Provider when not set.
The engine (or test harness) should set this to thread a transaction-scoped provider.
Protected_The metadata provider this driver uses. Falls back to the global Metadata.Provider
when the engine hasn't injected an explicit provider — multi-tenant servers should
always set this so each driver run binds to the right database connection.
ProtectedAnswerRuns every case through the pipeline in one batch and reads each answer's comparable values and the batch's cost. A record rejected for breaking its constraints is a wrong answer; any other failure fails the evaluation.
ProtectedAnswerThe comparable value of each output in a result payload. unknown because the payload is the model's structured
answer, whose shape the pipeline's spec declares at run time; each output is read at its spec Ref.
ProtectedBuildOne oracle per output (its value against the threshold), one for the required cases (when the set has any), plus the activation outcome, which decides the status.
ProtectedBuildA dry run (no writes) with the cache bypassed: the evaluation measures the model, not cached answers.
ProtectedcalculateProtectedCalculate overall score from oracle results.
If weights are provided, calculates weighted average. Otherwise, calculates simple average.
Results from oracle evaluations
Optionalweights: ScoringWeightsOptional scoring weights by oracle type
Overall score from 0.0 to 1.0
ProtectedcreateProtectedCreate a log message for the test execution log.
Log level
Log message
Optionalmetadata: Record<string, unknown>Optional metadata
TestLogMessage object
ProtecteddetermineProtectedDetermine overall test status from oracle results.
Test passes only if ALL oracles pass.
Results from oracle evaluations
'Passed' if all oracles passed, 'Failed' otherwise
ProtectedDisplayThe display name the constraint shows the model for a value, else the value: the text before the first " — " of the lookup's display text (its first display field).
ProtectedEvaluateRuns the cases under the pipeline's current pin, scores them and stores the score on that pin's version.
Evaluates the pipeline on its golden set; every reason not to store a result is one Error result.
ProtectedgetProtectedGet the effective timeout for a test.
Priority (highest to lowest):
The test entity
Optionalconfig: { maxExecutionTime?: number }Parsed configuration object (optional)
Timeout in milliseconds
ProtectedLoadThe pipeline's MJ: Record Processes row, read on this driver's provider; null when it does not exist.
ProtectedlogProtectedLog execution progress.
Log message
OptionalverboseOnly: booleanWhether to only log in verbose mode (default: false)
ProtectedlogProtectedLog errors.
Error message
Optionalerror: ErrorOptional error object
ProtectedlogProtectedLog a message to both the console (if verbose) and accumulate for test run log.
Driver execution context
Log level
Log message
Optionalmetadata: Record<string, unknown>Optional metadata
ProtectedparseProtectedParse and validate Configuration JSON.
Helper method for drivers to parse their configuration with type safety. Throws if configuration is invalid.
The configuration type
The test containing the configuration
Parsed configuration
ProtectedparseProtectedParse and validate ExpectedOutcomes JSON.
The expected outcomes type
The test containing the expected outcomes
Parsed expected outcomes
ProtectedparseProtectedParse and validate InputDefinition JSON.
The input definition type
The test containing the input definition
Parsed input definition
ProtectedPreflightWhy a real run of the pipeline would not compute at all; null when it would. The processor's read-only Preflight on the dry-run context, where the version gate's refusal is only a warning (evaluating is how it is cleared).
ProtectedPrepareParses the test, loads its pipeline and checks that the pipeline names this test as its golden set and that the labels may gate it, before any model call.
ProtectedPrimaryAn output's value as text; for an array, the item with the lowest orderBy value (else the first item), read at
the constraint's ItemRef. unknown: part of the model's answer, as in AnswerValues.
Suite-scoped setup. Runs ONCE after the suite run is created and before the
first test's Execute, when the test runs inside a suite (mj test suite).
Provision suite-shared fixtures here — discover/create users, create
Query/Category rows, refresh engine caches — and stash them on
context.Data / context.CreatedRecords so every Execute of the suite
(which receives this same context via DriverExecutionContext.fixtures) can
read them, and TeardownSuite can clean them up.
Default is a no-op, so existing drivers (AgentEval, Computer Use) are
unaffected. Does NOT fire for the standalone mj test run path (no suite).
Per-suite-run fixture bag, keyed by SuiteRunID
User context for data access
ProtectedStoreStores the score on the version of the pin the cases ran under, with the per-case answers and the label-quality record as its details.
Whether this driver supports cancellation via AbortSignal.
Drivers should override this to return true if they properly handle cancellation tokens. When a driver doesn't support cancellation, timeout will still mark the test as failed but the underlying execution may continue in the background.
true if driver supports cancellation, false otherwise
Suite-scoped teardown. GUARANTEED by TestEngine.RunSuite via a finally, so
it runs on pass, fail, a thrown Execute, and after a 'Timeout' result.
MUST be best-effort: never throw (the engine logs and ignores any throw),
only clean up. Delete what SetupSuite created (the context.CreatedRecords
list and any driver-specific payload under context.Data).
Default is a no-op. Does NOT fire for the standalone mj test run path.
The same per-suite-run fixture bag passed to SetupSuite
User context for data access
Validate test configuration.
Called when creating or updating a test to ensure the configuration is valid for this test type. Override to add type-specific validation.
The test being validated
Validation result with errors and warnings
Protected StaticAssertFails when a case's result is missing, skipped, or failed for a reason other than RejectedInvalid (a broken
constraint is the model's answer and is scored).
Protected StaticCompareOrders two array items by their numeric orderBy value, ascending; an item without one sorts last.
Protected StaticOrderAn item's numeric orderBy value; +Infinity when it is missing, empty or not a number.
StaticPromptWhat a batch's model calls cost: the sum of TotalCost over the distinct AI Prompt Runs the results name (records
that share a cache key share one prompt run); 0 when none does.
Protected StaticRequiredThe required cases' oracle: passed when none is wrong, else naming each wrong case and its wrong outputs.
Test driver of the "Feature Pipeline Golden Set" test type. Evaluates the version the pipeline would run now (its current pin) on the test's golden cases and stores the result on that version through the version registry, which activates it when the result meets the threshold the pipeline's
Activationdeclares, and refuses it otherwise.