Class IntegrationTestDriver

Abstract base class for test driver implementations.

Each TestType in the database has a corresponding DriverClass that extends this base. The driver is responsible for:

Parsing test-specific configuration from Configuration JSON
Executing the test with appropriate logic
Running oracles to evaluate results
Calculating scores and determining pass/fail status
Returning structured results

BaseTestDriver handles common functionality:

Configuration parsing
Score calculation
Status determination
Logging
Error handling

Follows pattern from BaseScheduledJob and BaseAgent.

Example

@RegisterClass(BaseTestDriver, 'AgentEvalDriver')
export class AgentEvalDriver extends BaseTestDriver {
    async Execute(context: DriverExecutionContext): Promise<DriverExecutionResult> {
        const config = this.parseConfig<AgentEvalConfig>(context.test);
        // Execute test logic
        return result;
    }
}

Hierarchy (View Summary)

BaseTestDriver
- IntegrationTestDriver

Index

Constructors

constructor

new IntegrationTestDriver(): IntegrationTestDriver
Returns IntegrationTestDriver
Inherited from BaseTestDriver.constructor

Properties

`Protected`_provider

_provider: IMetadataProvider | null

Metadata provider used by the driver for entity access. Set explicitly via the Provider setter; falls back to the global Metadata.Provider when not set. The engine (or test harness) should set this to thread a transaction-scoped provider.

Accessors

`Protected`_metadata

get _metadata(): Metadata
Returns Metadata
Deprecated
Use _provider (IMetadataProvider) instead. Retained for backward compatibility with subclasses that referenced the previous _metadata: Metadata field directly. Returns the same underlying provider cast to the legacy Metadata shape.
Inherited from BaseTestDriver._metadata
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:68

Provider

get Provider(): IMetadataProvider
The metadata provider this driver uses. Falls back to the global Metadata.Provider when the engine hasn't injected an explicit provider — multi-tenant servers should always set this so each driver run binds to the right database connection.

Returns IMetadataProvider
Inherited from BaseTestDriver.Provider
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:57
set Provider(value: IMetadataProvider | null): void
Parameters
- value: IMetadataProvider | null
Returns void
Inherited from BaseTestDriver.Provider
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:58

Methods

`Protected`calculateScore

calculateScore(oracleResults: OracleResult[], weights?: ScoringWeights): number
Protected
Calculate overall score from oracle results.

If weights are provided, calculates weighted average. Otherwise, calculates simple average.
Parameters
- oracleResults: OracleResult[]
  Results from oracle evaluations
- Optionalweights: ScoringWeights
  Optional scoring weights by oracle type
Returns number
Overall score from 0.0 to 1.0
Inherited from BaseTestDriver.calculateScore
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:110

`Protected`createLogMessage

createLogMessage(
    level: "debug" | "error" | "info" | "warn",
    message: string,
    metadata?: Record<string, unknown>,
): TestLogMessage
Protected
Create a log message for the test execution log.
Parameters
- level: "debug" | "error" | "info" | "warn"
  Log level
- message: string
  Log message
- Optionalmetadata: Record<string, unknown>
  Optional metadata
Returns TestLogMessage
TestLogMessage object
Inherited from BaseTestDriver.createLogMessage
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:235

`Protected`determineStatus

determineStatus(oracleResults: OracleResult[]): "Passed" | "Failed"
Protected
Determine overall test status from oracle results.

Test passes only if ALL oracles pass.
Parameters
- oracleResults: OracleResult[]
  Results from oracle evaluations
Returns "Passed" | "Failed"
'Passed' if all oracles passed, 'Failed' otherwise
Inherited from BaseTestDriver.determineStatus
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:120

Execute

Execute(context: DriverExecutionContext): Promise<DriverExecutionResult>
Execute the test.

This is the main entry point for test execution. The driver should:
1. Parse Configuration, InputDefinition, ExpectedOutcomes from test entity
2. Perform test-specific execution (e.g., run agent, execute workflow)
3. Run oracles to evaluate results
4. Calculate score and determine status
5. Return structured DriverExecutionResult
The base engine will handle:
- Creating/updating TestRun entity
- Logging to database
- Error handling
- Timing and cost tracking
Parameters
- context: DriverExecutionContext
  Execution context including test, run, user, options
Returns Promise<DriverExecutionResult>
Promise resolving to execution result
Overrides BaseTestDriver.Execute
- Defined in TestingFramework/testing-integration/src/IntegrationTestDriver.ts:113

`Protected`getEffectiveTimeout

getEffectiveTimeout(
test: MJTestEntity,
config?: { maxExecutionTime?: number },
): number
Protected
Get the effective timeout for a test.

Priority (highest to lowest):
1. Configuration JSON maxExecutionTime field (backward compatibility)
2. Test.MaxExecutionTimeMS column
3. DEFAULT_TEST_TIMEOUT_MS constant (5 minutes)
Parameters
- test: MJTestEntity
  The test entity
- Optionalconfig: { maxExecutionTime?: number }
  Parsed configuration object (optional)
Returns number
Timeout in milliseconds
Inherited from BaseTestDriver.getEffectiveTimeout
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:223

`Protected`log

log(message: string, verboseOnly?: boolean): void
Protected
Log execution progress.
Parameters
- message: string
  Log message
- OptionalverboseOnly: boolean
  Whether to only log in verbose mode (default: false)
Returns void
Inherited from BaseTestDriver.log
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:161

`Protected`logError

logError(message: string, error?: Error): void
Protected
Log errors.
Parameters
- message: string
  Error message
- Optionalerror: Error
  Optional error object
Returns void
Inherited from BaseTestDriver.logError
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:169

`Protected`logToTestRun

logToTestRun(
    context: DriverExecutionContext,
    level: "debug" | "error" | "info" | "warn",
    message: string,
    metadata?: Record<string, unknown>,
): void
Protected
Log a message to both the console (if verbose) and accumulate for test run log.
Parameters
- context: DriverExecutionContext
  Driver execution context
- level: "debug" | "error" | "info" | "warn"
  Log level
- message: string
  Log message
- Optionalmetadata: Record<string, unknown>
  Optional metadata
Returns void
Inherited from BaseTestDriver.logToTestRun
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:245

`Protected`parseConfig

parseConfig<T>(test: MJTestEntity): T
Protected
Parse and validate Configuration JSON.

Helper method for drivers to parse their configuration with type safety. Throws if configuration is invalid.
Type Parameters
- T
  The configuration type
Parameters
- test: MJTestEntity
  The test containing the configuration
Returns T
Parsed configuration

Throws
Error if configuration is missing or invalid
Inherited from BaseTestDriver.parseConfig
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:133

`Protected`parseExpectedOutcomes

parseExpectedOutcomes<T>(test: MJTestEntity): T
Protected
Parse and validate ExpectedOutcomes JSON.
Type Parameters
- T
  The expected outcomes type
Parameters
- test: MJTestEntity
  The test containing the expected outcomes
Returns T
Parsed expected outcomes

Throws
Error if expected outcomes is missing or invalid
Inherited from BaseTestDriver.parseExpectedOutcomes
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:153

`Protected`parseInputDefinition

parseInputDefinition<T>(test: MJTestEntity): T
Protected
Parse and validate InputDefinition JSON.
Type Parameters
- T
  The input definition type
Parameters
- test: MJTestEntity
  The test containing the input definition
Returns T
Parsed input definition

Throws
Error if input definition is missing or invalid
Inherited from BaseTestDriver.parseInputDefinition
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:143

SetProvider

SetProvider(provider: IMetadataProvider): void
Parameters
- provider: IMetadataProvider
Returns void
Deprecated
Use the Provider setter instead. Retained for backward compatibility.
Inherited from BaseTestDriver.SetProvider
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:62

SetupSuite

SetupSuite(context: SuiteFixtureContext, _contextUser: UserInfo): Promise<void>
Suite-scoped setup (D6). Discovers the two-user RLS fixture ONCE per suite run and stashes it on the shared SuiteFixtureContext so every test of the suite (and the rls-isolation bundle's checks) reads the same pair. Best-effort: discovery has nothing to delete, so failures are non-fatal — Execute lazily re-discovers when the fixture is absent (e.g. the mj test run path has no suite). Only meaningful on the server transport (the user cache is a SQLServerDataProvider concept).
Parameters
- context: SuiteFixtureContext
- _contextUser: UserInfo
Returns Promise<void>
Overrides BaseTestDriver.SetupSuite
- Defined in TestingFramework/testing-integration/src/IntegrationTestDriver.ts:76

supportsCancellation

supportsCancellation(): boolean
The driver arms its own timeout and breaks the check loop when it fires.

Returns boolean
Overrides BaseTestDriver.supportsCancellation
- Defined in TestingFramework/testing-integration/src/IntegrationTestDriver.ts:64

TeardownSuite

TeardownSuite(
_context: SuiteFixtureContext,
_contextUser: UserInfo,
): Promise<void>
Suite-scoped teardown (D6). No-op: the RLS fixture is DISCOVERED (not created), so there is nothing to delete — the safest possible teardown. Present so the engine's suite-hook lifecycle is exercised end-to-end by this driver.
Parameters
- _context: SuiteFixtureContext
- _contextUser: UserInfo
Returns Promise<void>
Overrides BaseTestDriver.TeardownSuite
- Defined in TestingFramework/testing-integration/src/IntegrationTestDriver.ts:92

Validate

Validate(test: MJTestEntity): Promise<ValidationResult>
Validate test configuration.

Called when creating or updating a test to ensure the configuration is valid for this test type. Override to add type-specific validation.
Parameters
- test: MJTestEntity
  The test being validated
Returns Promise<ValidationResult>
Validation result with errors and warnings
Inherited from BaseTestDriver.Validate
- Defined in TestingFramework/Engine/dist/drivers/BaseTestDriver.d.ts:98

Class IntegrationTestDriver

Example

Hierarchy (View Summary)

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

Returns IntegrationTestDriver

Properties

Protected_provider

Accessors

Protected_metadata

Returns Metadata

Deprecated

Provider

Returns IMetadataProvider

Parameters

Returns void

Methods

ProtectedcalculateScore

Parameters

Returns number

ProtectedcreateLogMessage

Parameters

Returns TestLogMessage

ProtecteddetermineStatus

Parameters

Returns "Passed" | "Failed"

Execute

Parameters

Returns Promise<DriverExecutionResult>

ProtectedgetEffectiveTimeout

Parameters

Returns number

Protectedlog

Parameters

Returns void

ProtectedlogError

Parameters

Returns void

ProtectedlogToTestRun

Parameters

Returns void

ProtectedparseConfig

Type Parameters

Parameters

Returns T

Throws

ProtectedparseExpectedOutcomes

Type Parameters

Parameters

Returns T

Throws

ProtectedparseInputDefinition

Type Parameters

Parameters

Returns T

Throws

SetProvider

Parameters

Returns void

Deprecated

SetupSuite

Parameters

Returns Promise<void>

supportsCancellation

Returns boolean

TeardownSuite

Parameters

Returns Promise<void>

Validate

Parameters

Returns Promise<ValidationResult>

Settings

On This Page

`Protected`_provider

`Protected`_metadata

`Protected`calculateScore

`Protected`createLogMessage

`Protected`determineStatus

`Protected`getEffectiveTimeout

`Protected`log

`Protected`logError

`Protected`logToTestRun

`Protected`parseConfig

`Protected`parseExpectedOutcomes

`Protected`parseInputDefinition