Interface ComputerUseExpectedOutcomes

Expected outcomes for a Computer Use test. Stored as JSON in TestEntity.ExpectedOutcomes.

interface ComputerUseExpectedOutcomes {
    finalUrlPattern?: string;
    goalCompleted?: boolean;
    judgeValidationCriteria?: string[];
    maxSteps?: number;
    minConfidence?: number;
}

Properties

finalUrlPattern?: string

Regex pattern that the final URL must match. Tested with new RegExp(pattern).test(finalUrl).

goalCompleted?: boolean

Whether the goal should be completed (default: true)

judgeValidationCriteria?: string[]

Custom validation criteria for LLM judge oracles

maxSteps?: number

Maximum number of steps allowed (exceeding = fail)

minConfidence?: number

Minimum judge confidence for a passing result (0.0 to 1.0)