Member Junction
    Preparing search index...

    Interface SuiteRunOptions

    Options for running a test suite

    interface SuiteRunOptions {
        agentEvaluationRubricId?: string;
        agentVersion?: string;
        configOverride?: Record<string, unknown>;
        delayBetweenTests?: number;
        dryRun?: boolean;
        environment?: string;
        failFast?: boolean;
        gitCommit?: string;
        logCallback?: (message: TestLogMessage) => void;
        maxParallel?: number;
        parallel?: boolean;
        progressCallback?: (progress: TestProgress) => void;
        publishedRubricVersionId?: string;
        repeatCountOverride?: number;
        rubricId?: string;
        rubricVersionId?: string;
        selectedTestIds?: string[];
        sequence?: number[];
        sequenceEnd?: number;
        sequenceStart?: number;
        shard?: { index: number; total: number };
        tags?: string;
        transport?: "server" | "client" | "all";
        variables?: TestRunVariables;
        verbose?: boolean;
    }

    Hierarchy (View Summary)

    Index

    Properties

    agentEvaluationRubricId? agentVersion? configOverride? delayBetweenTests? dryRun? environment? failFast? gitCommit? logCallback? maxParallel? parallel? progressCallback? publishedRubricVersionId? repeatCountOverride? rubricId? rubricVersionId? selectedTestIds? sequence? sequenceEnd? sequenceStart? shard? tags? transport? variables? verbose?

    Properties

    agentEvaluationRubricId?: string

    The target agent's default Evaluation rubric, used when nothing earlier names one.

    agentVersion?: string

    Agent/system version being tested

    configOverride?: Record<string, unknown>

    Override test configuration

    delayBetweenTests?: number

    Delay in milliseconds between test executions. Useful for avoiding rate limits (e.g., Auth0 brute-force protection) when tests perform repeated logins from the same IP.

    dryRun?: boolean

    Validate configuration without executing

    environment?: string

    Environment context (dev, staging, prod)

    failFast?: boolean

    Stop on first failure

    gitCommit?: string

    Git commit SHA for versioning

    logCallback?: (message: TestLogMessage) => void

    Log callback for streaming execution details to the test run log

    maxParallel?: number

    Maximum parallel tests (if parallel=true)

    parallel?: boolean

    Run tests in parallel

    progressCallback?: (progress: TestProgress) => void

    Progress callback for real-time updates

    publishedRubricVersionId?: string

    Latest published version observed at suite-run start, used to pin the suite.

    repeatCountOverride?: number

    Override the test's RepeatCount field at runtime. Useful for flaky-test detection: forces every test in the run to execute N times so the suite runner can compute score variance. If undefined, the test's own RepeatCount is used.

    rubricId?: string

    Experiment override for the rubric that judges this run. A version here is explicit and is not replaced by the suite pin.

    rubricVersionId?: string

    Explicit rubric version id. When omitted, the suite run pins the latest published version.

    selectedTestIds?: string[]

    Run only specific tests by their IDs. If provided, only tests with matching IDs will be executed.

    sequence?: number[]

    Run only specific sequence numbers (e.g., [1, 3, 5] runs tests at those positions)

    sequenceEnd?: number

    Stop execution at this sequence number (inclusive). Tests with sequence numbers greater than this value will be skipped.

    sequenceStart?: number

    Start execution from this sequence number (inclusive). Tests with sequence numbers less than this value will be skipped.

    shard?: { index: number; total: number }

    Deterministic shard specification for partitioning a suite across multiple isolated worker processes (1 <= index <= total). Applied after all other test filters using duration-aware Longest Processing Time (LPT) bucketing while preserving original suite sequence order within each shard.

    Type Declaration

    • index: number

      1-based shard index (1 <= index <= total).

    • total: number

      Total number of shards (total >= 1).

    tags?: string

    Tags to apply to the test run (JSON string array)

    transport?: "server" | "client" | "all"

    Filter tests by transport layer ('server' | 'client' | 'all'). Reads Configuration.transport off each test (falling back to bundle-based inference or 'server' when omitted). Server-side wire-companion bundles that seed state for or connect a secondary GraphQL client to MJAPI (realtime-session-guard, calendar-event-links) are grouped with 'client' so they run in the MJAPI-attached worker process ahead of client-transport bundles.

    variables?: TestRunVariables

    Variable values to use for this run. These values take highest priority in the resolution order: run > suite > test > type

    verbose?: boolean

    Verbose logging

    Settings

    Member Visibility

    On This Page

    Properties