Member Junction
    Preparing search index...

    Interface LLMConformanceSuiteConfig

    Configuration for RunLLMConformanceSuite. The Script* functions program the provider test file's vendor-SDK mock; the suite tells the adapter WHAT the vendor should do, the adapter translates that into the vendor's native wire shapes.

    interface LLMConformanceSuiteConfig {
        CreateLLM: () => BaseLLM;
        KnownDeviations?: LLMConformanceDeviation[];
        NonStreamingFailureMode: "throws" | "failedResult";
        PreAbortedStreamingBehavior?: "resolvesCancelled" | "rejectsErrorResult";
        ProviderName: string;
        ScriptFailure: (
            error: Error,
            at: FailureSite,
            options?: { ChunkBeforeError?: string },
        ) => void;
        ScriptNonStreamingSuccess: (
            content: string,
            usage: ExpectedUsageCounts,
        ) => void;
        ScriptStreamingCancellation: (
            chunksBeforeAbort: string[],
            controller: AbortController,
        ) => void;
        ScriptStreamingSuccess: (
            chunks: string[],
            usage: ExpectedUsageCounts,
        ) => void;
        SupportsStreaming: boolean;
    }
    Index

    Properties

    CreateLLM KnownDeviations? NonStreamingFailureMode PreAbortedStreamingBehavior? ProviderName ScriptFailure ScriptNonStreamingSuccess ScriptStreamingCancellation ScriptStreamingSuccess SupportsStreaming

    Properties

    CreateLLM: () => BaseLLM

    Construct a fresh driver instance wired to the scriptable vendor mock. Called in beforeEach, so every test gets a clean instance (and clean streaming state).

    KnownDeviations?: LLMConformanceDeviation[]

    Documented deviations from the ideal contract — see LLMConformanceDeviation.

    NonStreamingFailureMode: "throws" | "failedResult"

    How the driver surfaces NON-cancellation vendor failures on the NON-streaming path. BaseLLM imposes no rule here (ChatCompletions, plural, is the normalizing layer):

    • 'throws': the vendor error propagates out of ChatCompletion (OpenAI family, Groq, Cerebras, Mistral).
    • 'failedResult': the driver catches and resolves ChatResult{success:false} (Ollama, Azure).
    PreAbortedStreamingBehavior?: "resolvesCancelled" | "rejectsErrorResult"

    What happens when a STREAMING request is made with an already-aborted token. Required when SupportsStreaming is true.

    • 'resolvesCancelled': the driver detects the pre-abort itself (empty stream + cancelled finalize) and the promise resolves with the documented cancelled shape (Groq, Cerebras).
    • 'rejectsErrorResult': createStreamingRequest throws (driver pre-check or SDK abort error), and BaseLLM's outer catch — which does NOT special-case cancellation — rejects with a generic statusText:'error' ChatResult instead of the cancelled shape (OpenAI family, Mistral, Ollama, Azure). Pinned under a [KNOWN DEVIATION] test name.
    ProviderName: string

    Human-readable provider name, used in describe() labels.

    ScriptFailure: (
        error: Error,
        at: FailureSite,
        options?: { ChunkBeforeError?: string },
    ) => void

    Program the vendor mock: the next call fails with error at the given site. For 'midStream', the stream must first emit options.ChunkBeforeError (as a normal text chunk) and then throw, so the suite can prove partial content is discarded.

    ScriptNonStreamingSuccess: (content: string, usage: ExpectedUsageCounts) => void

    Program the vendor mock: the next non-streaming call succeeds with this content + usage.

    ScriptStreamingCancellation: (
        chunksBeforeAbort: string[],
        controller: AbortController,
    ) => void

    Program the vendor mock for a mid-stream cancellation: emit chunksBeforeAbort as normal text chunks, then abort controller, then end the stream the way the real SDK ends an aborted stream (throw its abort error, or end iteration silently — whichever is faithful).

    ScriptStreamingSuccess: (chunks: string[], usage: ExpectedUsageCounts) => void

    Program the vendor mock: the next streaming call emits these text chunks (one vendor chunk per entry, in order) and then completes, reporting usage in the vendor's native place for stream usage (final chunk / usage event) even if the driver is known to drop it.

    SupportsStreaming: boolean

    Whether the driver declares streaming support (asserted against SupportsStreaming).

    Settings

    Member Visibility

    On This Page

    Properties