Member Junction
    Preparing search index...

    Interface PromptEvalConfig

    Configuration for a Prompt Eval test.

    interface PromptEvalConfig {
        agentId?: string;
        agentName?: string;
        configurationId?: string;
        effortLevel?: string | number;
        maxExecutionTime?: number;
        modelId?: string;
        nativeControlFlow?: "envelope" | "implicit";
        nativeToolResults?: boolean;
        oracles: PromptEvalOracleConfig[];
        promptId?: string;
        scoringWeights?: Record<string, number>;
        systemPromptOverride?: string;
        toolCallingMode?: "envelope" | "native";
        toolChoice?: ChatToolChoice;
        vendorId?: string;
    }
    Index

    Properties

    agentId? agentName? configurationId? effortLevel? maxExecutionTime? modelId? nativeControlFlow? nativeToolResults? oracles promptId? scoringWeights? systemPromptOverride? toolCallingMode? toolChoice? vendorId?

    Properties

    agentId?: string

    Resolve the prompt from an agent's active prompt binding instead of naming it.

    agentName?: string

    Same, by agent NAME. The corpus states its agent by name — golden files stay readable and diffable, and the suite generator runs offline with no database to resolve a GUID against.

    configurationId?: string
    effortLevel?: string | number

    Per-cell reasoning effort. Numeric 1-100 is MJ's cross-provider scale; a provider-named level (OpenAI's 'xhigh' / 'none') is passed through verbatim for the levels that scale cannot express. Part of the cell identity, so two effort levels are two cells, not noise pooled into one.

    maxExecutionTime?: number
    modelId?: string

    Per-cell model pinning — this is the matrix axis (§3.3).

    nativeControlFlow?: "envelope" | "implicit"

    Control-flow arm for a native cell. 'implicit' keeps the control tools the composer declared (the model's catalog must also say NativeControlFlow: 'implicit' for the runner to send them); 'envelope' (default) strips them here so the cell measures the hybrid on a model whose catalog has been switched to implicit for the gate run.

    nativeToolResults?: boolean

    Native-tool-results arm: send tool-form history (assistant toolCalls + tool turns) as native tool turns. Otherwise the driver renders that history as the corpus's [Action Result] … prose, which is what the loop shows models that do not return results natively.

    oracles: PromptEvalOracleConfig[]
    promptId?: string

    The prompt to run. Exactly one of promptId / agentId is required.

    scoringWeights?: Record<string, number>
    systemPromptOverride?: string

    Replaces the composed system prompt wholesale. Used by the trimmed-prompt variant cells.

    toolCallingMode?: "envelope" | "native"

    Which wire encoding this cell observes.

    The composer attaches the agent's Actions as native tool declarations unconditionally, because that is what BaseAgent does and the whole point of composing through it is to send what production sends. A cell measuring the ENVELOPE therefore has to take them back off: toolsProvided is one of the three terms in the runner's gate, so withholding declarations is exactly how a real envelope-mode call differs from a native one. Nothing else about the request changes, which is what keeps the two arms comparable.

    Defaults to 'envelope', so a record written before this axis existed — keeps measuring the baseline it was generated to measure.

    toolChoice?: ChatToolChoice

    tool_choice for a native cell. Omitted accepts the agent's own default ('auto').

    vendorId?: string

    Settings

    Member Visibility

    On This Page

    Properties