Member Junction
    Preparing search index...

    Interface AgentEvalConfig

    Configuration for Agent Evaluation tests.

    interface AgentEvalConfig {
        agentId: string;
        evaluationStrategy?:
            | "final-turn-only"
            | "each-turn"
            | "all-turns-aggregate";
        judgeEvidence?: AgentEvalJudgeEvidenceConfig;
        maxExecutionTime?: number;
        oracles: {
            config?: Record<string, unknown>;
            type: string;
            weight?: number;
        }[];
        scoringWeights?: Record<string, number>;
    }
    Index

    Properties

    agentId evaluationStrategy? judgeEvidence? maxExecutionTime? oracles scoringWeights?

    Properties

    agentId: string

    Agent to test

    evaluationStrategy?: "final-turn-only" | "each-turn" | "all-turns-aggregate"

    When to evaluate with oracles in multi-turn tests

    • "final-turn-only": Run oracles only on final turn (default)
    • "each-turn": Run oracles after each turn
    • "all-turns-aggregate": Evaluate all turns together at the end
    judgeEvidence?: AgentEvalJudgeEvidenceConfig

    What judge oracles (llm-judge, decision-judge, rubric) see besides the bare output. By default they see the agent's final message, final payload and output artifacts — with registry component manifests resolved to full specs — whenever that adds to the payload alone.

    maxExecutionTime?: number

    Maximum execution time in milliseconds

    oracles: { config?: Record<string, unknown>; type: string; weight?: number }[]

    Oracles to run for evaluation

    scoringWeights?: Record<string, number>

    Scoring weights by oracle type

    Settings

    Member Visibility

    On This Page

    Properties