Member Junction
    Preparing search index...

    Class DecisionJudgeOracle

    Decision Judge Oracle.

    Judges output against the same criteria, and the same trace, as LLMJudgeOracle, but asks a typed decision model instead of parsing an LLM's JSON. Each criterion becomes one Likelihood question, "The response satisfies: ", and every question travels in one AIDecisionRunner call. The pass rule and the weighting live here, in code, not in the prompt.

    Criteria come from expectedOutput.judgeValidationCriteria, or else from config.criteria, as for the LLM judge. Each is either a string or { "criterion": "...", "weight": 2 }. A weight of 0 keeps a criterion out of the score but not out of the pass rule; at least one weight must be above 0.

    Configuration:

    • criteria: Array of criteria, used when the expected output has none
    • passThreshold: The probability every criterion must reach to pass (default: 0.5). The default is a placeholder: calibration (Task 2.4) sets real values.
    • promptName: The decision prompt whose model bindings choose the decision model (default: 'Default Decision')
    • timeoutMS: How long to wait for each decision model call, in milliseconds (default: 120000, two minutes). A call that runs over fails the oracle.

    Result:

    • passed: every criterion's probability is at or above passThreshold
    • score: the weighted mean of the probabilities (weights default to 1)
    • details: criteriaProbabilities (one entry per criterion), passThreshold, promptName, decisionRunId (the MJ: AI Prompt Runs row), decisionModel, and llmCost, the decision call's cost, under the key the LLM judge uses for its own. It is saved with the result details (TestRun.ResultDetails) but, as for the LLM judge, it is not added to TestRun.CostUSD.

    A failed decision call gives a failed result with the error message. The oracle never throws.

    const oracle = new DecisionJudgeOracle();
    const result = await oracle.evaluate({
        test,
        actualOutput: { response: 'Sales by region report created successfully' },
        expectedOutput: {
            judgeValidationCriteria: [
                'Response accurately answers the user\'s question',
                { criterion: 'Response includes actionable information', weight: 2 }
            ]
        },
        contextUser
    }, {
        passThreshold: 0.7
    });

    Implements

    Index

    Constructors

    constructor

    Properties

    type DEFAULT_PASS_THRESHOLD DEFAULT_PROMPT_NAME

    Methods

    evaluate

    Constructors

    Properties

    type: "decision-judge" = 'decision-judge'

    Unique type identifier for this oracle. Used in test configuration to specify which oracles to run.

    DEFAULT_PASS_THRESHOLD: 0.5

    The per-criterion pass probability used when the config sets none, until calibration (Task 2.4) sets real values.

    DEFAULT_PROMPT_NAME: "Default Decision" = 'Default Decision'

    The decision prompt used when the config names none.

    Methods

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Methods