Member Junction
    Preparing search index...

    Class LLMJudgeOracle

    LLM Judge Oracle.

    Uses an LLM to evaluate output quality based on custom criteria. Provides semantic evaluation beyond deterministic checks.

    Configuration:

    • criteria: Array of validation criteria (required)
    • model: Model to use for judging (default: from prompt or default model)
    • temperature: Temperature for LLM (default: 0.1 for consistency)
    • promptTemplate: Custom prompt template (optional, uses default if not provided)
    • strictMode: Require all criteria to pass (default: false, uses weighted scoring)
    const oracle = new LLMJudgeOracle();
    const result = await oracle.evaluate({
        actualOutput: { response: 'Sales by region report created successfully' },
        expectedOutput: {
            judgeValidationCriteria: [
                'Response accurately answers the user\'s question',
                'Response includes actionable information',
                'Response is professional and clear'
            ]
        },
        contextUser
    }, {
        model: 'claude-sonnet-4',
        temperature: 0.1
    });

    Implements

    Index

    Constructors

    constructor

    Properties

    type

    Methods

    evaluate

    Constructors

    Properties

    type: "llm-judge" = 'llm-judge'

    Unique type identifier for this oracle. Used in test configuration to specify which oracles to run.

    Methods

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Methods