@memberjunction/testing-engine

Core test execution and evaluation engine for the MemberJunction Testing Framework. Provides the driver/oracle architecture for running tests, evaluating results with multiple oracle strategies, and recording outcomes.

Architecture

graph TD
    subgraph "@memberjunction/testing-engine"
        A[TestEngine] --> B[Test Drivers]
        A --> C[Oracles]
        A --> D[Utilities]

        B --> E[BaseTestDriver]
        E --> F[AgentEvalDriver]

        C --> G[ExactMatchOracle]
        C --> H[LLMJudgeOracle]
        C --> I[SchemaValidatorOracle]
        C --> J[SQLValidatorOracle]
        C --> K[TraceValidatorOracle]

        D --> L[Variable Resolver]
        D --> M[Scoring Calculator]
        D --> N[Cost Calculator]
        D --> O[Result Formatter]
        D --> P[Execution Context]
    end

    subgraph "Base Layer"
        Q["TestEngineBase
(from engine-base)"]
    end

    A -->|extends| Q

    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
    style E fill:#2d8659,stroke:#1a5c3a,color:#fff
    style F fill:#b8762f,stroke:#8a5722,color:#fff
    style G fill:#7c5295,stroke:#563a6b,color:#fff
    style H fill:#7c5295,stroke:#563a6b,color:#fff
    style I fill:#7c5295,stroke:#563a6b,color:#fff
    style J fill:#7c5295,stroke:#563a6b,color:#fff
    style K fill:#7c5295,stroke:#563a6b,color:#fff
    style Q fill:#2d6a9f,stroke:#1a4971,color:#fff
Copy

Overview

This is the server-side execution engine for MemberJunction tests. It implements a plugin-based architecture where:

• Drivers handle test execution (running agents, calling APIs, etc.)
• Oracles evaluate test outputs against expected outcomes
• Utilities handle variable resolution, scoring, cost tracking, and result formatting

Key capabilities:

• Execute individual tests or full test suites with parallel/sequential modes
• Multiple oracle strategies for evaluating test results
• Variable resolution system with priority cascading (run > suite > test > type)
• Automatic cost and duration tracking
• Progress callbacks for real-time execution monitoring
• Result persistence to MJ: Test Runs and MJ: Test Suite Runs entities

Installation

npm install @memberjunction/testing-engine
Copy

Core Concepts

Driver/Oracle Pattern

graph LR
    A[Test Definition] --> B[Driver]
    B -->|Executes| C[Target System]
    C -->|Output| D[Oracle 1]
    C -->|Output| E[Oracle 2]
    C -->|Output| F[Oracle N]
    D --> G[Combined Score]
    E --> G
    F --> G

    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
    style B fill:#2d8659,stroke:#1a5c3a,color:#fff
    style D fill:#7c5295,stroke:#563a6b,color:#fff
    style E fill:#7c5295,stroke:#563a6b,color:#fff
    style F fill:#7c5295,stroke:#563a6b,color:#fff
    style G fill:#b8762f,stroke:#8a5722,color:#fff
Copy

1. The driver executes the test target (e.g., runs an AI agent)
2. Multiple oracles independently evaluate the output
3. Scores are combined using configurable weights

Test Drivers

Oracles

A test run picks one rubric. The first source that names one wins: the run override, the rubric oracle's own config, Test.RubricID, TestSuite.RubricID walking up ParentID, then the agent's default Evaluation rubric. The published version is pinned when the suite run starts, so a publish during the run does not split the suite. An explicitly named version is used as given. When a rubric is resolved and the test did not configure a rubric oracle, the driver adds one. A test that already has an llm-judge oracle keeps that list, and the agent's Evaluation rubric is not added. The added rubric oracle always gates status. It contributes to the score when scoringWeights is absent or already names rubric.

Variable Resolution

Variables cascade through four levels with the highest priority winning:

Run Variables > Suite Variables > Test Variables > Type Variables
Copy

The resolver tracks the source of each resolved value for auditing.

Usage

Running a Single Test

import { TestEngine } from '@memberjunction/testing-engine';

const engine = TestEngine.Instance;
await engine.Config(false, contextUser);

const result = await engine.RunTest(testId, contextUser, {
    verbose: true,
    variables: { AIConfiguration: 'gpt-4o' },
    progressCallback: (p) => console.log(`${p.percentage}%: ${p.message}`)
});

console.log(`Status: ${result.status}, Score: ${result.score}`);
Copy

Running a Test Suite

const suiteResult = await engine.RunSuite(suiteId, contextUser, {
    parallel: true,
    maxParallel: 5,
    failFast: false,
    variables: { Temperature: 0.3 }
});

console.log(`Passed: ${suiteResult.passedTests}/${suiteResult.totalTests}`);
console.log(`Average Score: ${suiteResult.averageScore}`);
Copy

Utilities

Testing

npm test
npm run test:coverage
Copy

Dependencies

License

Business Source License 1.1 — see LICENSE for details.