@memberjunction/testing-engine-base

Base engine for MemberJunction’s testing framework. Provides metadata caching for test types, tests, suites, rubrics, and shared type definitions. UI-safe — contains no execution logic.

Architecture

graph TD
    subgraph "@memberjunction/testing-engine-base"
        A[TestEngineBase] --> B[Test Types Cache]
        A --> C[Tests Cache]
        A --> D[Test Suites Cache]
        A --> E[Test Rubrics Cache]
        A --> F[Suite Tests Cache]
        G[Types Module] --> H[TestRunOptions]
        G --> I[SuiteRunOptions]
        G --> J[TestRunResult]
        G --> K[TestSuiteRunResult]
        G --> L[Variable System Types]
    end

    subgraph "Data Sources"
        M["MJ: Test Types"]
        N["MJ: Tests"]
        O["MJ: Test Suites"]
        P["MJ: Test Rubrics"]
        Q["MJ: Test Suite Tests"]
    end

    M --> B
    N --> C
    O --> D
    P --> E
    Q --> F

    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
    style B fill:#2d8659,stroke:#1a5c3a,color:#fff
    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
    style D fill:#2d8659,stroke:#1a5c3a,color:#fff
    style E fill:#7c5295,stroke:#563a6b,color:#fff
    style G fill:#b8762f,stroke:#8a5722,color:#fff

Overview

This package serves two purposes:

1. Metadata Caching: TestEngineBase is a singleton engine that loads and caches all testing metadata, providing fast lookups by ID, name, type, and tag
2. Shared Types: Comprehensive type definitions for test execution, results, variables, and configuration used across the entire testing stack

Key capabilities:

• Cached access to test types, tests, suites, rubrics, and suite-test associations
• Lookup methods by ID, name, type, and tag
• Active-only filtering for tests and suites
• Suite test sequencing with proper sort order
• Complete type system for test execution, results, validation, and variables
• Progress and logging callback interfaces for real-time execution updates

Installation

npm install @memberjunction/testing-engine-base

Usage

Loading Metadata

import { TestEngineBase } from '@memberjunction/testing-engine-base';

const engine = TestEngineBase.Instance;
await engine.Config(false, contextUser);

Querying Tests

// By type
const agentTests = engine.GetTestsByType(typeId);

// By tag
const regressionTests = engine.GetTestsByTag('regression');

// By name
const test = engine.GetTestByName('Agent Summarization Test');

// Active only
const activeTests = engine.GetActiveTests();
const activeSuites = engine.GetActiveTestSuites();

// Tests in a suite (sorted by sequence)
const suiteTests = engine.GetTestsForSuite(suiteId);

Using Types

import {
    TestRunOptions,
    SuiteRunOptions,
    TestRunResult,
    TestSuiteRunResult,
    OracleResult,
    TestRunVariables
} from '@memberjunction/testing-engine-base';

// Configure a test run
const options: TestRunOptions = {
    verbose: true,
    environment: 'staging',
    gitCommit: 'abc123',
    variables: { AIConfiguration: 'gpt-4o', Temperature: 0.7 },
    progressCallback: (progress) => console.log(progress.message)
};

// Suite run options add parallel/failFast controls
const suiteOptions: SuiteRunOptions = {
    ...options,
    parallel: true,
    maxParallel: 5,
    failFast: false
};

Type Reference

Core Result Types

Type	Description
TestRunResult	Complete result from a single test execution
TestSuiteRunResult	Aggregate result from a suite execution
OracleResult	Result from an individual oracle evaluation
ValidationResult	Configuration validation result with errors/warnings

Execution Options

Type	Description
TestRunOptions	Options for running a single test (verbose, dryRun, variables, etc.)
SuiteRunOptions	Options for running a suite (parallel, failFast, sequence filters)
TestProgress	Progress callback data (step, percentage, message)
TestLogMessage	Log message from test execution (level, message, metadata)

Variable System

Type	Description
TestTypeVariablesSchema	Schema defining available variables for a test type
TestVariableDefinition	Single variable definition (name, dataType, valueSource, etc.)
TestVariablesConfig	Variable overrides at the test level
TestSuiteVariablesConfig	Variable values at the suite level
ResolvedTestVariables	Final resolved values with source tracking
TestRunVariables	Variable values provided at run time

Context Types

Type	Description
RunContextDetails	Execution environment details (OS, Node.js, CI/CD, git info)
OracleConfig	Oracle-specific configuration properties
ScoringWeights	Weights for different oracle evaluation dimensions

Relationship to Other Testing Packages

graph LR
    A["testing-engine-base<br/>(Metadata + Types)"] --> B["testing-engine<br/>(Execution)"]
    B --> C["testing-cli<br/>(CLI Interface)"]

    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
    style B fill:#2d8659,stroke:#1a5c3a,color:#fff
    style C fill:#7c5295,stroke:#563a6b,color:#fff

Dependencies

Package	Purpose
@memberjunction/core	BaseEngine, UserInfo, IMetadataProvider
@memberjunction/core-entities	Test entity types
@memberjunction/global	MJGlobal utilities
rxjs	Observable patterns
debug	Debug logging

License

ISC