Integration Testing Quickstart

MemberJunction’s integration-test tier exercises the real server stack — live database, real data providers, real cache managers, real engines — from a headless Node process, with no browser and (by default) no LLM calls. It proves the seams between packages: the places unit tests mock away and browser tests traverse but cannot assert.

This guide covers the architecture, every way to run the tier, and every way to add coverage.

---

1. Where this tier sits

Tier	What’s real	What stands in	Tooling	Speed
Unit tests	The function under test	Providers, DB, transport (all mocked)	Vitest	ms
Integration tests	DB, providers, cache managers, engines, GraphQL transport	Only the top layer — the test process stands in for MJAPI/MJExplorer	mj test (the Testing Framework CLI)	~10–60s/suite
Browser / computer-use regression	Everything, end to end	Nothing	Playwright / CU	minutes

The guiding principle is “mock the top layer, keep everything else real.” A mocked provider never proves the real cache fingerprint is correct; a browser never notices that a cached payload came back with 2 columns instead of 5, and it can’t count how many times the cache was read. This tier asserts exactly those things — against the live dev database, deterministically.

The rule for shipping: no server-side feature is done until the deterministic integration tier passes headless (npm run test:integration), and new server capability comes with new integration coverage.

---

2. Architecture

2.1 Framework and content, one execution front-end

Since the July-2026 restructure, the tier is split into a framework package (published, content-free) and a content package (private, never published) — and there is exactly one way to execute the catalog: mj test.

  ┌───────────────────────────────────────────┐   ┌──────────────────────────────────────────┐
  │   @memberjunction/testing-integration     │   │  @memberjunction/integration-test-suite  │
  │   (published — FRAMEWORK ONLY)            │   │  (private: true — MJ's OWN test content) │
  │                                           │   │                                          │
  │  IntegrationCheckRegistry (BaseSingleton) │◀──┤  src/checks/  · 52 bundles · ~365 checks │
  │  NamedCheck / BundleLifecycle contracts   │   │  src/index.ts barrel (side-effect-       │
  │  bootstrap  (server / client / PostgreSQL)│   │    imports every bundle → registers all) │
  │  InstrumentedLocalStorageProvider         │   │  src/__tests__/  count table + parity    │
  │  tiers  (IsTierEnabled — the ONE gate)    │   │  rigs/  standalone tsx scripts           │
  │  TestRunner + Assert helpers · ai-verify  │   │        (NOT catalog entry paths)         │
  │  IntegrationTestDriver  (@RegisterClass)  │   └──────────────────────────────────────────┘
  └─────────────────────▲─────────────────────┘                       ▲
                        │                                             │ loaded at runtime via
  ┌─────────────────────┴─────────────────────────────────────────────┴──────┐
  │  The ONE front-end: mj test                                              │
  │  TestEngine → "Integration Test" TestType → IntegrationTestDriver        │
  │  mj test run / mj test suite  ·  results → MJ: Test Runs                 │
  │  mj.config.cjs testing.checkModules (or --checks-module) loads bundles   │
  └──────────────────────────────────────────────────────────────────────────┘

• The framework (packages/TestingFramework/testing-integration/) ships to customers — @memberjunction/server-bootstrap’s manifest imports IntegrationTestDriver from its barrel — so it deliberately carries no check bundles and no heavy dependencies.
• The content (packages/TestingFramework/integration-test-suite/, private: true) holds every check bundle plus the heavy dependency tree (ai-agents, record-set-processor, predictive-studio, …). Importing its barrel registers the full catalog on the shared registry as a side effect.
• The loading seam: the published mj CLI cannot depend on the private package, so mj test run / mj test suite side-effect-import it at startup via the repo root mj.config.cjs — testing: { checkModules: ['@memberjunction/integration-test-suite'] } — after the instrumented-cache install and before provider setup. Ad-hoc override: --checks-module <specifier>. External adopters point checkModules at their own check packages — that is the extension point.
• Execution is metadata-driven: a MJ: Test Types row named Integration Test points at DriverClass: "IntegrationTestDriver"; MJ: Tests rows select bundles via their Configuration JSON; suites group tests; results persist as MJ: Test Runs rows browsable in the Explorer Testing dashboard.

(The old second front-end — per-suite tsx dispatchers + a run-all.ts aggregator under packages/MJServer/integration-test-scripts/ — was deleted by design; that folder is now a pointer README. One registry, one driver, one entry path: coverage cannot drift because there is nothing left to drift.)

2.2 The check contract

A check is an async function that throws on failure and returns on pass (the Assert* helpers throw). There is no per-check result interface — the harness wraps each check in try/catch and records the outcome.

packages/TestingFramework/testing-integration/src/check.ts

export type IntegrationCheckFn = (ctx: IntegrationCheckContext) => Promise<void>;

export interface NamedCheck {
    Id: string;                    // '<bundle>.<localId>', e.g. 'server-cache.S1'
    Name: string;                  // human-readable, shown in output / TestRun details
    Fn: IntegrationCheckFn;
    RequiresMutation?: boolean;    // runs only when the mutation tier is enabled
    RequiresLiveModel?: boolean;   // runs only when the live-model tier is enabled
}

Checks are grouped into bundles — ordered arrays registered under a shared prefix (server-cache, rls-isolation, ai-skills, …). Order is load-bearing: several checks intentionally build on cache state from earlier ones (e.g. one check warms a cache slot that the next asserts a hit on), so the driver runs a bundle’s checks strictly in array order.

Every check receives one shared IntegrationCheckContext:

Field	What it is
User	Resolved context user (MJ_TEST_USER_EMAIL override → Owner-type → first user)
Provider	The run-scoped real provider — SQLServerDataProvider (server) or GraphQLDataProvider (client)
Storage	The InstrumentedLocalStorageProvider — per-category cache counters (§2.4)
Pool	Raw mssql pool for fixture SQL (server transport only)
Schema	Core schema (e.g. __mj)
Config	The opaque per-bundle config bag from the selector (e.g. datasetName, entityName)
Fixtures / RlsFixture / AiSkillsFixture / …	Typed slots populated by the bundle’s lifecycle or the driver

Mutating bundles register a BundleLifecycle — Setup creates the bundle’s shared fixture (throwaway rows, prefixed names) and assigns it onto the context; Teardown deletes everything in FK-safe order. The driver wraps the bundle’s checks in Setup → run → Teardown (finally), and teardown is best-effort (it never throws, so a failing check still cleans up).

The registry itself (check-registry.ts) is a BaseSingleton with Register, Get(id), GetBundle(prefix), RegisterLifecycle, and GetLifecycle — it lives in the framework package. Bundle modules self-register on import — the suite package’s index.ts exports every checks/*.checks.ts, so importing @memberjunction/integration-test-suite (which the CLI does via the checkModules seam) loads the full catalog. The driver’s @RegisterClass decorator fires from the framework package’s own barrel.

2.3 The bootstrap and the dedicated-process rule

Every integration run owns its process. The load-bearing invariant:

The InstrumentedLocalStorageProvider must be installed as the first caller of LocalCacheManager.Instance.Initialize(...) — before any provider setup. Initialize is first-caller-wins: installed late, instrumentation is a silent no-op (caching still works against the provider’s own storage, but the counters never see traffic, which looks exactly like a product bug).

bootstrap.ts enforces this with three install paths that share one process-global instrumented storage:

Function	Use
bootstrapIntegrationServer(opts?)	Owns the process: instrumented cache first, then the real database provider (SQL Server by default; PostgreSQL when DB_PLATFORM=postgresql), user-cache refresh, context-user resolution. Used by the smoke test, the standalone rigs, and as the driver’s self-bootstrap fallback.
bootstrapIntegrationClient()	Same first-caller discipline, then setupGraphQLClient against a running MJAPI (preflights the endpoint first; authenticates via MJ_API_KEY → x-mj-api-key).
installInstrumentedCacheFirst()	Installs ONLY the cache — for the testing CLI, which sets up its own provider afterwards. Triggered by MJ_INTEGRATION_TEST=1.

Configuration is resolved from the repo root’s .env / mj.config.cjs (databaseSettings takes precedence over DB_* env vars) — never hardcoded secrets — which is why everything runs from the repo root.

Two consequences:

1. Server-transport suites cannot run inside a live MJAPI — its StartupManager already initialized the cache. The driver detects this (serverProcessAlreadyClaimed()) and fails fast with an actionable Error result telling you to run via the CLI in a dedicated process. The Explorer Testing dashboard therefore browses integration runs and their per-check results, but execution belongs to the CLI / scripts.
2. Integration suites run strictly serially. Bundles share process-global singletons (LocalCacheManager, the instrumented counters, UserCache, Metadata.Provider), so the CLI forces parallel: false whenever MJ_INTEGRATION_TEST=1, overriding any --parallel.

PostgreSQL: DB_PLATFORM=postgresql routes the same bootstrap (and the same downstream check code) through @memberjunction/postgresql-dataprovider (an optional dependency, loaded only on that path). Context-user resolution on PG requires a populated user cache; without one the bootstrap throws a clear, actionable error rather than fabricating a user.

2.4 The two proof techniques

These make integration checks assertion-precise rather than vibes-based (instrumented-cache.ts):

1. UniqueFilter(column, tag) → Name <> 'zzz-cache-test-<tag>' — an always-true filter that is textually unique per tag. Because ExtraFilter is part of the cache fingerprint, every tag yields a guaranteed-cold cache entry while matching the same rows. Cold-cache determinism with zero data mutation — critical against a live shared dev database.

2. InstrumentedLocalStorageProvider — wraps the real in-memory cache storage with per-category Get/Set counters. Checks don’t guess whether the cache was used; they prove it: a miss shows a RunViewCache write, a hit shows none, a dedup/linger-served result shows zero storage traffic at all, and BypassCache must leave the counters untouched. Scope assertions to the 'RunViewCache' category (Storage.SetCount('RunViewCache')) — LocalCacheManager persists its registry index asynchronously in a different category, so global counters are noisy.

2.5 Tiers and gating

Every check (and every metadata Test) belongs to a tier (tiers.ts):

Tier	Contract	Gate
deterministic	Credential-free, read-only or self-cleaning; the blocking CI gate	none — always runs
mutation	Writes to the DB and cleans up unconditionally	RUN_MUTATION_TESTS=1
live-model	Real LLM calls — costs tokens, needs model credentials	default-ON; opt out with RUN_AGENT_TESTS=0

IsTierEnabled(tier) is the single predicate every consumer (driver, rigs) calls, so a gate is honored identically everywhere. Gating exists at two granularities:

• Whole-test: a metadata Test’s Configuration.tier (or an explicit requiresEnv env-var override). When gated off, the driver skips-as-Passed with one gate oracle whose message reads Skipped: <VAR> not set (tier '<tier>').
• Per-check: RequiresMutation / RequiresLiveModel flags on individual NamedChecks inside an otherwise-deterministic bundle (e.g. the save/delete invalidation checks in the server-cache bundle). A bundle selector can also opt mutation checks in declaratively via config.runMutationTests: true, independent of env.

Adjacent gates outside the tier enum: the Predictive Studio flow rigs (integration-test-suite/rigs/ps-*.ts) run only under PS_INTEGRATION=1 (they also need the Python sidecar), the cross-server rig under RUN_CROSS_SERVER=1, and client-transport tests need a reachable MJAPI + MJ_API_KEY.

2.6 Transports

Transport	Stack under test	Needs
server (default)	SQLServerDataProvider (or PostgreSQL) directly — server-side cache semantics (TrustLocalCacheCompletely = true)	Database only
client	GraphQLDataProvider → a running MJAPI — client cache semantics (TrustLocalCacheCompletely = false, CacheLocal opt-in, smart-cache-check round-trips)	MJAPI up + MJ_API_KEY

A metadata Test declares Configuration.transport explicitly, or the driver infers it from the selected bundles (the client-transport bundles are client-cache, rls-isolation-client, and remote-op-wire-progress).

2.7 The metadata layer

The Testing Framework (packages/TestingFramework/) is metadata-driven end to end:

MJ: Test Types ──▶ "Integration Test"  { DriverClass: "IntegrationTestDriver", Status: Active }
      │                                  metadata/test-types/.integration-test-type.json  (normal metadata — inert type def)
      ▼
MJ: Tests ───────▶ IT01…IT66            Configuration selects bundles + tier + transport
      │                                  metadata-optional/integration-test/tests/integration/.IT*.json
      ▼
MJ: Test Suites ─▶ "Integration Tests"  (parent — 0 members; running it errors, exit 1)
                   ├─ "Integration Tests — Deterministic"   IT01–IT15, IT20–IT52, IT64–IT66  (52 members, the blocking tier)
                   └─ "Integration Tests — Live Model"      IT16–IT19, IT53–IT63  (15 members)
                                         metadata-optional/integration-test/test-suites/.integration-suite.json
      ▼
MJ: Test Runs ───▶ one row per execution; ResultDetails = the per-check OracleResult[]

How execution works: the CLI first side-effect-imports the configured checkModules (populating the registry — a missing/unbuilt suite package surfaces as an Unknown integration check bundle failure), then TestEngine resolves the driver from the TestType’s DriverClass via the ClassFactory (CreateInstance(BaseTestDriver, 'IntegrationTestDriver')) and calls Execute(). The IntegrationTestDriver:

1. parses the Test’s Configuration (below);
2. applies the whole-test tier gate (skip-as-Passed when gated);
3. infers/uses the transport and obtains the instrumented provider stack (the one the CLI installed first-caller, or a self-bootstrap in a dedicated process — with the fail-fast host check from §2.3);
4. runs each selected bundle’s checks in order against one shared context, wrapping each bundle in its registered lifecycle (Setup → checks → Teardown in finally) and each check in try/catch — a thrown check becomes one failing OracleResult, never a re-throw (a re-throw would leave the TestRun stuck 'Running');
5. arms its own timeout (Configuration.maxExecutionTime → Test.MaxExecutionTimeMS → 5-minute default) and reports Timeout with partial results if it fires;
6. maps outcomes onto the framework result: status = Passed | Failed | Error | Timeout, score = passedChecks / totalChecks, one OracleResult per check with oracleType = '<bundle>.<id>'.

The engine persists TestRun.ResultDetails as a bare OracleResult[]; the Explorer Test Run form parses it into a per-check pass/fail breakdown.

The Configuration shape (parsed off MJ: Tests.Configuration, types.ts):

{
  "tier": "deterministic",          // or "mutation" | "live-model" (whole-test gate)
  "transport": "server",            // or "client"; optional — inferred from bundles
  "checks": [                       // ORDERED list of check bundles
    {
      "type": "aggregates-cache",   // bundle name, expanded via GetBundle at runtime
      "config": {                   // optional per-bundle knobs, surfaced as ctx.Config
        "entityName": "MJ: User Settings"
      }
    }
  ]
}

The metadata layer is intentionally thin: metadata selects, TypeScript asserts. There is no JSON assertion DSL — a Test row picks which registered bundles run (and with what knobs); the assertions live in the bundle code.

Suite mechanics: suite membership rows (MJ: Test Suite Tests) carry a Sequence (tests run in order — integration suites are serial) and a Status — members marked Skip are excluded from suite runs (IT03, IT15, and IT23 ship as Skip in the deterministic suite; run them individually with MJAPI up). For suites, the engine also gives drivers suite-scoped fixtures: it builds one SuiteFixtureContext ({ SuiteRunID, Data, CreatedRecords }) per suite run, calls each distinct driver’s SetupSuite once before the tests and TeardownSuite in a guaranteed finally, and threads the context into every Execute as context.fixtures. The IntegrationTestDriver uses this to discover the RLS two-user fixture once per suite run; on the no-suite mj test run path it lazily re-discovers.

Server registration: @memberjunction/testing-integration is part of the ServerBootstrap class-registration manifest, so any process that boots through it (MJAPI included) can resolve the driver — but remember §2.3: inside a live MJAPI, server-transport tests refuse to execute by design.

2.8 Seeded RLS fixtures

The RLS isolation coverage is the security core of this tier: prove one user’s Row-Level-Security-filtered cache entry can never serve a different user. It uses two fixture strategies together:

• Discovery (discoverRlsFixture) — finds two users with different effective RLS predicates from the live user cache + provider RLS filters. Nothing is created, so teardown is a no-op; on databases with only RLS-exempt admins the dependent checks degrade to skip-as-pass with a note.
• Seeded, purpose-built users — these live in the optional sibling root metadata-optional/integration-test/, NOT the default-pushed metadata/ tree, so the synthetic IsActive accounts never land in a production DB that only syncs metadata/: metadata-optional/integration-test/users/.integration-test-users.json, metadata-optional/integration-test/roles/.integration-test-roles.json, metadata-optional/integration-test/entity-permissions/.integration-test-permissions.json. it-rls-a@integration.test / it-rls-b@integration.test each hold ONLY the Integration Test: RLS Scoped Reader role (read on MJ: AI Agent Runs, scoped to the caller’s own UserID via the UI: Own AI Agent Runs RLS filter) — genuinely non-exempt users for the deterministic multi-user isolation checks. it-nogrant@integration.test has no roles at all — the negative check that a user with no grant is served no rows (cached or not). When the seed isn’t pushed, those checks skip-as-pass.

---

3. Running the tier

Everything runs from the repo root (.env / mj.config.cjs resolution is cwd-relative). Exit codes are uniform across every entry point: 0 passed (or cleanly skipped) · 1 failures · 2 bootstrap/connectivity error.

3.1 One-time setup — seed the metadata (load-bearing)

mj test is metadata-driven: it discovers tests from MJ: Tests / MJ: Test Suites records in the database. Without them it has nothing to dispatch — so every environment that runs the tier (a fresh dev DB, CI) must seed them once. The “Integration Test” TestType lives in the normal metadata/ tree (an inert type definition), so it lands with a normal metadata/ push; the IT tests, the suite hierarchy, and the RLS fixture users/roles/permissions live under the optional sibling root:

npx mj sync push --dir=metadata                             # includes the Integration Test TestType
npx mj sync push --dir=metadata-optional/integration-test   # IT tests + suite + RLS fixtures

Two more things are load-bearing on every run:

• MJ_INTEGRATION_TEST=1 makes the CLI install the instrumented cache first-caller before its own provider setup (otherwise counters silently see nothing) and forces the suite serial.
• Use the workspace-local mj (./node_modules/.bin/mj; npm run scripts and npx mj from the repo root resolve it automatically). A globally-installed mj cannot load the private suite package, so every bundle dispatch fails with Unknown integration check bundle.

The bundles themselves reach the CLI through the checkModules seam (§2.1): the repo root mj.config.cjs carries testing: { checkModules: ['@memberjunction/integration-test-suite'] }, and the suite package must be built (npm run build covers it). If mj test reports an unknown bundle, check — in order — metadata seeded, suite package built, checkModules configured.

3.2 The whole tier

npm run test:integration                      # deterministic suite (gated checks skip-as-pass)
RUN_MUTATION_TESTS=1 npm run test:integration # + mutation-gated checks inside the bundles

npm run test:integration is exactly MJ_INTEGRATION_TEST=1 mj test suite "Integration Tests — Deterministic" — the suite runs serially, one MJ: Test Runs row per test, one exit code for CI.

3.3 One test / one suite — mj test

The per-bundle iteration loop is mj test run with the bundle’s IT record name:

# One test (the old "run one suite's dispatcher" loop)
MJ_INTEGRATION_TEST=1 npx mj test run --name "IT01 - Server RunView Cache Integrity"

# A whole suite (runs serially under MJ_INTEGRATION_TEST=1 regardless of --parallel)
MJ_INTEGRATION_TEST=1 npx mj test suite --name "Integration Tests — Deterministic"

# The live-model suite (also needs the tier gate)
RUN_AGENT_TESTS=1 MJ_INTEGRATION_TEST=1 npx mj test suite --name "Integration Tests — Live Model"

# Client-transport tests — start MJAPI first: (cd packages/MJAPI && npm run start)
MJ_INTEGRATION_TEST=1 npx mj test run --name "IT03 - Client GraphQL Cache Integrity"

# Validate definitions without executing (driver resolvable, Configuration parses)
npx mj test validate --type "Integration Test"
npx mj test run --name "IT01 - Server RunView Cache Integrity" --dry-run

Useful extras: mj test list, mj test history, mj test suite --flaky-check 3 (runs each test N times and reports variance), and --checks-module <specifier> to preload an ad-hoc check package on top of the configured checkModules.

3.4 Browsing results — the Explorer Testing dashboard

Integration TestRun rows appear in the Testing dashboard like any other test type, with the per-check breakdown parsed from ResultDetails (each check’s <bundle>.<id>, pass/fail, and message) on the Test Run form. Execution from inside a live MJAPI is deliberately refused for server-transport tests (§2.3) — the run records an Error result whose message points you at the CLI invocation. Treat the dashboard as the observability surface; the CLI and scripts are the execution surface.

3.5 Special rigs — standalone by design

Coverage that can’t live inside the one-process mj test catalog runs as standalone tsx scripts under packages/TestingFramework/integration-test-suite/rigs/ (with their shared lib/harness.ts + lib/ai-bootstrap.ts). They are deliberately NOT catalog entry paths.

Bootstrap smoke test — proves the first-caller invariant end to end (a cold RunView must show an instrumented RunViewCache write); lives with the framework package:

npx tsx packages/TestingFramework/testing-integration/smoke/bootstrap-smoke.ts

Cross-server cache invalidation — two MJAPI processes sharing one DB and one Redis; a save through server A must invalidate server B’s cache (Redis pub/sub → remote-invalidate). Uses a compose overlay on the regression stack:

docker compose -f docker/regression/docker-compose.test.yml \
               -f docker/regression/docker-compose.cross-server.yml \
               --profile full up -d --wait sqlserver db-setup mjapi mjapi-b

MJAPI_A_URL=http://localhost:14000/ MJAPI_B_URL=http://localhost:14001/ \
MJ_API_KEY=<system-api-key> \
npx tsx packages/TestingFramework/integration-test-suite/rigs/cross-server-invalidation-tests.ts

Other rigs: agent-memory-tests.ts (live-model, RUN_AGENT_TESTS=1), runview-matrix-tests.ts (client-first RunView sweep across every entity — needs MJAPI + MJ_API_KEY), and the ten ps-inproc-* / ps-live-* Predictive Studio flows (PS_INTEGRATION=1 + the Python sidecar). All run the same way:

PS_INTEGRATION=1 npx tsx packages/TestingFramework/integration-test-suite/rigs/ps-inproc-operate-flow.ts

Outcome files — a run can emit per-check outcome JSON via EMIT_OUTCOMES=<path> ({name, passed, durationMs, error?}[]); scripts/integration-golden-diff.mjs diffs two such files by check id and fails on missing/extra/pass-mismatch (timing differences only warn) — originally the front-end-equivalence proof during the migration, still useful for comparing two runs.

The packages’ own unit tests (registry semantics, config parsing, tier gating, driver mapping in the framework; per-bundle count table + sibling parity in the suite — mocked, no DB):

cd packages/TestingFramework/testing-integration && npm run test
cd packages/TestingFramework/integration-test-suite && npm run test

3.6 Environment reference

Variable	Consumed by	Meaning
DB_HOST / DB_PORT / DB_USERNAME / DB_PASSWORD / DB_DATABASE	server bootstrap	SQL connection (mj.config.cjs databaseSettings takes precedence)
DB_PLATFORM	server bootstrap	sqlserver (default) or postgresql
MJ_TEST_USER_EMAIL	server bootstrap	Context-user override (default: Owner-type user, else first user)
RUN_MUTATION_TESTS=1	tier gate	Enables the mutation tier / RequiresMutation checks
RUN_AGENT_TESTS	tier gate	Live-model tier / RequiresLiveModel checks. Default-ON — IsTierEnabled returns RUN_AGENT_TESTS !== '0', so =1 is a no-op kept for back-compat and only =0 disables. Note RUN_MUTATION_TESTS is the opposite: strictly === '1'
PS_INTEGRATION=1	ps-* rigs	Enables the Predictive Studio flow rigs
MJ_API_KEY	client bootstrap	System API key MJAPI accepts via x-mj-api-key
GRAPHQL_PORT / GRAPHQL_ROOT_PATH / MJAPI_URL	client bootstrap	Endpoint resolution; MJAPI_URL overrides the composed localhost URL
MJ_INTEGRATION_TEST=1	testing CLI	Install instrumented cache first-caller + force serial suite execution
EMIT_OUTCOMES=<path>	driver / TestRunner	Write the per-check outcomes JSON (golden-diff format)
MJ_TEST_DATASET	dataset-cache bundle	Dataset name override (default MJ_Metadata)
MJ_CORE_SCHEMA	client transport	Core schema override (default __mj)
RUN_CROSS_SERVER=1 / MJAPI_A_URL / MJAPI_B_URL	cross-server rig	Gate + the two MJAPI endpoints

3.7 CI

.github/workflows/integration.yml is the tier’s CI home — a blocking PR gate into next (path-filtered to packages/**, the integration metadata dirs, and the workflow itself; also workflow_dispatch). The job:

1. starts SQL Server 2022 in Docker on the runner and creates an empty test database;
2. npm ci, then builds the needed graph with turbo;
3. mj migrate (applies committed migrations — no live CodeGen), then mj sync push --dir=metadata --ci (the default metadata, including the inert Integration Test TestType) and mj sync push --dir=metadata-optional/integration-test --ci (the IT tests, suites, and RLS fixtures — the metadata mj test dispatches from);
4. verifies the RLS fixture users actually landed (a silent no-op seed would let the strongest checks skip-as-pass while the gate stays green);
5. npm run test:integration — the deterministic suite via mj test, pass/fail on its exit code.

PS_INTEGRATION and RUN_AGENT_TESTS are deliberately unset in CI (no token cost, no flakiness) and no MJAPI is up, so the deterministic tier is the gate and client-transport tests are parked/skipped.

---

4. Writing new tests — pick your method

You want to…	Method
Add an invariant to an area that already has a bundle	1 — add a check to the bundle
Cover a new area (new engine, new subsystem)	2 — create a new bundle (+ Test row)
Re-target or recombine existing checks (different dataset/entity, new grouping, new gate)	3 — metadata only
Exercise infrastructure that can’t share one process (multi-server), or prototype freely	4 — standalone rig

Rules of the road (all methods)

• Deterministic by default. No credentials, no LLM calls, no reliance on specific business data. Gate anything else behind the right tier flag.
• Self-cleaning fixtures only. Create your own throwaway rows (clearly-prefixed names, e.g. mj-frbu-test-*, or tagged (mj-integration-test — safe to delete)) and delete them in teardown, FK-safe order. Be reference-only toward pre-existing records.
• Checks throw on failure — use Assert / AssertEqual / AssertRowShape / AssertKeysInclude / AssertKeysExclude from the package.
• Construct fresh RunView param objects per call. The pipeline widens params.Fields in place on cacheable calls; reusing a params object silently turns the second call into an all-fields request. Use a makeParams() factory.
• Fresh UniqueFilter tag for every check that needs a cold cache entry; share a tag across checks only when the warm/hit chain is the thing under test.
• Scope counters to the category (Storage.SetCount('RunViewCache')), and Storage.ResetCounts() at the start of a counter-asserting check.
• await settle(ms) after fire-and-forget saves (run/step/log finalization goes through the async save queue) before reading rows back; outlive the ~5s in-flight dedup linger window (sleep ~5.2s) when the second call must genuinely reach the cache/DB.
• Teardown never throws. Best-effort deletes (.catch(() => undefined)) so a failing check still cleans up.
• Read BypassCache: true when a check must observe true DB state.
• A new bundle needs its metadata sibling — an IT Test record joined to the suite. The bundle is the single source of truth; the metadata record (.IT##-<bundle>.json + its membership in .integration-suite.json) is a thin pointer to it. The suite package’s sibling-parity.test.ts drift-check fails the build if a bundle lacks its IT record / suite membership, if a record points at a non-existent bundle, or if mj.config.cjs stops loading the suite package via testing.checkModules. (The old third sibling — a per-bundle tsx dispatcher — no longer exists.)

Method 1 — add a check to an existing bundle

Find the bundle in packages/TestingFramework/integration-test-suite/src/checks/ and append a NamedCheck to its exported array (the file’s registration loop picks it up):

// in src/checks/dataset-cache.checks.ts — appended to DatasetCacheChecks
{
    Id: 'dataset-cache.DS4',                       // '<bundle>.<next local id>'
    Name: 'DS4: <the invariant, stated as a sentence>',
    Fn: async (ctx: IntegrationCheckContext) => {
        const md = new Metadata();
        // arrange … act … assert (throw on failure)
        Assert(await md.IsDatasetCached(datasetName(ctx)), 'DS4: expected the dataset cached');
    }
    // RequiresMutation: true,   // ← only if it writes (runs under RUN_MUTATION_TESTS / runMutationTests)
    // RequiresLiveModel: true,  // ← only if it calls a model (runs under RUN_AGENT_TESTS)
}

That’s almost the whole change — the corresponding ITxx Test expands the bundle at runtime, so the new check runs with no metadata edit. Mind the ordering: add stateful checks after the state they depend on, and keep them self-contained (reset your own counters, restore anything you flip). Update the bundle’s row in the count table in check-registry.test.ts (the coverage-loss guard). Then verify:

cd packages/TestingFramework/integration-test-suite && npm run build && npm run test
MJ_INTEGRATION_TEST=1 npx mj test run --name "IT07 - Dataset Cache (DatasetCache category)"

Method 2 — create a new bundle (new coverage area)

1. Write the bundle — src/checks/<area>.checks.ts in the suite package (packages/TestingFramework/integration-test-suite/), importing the contracts from the framework:

import { Assert, IntegrationCheckRegistry } from '@memberjunction/testing-integration';
import type { NamedCheck, IntegrationCheckContext } from '@memberjunction/testing-integration';

export const MyAreaChecks: NamedCheck[] = [
    {
        Id: 'my-area.MA1',
        Name: 'MA1: <invariant one>',
        Fn: async (ctx: IntegrationCheckContext) => { /* … Assert(...) … */ }
    },
    {
        Id: 'my-area.MA2',
        Name: 'MA2: <invariant two>',
        Fn: async (ctx: IntegrationCheckContext) => { /* … */ }
    }
];

for (const check of MyAreaChecks) {
    IntegrationCheckRegistry.Instance.Register(check);
}

If the checks share created fixtures, register a lifecycle in the same file — the driver runs it around the bundle (setup before the checks, teardown in finally):

IntegrationCheckRegistry.Instance.RegisterLifecycle('my-area', {
    Setup: async (ctx) => { /* create throwaway rows; assign ctx.<MyAreaFixture> */ },
    Teardown: async (ctx) => { /* delete them FK-safe; never throw */ }
});

(For a typed fixture slot, add an optional property to IntegrationCheckContext in check.ts alongside the existing ones. Ad-hoc knobs can ride the untyped ctx.Config bag instead.)

2. Export the module from the suite package’s src/index.ts (this is what registers it on import):

export * from './checks/my-area.checks';

3. Build: cd packages/TestingFramework/integration-test-suite && npm run build.

4. Add the Test row — metadata-optional/integration-test/tests/integration/.IT31-my-area.json (omit primaryKey/sync; the sync tool populates them):

{
  "fields": {
    "TypeID": "@lookup:MJ: Test Types.Name=Integration Test",
    "Name": "IT31 - My Area",
    "Description": "What the checks prove, stated as invariants.",
    "InputDefinition": {},
    "ExpectedOutcomes": { "summary": "One-line statement of the proven behavior." },
    "Configuration": {
      "tier": "deterministic",
      "transport": "server",
      "checks": [ { "type": "my-area" } ]
    },
    "Status": "Active"
  }
}

For a client-transport bundle, set "transport": "client" explicitly (or add the bundle name to the driver’s CLIENT_BUNDLES set so inference covers it).

5. Add suite membership — in metadata-optional/integration-test/test-suites/.integration-suite.json, append to the right child suite’s MJ: Test Suite Tests:

{ "fields": { "SuiteID": "@parent:ID", "TestID": "@lookup:MJ: Tests.Name=IT31 - My Area", "Sequence": 27, "Status": "Active" } }

6. Update the count table — add the bundle’s row (name, checks array, expected count) to check-registry.test.ts; sibling-parity.test.ts will already be asserting the IT record + suite membership exist.

7. Push and run:

npx mj sync push --dir=metadata-optional/integration-test
MJ_INTEGRATION_TEST=1 npx mj test run --name "IT31 - My Area"

No dispatcher, no aggregator registration — the metadata record IS the entry point.

Method 3 — metadata only (no code)

Because Tests select registered bundles, you can add coverage variants without touching TypeScript:

• Re-target a parameterized bundle — e.g. a Test that runs dataset-cache against a different dataset, or aggregates-cache against a different entity:

  "checks": [ { "type": "dataset-cache", "config": { "datasetName": "MJ_Skills" } } ]

  Per-bundle knobs today: datasetName (dataset-cache), entityName (aggregates-cache), runMutationTests (server-cache / client-cache), requireTwoDistinctUsers (rls-isolation).

• Compose several bundles into one Test — checks is an ordered array; all selected bundles run in one Execute() against one bootstrapped context:

  "checks": [ { "type": "dataset-cache" }, { "type": "aggregates-cache" } ]

• Declare mutation coverage on for a scheduled (non-CI) run profile via "config": { "runMutationTests": true } — declarative, no env needed.

• Regroup with suites — new MJ: Test Suites rows (optionally under the Integration Tests parent) with any membership/ordering; Status: "Skip" on a membership row parks a test without deleting it.

• Gate specially with "requiresEnv": "MY_FLAG" — the driver then skip-passes unless MY_FLAG=1, overriding the tier-derived gate.

Push with npx mj sync push --dir=metadata-optional/integration-test and run via mj test.

Method 4 — standalone rig

For coverage that can’t live inside the one-process model — multi-process topologies (cross-server invalidation), sidecar-dependent end-to-end flows (ps-*), or exploratory work that isn’t ready to be a bundle — write a self-contained script in packages/TestingFramework/integration-test-suite/rigs/. Use the shared harness + the framework bootstrap so the process discipline stays right:

import { TestRunner, Assert, bootstrapIntegrationServer } from './lib/harness';

async function main(): Promise<void> {
    // Self-skip protocol: exit 0 with a clear SKIPPED message when prerequisites are
    // absent, so ad-hoc CI wrappers stay green on boxes without the rig's topology.
    if (process.env.MY_PREREQ !== '1') {
        console.log('my-area-tests: SKIPPED — MY_PREREQ not set.');
        process.exit(0);
    }

    const ic = await bootstrapIntegrationServer({ ContextUserEmail: process.env.MJ_TEST_USER_EMAIL });
    const suite = new TestRunner('My area (standalone)');
    suite.Test('does the thing against the real stack', async () => {
        Assert(true, 'assertion message');
    });

    const failures = await suite.Run();
    await ic.ClosePool();
    process.exit(failures > 0 ? 1 : 0);
}

main().catch(err => { console.error(`\nBootstrap error: ${err}`); process.exit(2); });

Honor the exit-code contract (0/1/2) and keep it self-cleaning. Rigs are not part of the mj test catalog or npm run test:integration — they run only when invoked directly. When a rig’s coverage stabilizes (and it can live in one process), prefer folding it into a bundle (Method 2) so the metadata path runs it too.

---

5. Reference

5.1 Bundle inventory

All bundles live in packages/TestingFramework/integration-test-suite/src/checks/ (the per-bundle count table in that package’s check-registry.test.ts guards against silent coverage loss):

Bundle	Checks	Transport	Lifecycle fixture	Selected by	Notes
server-cache	32 (S17/S23/S24/S29/S30/S31b mutation-gated)	server	—	IT01	RunView server cache: shape parity, fingerprint identity, dedup/linger, BypassCache, invalidation
runquery-cache	Q1–Q12 (12)	server	Query Category + TTL/validated Queries	IT02	RunQuery caching (TTL + smart validation); the bundle mutates by design
client-cache	C1–C13 (13; C10 mutation-gated)	client	—	IT03	CacheLocal opt-in caching + smart cache-check over GraphQL
record-process	RP1–RP8 (8)	server	—	IT04	RecordSetProcessor substrate: persistence, isolation, circuit breaker, concurrency
record-process-facade	RPF1–RPF2 (2)	server	Record Process definition	IT05	RecordProcessExecutor facade (Run/RunByID)
rls-isolation	RLS1–RLS6, RLS8–RLS10 (9)	server	discovered + seeded users	IT06	RLS token substitution, predicate divergence, fingerprint no-leak, live scoping, no-grant negative
rls-isolation-client	RLS7 (1)	client	—	IT23	The client-transport RLS leg
dataset-cache	DS1–DS3 (3)	server	—	IT07	GetAndCacheDatasetByName + status APIs (datasetName knob)
aggregates-cache	AGG1–AGG3 (3)	server	—	IT08	Aggregates in the fingerprint + round-trip + ordering (entityName knob)
scheduled-jobs	SJ1–SJ2 (2)	server	Scheduled Job row	IT09	Run lifecycle + distributed lease
field-rules-bulk-update	FR1–FR3 (3)	server	3 Action Categories	IT10	FieldRules dry-run/apply/conditional gating
remote-operations	RO1–RO7 (7)	server	Template + Record Process + categories	IT11	The Remote Operations primitive, headless full-stack
ai-skills	AS1–AS21 (21)	server	4 skills + grants/junctions	IT12	Skills gates, resolution, SKILL.md round-trip, activation governance — no LLM
api-keys	AK1–AK3 (3)	server	key fixtures	IT13	API Keys engine Config + end-to-end Authorize allow/deny
predictive-studio	PS1–PS5 (5)	server	Pipeline → Model → Binding chain	IT14	PS stack seams (entities, work-type registration, Actions) — no sidecar
remote-op-wire-progress	WIRE1 (1)	client	over-the-wire fixtures	IT15	Remote Operation progress streamed over GraphQL
prompt-runner	PR1 (1)	server	lifecycle	IT16 (live-model)	Real prompt run + persisted MJ: AI Prompt Runs verification
agent-runner	AR1 (1)	server	lifecycle	IT17 (live-model)	Real agent run, deep-verified (steps → prompt runs → action logs → sub-agents)
concurrent	CC1–CC2 (2)	server	lifecycle	IT18 (live-model)	N concurrent prompt/agent runs persist independently
remote-op-ai-authoring	RO4-1→RO4-3 (3)	server	AI-generated Remote Operation	IT19 (live-model)	The AI-authored operation loop (save → approve → emit)
lists	LS1–LS3 (3)	server	List fixture	IT20	Lists substrate + the ListSource keyset-pagination change
open-app-teardown	OAT1–OAT2 (2)	server	seeded Open-App metadata	IT21	Open-App metadata teardown seam
user-routines	UR1–UR16 (16)	server	routine fixtures	IT22	User Routines entity servers + dispatcher, end to end
metadata-consistency	7	server	—	IT24	Read-only audit: entity metadata vs the physical DB catalog
view-execution	9	client	—	IT25	Viewing-system deterministic tier, over the wire
app-wiring	AW1–AW10 (10)	client	—	IT26	Every shipped app wired correctly (parameterized over all apps)
entity-writes	EW1–EW8 (8)	client	—	IT27	Core data write-side contract over GraphQL
permission-engine	14	client	—	IT28	Unified PermissionEngine + scope enforcement, live
cache-gauntlet	8	server	—	IT29	The subset-slot × mutation cell that shipped two production cache bugs
conversation-compaction	CC1–CC12 (12)	server	—	IT30	Compaction assembly layer (graduated from PR #2732)
self-test	cache-warm (1)	server	—	smoke script	Proves the instrumented cache observes RunView traffic (lives in the framework package)

There are no script-only suites anymore — the former lists / user-routines / open-app-teardown scripts graduated to bundles (IT20–IT22), and the multi-process / sidecar-dependent scripts (cross-server-invalidation-tests.ts, agent-memory-tests.ts, runview-matrix-tests.ts, the ten ps-* flows) live on as standalone rigs (§3.5).

Some checks are executable bug reproductions: they encode the correct invariant and stay red until the product fix lands. Each documents its symptom, root cause, and proposed fix inline in the bundle source.

5.2 Key paths

Path	What
packages/TestingFramework/testing-integration/	The framework (published): driver, registry, check contracts, bootstraps, tiers, instrumented cache
packages/TestingFramework/integration-test-suite/	The content (private, never published): all 30 check bundles, their unit tests, and the standalone rigs
metadata/test-types/.integration-test-type.json	The Integration Test TestType — an inert type definition, kept in the normal metadata/ tree
metadata-optional/integration-test/	The optional sibling root — the IT01–IT66 Tests (67 records), the suite hierarchy, the seeded RLS test users/role/permission, AND the synthetic AI stack the live tier drives (14 IT: * agents, 14 prompts, 42 model bindings, a skill, a search scope). One push seeds 242 records. Kept out of the default-pushed metadata/ tree so these test-only records never reach production. Must be seeded once per environment
mj.config.cjs → testing.checkModules	The runtime seam that loads the private suite package (or a consumer’s own check packages) into mj test
packages/TestingFramework/Engine/	TestEngine, BaseTestDriver, suite fixture lifecycle
packages/TestingFramework/CLI/	mj test run / suite / list / validate / history
.github/workflows/integration.yml	The CI gate
scripts/integration-golden-diff.mjs	Per-check outcome diff (EMIT_OUTCOMES format)

5.3 Further reading

• The coverage index: packages/TestingFramework/testing-integration/CATALOG.md — the living catalog: transport doctrine, current bundles, expansion roadmap.
• Per-sub-suite deep docs: packages/TestingFramework/integration-test-suite/docs/ — mechanism + per-check inventory for every shipped bundle family (cache, data-access, security, writes/transactions, AI, platform), plus the candidate test-catalog.
• The restructure rationale: plans/integration-test-expansion/framework-restructure-proposal.md (executed 2026-07-20) and the pointer README at packages/MJServer/integration-test-scripts/README.md for where the old scripts went.
• Test metadata authoring: metadata-optional/integration-test/tests/integration/README.md.
• The Testing Framework generally: packages/TestingFramework/README.md — the TestType / Test / TestSuite / TestRun model shared by every test type (agent evals, computer use, integration).