@memberjunction/predictive-studio-truth-suite
Predictive Studio planted-truth datasets and the harness the containment gate checks use. Repo-local test content (private: true); never published.
The gates themselves are MJ Tests in the integration test suite (packages/TestingFramework/integration-test-suite/src/checks/ps-gate-*.checks.ts). This package supplies what they run on: seeded datasets whose true data-generating process is known before any model is fit, loaded into an isolated pstruth SQL Server schema, and a harness that drives Predictive Studio through MJAPI the way the browser does.
1. What the package holds
Section titled “1. What the package holds”Datasets
Section titled “Datasets”The catalog (src/data/catalog.ts) has 24 entries. With their base seeds that makes 68 datasets.
| Kind | IDs | Base seeds |
|---|---|---|
| Worlds | W1-signal, W3-leak-row, W4-pit, W5-censor, W6-drift, W7-small-imbalanced, W8-highcard, W9-multiclass | 42, 43, 44 and the fresh seed 9042 (4 each) |
| Worlds | W2-noise | 100–119 and the fresh seed 9100 (21) |
| Scenarios | C2, C4, D1c, D1r, D2, N1c, N1r, R1, R2p, R3, T1p, T2p, T3, T4 | one seed each, from the spec file |
| Derived scenario | T3-lag (generated from T3’s output) | T3’s seed |
- World specs are in
fixtures/worlds/and scenario specs infixtures/scenarios/specs/. - The generators are in
src/generators/.fixtures/golden-hashes.jsonpins therows.csvandtruth.csvof every base seed byte for byte (T3-lag included). The W2-noise seeds s120–s159 that G0.2 also loads have no pin. pnpm run truth:datagenerates datasets into the cache (.data/, or$TRUTH_DATA_DIR). The cache key covers the spec, the generator, the files it reads (the worlds domainfixtures/worlds-domain/domain.jsonfor worlds,make_scenario.tsfor T3-lag), the shared generator modules and the Loom pin.
C1 corpus
Section titled “C1 corpus”fixtures/c1-corpus/ holds the C1 corpus: the MoreCheese person-periods (6,673 final-valid dues periods for 2,015 payers, of which 4,873 are the reference’s resolved person-years, listed in person_years_keys.csv) with their orders, non-dues lines and dated activity, one CSV per table, pinned by SHA256SUMS.
python/fixtures/build_c1_corpus.pybuilds it from the pinned MoreCheese corpus and ps-review’sperson_years.parquet. IDs are UUIDv5 under a fixed namespace, so a rebuild is byte-identical, and the builder refuses a build whose counts differ from the reference.migrations/V202610201300__pstruth_c1_corpus.sqlcreates the eightpstruth.C1*tables andmetadata/entities/.pstruth-c1-entities.jsongives their entities the direct-SQL opt-in (node scripts/gen-pstruth-ddl.mjs --metadatawrites it).C1CorpusFixture.LoadC1Corpus(src/fixtures/c1-corpus-loader.ts) replaces the C1 tables’ contents with the fixture after checking the sums and the opt-in.ImmutableFieldDeclarations.Apply(src/fixtures/immutable-fields.ts) setsAllowUpdateAPI = falseon the fieldsTRUTH_IMMUTABLE_FIELDSdeclares fixed at creation.
Harness
Section titled “Harness”src/harness/ is the TruthHarness the gate checks get from RunFixtures.HarnessFor(provider, user). It does the following:
- loads datasets into
pstruth, after checking a base seed’s generatedrows.csvandtruth.csvagainstfixtures/golden-hashes.json(a mismatch throws before anything loads) - creates pipelines and trains them
- runs the Pipeline Builder and Model Development agents through MJAPI
- promotes and scores models
- runs the
holdout_rebuildPython judge
It records the ID of every pipeline and model it creates, and the exact names of every agent run’s pipelines. RunFixtures.CleanupRun looks those names up again (an agent run can create its pipelines after the client call returns), deletes exactly the recorded rows, and closes the harness’s database pool.
Training stores model artifacts through the File Storage Provider PS_STORAGE_PROVIDER names in MJAPI’s environment (default Local File System, rooted at STORAGE_LOCAL_ROOT), whatever its IsActive flag, so the harness switches no provider on. A provider whose driver is not configured on the server refuses every training run before the sidecar is called.
Python
Section titled “Python”python/truthsuite/judges/holdout_rebuild.py is the one Python module a gate calls: G0.12 runs it through h.Judge('holdout_rebuild', ['--model', <artifact>]) to get the always-majority rate of a model’s locked holdout. It reads the class counts training recorded in the model’s lineage (LineageJSON → lockedHoldoutClassCounts) and refuses a model without them. Its tests are in python/tests/test_judges.py.
2. Setup
Section titled “2. Setup”Your .env in the repository root must define DB_HOST, DB_PORT, DB_DATABASE, DB_USERNAME, DB_PASSWORD, CODEGEN_DB_USERNAME, CODEGEN_DB_PASSWORD, GRAPHQL_PORT and MJ_API_KEY. truth:provision and test:db load that file themselves, whatever the current directory; a variable already exported in your shell wins. The database must be a private one (MJ_PS_PR<n>_<agent>, or CI’s test) with the core schema migrated: truth:provision refuses any other name before it runs mj migrate, and the scoped CodeGen config refuses it again. There is no override.
cd packages/AI/PredictiveStudio/TruthSuitepnpm run buildpnpm run truth:loom # clone the Loom pin (fixtures/loom.json) under .loom/pnpm run setup:python # create .venv from python/requirements.txtpnpm run truth:provision # create the pstruth schema, register its entities, push their direct-SQL opt-inspnpm run truth:sidecar # start the single-threaded sidecar; writes its URL to .run/sidecar.urlTo load the C1 corpus, run npx tsx scripts/load-c1-corpus.ts with MJAPI running (it loads the C1 tables, then declares the immutable fields through MJAPI), and restart MJAPI afterwards so it reads the changed field metadata.
Start MJAPI with PREDICTIVE_STUDIO_SIDECAR_URL set to the URL in .run/sidecar.url. After truth:provision, restart MJAPI so it loads the pstruth entities.
3. Running the gates
Section titled “3. Running the gates”MJ_INTEGRATION_TEST=1 RUN_MUTATION_TESTS=1 pnpm mj test suite "Predictive Studio Gates — Containment"The suite first loads the 68 datasets (the loader checks each against its golden hash) and checks that MJAPI reads back each table’s loaded row count, then runs the nine deterministic PR 2 gates. The separate suite “Predictive Studio Gates — Pending” holds the gates that do not block a merge: G0.2 and G0.3 (owned by PR 4), and G0.6, an LLM judge whose verdict can differ between runs on the same text (it needs RUN_AGENT_TESTS=1 and AI_VENDOR_API_KEY__GeminiLLM).
4. Package tests
Section titled “4. Package tests”pnpm test # unit testspnpm run test:db # dataset loading and pstruth isolation against the databasepnpm run test:regeneration # every base seed, generated into a fresh empty cache, matches fixtures/golden-hashes.jsonpnpm run test:python # pytest, then mypy --strict over python/