Skip to content

Predictive Studio Guide

When to read this: Before you touch anything under packages/AI/PredictiveStudio/**, the Predictive Studio dashboard (packages/Angular/Explorer/dashboards/src/PredictiveStudio/), the Knowledge Hub Feature Pipelines panel, the MJ: ML * / MJ: Experiment* entities, the 7 PredictiveStudio.* Remote Operations, the Model Development Agent, or before you add a trained-model capability (feature assembly, training, scoring, experiment search, maintenance). This is the canonical reference for how MJ trains predictive models on a client’s own data and scores records with them.

MJ already runs off-the-shelf models well — embeddings, LLMs, image/audio inference via MJ: AI Models. It does not train models on a client’s own data. Predictive Studio closes that gap: a reasonably technical business user (not a career data scientist) can assemble features from across MJ’s entire data surface, train a genuinely useful predictive model (member retention, renewal, lapse risk, lead scoring, attendee-return), and score who is likely to renew/lapse/return — and the Model Development Agent drives the same object model end to end.

It is core MJ, not an OpenApp. It composes on substrates MJ already has — entities, MJ: Queries, vectors, Record Set Processing, Remote Operations, Actions, Agents, Artifacts — and adds a small, opinionated set of objects + a self-managing Python ML sidecar + a world-class Studio UI + an agent. The differentiation is data assembly + agentic search over MJ’s entire data surface, not algorithmic innovation.


What it isA core MJ capability for training predictive models on a client’s own data and scoring records with them — feature assembly, training, scoring, a budgeted agentic experiment search (built, not wired), and a maintenance/retrain loop (built, not wired).
Who it’s forA reasonably technical business user (renewal / retention / lead-scoring analyst) and an agent — the same object model + the same 7 Remote Operations drive both.
What it is notSageMaker / Databricks. No GPU. No training embeddings from scratch (it uses pre-trained embeddings as features). Deliberately rigid about algorithms (a fixed 6-catalog), flexible about data.
ComputeCPU-bound, no GPU. Gradient boosting / logistic / random forest / small MLP on tabular data train in seconds-to-minutes — matches MJ’s API-runtime infra.
Invocation7 typed Remote Operations (predictive:execute scope) + 6 Actions, delegating to one shared engine layer. The browser, Skip, and the agent all call the same code.
StatusEnd-to-end shipped — sidecar, FeatureAssembly, training, scoring, training under a Prediction Unit and an Evaluation Protocol with a stored three-outcome gate decision, the 7 Remote Ops + 6 Actions, Feature Pipelines, the Model Development Agent + artifact + viewer, multimodal vision-as-feature, the Studio UI, and a live integration test. Two pieces are built, not wired: experiment sessions and the maintenance/retrain loop (see the notes in the experiment-session and maintenance sections). The one explicitly-deferred item is SP5 materialized prediction columns, gated on PR #2770 (see Roadmap).

Proof it works — the live integration test (live-train-score.integration.test.ts, real managed sidecar, generated yearly membership rows with a learnable signal, no DB): a Prediction Unit pipeline trains through the real sidecar trainer under its Evaluation Protocol and the test asserts that

  • the run completes and stores a gate record, counting the 400 locked-holdout rows the protocol carved;
  • a decision that ships creates the MJ: ML Models row from the shipped artifact (ps-predictor/v2, with the Training Snapshot’s ID in its lineage), and a no-signal decision creates none.

The statistical claims (a planted signal is found, a planted null ships nothing) are made by the planted-truth gates (§19.5), not by thresholds in this test.

This guide ties together the packages that each document one layer:

LayerPackageWhat it is
Type contracts@memberjunction/predictive-studio-coresidecar and trainer contracts, pipeline spec, Prediction Unit and Evaluation Protocol shapes, the gate and trust verdict, feature-step DAG, modeling plan, Zod plan/budget validators, shared metric-direction util
Python sidecar@memberjunction/predictive-studio-sidecarMLSidecar — self-managing, bundled FastAPI ML service (managed-spawn default; no Docker)
Server engine@memberjunction/predictive-studioAsOfFeatureAssembler, FeatureAssemblyExecutor, TrainingEngine, MLModelInferenceProcessor, ExperimentOrchestrator, FeaturePipelineEngine, MaintenanceEngine, the 7 Remote Ops + 6 Actions
Studio UI@memberjunction/ng-dashboards (PredictiveStudio/)the lazy-loaded Explorer dashboard + embedded copilot; the Knowledge Hub Feature Pipelines panel
Artifact viewer@memberjunction/ng-artifactsMLExperimentResultsViewerComponent — leaderboard / metrics / importance + drill-through

  1. The four-layer architecture
  2. The self-managing Python sidecar
  3. The type contracts (predictive-studio-core)
  4. The FeatureAssembly executor — the correctness backbone
  5. Training — immutable versioned models judged under an Evaluation Protocol
  6. Scoring — a Record Set Processing work type
  7. The experiment engine — a generic agentic-search primitive
  8. The data model — entities and relationships
  9. The guidance layer — algorithm catalog + the 6×7 matrix
  10. The invocation surface — Actions vs Remote Operations
  11. Feature Pipelines — the category route (no new entity)
  12. The Model Development Agent + artifact + viewer
  13. The security model
  14. The maintenance loop — staleness + challenger-vs-incumbent
  15. Phase 2 — Model as a First-Class Primitive (Reach)
  16. The Business-User Experience
  17. Roadmap / future phases
  18. The Studio UI
  19. Getting started — train and score a model
  20. Quick reference — what’s built

The six invariants this system is built to protect. Each is enforced by code, not convention — they recur throughout the guide:

  1. Anti train/serve skew — one assembly code path for train and score; preprocessing is fit once, applied everywhere, and every fit during training (each inner fold, the finalist refit, each baseline) fits its encoders on that fit’s own training rows only (§4.2, §5.1).
  2. Locked holdout — the protocol’s date-carved (or rolling-origin) holdout, frozen into the Training Snapshot before any candidate trains and read once, for the finalist (§5.1).
  3. Leakage gate — deny-list at assembly + single-feature-dominance flag post-train, blocking gate promotion without sign-off (§4.4, §13).
  4. Point-in-time guard — a Prediction Unit decides each row’s decision date and features are read as of it; a source whose history does not cover the date is refused, and the retired AsOfStrategy modes and asOf input are refused by name (§4.3).
  5. Model immutability — every successful run yields a new versioned MJ: ML Models row, never a mutation (§5.2).
  6. One engine, thin surfaces — every server capability lives in an engine; the 7 Remote Operations and 6 Actions delegate to the shared engine layer (§10).

Four layers, the first of which already existed in MJ. Data flows down; the agent drives across.

flowchart TB
    subgraph DATA["1 · Data layer (existing)"]
        E[Entities]
        Q["MJ: Queries<br/>Status='Approved' = ground truth"]
        EXT["External entities · #2449"]
        VEC[Vectors / Knowledge Hub]
        DOC[DBAutoDoc descriptions]
    end
    subgraph FEAT["2 · Feature layer"]
        FP["Feature Pipelines<br/>(categorized Record Processes)<br/>LLM/derived → write back"]
        FA["AsOfFeatureAssembler<br/>(unit rows + FeatureSteps, as of each decision date) → typed matrix<br/>FeatureAssemblyExecutor assembles score rows for models trained before Prediction Units"]
    end
    subgraph MODEL["3 · Model layer"]
        TRAIN["TrainingEngine (TS orchestration)"]
        SIDE["MLSidecar → Python (CPU)<br/>XGBoost · LightGBM · sklearn"]
        MM["MJ: ML Models<br/>immutable · versioned · lineage"]
    end
    subgraph INFER["4 · Inference layer"]
        RSP["Record Set Processing<br/>MLModelInferenceProcessor ('ML Model' work type)"]
        MAT["Materialization · #2770 (deferred)<br/>physical prediction columns"]
    end
    DATA --> FEAT
    FP -->|persist features| E
    FEAT --> TRAIN
    TRAIN <-->|"/trainer · /predict"| SIDE
    TRAIN --> MM
    MM --> RSP
    MM -.deferred.-> MAT
    RSP -->|optional write-back| E
    AGENT["Model Development Agent<br/>Remote Ops + Actions"] -.drives.-> FEAT & MODEL & INFER
    style FEAT fill:#ecfeff,stroke:#0891b2
    style MODEL fill:#ede9fe,stroke:#6d28d9

The differentiation is data assembly + agentic search over MJ’s entire data surface, not algorithmic innovation. We are deliberately rigid about algorithms (a fixed 6-algorithm catalog) and flexible about data. This is CPU-bound, no GPU — gradient boosting / logistic regression / random forest / small MLP on tabular data train in seconds-to-minutes on CPU, matching MJ’s API-runtime infra.

LayerOwns§
1 · Data (existing)Entities, MJ: Queries, external entities (#2449), vectors, DBAutoDoc — all become feed-ins via RunView/RunQuery—
2 · FeatureAsOfFeatureAssembler (training and unit-model scoring read every row as of its decision date) + FeatureAssemblyExecutor (scoring of models trained before Prediction Units) + Feature Pipelines (persisted derived features)§4, §11
3 · ModelMLSidecar + TrainingEngine (trains under a Prediction Unit and an Evaluation Protocol) → immutable versioned MJ: ML Models when the gate ships one§2, §5
4 · InferenceMLModelInferenceProcessor (RSP work type); population-wide indexed columns are deferred to #2770§6, §17

Node is poor for ML training; Python is excellent. So MJ (TypeScript) assembles the matrix and orchestrates, and a Python sidecar does the CPU-bound fitting and inference. The key engineering choice: the sidecar is self-managing and Docker-free by default.

2.1 MLSidecar — managed-spawn is the default

Section titled “2.1 MLSidecar — managed-spawn is the default”

MLSidecar (packages/AI/PredictiveStudio/Sidecar/src/ml-sidecar.ts) follows the @memberjunction/sqlglot-ts pattern: the Python microservice is bundled inside the npm package (src/python/) and the TypeScript class spawns it as a child process on demand. No Docker is required for local or embedded use.

flowchart LR
    subgraph TS["TypeScript (MJ process)"]
        MS["MLSidecar"]
    end
    subgraph PY["bundled Python (src/python/)"]
        SRV["server.py → uvicorn"]
        APP["app/main.py (FastAPI)<br/>/trainer · /predict · /health"]
    end
    MS -->|"spawn() on 127.0.0.1<br/>ephemeral port"| SRV
    SRV -->|"stdout: PREDICTIVE_STUDIO_SIDECAR_PORT=N"| MS
    MS -->|"poll /health until ready"| APP
    MS <-->|"HTTP trainer/predict"| APP
    style MS fill:#0891b2,color:#fff

Two topologies, chosen automatically:

ModeWhenWhat Start() does
Managed (default)nothing configuredSpawns the bundled FastAPI service on 127.0.0.1 with an OS-assigned ephemeral port, reads PREDICTIVE_STUDIO_SIDECAR_PORT=<n> from its stdout, polls /health until ready, and registers SIGINT/SIGTERM/exit cleanup so the child dies with the parent.
Remoteurl option or PREDICTIVE_STUDIO_SIDECAR_URL env setConnects only — no child process — and just verifies /health. Use for a containerized/horizontally-scaled sidecar. Stop() is then a no-op (this client never owned the process).

The public surface (Sidecar/src/index.ts): MLSidecar, SidecarError, TrainerResultMalformedError, plus the MLSidecarOptions and SidecarHealthResponse types.

import { MLSidecar } from '@memberjunction/predictive-studio-sidecar';
const s = new MLSidecar(); // managed mode by default
await s.Start(); // spawns the bundled Python service
const trained = await s.RunTrainer({ /* TrainerRequest */ }); // POST /trainer
const { predictions } = await s.Predict({ /* PredictRequest */ });
await s.Stop(); // SIGTERM the child (no-op in remote mode)

MLSidecar exposes IsRemote, IsRunning (remote always true; managed needs a live child), and Port (ephemeral in managed, null in remote). On macOS it automatically appends DYLD_LIBRARY_PATH=/opt/homebrew/opt/libomp/lib to the spawn environment so XGBoost/LightGBM find the OpenMP runtime.

2.2 npm run setup:python — the one-time bootstrap

Section titled “2.2 npm run setup:python — the one-time bootstrap”

One-time setup (fresh clone). Predictive Studio’s packages depend on other workspace packages, so build them together, never one by one (a per-package tsc on a fresh clone fails with TS2307 Cannot find module '@memberjunction/core'):

Terminal window
pnpm install # repo root
pnpm run build # repo root: builds PS and everything it depends on, in dependency order
pnpm --filter @memberjunction/predictive-studio-sidecar run setup:python

setup:python needs CPython 3.11 or 3.12 (numpy 1.26.4 has no wheels for 3.13+); it tries python3.11, python3.12, python3, python and stops with the list it probed if none fits. Pass -- --python <path> after the setup:python command to choose one. On macOS also brew install libomp (XGBoost/LightGBM). To run the sidecar or its tests on another interpreter, set PREDICTIVE_STUDIO_PYTHON=<path to python>.

The pinned requirements.txt includes FastAPI + uvicorn, xgboost, lightgbm, scikit-learn, numpy, scipy, pandas, joblib, and the test deps. The setup script (scripts/setup-python.mjs) creates the venv, upgrades pip, and installs requirements; MLSidecar picks its interpreter in this order: an explicit pythonPath option, then PREDICTIVE_STUDIO_PYTHON, then that bundled .venv, then python3.

The contract is defined once in @memberjunction/predictive-studio-core (trainer-contract.ts for training, sidecar-contract.ts for scoring) and mirrored by the Python Pydantic models (app/trainer_schemas.py, app/schemas.py). MJ freezes the matrix into a Training Snapshot and orchestrates; the sidecar selects, scores and serves.

EndpointRequest → ResponseRole
GET /health→ SidecarHealthResponse (status, algorithms[], cached_models)liveness + registered driver keys + warm-cache depth
POST /trainerTrainerRequest → TrainerResult (completed, or failed with its stage)one protocol-driven training: downloads the snapshot’s matrix and row manifest (SHA-256 verified), selects a finalist on the inner folds of the development rows, scores the locked holdout once, fits the baselines, and returns the scores with intervals, calibration, leakage-audit flags, permutation importance and the artifacts. A request that must not run (for example a primary score of another task family) is a 400
POST /predictPredictRequest → PredictResponseapplies a stored predictor to aligned rows and returns predictions[]: a ps-predictor/v2 artifact applies its own in-fold encoding; a model trained before Prediction Units applies its frozen fitted_preprocessing. Never re-fits

The retired /train route (and MLSidecar.Train) are gone: nothing trains outside the trainer, and models already trained keep scoring through /predict.

The Python side: server.py resolves the port (0 → OS picks free), prints PREDICTIVE_STUDIO_SIDECAR_PORT=<n> to stdout, and runs uvicorn; app/main.py is the FastAPI app; app/trainer.py, app/selection.py, app/evaluation.py, app/baselines.py, app/calibration.py, app/intervals.py and app/gate.py are the trainer’s stages; app/encoding.py is the typed in-fold encoder (§4.2); app/algorithms.py is the driver registry (the DriverClass values seeded in MJ: ML Algorithms: xgboost, lightgbm, logistic_regression, random_forest, ridge, mlp); app/preprocessing.py is the fit/transform core that models trained before Prediction Units still score through (§4.2); app/artifacts.py loads the legacy envelope and runs the warm LRU model cache (max 32, SHA-256 keyed) that keeps single-record interactive scoring fast, and app/artifacts_v2.py is the ps-predictor/v2 envelope.

Model artifact envelopes (base64 on the wire, stored in MJStorage). A model trained under a Prediction Unit is a ps-predictor/v2 envelope: { format: "ps-predictor/v2", version: 2, payloadB64, featureColumns, taskKind, classOrder, featureSchemaHash, driverBaseline, transformedColumns }, where the payload is one fitted trainer predictor (in-fold encoding, estimator and optional calibration map), so the encoding travels with the model. A model trained before Prediction Units keeps its { format: "joblib", version, payload_b64, fitted_preprocessing, feature_schema } envelope; its fitted preprocessing is the anti-skew payload (§4.2).


3. The type contracts (@memberjunction/predictive-studio-core)

Section titled “3. The type contracts (@memberjunction/predictive-studio-core)”

The shared vocabulary across the sidecar, the engine, the UI, and the agent. The package has more than thirty modules (its README lists them all); these are the ones the rest of this guide leans on:

FileDefines
sidecar-contract.tsthe /predict contract: PredictRequest/PredictResponse, Prediction, PredictionContribution, FeatureSchemaEntry, PreprocessingOp, MatrixData; scalars FeatureKind, ProblemType, ModelMetrics, FeatureImportance, ImportanceKind, FittedPreprocessing (the parameters a model trained before Prediction Units carries)
trainer-contract.tsthe /trainer contract: TrainerRequest, TrainerResult (completed / failed), CandidateSpec, PreAuthDownload, the returned artifacts, PREDICTOR_ARTIFACT_FORMAT
prediction-unit.ts, task-type.ts, unit-edit.tsthe Prediction Unit shapes (unit rows, decision date rule, label rule, censoring, change-history policy, task type) with their parser (PredictionUnit.ParsePredictionUnitRow), and the version-saving edit rules
assembly-contract.tsthe as-of assembly contract: the dated sources a feature reads (DatedSource: record-changes, immutable-field, feature-value, windowed-aggregate, series), how each column is encoded (ColumnEncoding), the change-history coverage report (ChangeHistoryCoverage), every assembly refusal (AssemblyRefusal), per-column and per-value provenance, and the snapshot’s row manifest (SnapshotRowKey)
snapshot-format.ts, holdout-carve.tsthe Training Snapshot file shapes (matrix header and cells, provenance line, reserved columns) and the locked-holdout carve each protocol declares (date-carved, seeded-random, rolling-origin)
evaluation-protocol.ts, evaluation-protocol-rules.ts, protocol-edit.tsthe Evaluation Protocol shapes (holdout, inner validation, primary proper score, baselines, calibration, intervals, promotion rule, leakage audit, exclusions, budgets) and their guard-field rules
selection.tsInnerScoreSelection: the check of the trainer’s selection report (every candidate scored on the inner folds, the holdout read once, for the finalist) and the leaderboard rows built from the inner-fold scores
gate.ts, run-evaluation.ts, training-run-results.tsthe three-outcome gate (GateRules, GateRecords), the stored gate record, and the ps-run-results/v2 envelope a training run stores
trust.tsTrustVerdicts.DeriveTrustVerdict: the stored gate decision rendered as plain copy, with the CanAct gate the catalog and the agent’s builder share
refusal-codes.tsPredictiveStudioRefusalCodes: the typed ResultCodes a refused Train or Score request returns (§10.4)
pipeline-spec.tsSourceBinding, LeakageGuard, ValidationStrategy (the declarative shape of a training pipeline) and the retired AsOfStrategy (the unit decides decision dates now)
feature-steps.tsthe visual FeatureStep DAG — a discriminated union on Kind (11 kinds: select/windowed-aggregate/series/impute/standardize/onehot/bin/embedding/llm-derived/flow-agent/vision-llm) + FeatureStepGraph
modeling-plan-spec.tsModelingPlanSpec (the Model Development Agent’s strongly-typed payload), Budget, LeaderboardEntry
modeling-plan-schema.tsZod ModelingPlanSpecSchema / BudgetSchema + ValidateModelingPlanSpec() / ValidateBudget() — the runtime plan/budget validators the Start-Experiment op and Run-Experiment action call before any iteration runs
metrics-util.tsMetricDirections.IsErrorMetric(metricKey) / MetricDirections.MetricDirectionOf(metricKey) — the lower-is-better set is read from Core’s METRIC_DIRECTION declaration (plus the legacy spellings such as mse/loss/logloss/log_loss) — lifted to Core so the leaderboard (§7) and the challenger-vs-incumbent comparison (§14) cannot drift on metric direction
import type { ModelingPlanSpec, TrainerRequest, FeatureStepGraph } from '@memberjunction/predictive-studio-core';
import { ValidateModelingPlanSpec, MetricDirections } from '@memberjunction/predictive-studio-core';

Note on dependencies: Core holds the contracts and the browser-safe runtime logic every layer must agree on (the gate, the trust verdict, the protocol and unit parsers and edit rules, the holdout carve, the metric directions, the validators). Its two runtime dependencies are zod (the validators of stored and wire JSON and of the modeling plan) and @memberjunction/global (UUID comparison, ordinal ordering, canonical JSON); it has no server or database dependency.

Import the sidecar contract types from here, not from the Sidecar package (which only re-uses them). MJ’s TypeScript rules (.claude/rules/typescript-style.md) forbid re-exporting another package’s shapes.


4. The FeatureAssembly executor — the correctness backbone

Section titled “4. The FeatureAssembly executor — the correctness backbone”

Two classes assemble features, one per generation of model, and each is the single code path of its generation:

  • AsOfFeatureAssembler (Engine/src/feature-assembly/as-of/as-of-assembler.ts) assembles the training matrix of every Prediction Unit pipeline (Context: 'train') and the score rows of the models trained that way (Context: 'score'). It reads every feature as of each unit row’s decision date (§4.3).
  • FeatureAssemblyExecutor (Engine/src/feature-assembly/feature-assembly-executor.ts) assembles the score rows of models trained before Prediction Units, from the recipe those models stored. Training no longer calls it. Its one public method —
public async assemble(params: FeatureAssemblyParams): Promise<FeatureAssemblyResult>

— turns (record set, frozen FeatureSteps spec) → feature matrix. It refuses windowed-aggregate and series steps, which need a Prediction Unit pipeline.

A model’s own Lineage decides which generation it belongs to: a Lineage that carries predictionUnitId (written only by Prediction Unit training) takes the unit path, and every other readable Lineage keeps scoring on the executor path, even after its pipeline has been migrated to a unit. That single-path-per-generation property is what prevents train/serve skew by construction.

flowchart LR
    UR["Unit rows + decision dates<br/>(the Prediction Unit)"] --> AA{{AsOfFeatureAssembler.Assemble}}
    SPEC[Pipeline FeatureSteps] --> AA
    AA -->|"Context 'train'"| SNAP["Training Snapshot<br/>typed matrix · row manifest · provenance<br/>(frozen, hashed)"]
    SNAP --> TR["sidecar /trainer<br/>in-fold encoding + estimator in one pipeline"]
    TR --> MODEL["MJ: ML Models<br/>ps-predictor/v2 artifact + Lineage<br/>(typed columns · kept levels · snapshot)"]
    MODEL -->|"the encoding travels inside the artifact"| SP
    AA -->|"Context 'score'"| SR["typed score rows"]
    SR --> SP["sidecar /predict<br/>APPLY only"]

Scoring runs the same assembler with the typed columns and kept categorical levels that training stored in the model’s Lineage (UnitModelLineage), so a feature is computed identically at training and at scoring time. For models trained before Prediction Units, FeatureAssemblyExecutor.AssemblyContext is 'train' | 'materialize' | 'on-demand' and scoring uses 'on-demand'.

4.2 In-fold encoding (anti train/serve skew)

Section titled “4.2 In-fold encoding (anti train/serve skew)”

For a Prediction Unit pipeline the assembler emits typed columns (numeric, one-hot categorical with at most 50 levels and the rest folded into one __other level, plus a missing-value indicator beside every imputed column). The sidecar’s app/encoding.py turns the snapshot’s per-column encodings into one unfitted ColumnTransformer that is always placed inside one sklearn Pipeline with the estimator. Every fit during training (each inner fold, the calibration fold, the finalist refit, each baseline) therefore fits its imputers, scalers, vocabularies and target encodings on that fit’s own training rows only, so no statistic is learned from rows the fit is later judged on. A non-numeric value in a numeric column is refused rather than mapped to 0. The whole fitted predictor is the artifact (ps-predictor/v2), so /predict applies the same encoding to a score row and never re-fits; MLModel.FittedPreprocessing is {} for these models. src/python/tests/test_encoding.py locks the in-fold rule down.

Models trained before Prediction Units keep the earlier split, because they keep scoring. FeatureAssemblyExecutor (partitionSteps) partitions the FeatureStep graph into two kinds of step:

flowchart TB
    G["FeatureStep graph"] --> SPLIT{{"executor partitions<br/>the graph"}}
    SPLIT -->|"data-assembly steps<br/>select · embedding · llm-derived · flow-agent · vision-llm"| RAW["RAW matrix<br/>built in TypeScript<br/>(RunView · Query · external · vectors · vision-LLM)"]
    SPLIT -->|"preprocessing steps<br/>impute · standardize · onehot · bin"| RECIPE["PreprocessingOp[] recipe<br/>NOT applied in TS — shipped to sidecar"]
    FITTED["FittedPreprocessing<br/>(frozen when the model was trained)"] -.stored with the model.-> MODEL["MJ: ML Models"]
    RAW --> PRED["sidecar /predict"]
    RECIPE -.same recipe.-> PRED
    MODEL --> PRED
    PRED -->|"transform() — APPLY frozen params only · never re-fit"| OUT["aligned predictions"]
    style FITTED fill:#fef3c7,stroke:#d97706
    style RECIPE fill:#ecfeff,stroke:#0891b2
  • Data-assembly steps (select, embedding, llm-derived, flow-agent, vision-llm) → produce the raw matrix in TypeScript, drawing from RunView/RunViews, Query bindings, external entities, persisted vectors, and (for vision-llm) a per-row vision prompt (§12 note: it’s a dedicated kind, distinct from llm-derived). For llm-derived steps, assembly reads the live column value on the source record without pinning a MJ: Feature Pipeline Versions row.
  • Preprocessing steps (impute, standardize, onehot, bin) → are NOT applied in TypeScript. They are emitted as a PreprocessingOp[] recipe and shipped to the sidecar.

Why: Stateful transforms (normalization mean/std, one-hot vocabulary, bin edges, imputation fill values) must be fit once on the training data and then only applied — never re-fit — at inference. For these models the sidecar (app/preprocessing.py) did exactly this: fit_transform() at the retired /train route learned the params and returned them as fitted_preprocessing; transform() at /predict only applies those frozen params (“APPLY ONLY, never re-fit”). The fitted preprocessing was serialized into MLModel.FittedPreprocessing alongside the weights and travels with the model. This is why FeatureSchema alone is insufficient: the fitted pipeline is part of the model’s identity. The Python golden test (src/python/tests/test_preprocessing_golden.py) locks this down — the same raw row produces an identical transformed vector at train and at predict.

Required raw columns are guaranteed present at score time (anti-skew, the data half; FeatureAssemblyExecutor path). The preprocessing split above guarantees the transforms match at train and score. A second skew vector is the raw data itself: on-demand / scheduled scoring receives its records from an upstream Record-Set-Processing scope, which may have loaded them with a narrow field projection that dropped a virtual / denormalized column (e.g. a value-list field joined into the entity view but absent from the base table). The model trained on that column; missing it at score time silently degrades every prediction toward a constant. assemble() now closes this at the single seam where it can enter: after building the column plan, hydrateMissingFeatureColumns re-reads any absent required column from the same entity view the model’s training read (dataAccess.fetchRows, keyed by primary key) — columns already present are untouched, so full entity_object loads pay no second read and no blob tax. Then assertRequiredColumnsPresent hard-fails if a required column is still absent: MLModelInferenceProcessor’s per-record try/catch marks that run Failed with a precise message that bubbles to the run history + Production-tab UI — no silent “Succeeded” with degenerate output. Locked down by a train/serve parity unit test (same rows inline-vs-hydrated → byte-identical matrix) and an integration assertion that on-demand predictions vary across a multi-type scope (COUNT(DISTINCT score) > 1). See plans/predictive-studio-scoring-virtualfield-skew.md for the full find→fix write-up.

For forward prediction, features are read as they were at the decision point rather than as they stand today. The Prediction Unit owns the decision date. Its DecisionDateRule is a column on the unit row, a column-offset (that column plus an offset in days), or a fixed date, which with advance: 'each-run' moves to the run’s UTC day before each training run by saving the next unit version (a second run on the same day saves nothing); scoring dates such a unit’s rows at the scoring run’s own UTC day. AsOfFeatureAssembler (feature-assembly/as-of/as-of-assembler.ts) then reads each feature for each unit row as of that row’s date:

Source of a columnRead as of the decision date from
a field the entity does not allow to be updated through the API (AllowUpdateAPI = false)the stored value (it cannot change)
any other field, and every __mj_ system column (MJ declares them not updatable, but __mj_UpdatedAt changes on every save)the MJ: Record Changes history of the record
a virtual field (a value the entity’s view joins from a related record)the stored value, only when every row is decided on the assembly’s day or later (a fixed decision date that advances each run, as the Pipeline Builder and the migration create); otherwise the read is refused (virtual-field), since the stored value is the related record’s value today; select the field from the related entity through its key (a select with Entity and KeyField)

Every column has a dated source (DatedSource, assembly-contract.ts): a field of the row’s entity or of an entity it joins to (record-changes or immutable-field, from the table above), a child table aggregated over a window that ends on the decision date (windowed-aggregate), a pinned Feature Value (feature-value), or a declared lag or rolling value of a forecast series (series). There is no “today’s value” source. Each column is typed from its field (ColumnEncoding: numeric, categorical, date parts, boolean, or an identifier that is excluded), and every column gets an imputer with a missing-value indicator (§4.2). The embedding, flow-agent and vision-llm steps have no as-of reading and are refused (unsupported-step-kind), an llm-derived step without a pinned Feature Pipeline version is refused (unpinned-feature-version), and a read that returns fewer rows than it counted is refused (truncated-read): there is never a partial matrix.

Coverage. Every Record Changes history the assembly reads is reported in its result’s Coverage, one entry per entity and field: where the history starts and how many records its continuity check replayed and found mismatched. A unit row whose replayed value cannot be read (the history does not reach its decision date, or reaches it but cannot be replayed) is uncovered, and the unit’s ChangeHistoryPolicy.onUncoveredRow decides what happens to it: refuse refuses the whole assembly, and exclude-and-count leaves the row out and counts it per entity and field on the Training Snapshot (UncoveredHistoryRowsExcluded). A stored value read directly that the history cannot vouch for is refused whatever the policy (below).

A feature the assembler cannot read as of a date is refused, never approximated: AS_OF_SOURCE_UNDATED carries decision-date-before-change-history (the history does not reach back to the date) or select-changes-after-decision-date (a stored value changed after the date). On a tracked entity, a read for a decision day on or after the assembly day uses the history whenever it covers that day, and the stored value only where there is no covering history.

AsOfStrategy (none / column / offset) is the retired pipeline-level form of the same idea. Training refuses a pipeline whose stored AsOfStrategy.Mode is not none (input-not-accepted, because the unit decides decision dates), and PredictiveStudio.TrainModel refuses any asOf input, whatever its mode (AS_OF_NO_EFFECT, §10.4). The “Migrate Pipelines to Prediction Units and Evaluation Protocols” Action (§10.2) moves a column as-of into the unit’s decision date and resets the pipeline’s AsOfStrategy to none; it archives a pipeline whose as-of is an offset.

An automated feature-search agent will relentlessly exploit target leakage (a field that’s a proxy for the label → AUC ~0.99 garbage). Complementary defenses (the first two live in feature-assembly/leakage-guard.ts):

  1. Deny-list enforcement (assembly-time). On the unit path the union of the pipeline’s LeakageGuard deny lists and the bound protocol’s exclusions keeps fields and sources out of the matrix (TrainingEngine.EffectiveExclusions), so neither list can bring back a column the other keeps out. LeakageGuardEnforcer normalizes the LeakageGuard.DenyFields / DenySources into case-insensitive sets; deny-listed fields/sources are filtered out of the matrix before any column is produced (IsFieldAllowed, IsSourceAllowed, PartitionColumns).
  2. Single-feature-dominance detection (post-train). DetectSingleFeatureDominance(featureImportance, threshold) normalizes the finalist’s held-out permutation importance to shares of total importance (using magnitudes, so signed coefficients work) and flags when the top feature’s share exceeds SingleFeatureDominanceThreshold (e.g. 0.6). The flagged run carries a LEAKAGE WARNING note and its model stays Draft.
  3. The protocol’s leakage audit (in the trainer). The Evaluation Protocol’s LeakageAudit (a single-feature held-out AUC ceiling, a time-shifted-label test, a drop-column test, per-source-column importance, a high-cardinality-identifier check) runs inside /trainer; its flags come back in the trainer result and the Studio’s evaluation panel lists them.

When a run is flagged, the system does not silently proceed and does not auto-promote. It surfaces a clear, business-person-friendly warning — “One field is doing almost all the predicting — this often means we’re accidentally peeking at the answer. The model is kept in Draft until a promote request records a sign-off with a reason.” — and blocks promotion until a promote request records a sign-off with an auditable reason (the promotion sign-off gate, §13).


5. Training — immutable versioned models judged under an Evaluation Protocol

Section titled “5. Training — immutable versioned models judged under an Evaluation Protocol”

TrainingEngine (Engine/src/training/training-engine.ts) orchestrates the whole train path:

public async trainModel(input: TrainModelInput, deps: TrainingDeps): Promise<TrainModelResult>

TrainModelInput is { pipelineId, experimentSessionIterationId?, sidecarVersion?, candidates?, timeoutSeconds? }; the result is { model, run }, and model is null when the stored gate decision ships no artifact.

A run passes through six objects in order: the Prediction Unit decides which rows exist, their decision dates and their labels; the as-of assembler reads every feature of every row as of its decision date (§4.3); the Training Snapshot freezes the assembled rows into fingerprinted files; the Evaluation Protocol decides the locked holdout carved into that snapshot and how candidates are judged; the trainer (the sidecar’s /trainer) selects a finalist on the development rows and scores it once on the holdout; and the gate turns the stored scores into one of three outcomes. The steps:

  1. Resolve the pipeline and refuse before any row is written — a pipeline without a Prediction Unit (unit-required) or without a readable Evaluation Protocol (protocol-required), more candidates than the protocol’s budget allows, a pipeline that declares a time index (ValidationStrategy.TimeIndexColumn) under a protocol whose holdout does not respect time (time-index-needs-time-holdout), and a stored AsOfStrategy that is not none (input-not-accepted) all stop here, and so do deps without a snapshot file store or a trainer. No run row and no unit version exists yet.
  2. Advance the decision date and create the run row — a pipeline that passed step 1 first moves a fixed decision date with advance: 'each-run' to the run’s UTC day (a new unit version), then creates the MJ: ML Training Runs row (Status='Running'). The protocol row must predate the run row, and a protocol that does not fit the unit is refused on the run.
  3. Read the unit rows — every row of the Prediction Unit; censored and label-known rows are excluded and counted, and the target is validated (a single-valued target is refused).
  4. Assemble as of each decision date — AsOfFeatureAssembler (Context: 'train') builds the typed matrix (§4).
  5. Freeze a Training Snapshot — the matrix, the row manifest and the provenance are stored as files and hashed into an immutable MJ: ML Training Snapshots row; the locked holdout is the one the protocol declares, carved on the protocol’s cutoff before anything is fitted; identical content reuses the existing snapshot.
  6. Call the sidecar trainer (/trainer) — it is sent pre-authorised downloads of the snapshot files and returns the inner-fold scores, the baselines, the finalist’s single read of the holdout, calibration, leakage flags and the artifacts.
  7. Decide the gate in TypeScript (GateRecords.BuildStoredGateRecord) — a sidecar decision that differs from the transcription refuses the run (sidecar-gate-mismatch).
  8. Persist what ships — each returned artifact goes to MJStorage (MJ: Files) via an injected store, and the immutable MJ: ML Models row (Status='Draft') is created from the artifact the decision ships only: the finalist’s for model, the kept simple baseline’s for keep-simple-baseline, none for no-signal or a kept heuristic rule. Its Lineage names the unit version, the typed feature columns, the kept categorical levels scoring reuses, the snapshot’s ID and content hash, and the protocol.
  9. Leakage check — a single feature dominating the finalist’s importance flags the run (a LEAKAGE WARNING note) and the model stays Draft until a promote request records a sign-off.
  10. Finalize the run (Completed, or Failed with the refusal or failure recorded) — ValidationResults holds the ps-run-results/v2 envelope (§8); ComputeSeconds and CodeVersion are recorded.
sequenceDiagram
    participant C as Caller
    participant TE as TrainingEngine
    participant FA as AsOfFeatureAssembler
    participant SC as MLSidecar (Python)
    participant ST as MJStorage (MJ: Files)
    participant DB as MJ: ML Training Snapshots / ML Models / Training Runs
    C->>TE: trainModel(input, deps)
    TE->>TE: refuse unit-less, protocol-less, time-index and as-of pipelines (no rows written)
    TE->>DB: create Training Run (Status='Running')
    TE->>FA: assemble(Context='train') as of each decision date
    FA-->>TE: typed matrix + provenance
    TE->>ST: freeze Training Snapshot (matrix · row manifest · provenance)
    TE->>SC: POST /trainer (snapshot downloads + candidates + protocol)
    SC->>SC: inner-fold selection on development rows · baselines · holdout scored ONCE
    SC-->>TE: TrainerResult (scores + intervals · calibration · leakage flags · artifacts)
    TE->>TE: decide the gate (model / keep-simple-baseline / no-signal)
    TE->>ST: persist the artifact the decision ships
    TE->>DB: create immutable ML Model (Status='Draft') when one ships
    TE->>DB: finalize Training Run (ps-run-results/v2 envelope)
    TE-->>C: { model | null, run }

The engine is built on dependency-injection seams (src/training/types.ts + src/training/seams.ts): IEntityFactory, IRecordLoader, IArtifactStore, and on the unit path the snapshot file store, ITrainerEvaluator, and the Prediction Unit, Evaluation Protocol and Training Snapshot stores. Production wires MetadataEntityFactory, RunViewRecordLoader, StorageProviderArtifactStore and MJSidecarTrainer; tests substitute in-memory fakes (which is what makes the engine unit-testable with no DB and no live sidecar — see the live integration test in §19.4).

5.1 Evaluation Protocol and the locked holdout

Section titled “5.1 Evaluation Protocol and the locked holdout”

Every training run uses the Prediction Unit and the Evaluation Protocol its pipeline binds; a pipeline without either is refused. Both are versioned as immutable rows: “editing” saves the next version, and a run, its snapshot and its model point at the exact version rows they used. The protocol, saved before the first candidate trains, declares the locked holdout (date-carved: every row whose date is on or after the cutoff, optionally before an end date; or rolling-origin for forecasts, at least five origins; or, for a unit with one fixed decision date, a seeded-random row-hash fraction, which a pipeline that declares a time index is refused under), the inner validation over development rows (repeated or grouped folds; blocked in time with a gap of at least the horizon for series), the primary proper score of the unit’s task family (log loss, multiclass log loss, RMSE/MAE, Poisson or Tweedie deviance, MASE), the baselines, calibration, intervals, the leakage audit and the columns that never become inputs, slices, the promotion margin and hard-stopping budgets. The promotion rule, leakage audit, holdout, exclusions and budgets are guard fields: they are set in protocol configuration — the UI, a non-agent API caller, a scheduled job, the defaults or shipped metadata — and never by an agent tool call; a protocol can optionally restrict them to users (GuardFieldsEditableBy: human-only). The rule reads each version’s AuthoredBy, so the server refuses a version whose AuthoredBy names a user other than the one saving it, and a pipeline rebound to a protocol of another lineage, or back to an earlier version of its own lineage, is held to the same rule: when the guard fields differ, both the new protocol’s author and whoever rebinds (the saving user; the system when there is none) must be ones the previous protocol’s setting allows. Moving a pipeline forward within its lineage needs no check, because each later version’s save applied the rule. Protocol, unit and snapshot rows also refuse a ReplayOnly save once saved, since that option skips validation but still writes.

Candidates are ranked by their inner-fold mean with a 95% interval; the winner’s inner score is optimistic and labelled so. The locked holdout is read once, for the finalist, and the run stores the paired intervals against the flat guess (the dummy baseline) and the simple baselines, then the three-outcome gate: model (the model beats the flat guess and the best simple baseline), keep-simple-baseline (a simple baseline carries the signal), or no-signal. Nothing ships when the rarest class has fewer test rows than the protocol’s minimum (20 by default) or a binary model’s AUC interval contains 0.5. The decision and its reason are stored on the training run; publishing is a separate, automated step that reads that stored decision.

The seed in the protocol fixes every random state and the run uses one thread, so the same rows and settings give the same model. The pipeline’s ValidationStrategy.MajorityBaselineRule ("enforce" or "warn") is still checked before training (an unknown value is refused with INVALID_EVALUATION_SETTING) and recorded in the model’s Lineage.evaluation, but the stored gate decision does not read it. Training also records which kind of feature importance the trainer produced (ImportanceKind); a model’s global drivers are shown only for coefficient and held-out permutation importances, never for split or impurity counts.

The integration test (§19.4) exercises this end to end through the real sidecar trainer: the protocol’s holdout is carved into the Training Snapshot before training, and the run stores its gate decision with the holdout row count.

5.2 MJ: ML Models is distinct from MJ: AI Models

Section titled “5.2 MJ: ML Models is distinct from MJ: AI Models”
MJ: AI ModelsMJ: ML Models
Whatoff-the-shelf foundation models we call (LLMs, embeddings, image-gen)predictive models we train from client data
Inference pathvendor API + driver classthe Python sidecar
Produced byseeded vendor metadataa TrainingEngine run whose gate decision ships an artifact

An ML Model is immutable + versioned — each run that ships an artifact yields a new row, never a mutation (a no-signal run yields none). It may reference an AI Model in its Lineage (e.g. the embedding model used to build features) — a pointer, not membership. The generated entity classes are MJMLModelEntity / MJMLTrainingPipelineEntity in @memberjunction/core-entities. Its lifecycle is the Draft → Validated → Published → Archived state machine enforced at promotion (§13).


6. Scoring — a Record Set Processing work type

Section titled “6. Scoring — a Record Set Processing work type”

Scoring composes onto Record Set Processing (see the Record Set Processing Guide) rather than re-implementing batching/concurrency/audit. MLModelInferenceProcessor (Engine/src/scoring/ml-model-inference-processor.ts) is a new work type alongside the substrate’s built-in FieldRules / Action / Agent / Infer:

@RegisterClass(MLModelInferenceProcessor, 'ML Model')
export class MLModelInferenceProcessor implements IRecordProcessor {
public async ProcessRecord(record: RecordRef, context: RecordProcessorContext): Promise<RecordResult>
public async ProcessBatch(records: RecordRef[], context: RecordProcessorContext): Promise<RecordResult[]>
}

⚠️ Disambiguation: the existing Infer work type runs an AI Prompt per record — that is LLM inference, not ML inference. The new ML Model work type (MLModelInferenceProcessor) runs a trained ML model through the sidecar. Don’t confuse them. (The Record Set Processing Guide §4 carries the same disambiguation note.)

6.1 How it plugs in without forking the substrate

Section titled “6.1 How it plugs in without forking the substrate”

The record-set-processor base package must not depend on Predictive Studio. So instead of editing RecordProcessExecutor.buildProcessor(), the processor registers itself on the MJGlobal ClassFactory via @RegisterClass — primary key 'ML Model' (on the decorator) and alias 'MLModelInference' (registered in src/scoring/register.ts). A thin resolveMLInferenceProcessor(workType, options) pulls it back out by work-type key; isMLInferenceWorkType() gates membership; LoadMLModelInferenceProcessor() is the tree-shaking anchor. A supervisor wires this into the executor seam — no change to the base substrate. This is the generic pluggable-processor registry working exactly as designed: a new work type ships as a registered class, not a fork.

flowchart TB
    M["MJ: ML Models"] --> P[MLModelInferenceProcessor]
    OD["On-demand<br/>RunNow + scope override<br/>records / view / list / filter / single"] --> P
    SCH["Scheduled<br/>Schedule trigger"] --> P
    subgraph RSP["Record Set Processing"]
        P --> A["assemble features<br/>context='on-demand' · transform-only (frozen)"]
        A --> S["sidecar /predict"]
        S --> OUT{"OutputMapping present?"}
    end
    OUT -->|no| EPH["Ephemeral scores<br/>sortable within the result set"]
    OUT -->|yes| WB["WriteBackProcessor<br/>score column / child rows<br/>sortable · filterable · reportable"]

The processor warm-loads the model once and shares it across the batch (an in-flight loadPromise deduplicates concurrent records). It resolves the assembly config from the model’s Lineage (or the related pipeline), assembles features in 'on-demand' context (transform-only — never re-fits), calls the sidecar /predict, and returns a RecordResult with an MLInferenceResultPayload. Those payload fields are the refs available to an OutputMapping (read with $.):

refmeaning
$.scorenumeric output — probability (classification) or value (regression)
$.classpredicted class label (classification only)
$.scoredAtISO-8601 UTC timestamp of when the prediction was produced
$.modelId / $.target / $.problemTypelineage (which model, its target, classification/regression)

Ephemeral by default; write-back when an OutputMapping is present. The processor itself only produces the result payload (sortable within the result set). The substrate’s shared WriteBackProcessor decorator — not the ML processor — applies the OutputMapping when configured, writing a prediction column to the entity (or a child record). Written-back predictions become sortable/filterable/reportable like any column. This is the same write-back mechanism every Action/Agent/Infer work type uses, so the ML scorer inherits it for free.

Who may score. Only Validated and Published models score. A Draft or Archived model is refused once, before the first record (ModelNotScorableError from Preflight), and nothing is written. Dry runs: PredictiveStudio.ScoreRecordSet with dryRun: true computes the predictions and returns them without writing back, even when a write-back mapping is supplied. The Score Record Set Action has no dry-run parameter.

The sidecar doesn’t care about cardinality — single-record serves interactive/agent needs, batch serves bulk. On-demand = RecordProcess.RunNow + a runtime scope override (records/view/list/filter/single). Scheduled = the Schedule trigger, with an MJ: ML Model Scoring Binding (upsertScoringBinding) recording lineage (LastScoredAt / LastRowCount) for staleness detection and retraining (§14). Materialized prediction columns (population-wide, indexed) are the deferred later optimization gated on #2770 (§17); scoring + write-back ship with zero dependency on it.

6.5 Where predictions go — the four storage paths

Section titled “6.5 Where predictions go — the four storage paths”

These are four distinct, orthogonal things. The common confusion is to assume column write-back needs materialization (#2770), or that “Materialized Results” is an ML run-history store — neither is true. All “store a prediction” needs are met by paths A–C today; #2770 (D) is an optional future auto-refresh optimization, nothing more.

PathWhat it isStatusWhen to use
AColumn write-back — OutputMapping.fieldsWrites the prediction into a real column on the target entity, overwritten each run (the current value). fields is a map, so one run can write several columns.✅ shipped”Put each member’s latest renewal probability on the member record.” e.g. { "RenewalProbability": "$.score", "LastScoredAt": "$.scoredAt" }
BMJ: Process Run DetailsAutomatic per-record transactional history — every scoring run writes one detail row per record carrying the full prediction ResultPayload (score/class/scoredAt/lineage), RecordID, Status, CompletedAt. No config.✅ shipped (automatic)Audit / replay / “what did the model say for this member on each run” — for free, even with no OutputMapping.
COutputMapping.childRecord — typed time-seriesCreates a child record per scored record per run in an entity you define (e.g. Member Renewal Scores with MemberID + Probability + ScoredAt). A clean, queryable, append-only history of your own shape.✅ shipped”Keep a typed history table / time-series of scores” (cleaner than the generic JSON in path B).
D#2770 Materialized ResultsA generic, not-yet-built primitive for materializing any query/entity result set as a cached, auto-refreshed physical table. Not ML-specific, not run history. ML can later ride it for population-wide indexed prediction columns.⏳ deferred (design-doc; gated on PR #2770)A future optimization for indexed, population-wide querying — never required to store predictions.

#2770 is NOT “ML Model Run Results.” It’s view/entity-materialization infrastructure that ML could optionally use later. Per-run prediction history already lives in path B (MJ: Process Run Details) and, if you want a typed table, path C (childRecord). Writing a prediction into an entity column (path A) is a plain MJ column write — it needs nothing from #2770.

6.x “Needs training” and opt-in auto-train

Section titled “6.x “Needs training” and opt-in auto-train”

A model row can exist without a trained artifact — most often because an app seeds model metadata but the serialized model only ever lived on the machine that trained it. Artifact bytes are per-host (§5) and training always creates a new immutable model row, so a shipped model id can never be made scoreable in place.

  • Refuse up front, clearly. Record Set Processing has an optional run-level IRecordProcessor.Preflight(context) seam, called once before the first batch. MLModelInferenceProcessor.Preflight warm-loads the model; a model that is missing, has no ArtifactFileID, or whose artifact bytes aren’t on this server throws a ModelNeedsTrainingError (”… needs training …”). The run is recorded as Failed with that message and no record is processed — instead of every record failing into the error-rate circuit breaker. ScoreRecordSet uses the same preflight.

  • Never publish an untrained model. The promotion gate refuses → Published for a model with no artifact (needs-training outcome; Actions return MODEL_NEEDS_TRAINING, Remote Ops throw the message).

  • Never publish a model /predict does not serve. A forecast whose shipped artifact is a series forecaster (a series model, ar1_const / trend_season_ols / trend_season_ar1, or a series baseline, naive / seasonal-naive / mean, named by the model Lineage’s shippedDriver) is refused by the sidecar’s /predict, so the promotion gate refuses → Published (not-servable outcome; Actions return MODEL_NOT_SERVABLE) and auto-train never reuses one (ModelReadiness.IsServedByPredict). A forecast unit’s tabular finalist is a trainer predictor and is served.

  • Studio shows it. The Production panel marks such a model Needs training (amber) instead of Bound/Scheduled and disables Operate.

  • Opt-in auto-train. An 'ML Model' Record Process can declare, in its Configuration:

    { "primaryKeyField": "ID",
    "autoTrain": { "pipelineId": "<ML Training Pipeline id>",
    "binding": { "targetColumn": "RenewalProbability", "mode": "Scheduled" } } }

    (modelId may then be omitted.) When a run’s model needs training, PipelineAutoTrainer reuses the newest Published model of that pipeline whose artifact loads on this server, or trains the pipeline through the real TrainingEngine and publishes it through the promotion gate (Draft → Validated → Published — a leakage-flagged model is refused and stays Draft; the gate promotes it only on a request that records a sign-off with a reason). Auto-train does not read the trust verdict; the training gate decides what it can publish. A run whose stored decision ships no model (no-signal, or the existing rule kept) stops auto-train with an error that names the stored reason, and only the promotion gate’s leakage refusal stops a model that does ship. It then writes the model id back into the process Configuration, repoints the process’s bindings (and any still on the superseded model), creates the declared binding if none exists, archives the superseded Published model, and the same run goes on to score. Processes that don’t declare autoTrain are unaffected. This is how an Open App ships a scoring process that works on a fresh install with no hand steps — ship the pipeline + Record Process, not model rows.

Training has no row-cap input: the unit’s rows are read through a data access that sets IgnoreMaxRows unless a cap is given, so a read is never silently capped at the entity’s UserViewMaxRows (1000 by default).


7. The experiment engine — a generic agentic-search primitive

Section titled “7. The experiment engine — a generic agentic-search primitive”

Built, not wired. Experiment sessions are switched off: the production session runner has no IPipelineResolver, so it cannot turn a proposed experiment into a training pipeline, and every session it ran trained nothing while reporting Completed. The Run Experiment Session Action and PredictiveStudio.StartExperimentSession refuse with “Experiment sessions are switched off…”, the Studio shows past sessions read-only, and the Model Development Agent trains through its Pipeline Builder (§12.4). The description below is the design the orchestrator code implements.

Experiment → ExperimentSession → ExperimentSessionIteration are deliberately GENERIC, ML-agnostic, reusable primitives: a budgeted, plan-then-execute-then-refine agentic search that groups N iterations with a leaderboard and an owning agent run. Predictive Studio is the first consumer (the ML leaf is MJ: ML Training Runs, which FKs into an iteration), but the same three tables are intended to back prompt-optimization, agent-config search, and eval sweeps — each with its own leaf run table.

erDiagram
    Experiments ||--o{ Experiment_Sessions : "executions over time"
    Experiment_Sessions ||--o{ Experiment_Session_Iterations : "attempts (leaderboard unit)"
    Experiment_Session_Iterations ||--o| ML_Training_Runs : "ML leaf (nullable FK)"
  • MJ: Experiments — the durable “what we’re optimizing,” independent of any execution. ExperimentType is an open NVARCHAR (no CHECK) so new consumers add types without a migration. Re-running monthly creates new sessions under the same Experiment for comparison over time.
  • MJ: Experiment Sessions — one execution. Carries the Budget (JSON), the approved PlanSpec (for PS, the ModelingPlanSpec), a Leaderboard snapshot, and AgentRunID.
  • MJ: Experiment Session Iterations — one attempt, the leaderboard unit. Carries Sequence, Status, the normalized Score (the Experiment’s TargetMetric), ComputeCost/TokensUsed, Rationale, and the driving AIAgentRunID.

ExperimentOrchestrator (Engine/src/experiment/experiment-orchestrator.ts) is deterministic TS that executes an already-approved ModelingPlanSpec:

public async runSession(plan: ModelingPlanSpec, deps: ExperimentDeps, options?: ExperimentRunOptions): Promise<ExperimentSessionResult>

It asserts plan.Approved === true (the approval gate is the agent’s job, not the orchestrator’s) and runs iterations in waves in an in-process bounded pool (runBounded, experiment/concurrency.ts) — no Record Set Processing run is created:

flowchart LR
    START([approved ModelingPlanSpec]) --> GEN["generate wave<br/>(IWaveStrategist)"]
    GEN --> TRAIN["train wave<br/>(runBounded · bounded concurrency<br/>budget gate INSIDE via shouldStop)"]
    TRAIN --> EVAL["evaluate<br/>(normalized Score)"]
    EVAL --> LB["update leaderboard"]
    LB --> PRUNE["prune<br/>(top-K / threshold)"]
    PRUNE --> GATE{"between-wave budget gate<br/>MaxRuns · MaxComputeCost ·<br/>MaxWallclockMinutes?"}
    GATE -->|within budget & more to try| GEN
    GATE -->|bound hit| PAUSE(["Status='Paused'<br/>(a status label; resume relabels only)"])
    GATE -->|plan exhausted| DONE(["Status='Completed'"])
    style GATE fill:#fef3c7,stroke:#d97706
ConcernHow
Wave generationthe IWaveStrategist seam — default PlanOrderWaveStrategist (priority order); swap in an LLM-backed strategist for adaptive “what next” without touching the orchestrator
Leaderboard (src/experiment/leaderboard.ts)LeaderboardEntry[] ranked by a normalized score (prefers HoldoutMetrics, falls back to training metrics; error metrics negated via Core’s MetricDirections.IsErrorMetric so higher always = better)
PruningselectPrunedIterationIds() — top-K / relative-threshold rules; pruned iterations marked Pruned + removed from the leaderboard
Budget gateMaxRuns / MaxComputeCost / MaxWallclockMinutes checked both between waves and inside the bounded runner (see below)
ConcurrencyrunBounded() (src/experiment/concurrency.ts) with a configurable max

The budget gate is enforced inside the bounded runner, not just between waves. As each iteration completes within a wave, foldOutcome folds its cost into budgetState immediately and re-checks checkBudget(...); if a bound trips it sets stopReason, which feeds runBounded’s shouldStop predicate (each worker calls if (shouldStop && shouldStop()) return; before claiming its next task). The compute of every attempt counts: an attempt that fails after its run row exists still charges the compute that run recorded against the session (TrainingAttemptFailedError carries its seconds and cost). Each completed run records ComputeSeconds (wall-clock seconds) and, only when PS_COMPUTE_USD_PER_SECOND is set, a ComputeCost priced from them (NULL otherwise, folded in as 0). So MaxComputeSeconds always bounds a session, while MaxComputeCost bounds it only when that rate is configured. Under an Evaluation Protocol the protocol’s maxCandidates and maxComputeSeconds tighten MaxRuns and MaxComputeSeconds. Either compute bound stops the session as budget-maxComputeCost.

A session creates no Process Run; waves run in the in-process pool above. Like the other engines, the orchestrator is built on injectable seams (IExperimentEntityFactory, IClock, IExperimentTrainer, IWaveStrategist — src/experiment/types.ts / seams.ts).


8. The data model — entities and relationships

Section titled “8. The data model — entities and relationships”

13 core entities. CodeGen generates timestamps, FK indexes, sprocs, views, and the strongly-typed entity classes (MJMLModelEntity, MJMLTrainingPipelineEntity, …) in @memberjunction/core-entities. Names follow the MJ “MJ: ” prefix convention. The three Experiment* tables are generic (ML-agnostic); everything ML * is the Predictive Studio leaf, as are MJ: ML Prediction Units, MJ: ML Evaluation Protocols and MJ: ML Training Snapshots.

erDiagram
    ML_Algorithms ||--o{ ML_Algorithm_Use_Case_Rankings : "ranked in"
    ML_Algorithm_Use_Cases ||--o{ ML_Algorithm_Use_Case_Rankings : "scored for"
    ML_Algorithms ||--o{ ML_Training_Pipelines : "uses"
    ML_Training_Pipelines ||--o{ ML_Training_Runs : "produces"
    ML_Training_Pipelines ||--o{ ML_Models : "versions"
    ML_Training_Runs ||--o| ML_Models : "yields (nullable)"
    Experiments ||--o{ Experiment_Sessions : "executions over time"
    Experiment_Sessions ||--o{ Experiment_Session_Iterations : "attempts"
    Experiment_Session_Iterations ||--o| ML_Training_Runs : "ML leaf (nullable)"
    Prediction_Units ||--o{ ML_Training_Pipelines : "decides rows · dates · labels"
    Evaluation_Protocols ||--o{ ML_Training_Pipelines : "judges"
    Training_Snapshots ||--o{ ML_Training_Runs : "trained on"
    Evaluation_Protocols ||--o{ ML_Training_Runs : "judged under"
    ML_Models ||--o{ ML_Model_Scoring_Bindings : "scored via"
    ML_Model_Scoring_Bindings }o--|| Record_Processes : "runs through"

(Entity names shown without the “MJ: ” prefix for diagram clarity. Record_Processes is the existing Record Set Processing entity — Feature Pipelines are categorized rows of it (§11), not a separate entity. MaterializedResultID on a scoring binding is a soft reference to #2770’s table — not a FK until that table exists.)

EntityRoleKey fields
MJ: ML Algorithmsthe fixed 6-algorithm catalogProblemTypes, DriverClass (sidecar key), HyperparameterSchema, DefaultHyperparameters, SupportsFeatureImportance
MJ: ML Algorithm Use Cases7 decision-relevant scenariosProblemTypeScope, Guidance, DisplayOrder
MJ: ML Algorithm Use Case Rankingsthe 6×7 join (42 rows)SuitabilityScore (1–5), RecommendationLevel, Rationale; UNIQUE(Algorithm, UseCase)
MJ: ML Training Pipelinesdeclarative training definitionTargetEntityID, TargetVariable, ProblemType, AlgorithmID, SourceBindings, FeatureSteps, LeakageGuard, ValidationStrategy, PredictionUnitID, EvaluationProtocolID (the exact versions it trains under); the AsOfStrategy column is retired (the unit decides decision dates)
MJ: ML Modelsimmutable, versioned trained modelFittedPreprocessing, FeatureSchema, Metrics, HoldoutMetrics, FeatureImportance, Lineage, ArtifactFileID, Status (Draft/Validated/Published/Archived)
MJ: ML Training Runsone training attempt (iteration leaf)PipelineID, ResultingModelID (nullable), ExperimentSessionIterationID (nullable), TrainingSnapshotID, EvaluationProtocolID, ValidationResults (the ps-run-results/v2 envelope), CodeVersion, ComputeSeconds, ComputeCost, TokensUsed
MJ: ML Prediction Unitswhat one training row is: the unit rows and their decision date, the outcome rule, window and censoring, the declared task. Immutable version rows (a new version is a new row under the same LineageRootID)LineageRootID, Version, UnitRows, LabelRule, LabelWindowDays, LabelGapDays, Censoring, TaskType, LabelVersion, ChangeHistoryPolicy, GroupKey, TemporalSpec, DecisionSpec, IncumbentHeuristic, AuthoredBy
MJ: ML Evaluation Protocolshow a run is judged (§5.1). Immutable version rows, saved before the first candidate trainsLineageRootID, Version, Holdout, InnerValidation, Seed, PrimaryScore, Baselines, Calibration, Intervals, Promotion, LeakageAudit, Exclusions, Slices, Budgets, MinMinorityClassRows, GuardFieldsEditableBy, AuthoredBy
MJ: ML Training Snapshotsthe frozen, fingerprinted rows and values a run learned from: the matrix, the row manifest and the provenance, as files in MJ storageContentHash, PipelineID, PredictionUnitID, EvaluationProtocolID, MatrixFileID, RowManifestFileID, ProvenanceFileID, ManifestRowCount, HoldoutRowCount, FeatureSpec, FeatureSchemaHash, ClassBalance, DataCutoff
MJ: Experimentsdurable “what we optimize” (generic)ExperimentType (open NVARCHAR), Goal, TargetMetric, PlanSpecTemplate
MJ: Experiment Sessionsone execution (generic)ExperimentID, Budget, PlanSpec, Leaderboard, AgentRunID
MJ: Experiment Session Iterationsone attempt / leaderboard unit (generic)Sequence, Status, Score, ComputeCost, TokensUsed, Rationale, AIAgentRunID
MJ: ML Model Scoring Bindingslineage for scoring/retrainingMLModelID, RecordProcessID, TargetEntityID/TargetColumn, Mode, MaterializedResultID (soft), LastScoredAt, LastRowCount

MJ: ML Training Runs.ValidationResults holds the ps-run-results/v2 envelope (TrainingRunResults, Core/src/training-run-results.ts). A finished run stores status: 'completed' with unitResolution (the counts of unit rows read and excluded), trainer (the stored trainer result, its artifacts as file references, never their bytes), trainerRequest (the settings a retrain replays), carve (the holdout carve summary) and gate (the stored gate record: outcome, branch, reason, intervals and guards); a refused run stores status: 'refused' with the refusal. The Studio’s results widget, the trust verdict and the TrainModel output all read this one envelope.

Reminder: don’t write code against new fields until the migration + CodeGen have run. All ML reference data (ml-algorithms/, ml-algorithm-use-cases/, ml-algorithm-use-case-rankings/) is seeded via metadata files + mj sync push, never SQL INSERTs.


9. The guidance layer — algorithm catalog + the 6×7 matrix

Section titled “9. The guidance layer — algorithm catalog + the 6×7 matrix”

Neither the agent nor a non-expert user should have to guess which algorithm fits. Three seeded metadata entities encode evidence-based defaults:

  • MJ: ML Algorithms — the fixed, curated catalog (6 algorithms): XGBoost, LightGBM, Logistic Regression, Random Forest, Linear/Ridge Regression, MLP. Each row declares ProblemTypes, DriverClass (the sidecar key), HyperparameterSchema, DefaultHyperparameters, and SupportsFeatureImportance.
  • MJ: ML Algorithm Use Cases — decision-relevant scenarios that genuinely differentiate algorithms (7 of them), NOT business labels (churn/renewal/attendee-return are all the same binary classification shape and so don’t differentiate). E.g. “Binary classification”, “Regression”, “Interpretability required”, “Minimal tuning (business-user)”, “Large/wide dataset (speed)”, “Embedding/LLM-feature-heavy”, “Small dataset”.
  • MJ: ML Algorithm Use Case Rankings — the 6×7 join (42 rows). Each cell carries a SuitabilityScore (1–5), a RecommendationLevel (Primary / Strong / Viable / Weak / NotRecommended), and the real payoff: a Rationale (agent- and human-readable, e.g. “Gives feature importances but not simple coefficients — if a stakeholder needs to see exactly why each prediction was made, prefer Logistic/Ridge.”).

The seeded recommendation matrix (the per-cell Rationale is the real payoff):

Use case ↓ / Algo →XGBoostLightGBMLogistic RegRandom ForestLinear/RidgeMLP
Binary classificationPrimaryStrongViableStrongNotRecViable
RegressionPrimaryStrongNotRecStrongStrongViable
Interpretability requiredWeakWeakPrimaryViablePrimaryNotRec
Minimal tuning (business-user)ViableViableStrongPrimaryStrongWeak
Large/wide dataset (speed)StrongPrimaryStrongViableStrongViable
Embedding/LLM-feature-heavyStrongStrongViableViableViablePrimary
Small datasetViableViablePrimaryStrongPrimaryWeak

All three are seeded via metadata files (metadata/ml-algorithms/, metadata/ml-algorithm-use-cases/, metadata/ml-algorithm-use-case-rankings/) with @lookup: refs and mj sync push — never SQL INSERTs. The dashboard’s catalog panel queries this matrix for ranked, rationale-bearing recommendations per scenario (the UI computes “best level across selected scenarios” via PredictiveStudioEngine.BestLevelsForScenarios; no agent reads the matrix directly).


10. The invocation surface — Actions vs Remote Operations

Section titled “10. The invocation surface — Actions vs Remote Operations”

Every server-side capability is a Remote Operation (Manual mode) + (for the five agent-facing ones) a thin Action, plus one Action with no operation (the pipeline migration, §10.2), so the browser, Skip, Query Builder, and agents all inherit it. See the Remote Operations Guide and Transport-Layer Architecture Guide. Crucially, both surfaces delegate through one shared path (operations/delegation.ts) — the production deps/seams wiring exists exactly once, so an Action and its sibling Remote Op can’t diverge.

All are @RegisterClass(BaseRemotableOperation, '<key>') Manual-mode subclasses of CodeGen-emitted bases in MJCoreEntities/src/generated/remote_operations.ts; the subclass bodies live in Engine/src/operations/. Every one carries RequiredScope = 'predictive:execute' and is seeded as an MJ: Remote Operations row (metadata/remote-operations/.remote-operations.json, category “Predictive Studio”, GenerationType='Manual', CodeApprovalStatus='Approved').

Operation keyServer classModeDelegates to
PredictiveStudio.TrainModelPredictiveStudioTrainModelServerOperationLongRunningTrainingDelegation.TrainModelViaEngine → TrainingEngine.trainModel
PredictiveStudio.ScoreRecordSetPredictiveStudioScoreRecordSetServerOperationLongRunningScoreRecordSetViaRunner → ProductionScoreRecordSetRunner → MLModelInferenceProcessor
PredictiveStudio.RunFeaturePipelinePredictiveStudioRunFeaturePipelineServerOperationLongRunningRecordProcessExecutor.RunByID (the RSP substrate — §11)
PredictiveStudio.StartExperimentSessionPredictiveStudioStartExperimentSessionServerOperationLongRunningRunExperimentSessionViaOrchestrator → ExperimentOrchestrator.runSession
PredictiveStudio.ControlExperimentSessionPredictiveStudioControlExperimentSessionServerOperationSynca lifecycle Status write on the MJ: Experiment Sessions row (pause/resume/cancel)
PredictiveStudio.PromoteModelPredictiveStudioPromoteModelServerOperationSyncPromoteModelViaGate → ProductionModelPromotionGate (§13)
PredictiveStudio.CreateScoringProcessPredictiveStudioCreateScoringProcessServerOperationSyncCreateScoringProcess (creates an on-demand 'ML Model' Record Process + scoring binding)

The four LongRunning ops emit context.emitProgress({ OperationKey, Processed, Total?, Status, Message }). RunFeaturePipeline is the richest — it forwards the RSP executor’s per-batch onProgress as typed progress, so attached and over-the-wire callers see live progress (mirroring RecordProcess.RunNow); Train/Score/StartExperiment emit coarse start/finish phases (the underlying engines expose no per-step hook). The wire input/output types are defined inline in the generated remote_operations.ts; Budget and ModelingPlanSpec come from Core and are runtime-validated by ValidateBudget / ValidateModelingPlanSpec before any work runs. Tree-shaking anchor: LoadPredictiveStudioOperations().

PredictiveStudio.TrainModel trains under the pipeline’s Prediction Unit and Evaluation Protocol and returns the run’s stored gate decision. Its input (PredictiveStudioTrainModelInput) is pipelineId, experimentSessionIterationId? and sidecarVersion?. Its output (PredictiveStudioTrainModelOutput) carries trainingRunId, status, leakageFlagged and holdoutMetrics?, and from the stored decision gateOutcome (model | keep-simple-baseline | no-signal), gateBranch, gateReason and shipped, plus the trainingSnapshotId, evaluationProtocolId and evaluationProtocolVersion the run used. modelId and version are optional: they are absent when the decision ships no model, and status is then NoSignal, or KeptExistingRule when the shipped member is the unit’s existing heuristic rule; otherwise status is the model’s own (Draft). A run whose stored results cannot be read fails rather than returning a guessed status. holdoutMetrics is the produced model’s stored test-set scores, or the finalist’s when no model is produced.

The operation refuses the retired asOf, labelEventDates, maxRows and primaryKeyField inputs by presence, before any progress is emitted or the engine is called: a key sent as {} or undefined, or asOf: { "Mode": "none" }, is refused too. asOf returns AS_OF_NO_EFFECT; the others return EXECUTION_ERROR with a message naming every retired key the call sent (§10.4). RetiredTrainModelInputs types a payload that still sends them.

For the agent / workflow / low-code boundary, six capabilities have thin Actions in Engine/src/actions/ (all extend BasePredictiveStudioAction → BaseAction, registered via @RegisterClass(BaseAction, '<DriverClass>'), seeded in metadata/actions/predictive-studio/):

Action (MJ: Actions)DriverClassDelegates to
Train ML ModelPredictiveStudioTrainModelActionTrainingEngine.trainModel, then TrainModelMapping (the same output mapping the operation uses)
Score Record SetPredictiveStudioScoreRecordSetActionProductionScoreRecordSetRunner → MLModelInferenceProcessor
Run Experiment SessionPredictiveStudioRunExperimentActionExperimentOrchestrator.runSession
Promote ML ModelPredictiveStudioPromoteModelActionProductionModelPromotionGate
Schedule Model ScoringPredictiveStudioScheduleModelScoringActioncreateScheduledModelScoring (§15.6)
Migrate Pipelines to Prediction Units and Evaluation ProtocolsPredictiveStudioMigratePipelinesActionPipelineMigration.MigratePipelines

Train ML Model takes PipelineID only. A supplied MaxRows is refused by name (TRAINING_FAILED: the row cap is a retired input). Its outputs are ModelID (null when no model ships), HoldoutMetrics, LeakageFlagged, GateOutcome, GateReason, TrainingSnapshotID and EvaluationProtocolID; a run whose decision ships no model succeeds with its stored reason as the message.

Migrate Pipelines to Prediction Units and Evaluation Protocols is a dry run by default (DryRun defaults to true; PipelineIDs narrows it). It binds every pipeline that lacks them to a version-1 Prediction Unit and a version-1 Evaluation Protocol mapped from its legacy settings, archives a pipeline that cannot be expressed (an offset as-of, a single-valued target, a sequence model) with the reason in its description, or holds it with the reason: a binary pipeline whose outcome polarity is not declared stays held until a later run finds it declared, and a pipeline that declares a time index stays held until a forecast unit with a rolling-origin protocol is bound. A held pipeline is refused by training. It reports Report, MigratedCount, ArchivedCount, HeldCount, SkippedCount and FailedCount; the release procedure, a documented step the release engineer executes after deploying, runs it (dry run first, then the real run). No MJ code invokes it, so an admin can also run the Action themselves, dry run first. Until a pipeline is migrated, training it is refused with UNIT_REQUIRED. It has no Remote Operation.

There is no “Control Experiment Session” or “Run Feature Pipeline” Action — those two exist only as Remote Operations (control is a UI/host concern; feature-pipeline running is the RSP facade) — and the migration Action is the one Action with no Remote Operation. Tree-shaking anchor: LoadPredictiveStudioActions().

10.3 Decision table — which surface do I reach for?

Section titled “10.3 Decision table — which surface do I reach for?”
You want to…UseWhy
Have an agent train / score / run an experiment / promotean ActionActions are the metadata-discoverable code→agent boundary (see root CLAUDE.md Actions philosophy)
Have the browser, Skip, or Query Builder invoke a capability with typed I/O + LongRunning progressa Remote Operationtyped, scope-gated (predictive:execute), marshalled identically client + server
Read ML models / runs / leaderboards / rankings for displayRunView / RunViews (or PredictiveStudioEngine in Angular)plain reads — no need for an op; the engine caches reference data
Score a set of records as a saved, scheduled, audited joba Record Process ('ML Model' work type) via RecordProcess.RunNowinherits batching/concurrency/budget/pause/resume/audit (§6)
Run a derived-feature pipeline that writes attributes backa Feature Pipeline = categorized Record Process, run via RunFeaturePipelinereuses the whole RSP substrate (§11)
Call training/scoring from other server-side codethe engine class directly (TrainingEngine, MLModelInferenceProcessor)never call an Action from code (root CLAUDE.md) — import the engine

When Predictive Studio refuses a Train, Score or experiment request, the Action and its sibling Remote Operation return the same typed ResultCode (Core/src/refusal-codes.ts, mapped in Engine/src/operations/refusals.ts). Callers branch on the code, never on the message text.

ResultCodeFires when
READ_CAP_EXCEEDEDA caller-supplied row cap (maxRows in a Score filter scope) is smaller than the rows in scope. Reads are uncapped by default: filter-scoped scoring reads every matching row, so a cap that would cut the scope short is refused instead of scoring on a sample. Training takes no row cap (a maxRows on Train is a retired input, below).
AS_OF_NO_EFFECTThe Train request carries an asOf input (any mode, { "Mode": "none" } included), the pipeline’s stored AsOfStrategy.Mode is not none, or a modeling plan asks for point-in-time assembly. Decision dates come from the pipeline’s Prediction Unit, so the input is refused by name (input-not-accepted) instead of being ignored (§4.3). The other retired Train keys (labelEventDates, maxRows, primaryKeyField) return EXECUTION_ERROR with a message naming every key the call sent.
UNIT_REQUIREDTraining a pipeline with no Prediction Unit. Every pipeline trains under a unit: run the “Migrate Pipelines to Prediction Units and Evaluation Protocols” Action or bind one (a held pipeline’s description names what to declare). Refused before any run row is written.
PROTOCOL_REQUIREDTraining a pipeline that names no Evaluation Protocol, or one that cannot be found. Refused before any run row is written.
TIME_HOLDOUT_REQUIREDA pipeline that declares a time index (ValidationStrategy.TimeIndexColumn) is bound to a protocol whose holdout is seeded-random (time-index-needs-time-holdout): that holdout would test it on rows interleaved in time with its training rows. Bind a forecast unit with a rolling-origin (or date-carved) protocol. Refused before any run row is written.
BUDGET_EXCEEDEDThe trainer stopped because the run reached its protocol’s compute budget.
AS_OF_SOURCE_UNDATEDA unit pipeline’s feature cannot be read as of a decision date: its Record Changes history does not cover the date (decision-date-before-change-history), or its stored value changed after it (select-changes-after-decision-date).
INVALID_EVALUATION_SETTINGTraining a pipeline whose ValidationStrategy.MajorityBaselineRule is not "enforce" or "warn" (§5.1); it is refused before any training row is written.
MODEL_NEEDS_TRAININGScoring or promoting a model with no trained artifact, or whose artifact is not on this server (§6.x).
MODEL_NOT_SCORABLEScoring a trained model whose status does not allow it: only Validated and Published models score, so a trained Draft must be promoted first and an Archived model cannot score.
MODEL_NOT_SERVABLEScoring a model whose artifact is a forecast’s series forecaster, which the sidecar’s /predict does not serve (§6.x). Never retrained by auto-train: a retrain would ship the same kind of artifact.
EXPERIMENT_SESSIONS_UNAVAILABLEAny Run Experiment Session Action or StartExperimentSession call: experiment sessions are switched off (§7).

11. Feature Pipelines — the category route (no new entity)

Section titled “11. Feature Pipelines — the category route (no new entity)”

A Feature Pipeline generalizes Content Autotagging (LLM classify → persisted tags): chain LLM prompt / Action / Agent / vision → transform → write an attribute back to the row. They have standalone analytics value, independent of any model, which is why they live in Knowledge Hub as well as feeding Predictive Studio.

The key shipped decision: a Feature Pipeline IS a categorized MJ: Record Processes row — there is NO standalone MJ: Feature Pipelines entity. (Verified: zero MJ: Feature Pipelines / FeaturePipelineEntity in generated code or migrations.) Reusing the Record Set Processing substrate means batching, resume, audit, and write-back all come for free; the only new thing is a category that marks a Record Process as a feature pipeline.

flowchart LR
    RPC["MJ: Record Process Categories<br/>'Feature Pipeline' (seeded)"] --> RP["MJ: Record Processes<br/>(categorized rows)"]
    RP --> FPE["FeaturePipelineEngine<br/>(BaseEngine cache)<br/>narrows by Category in memory"]
    FPE --> KH["Knowledge Hub panel<br/>FeaturePipelinesResourceComponent"]
    KH -->|Run| OP["PredictiveStudio.RunFeaturePipeline"]
    OP --> EX["RecordProcessExecutor.RunByID<br/>Infer/Action/Agent/vision + WriteBack"]
    EX -->|persist derived attribute| ENT[Entities]
    KH -->|Author| ED["mj-record-process-editor<br/>(reused, not rebuilt)"]
  • FeaturePipelineEngine (Engine/src/feature-pipelines/feature-pipeline-engine.ts) — a BaseEngine singleton that caches all MJ: Record Processes + recent MJ: Process Runs, then narrows in memory to feature pipelines by matching the denormalized Category view field against FEATURE_PIPELINE_CATEGORY_NAME = 'Feature Pipeline' (case-insensitive, isFeaturePipeline()). It projects FeaturePipelineSummary[] (target entity, the written OutputAttribute parsed from OutputMapping, last-run freshness). It’s a discovery/monitoring cache, not a registry.
  • Category seed — metadata/record-process-categories/.record-process-categories.json seeds the one MJ: Record Process Categories row named “Feature Pipeline”.
  • Run path — RunFeaturePipeline (the only op with no PS-specific logic) delegates straight to RecordProcessExecutor.RunByID(featurePipelineID, …). Running a feature pipeline is exactly running the underlying Record Process — full batching/resume/audit + true forwarded per-batch progress.
  • Knowledge Hub UI — FeaturePipelinesResourceComponent (@RegisterClass(BaseResourceComponent, 'FeaturePipelinesResource'), packages/Angular/Explorer/dashboards/src/KnowledgeHub/components/feature-pipelines/) lists pipelines from the engine, shows target/written-attribute/freshness, offers Run (via the provider’s RouteOperation seam → RunFeaturePipeline), and an authoring entry that opens the reusable mj-record-process-editor rather than rebuilding authoring.

The key. A MJ: Feature Values row is upserted on its key: the record, the feature name, the pipeline version, the prompt version hash (template text, outputs, constraint block, pipeline type, escalation settings), the primary AI model, the parameters hash and the input hash. ValueKeyHash (SHA-256 of the key) is unique per pipeline. A re-run with the same key leaves the row Unchanged; a changed value updates it; a changed key is a new row. The latest value of a feature for a record is the row with the newest ComputedAt.

Human values. Source is Pipeline or Human (a value a person entered on the target record). Write-back never overwrites a Human value unless the run sets overwriteHumanValues; HumanValuesPreserved counts the ones it kept.

  • Versions. Each distinct pin (prompt version hash, primary model, parameters hash, spec hash) is the next numbered MJ: Feature Pipeline Versions row: Candidate, Active, Retired or Refused.
  • Without an Activation block, an on-demand or manual run activates a changed pin; a scheduled or on-change run of a changed pipeline is refused until then, so a prompt edit never silently changes what an unattended job writes.
  • With Activation (GoldenSetTestID, Metric: 'accuracy', Threshold), a version runs only when its golden-set result meets the threshold; a failing result marks it Refused. Dry runs report this (WouldRefuse) and never register or activate.
  • The Feature Pipeline Version Steward (FeaturePipelineVersionSteward in @memberjunction/feature-pipelines-engine, exposed as the Steward Feature Pipeline Version Action and run per pipeline by the Feature Pipeline Version Steward MJ: Record Processes row of WorkType: 'Action' over WorkType='Infer' AND Status='Active', dispatched every 15 minutes by RecordProcessScheduledJobDriver) evaluates a gated pipeline’s unevaluated version and runs the one backfill of a changed version (the records whose latest value is under another version or none), returning each decision in Outcomes / Version / LastAction / Detail. A pipeline opts out per step with Unattended: { AutoEvaluate: false } / { AutoBackfill: false }.

Golden sets. A Feature Pipeline Golden Set test (FeaturePipelineGoldenSetDriver, GoldenSetScorer, and GoldenLabelQualityCheck in @memberjunction/feature-pipelines-engine) scores a pipeline’s answers against labelled cases. Model-made labels carry a stored label-quality record: two labelers from two vendors label every case under the same guide, each must agree with the existing labels at a Wilson 95% lower bound of at least 0.70, one adjudicates the other’s disagreements, and at most 5 of 52 cases may be excluded. The record is tied to the cases by hash, and the driver refuses a set that fails it before any model call. A case marked Required: true that is answered wrong fails the set whatever its accuracy (gate value 0, the cases listed in RequiredFailures). A case can expect an abstention for some outputs (ExpectAbstain). LabelSource: "specification" cases (prompt-injection attempts, non-titles) are fixed by the guide’s rule rather than labelled, must be Required, and stay outside the label-quality record.

Output constraints. A lookup constraint reads its entity’s rows (DisplayFields, Filter) and renders them into the prompt as the taxonomy; ItemRef checks an array output item by item. A violating answer is RejectedInvalid, nothing is written for it, and the run ends Failed.

Confidence floors. An output can declare MinConfidence with OnLowConfidence: 'null' (an LLM pipeline also needs ConfidenceRef). Below the floor the output is withheld: its Feature Value records the abstention with no value, write-back clears a value the pipeline wrote and keeps a person’s, and the result lists it in BelowConfidenceOutputs. The floor is part of the spec hash, so changing it is a new version.

The record as the user turn. An LLM pipeline that sets Context.RecordAsUserMessage: true sends the template as the system turn and the record’s context fields, as one JSON object, as the user turn, so a field value cannot pose as an instruction.

An Infer (InferProcessor in @memberjunction/feature-pipelines-engine, registered into RecordProcessorRegistry by FeaturePipelineStartup / RegisterInferRecordProcessor()) dry run writes nothing (no Dedup Cache entry, Feature Value, watermark, target or child row) but still calls the model, so its MJ: AI Prompt Runs are recorded and cost what a real run of those records would.

  • Without a scope or a single record it previews at most 25 records (DryRunSampleSize overrides it).
  • A record whose real write would fail is reported Failed with the write-back’s reason (PreviewErrors).
  • It reports, in WouldRefuse (via InferProcessor.BuildRunSummary), why a real run would be refused (a version that has not passed its golden set), and never registers or activates a version. RecordProcess.RunNow / PredictiveStudio.RunFeaturePipeline return it as wouldRefuse; the Bulk Operations dialog then offers no Apply.
  • Each record’s previewFields / previewChildren / previewTags hold only what the real write would change, so the dialog’s change count is what the following real run writes. Outputs withheld below their confidence floor are listed in withheldFields (and withheldKeptFields when the column does not allow null) and withheldChildEntities.
  • An output with no Target (“Feature value only”) writes nothing to the record; its dry-run result is previewFeatureValues (FeatureName, Previous, Proposed), listing the Feature Values a real run would store. The dialog shows one <Feature> (feature value) row per item, counted as a change like any other row; a record with no such rows counts as unchanged.
  • On every Feature Pipeline run, MJ: Process Runs.ResultSummary holds the FeaturePipelineRunResult JSON built by FeaturePipelineRunResults.Build: the pin the run used and the tallies Attempted, Upserted, Unchanged, Previewed, RejectedInvalid, BelowConfidence, Failed, Skipped, HumanValuesPreserved. Because InferProcessor sets FailRunOnRecordError = true, a Feature Pipeline run with any failed or rejected record ends Failed.

12. The Model Development Agent + artifact + viewer

Section titled “12. The Model Development Agent + artifact + viewer”

A Loop agent (conversational) mirroring Agent Manager: collaborate with the user to build a strongly-typed ModelingPlanSpec, get approval, then execute with deterministic code (the Pipeline Builder, §12.4: one pipeline, one Prediction Unit, one Evaluation Protocol, one search over every approved experiment), then report with an LLM-authored rich artifact. Seeded and shipped — metadata/agents/.model-development-agent.json.

flowchart TB
    U([User: business goal]) --> GA
    subgraph PLAN["Plan phase · LLM-heavy · Loop orchestrator"]
        GA[Goal Analyst] --> DS[Data Scout] --> ED[Experiment Designer]
    end
    GA -.writes Goal/TargetDefinition.-> SPEC[("ModelingPlanSpec<br/>payload-path guarded")]
    DS -.writes CandidateSources/Features/Leakage.-> SPEC
    ED -.writes ProposedExperiments/Validation/Budget.-> SPEC
    DS -->|reads| GT["Approved MJ: Queries · DBAutoDoc<br/>Agent Notes · existing ML Models"]
    SPEC --> GATE{"User approves plan?<br/>(parent writes Approved)"}
    GATE -->|edits| PLAN
    GATE -->|approved| EXEC
    subgraph EXEC["Execution phase · deterministic"]
        BUILDER[Pipeline Builder] --> RUNS["one search under the Evaluation Protocol<br/>inner-CV selection · holdout read once · three-outcome gate"]
    end
    RUNS --> LEAK{"Leakage / dominance?"}
    LEAK -->|suspicious| SIGN["Warn in plain language<br/>block promotion → human sign-off + reason"]
    LEAK -->|clean| REPORT
    SIGN -.->|signed off| REPORT
    REPORT["Report phase · LLM<br/>ML Experiment Results artifact<br/>→ drill-through to ML Models"] --> U
    style GATE fill:#fef3c7,stroke:#d97706
    style LEAK fill:#fee2e2,stroke:#dc2626

12.1 Topology — the agent and its three sub-agents

Section titled “12.1 Topology — the agent and its three sub-agents”

The parent Model Development Agent is a Loop agent (ModelSelectionMode='Agent', ArtifactCreationMode='Always', DefaultArtifactTypeID → ML Experiment Results, InjectNotes=true, MaxNotesToInject=10, ExposeAsAction=true). Its six wired Actions are five Predictive Studio Actions (Run Experiment Session, Train ML Model, Score Record Set, Promote ML Model, Schedule Model Scoring) plus Write Entity Field(s). Three sub-agents (all Loop, InvocationMode='Sub-Agent') refine the plan:

Sub-agentExecOrderWrites (its payload slice)
Goal Analyst1refines the business goal → Goal, TargetDefinition (target + problem type + success metric)
Data Scout2studies the data → CandidateSources, CandidateFeatures, LeakageNotes; flags leakage risks
Experiment Designer3proposes ranked experiments + validation + budget → ProposedExperiments, ValidationStrategy, ProposedBudget

Sequencing is declarative, not hardcoded control flow — it’s enforced by metadata payload-path guards. Each sub-agent has PayloadDownstreamPaths: ["*"] (it can read the whole plan) and an PayloadUpstreamPaths list scoped to exactly the slice it’s allowed to write back (e.g. Goal Analyst → ["Goal", "TargetDefinition", "TargetDefinition.*"]). The parent holds PayloadSelfWritePaths: ["*"]: it may write any part of the plan, and it is the one that stamps the approval gate (Approved).

12.2 Ground truth, drafted queries, memory

Section titled “12.2 Ground truth, drafted queries, memory”

The Data Scout reads ground truth only from approved sources, wired via MJ: AI Agent Data Sources:

  • APPROVED_QUERIES — RunView over MJ: Queries where Status = 'Approved' (the trusted semantic layer; “Never hand-write raw SQL for feature extraction”).
  • EXISTING_APPROVED_MODELS — MJ: ML Models where Status IN ('Published','Validated').
  • Agent Notes — prior learnings (what worked, what leaked), auto-injected (InjectNotes).

When the Scout needs a query that doesn’t exist, it may draft a new MJ: Query as Status='Pending' — usable for this exploration but not trusted ground truth until a human approves it into the shared semantic layer.

The agent emits an ML Experiment Results artifact — content type application/vnd.mj.ml-experiment-results (metadata/artifact-types/.ml-experiment-results-artifact-type.json, ParentID → JSON, DriverClass='MLExperimentResultsViewerPlugin', with ExtractRules for name/description/displayMarkdown/bestModelId/leaderboardSize).

The viewer is MLExperimentResultsViewerComponent (@RegisterClass(BaseArtifactViewerPluginComponent, 'MLExperimentResultsViewerPlugin'), packages/Angular/Generic/artifacts/src/lib/components/plugins/ml-experiment-results-viewer.component.ts, package @memberjunction/ng-artifacts). It renders the experiment Goal + headline best score, a ranked leaderboard (rank / algorithm / feature-set / score / modelId — the score column is the out-of-sample headline metric, shown as “not measured” when a run has none), normalized feature-importance bars, a winning-model card, and an extra “Report” tab for the agent-authored markdown. Drill-through to a model emits a NavigationRequest:

// OnDrillThroughToModel() / OnOpenRowModel(row)
{ appName: 'Predictive Studio', navItemName: 'Models', queryParams: { modelId } }

Stale-prose caveat: the artifact-type Description field still says the Angular viewer “is a follow-up; falls back to the parent JSON viewer.” That prose is out of date — the viewer is fully built, registered, and module-declared, and the DriverClass correctly points at it. Treat the description text as lagging reality.

12.4 The deterministic Pipeline Builder (publish-gated-on-trust)

Section titled “12.4 The deterministic Pipeline Builder (publish-gated-on-trust)”

The Model Development Agent is elevated to the Database-Designer / Agent-Manager domain-builder pattern: the conversation reasons in plain language, but the structure it commits to metadata is deterministic + type-safe, never LLM-emitted. Three pieces (in packages/AI/PredictiveStudio/Engine/src/agent/):

  • PredictiveStudioModelDevAgent (the parent’s DriverClass) — overrides determineNextStep: when shouldForceBuild matches (a payload with ProposedExperiments and either Approved: true or an approval phrase in the latest user message, provided no Pipeline Builder sub-agent step has already run), the orchestrator forces routing to the builder sub-agent with Approved: true. Every other decision stays LLM-driven via super.
  • PredictiveStudioPipelineBuilderAgent (the Pipeline Builder code sub-agent, executeAgentInternal, no LLM) — wraps:
  • PredictiveStudioPipelineBuilder (pipeline-builder.ts) maps the approved ModelingPlanSpec to one MJ: ML Training Pipelines row (ModelingPlanToPipelineConfig), binds it to a version-1 Prediction Unit mapped from the pipeline’s columns the way the pipeline migration maps one (PipelineMigration.BindMigratedUnit; a pipeline the mapping would archive or hold is deleted and the build fails with the reason) and to a version-1 Evaluation Protocol created from the defaults (EvaluationProtocolRules.DefaultEvaluationProtocolDraft) before anything trains, turns every approved experiment into a candidate of a single search, and trains through TrainingDelegation.TrainModelViaEngine. The leaderboard shows each candidate’s inner-fold mean with its 95% interval; the winner’s is marked optimistic. The builder’s publish step (autoPublish, on by default) publishes the model only when training was not leakage-flagged and its trust verdict is actionable (TrustVerdicts.DeriveTrustVerdict(...).CanAct: the run’s stored gate decision shipped a model or a kept simple baseline); otherwise the model stays Draft with a plain held reason. The run stores the gate decision, and a run that ends no-signal — including a binary model whose test-set AUC interval contains 0.5 — has no model row at all, so there is nothing to publish and the held reason is the stored one. A model whose Lineage records a calibrationHold (§13, point 6) is held with the calibration reason. Publishing goes through the promotion gate (ProductionModelPromotionGate, Draft → Validated → Published with no sign-off), so the gate’s state machine, artifact and /predict checks apply and the model gets its per-model scoring Action (§15.3); a gate refusal holds the model, at the status the gate left it, with the gate’s reason.

Proven by: unit tests (mapper, gate logic, projection helpers), an in-process integration test (ps-inproc-agent-builder.ts — builds + trains + asserts the publish gate), and an agent-loop test through the real AgentRunner (ps-inproc-agent-run.ts — the sub-agent runs end-to-end and the gate fires). ”+ New prediction” on the business surface opens this agent as a docked mj-conversation-chat-area co-pilot.


Predictive Studio touches client data, runs arbitrary trained models against record sets, and lets an agent draft queries — so security is treated as a first-class concern, enforced at code boundaries, not by convention.

1 · API scope gating (predictive:execute). Every one of the 7 Remote Operations declares RequiredScope = 'predictive:execute' (on the generated base and on the seeded MJ: Remote Operations row). The scope itself is defined as an APIScope (metadata/api-scopes/.predictive-scopes.json, FullPath: 'predictive:execute'). An API key without that scope can’t invoke train / score / run-feature-pipeline / experiment / promote. (Scope is a Remote Operations concept; Actions carry no scope — they run under the agent’s own permissions.)

2 · Primary-key field-NAME validation (injection guard) at the scoring boundary. When ProductionScoreRecordSetRunner builds a SQL filter from caller-supplied primary keys, it validates each key name (not just the value) against the entity’s real field list before interpolation (score-record-set.runner.ts → primaryKeyFilter):

const field = entity.Fields.find(f => f.Name.trim().toLowerCase() === k.trim().toLowerCase());
if (!field)
throw new Error(`Score Record Set: '${k}' is not a field on entity '${entityName}'; refusing to build a filter from an unknown key.`);
return `${field.Name}='${String(v).replace(/'/g, "''")}'`; // canonical field name + escaped value

An unknown/forged key name is refused outright rather than concatenated; the value is single-quote-escaped; and the resolved canonical field.Name (from metadata) is used, never the raw caller string.

3 · UUID-gated ids. Before any id is concatenated into SQL, it must match a strict UUID regex (promote-model.gate.ts):

const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;

loadModel returns null for a non-UUID modelId (“a non-UUID can’t match a real row and is refused rather than concatenated”); resolveThreshold falls back to the default for a non-UUID pipelineId. The id is interpolated only after the gate passes.

4 · The promotion state machine (Draft → Validated → Published → Archived). ProductionModelPromotionGate (actions/promote-model.gate.ts) is the single place that loads the model, detects leakage, enforces sign-off and the calibration hold, and transitions status; the Promote action and operation, opt-in auto-train and the agent’s Pipeline Builder (§12.4) all promote through it. The allowed transitions are an explicit table:

static readonly ALLOWED_TRANSITIONS = {
Draft: ['Validated'],
Validated: ['Published', 'Archived'],
Published: ['Archived'],
Archived: ['Published'],
};

An illegal jump (e.g. Draft → Published) is rejected before any mutation with { kind: 'invalid-transition' } → action error code INVALID_TRANSITION.

5 · Leakage sign-off requires an auditable reason. Promoting a model the leakage guard flagged (§4.4) is blocked unless the caller passes both signOff: true and a non-empty reason:

  • no signOff → { kind: 'refused-leakage' } → LEAKAGE_SIGNOFF_REQUIRED
  • signOff but blank reason → { kind: 'signoff-reason-required' } → SIGNOFF_REASON_REQUIRED
  • valid sign-off → recordSignOff() writes an auditable note (who / when / what-was-overridden / reason) before the transition proceeds.

A kept simple baseline stores {} as its FeatureImportance, so every trainer model’s Lineage also records the run’s leakage audit (trainingLeakageAudit: the finalist importance’s top feature and share). The gate flags a model when either its own importance or that audit exceeds the pipeline’s current dominance threshold; a model whose Lineage records no audit is judged by its importance alone.

6 · A calibration hold has no sign-off. When the gate ships the finalist of a probability task (binary, multiclass) and TrainerContract.CalibrationHidesFamily hides its family (holdout calibration slope outside 0.8–1.2, or no slope measured), training keeps the model Draft, records calibrationHold: { slope } on its Lineage and a CALIBRATION HOLD: sentence in the run notes. The gate refuses → Validated and → Published for it with { kind: 'refused-calibration' } → CALIBRATION_HOLD, whatever the sign-off; it may still be archived. A kept baseline member is not judged by the finalist’s calibration report.

The Remote Op surfaces these as clean EXECUTION_ERRORs with the same plain-language messages; the Action returns ActionResultSimple failures with the codes above. The leakage guard is a single-feature dominance check that misses restated and one-hot leaks; promoting a flagged model through the gate needs sign-off with a reason, whereas a model the guard does not flag (and that holds no calibration hold) is published through the gate by the agent’s Pipeline Builder or by opt-in auto-train with no sign-off.


14. The maintenance loop — staleness + challenger-vs-incumbent

Section titled “14. The maintenance loop — staleness + challenger-vs-incumbent”

Built, not wired. Nothing runs the maintenance pass: no Action exists for it and its scheduled job (“Predictive Studio Maintenance”) is Disabled. The design below is what the engine code implements.

MaintenanceEngine (Engine/src/maintenance/maintenance-engine.ts, entry RunMaintenancePass(...)) drives staleness detection, scheduled re-scoring, and retraining-with-comparison off the MJ: ML Model Scoring Bindings.

flowchart TB
    BIND["MJ: ML Model Scoring Bindings"] --> DET["DetectStaleness"]
    DET --> CAD{"cadence?<br/>(anchor: lastScored or TrainedAt<br/>older than cadenceDays)"}
    DET --> VOL{"data-volume?<br/>(growth > threshold)"}
    DET --> DRIFT{"drift?<br/>(IDriftDetector seam)"}
    CAD & VOL & DRIFT -->|any fires| STALE["stale → TriggerRetrainIfStale"]
    STALE --> CHAL["train challenger (TrainingEngine)"]
    CHAL --> CMP["CompareChallenger<br/>(metric-direction aware)"]
    CMP --> REC{"improvement ≥ margin?"}
    REC -->|yes| PROP["recommendation = 'promote'<br/>(PROPOSE only)"]
    REC -->|no / unknown| HOLD["recommendation = 'hold'"]
    PROP --> HUMAN["Human sign-off required<br/>(§13 promotion gate)<br/>NEVER auto-promotes"]
    style HUMAN fill:#fee2e2,stroke:#dc2626

Staleness — three triggers. DetectStaleness accumulates StalenessReason[]; the binding is stale if any fires:

  • Cadence — anchor is binding.LastScoredAt (when cadenceAnchor='lastScored') or model.TrainedAt; stale if older than cadenceDays. A never-scored/never-trained binding is treated as stale so it’s picked up on the first pass.
  • Data-volume — growth = (currentRowCount − model.TrainingRowCount) / model.TrainingRowCount; stale if growth > dataVolumeGrowthThreshold (current count via an injected rowCounter).
  • Drift — gated by policy.driftEnabled, delegated to an injected IDriftDetector seam.

Scheduled re-scoring (RescoreScheduledBinding, for Mode='Scheduled' bindings) stamps LastScoredAt = now and LastRowCount = scoredCount after each pass.

Challenger-vs-incumbent — respects metric direction, never auto-promotes. When a binding is stale, a challenger is trained and compared. Direction is handled via Core’s shared MetricDirections.IsErrorMetric:

private metricImprovement(metric, incumbentValue, challengerValue): number | null {
if (incumbentValue == null || challengerValue == null) return null;
return MetricDirections.IsErrorMetric(metric) ? incumbentValue - challengerValue // lower-is-better
: challengerValue - incumbentValue; // higher-is-better
}
private recommendPromotion(improvement, margin): PromotionRecommendation {
if (improvement == null) return 'hold'; // unknown → hold
return improvement >= margin ? 'promote' : 'hold';
}

Positive improvement always means “better,” whichever direction the metric runs. CompareChallenger returns only a recommendation ('promote' | 'hold') — it performs no promotion. The class doc is explicit: “It NEVER auto-promotes — promotion is a separate signed-off Action,” and the comparison detail always appends “Promotion still requires explicit human sign-off — nothing has been promoted automatically.” Promotion only ever happens through the §13 gate with its sign-off.

One nuance: the 'auto' comparison-metric selection (metrics.ts → ResolveComparisonMetric) only auto-picks keys whose direction MetricDirections.MetricDirectionOf declares higher- or lower-is-better (closer-to-one keys such as calibration slope and unknown keys are skipped), and metricImprovement compares either direction correctly.


15. Phase 2 — Model as a First-Class Primitive (Reach)

Section titled “15. Phase 2 — Model as a First-Class Primitive (Reach)”

Phase 1 (§1–§14) shipped the model entity and its core invocation surfaces — Actions, Remote Operations, and write-back into entity fields — proven live end-to-end. Phase 2 extends the model’s reach: the same trained scorer, plumbed into every MJ surface where a prediction is useful. None of this is new ML. There is exactly one scorer (MLModelInferenceProcessor, §6) and exactly one scoring entry point (PredictiveStudioScoreRecordSetAction / the PredictiveStudio.ScoreRecordSet Remote Op, which share one delegation path); Phase 2 is purely the plumbing that exposes that one scorer as a saved/scheduled Record Process, a discoverable per-model Action, a query column, and an interactive-component capability — and turns its output into ordinary entity data via write-back.

15.1 The unifying picture — one scorer, many surfaces

Section titled “15.1 The unifying picture — one scorer, many surfaces”
flowchart TB
    M["MJ: ML Models<br/>(one immutable, published model)"]
    M --> SC["The ONE scorer<br/>MLModelInferenceProcessor<br/>(PredictiveStudio.ScoreRecordSet)"]

    SC --> RP["PS2-1 · Saved 'ML Model'<br/>Record Process<br/>(on-demand + scheduled)"]
    SC --> ACT["PS2-2 · Per-model Action<br/>'Score with &lt;pipeline&gt; v&lt;n&gt;'<br/>(generated on Publish)"]
    SC --> QRY["PS2-3 · Scored Query column<br/>(RunQuery Enrichment)"]
    SC --> IC["PS2-4 · Interactive Component<br/>utilities.ml.score / .listModels"]

    RP --> WB["WriteBackProcessor<br/>OutputMapping → real entity column"]
    ACT --> WB
    QRY --> EPH1["ephemeral prediction column<br/>appended to the result rows"]
    IC --> EPH2["ephemeral predictions<br/>folded into charts / tables"]
    WB --> COL["RenewalScore / PredictedRenewalClass<br/>= ordinary, sortable / filterable /<br/>reportable entity data"]

Decoupling is the discipline that makes this safe. No base/types package gains a dependency on Predictive Studio. Every surface plugs in through a registry or the MJGlobal ClassFactory, resolved by string key at runtime, no-op when the providing package isn’t loaded:

SurfaceSeamLives in (PS-free)Provided by (PS)
Record ProcessRecordProcessorRegistry@memberjunction/record-set-processor-baseregisterMLScoringProcessor() at startup
Scored QueryQueryResultEnricherBase + ClassFactory@memberjunction/core (queryResultEnricher.ts)@RegisterClass(QueryResultEnricherBase, 'ML Model Score')
Interactive ComponentComponentUtilities.ml? (optional)@memberjunction/interactive-component-typesRuntimeUtilities.CreateSimpleMLTools() (ng-react)
Per-model ActionMJ: Actions metadata + ParentID— (pure metadata)ModelScoringActionGenerator on Publish

15.2 PS2-1 — Scoring as a saved (and scheduled) Record Process

Section titled “15.2 PS2-1 — Scoring as a saved (and scheduled) Record Process”

What it is. Phase 1’s MLModelInferenceProcessor is a Record Set Processing work type ('ML Model', §6), but a @RegisterClass decoration only makes it resolvable through the ClassFactory — it does not make the substrate route a saved MJ: Record Processes row with WorkType='ML Model' to it. PS2-1 closes that gap so a saved, replayable, schedulable Record Process scores a scope and writes a prediction column.

Key entry points.

  • registerMLScoringProcessor(deps: MLInferenceDeps): void (Engine/src/scoring/register.ts) — installs a factory into RecordProcessorRegistry.Instance for both 'ML Model' and the 'MLModelInference' alias. The factory reads the per-run MLScoringConfiguration ({ modelId, primaryKeyField?, datedSources? }) off the Record Process’s Configuration JSON and closes over the injected MLInferenceDeps (model loader / artifact loader / sidecar).
  • PredictiveStudioScoringStartup (Engine/src/scoring/startup-register.ts) — a @RegisterForStartup-decorated IStartupSink (a BaseSingleton) whose HandleStartup() calls registerMLScoringProcessor(BuildProductionMLInferenceDeps()) once at MJAPI boot. It warms no BaseEngine cache and touches no DB — the injected seams take their provider + user per call — so the registration is a cheap synchronous map write (not deferred). Mirrors RemoteBrowserEngine’s startup pattern.
  • The migration migrations/v5/V202606280215__v5.44.x__RecordProcess_WorkType_AddMLModel.sql adds 'ML Model' to the CK_RecordProcess_WorkType CHECK constraint — drop the old constraint + add the new one ('Action' | 'Agent' | 'Infer' | 'FieldRules' | 'ML Model') in a single migration. The CHECK is the source of truth: CodeGen derives the field’s value list, its ValueListType, and the generated TypeScript union from it, and the value validates at SAVE time (a typo’d work type is rejected up front). Work-type resolution still happens at execution time via RecordProcessExecutor.buildProcessor() → RecordProcessorRegistry.Resolve() (the registry is the pluggable seam) — the CHECK just guarantees the stored value is a known work type. To add further work types, extend the CHECK the same way + re-run mj codegen. (Never hand-edit EntityFieldValue / ValueListType to do this — that metadata is CodeGen-derived from the CHECK; see migrations/CLAUDE.md.)

Dry-run write-back fix. PS2-1 also hardened the substrate’s write-back so a dry-run of any wrapped work type (Action / Agent / Infer / ML Model) computes its effect without mutating data. applyOutputMapping(... dryRun: true) (packages/RecordSetProcessor/engine/src/writeBack.ts) now still resolves the mapped values and validates the target entity/field metadata, but saves nothing — returning the would-be values in previewFields / previewChild (WriteBackResult.dryRun). This mirrors the first-class ProcessRun.DryRun flag threaded through WriteBackProcessor.ts and RecordProcessExecutor.ts, and through the action’s ScoreRecordSetRequest.dryRun.

Usage — score a view and write a probability column on demand:

// A saved MJ: Record Processes row (created once, then runnable / schedulable):
// WorkType = 'ML Model' // set via rp.Set('WorkType', …) — see 15.6
// ScopeType = 'Filter' ScopeFilter = "IsActive = 1"
// Configuration = { "modelId": "<MJ: ML Models id>", "primaryKeyField": "ID" }
// OutputMapping = { "fields": { "RenewalScore": "$.score" } } // write-back
// Run it (same path the scheduled job driver uses):
await RecordProcessExecutor.RunByID(recordProcessId, { contextUser, provider });
// → each row scored by the model → RenewalScore column written back.

Proven live by ps-live-recordprocess-scoring.ts — a SAVED Record Process scoring over GraphQL against a running MJAPI, relying on the startup-registered scorer (it does not register the work type itself).

15.3 PS2-2 — A per-model Action generated on Promote

Section titled “15.3 PS2-2 — A per-model Action generated on Promote”

What it is. When a model is promoted to Published, the promote gate auto-generates an idempotent child Action Score with <pipeline> v<n>, so every model published through the gate becomes a first-class, discoverable Action (a model the Pipeline Builder publishes directly gets none, §12.4) — ActionSmith / CodeSmith pick it up for free, with zero per-model code or AI codegen.

Key entry points.

  • ModelScoringActionGenerator (Engine/src/actions/model-scoring-action-generator.ts):
    • generateForModel(model, contextUser, provider): Promise<void> — find-or-create the Action by its deterministic name; reconcile its params (delete + recreate to stay in sync with the parent contract).
    • disableForModel(model, contextUser, provider): Promise<void> — flip Status to 'Disabled' when the model is Archived.
  • The hook is in ProductionModelPromotionGate.syncScoringAction() (Engine/src/actions/promote-model.gate.ts), called from transition() after the lifecycle save: generate on Published, disable on Archived. It is a pure enhancement — any failure is logged and swallowed so it can never fail an otherwise-successful promotion.

Composition, not inheritance (CLAUDE.md Actions rule). The generated row is a child of the canonical Score Record Set parent — it carries the parent’s ParentID (metadata link only) and reuses the same DriverClass (SCORE_RECORD_SET_DRIVER_CLASS = 'PredictiveStudioScoreRecordSetAction'). That driver already reads ModelID / Scope / WriteBack, so the child needs no new implementation. The only thing model-specific is the ModelID input param’s DefaultValue, baked to model.ID — ActionEngine seeds each param from its DefaultValue before invoke, so omitting ModelID scores with this model. The Action is named from the model’s denormalized Pipeline view field (the model has no Name column) and Version, and grouped under a find-or-create Predictive Studio Models Action Category.

Proven live by ps-live-modelaction-generation.ts — promotes a model to Published over GraphQL, then reads MJ: Actions / MJ: Action Params back and asserts the generated Action name + the baked ModelID default.

What it is. A RunQuery post-query enrichment seam: a query run with an Enrichment directive has its result rows scored by the existing scorer and the prediction appended as a new column — post-query, not SQL-inline, so the model stays in the engine (portable, no DB-side model dependency).

Key types/entry points (all decoupled from PS).

  • QueryResultEnricherBase (abstract) + RunQueryEnrichment ({ EnricherKey: string; Config: Record<string, unknown> }) + resolveQueryResultEnricher(key) live in @memberjunction/core (src/generic/queryResultEnricher.ts). RunQueryParams.Enrichment?: RunQueryEnrichment is a runtime-only directive — there is intentionally no persisted per-query column (a saved annotation is a deliberate follow-up), so the feature is purely additive: no schema change, no CodeGen.
  • The hook is in GenericDatabaseProvider.InternalRunQuery → enrichQueryResults(...): after paging, if params.Enrichment?.EnricherKey is set, it resolves the enricher via the ClassFactory and awaits EnrichResults(...). Resolution checks GetRegistration before constructing (so it never instantiates the abstract base) and the whole call is wrapped in try/catch — when no enricher is registered (providing package not loaded) or anything fails, it returns the original rows. A scoring problem can never break the query.
  • MLModelScoreEnricher (Engine/src/scoring/ml-model-score-enricher.ts, @RegisterClass(QueryResultEnricherBase, 'ML Model Score')) is the PS half. Its MLModelScoreEnricherConfig is { modelId, outputField, primaryKeyField?, valueKind?: 'score' | 'class' }. Because the model determines the target entity, the enricher only needs each row’s primary key: it collects the PKs, scores them without write-back (ephemeral), and joins predictions back onto rows by id (O(1) map). LoadMLModelScoreEnricher() anchors the registration against tree-shaking.

Usage:

const result = await new RunQuery().RunQuery({
QueryName: 'Active Members',
Enrichment: {
EnricherKey: 'ML Model Score',
Config: { modelId: '<MJ: ML Models id>', outputField: 'RenewalScore', primaryKeyField: 'ID' },
},
});
// → each result row now carries a `RenewalScore` column (the model's numeric score).

Proven live by ps-inproc-scored-query.ts — runs in-process (the Enrichment param is not marshaled over GraphQL yet, by design), exercising the full path: RunQuery → resolveQueryResultEnricher('ML Model Score') → MLModelScoreEnricher → MLModelInferenceProcessor → DB + sidecar → new column.

15.5 PS2-4 — An ml capability on Interactive Components

Section titled “15.5 PS2-4 — An ml capability on Interactive Components”

What it is. An optional ml capability on the Interactive Components utilities surface, so Skip’s components can list trained models and score records and weave predictions (renewal likelihood, lead scores, churn risk) into charts / tables / dashboards. Push-to-generic: every interactive component gains it.

Key types/entry points.

  • SimpleMLTools (packages/InteractiveComponents/src/shared.ts) — the capability interface:
    • listModels(filter?: SimpleMLListModelsFilter, contextUser?): Promise<SimpleMLModelInfo[]> — newest version first; resilient (returns [] on failure).
    • score(modelId, records: Array<Record<string,unknown> | string>, options?): Promise<SimpleMLScoreResult> — ephemeral predictions (nothing written back).
    • Supporting shapes: SimpleMLModelInfo, SimpleMLPrediction, SimpleMLScoreResult, SimpleMLListModelsFilter.
  • ComponentUtilities.ml?: SimpleMLTools (runtime-types.ts) is optional — it can be undefined when the capability isn’t available in the current environment/security context, so component code must guard for that.
  • RuntimeUtilities.CreateSimpleMLTools() (packages/Angular/Generic/react/src/lib/utilities/runtime-utilities.ts) builds it: listModels reads the MJ: ML Models catalog via RunView (defaulting Status='Published'); score marshals the PredictiveStudioScoreRecordSetOperation (the PredictiveStudio.ScoreRecordSet Remote Op) over GraphQL to the server engine — because the Python sidecar lives server-side and cannot run in the browser. Returns undefined when there is no GraphQLDataProvider, so the ml capability degrades cleanly.

Usage (inside an interactive component):

if (utilities.ml) {
const models = await utilities.ml.listModels({ targetVariable: 'Renewed' });
const { predictions } = await utilities.ml.score(models[0].id, memberRows /* objects or pk strings */);
// fold predictions (by recordId) into the chart/table…
}

15.6 PS2-6 — The north-star: scheduled model scoring

Section titled “15.6 PS2-6 — The north-star: scheduled model scoring”

What it is. The end-to-end composition Phase 2 builds toward: “build a model, write the renewal probability back into the member record, and re-score it on the 1st of every month” — as a single call, with no new scheduling, dispatch, or write-back code. Everything underneath already exists (PS2-1 + the scheduled-job machinery); the helper just assembles one MJ: Record Processes row.

Key entry point. createScheduledModelScoring(opts: ScheduleModelScoringOptions): Promise<MJRecordProcessEntity> (Engine/src/scheduling/scheduled-model-scoring.ts). Required quartet: modelId, targetEntityName, outputField, and exactly one scope selector (filter | viewId | listId); sensible defaults otherwise ('Monthly' cadence = 0 0 1 * *, ID primary key, numeric 'score', UTC). It validates, maps the ScoringCadence to a cron (cadenceToCron), and populates the row:

  • WorkType='ML Model' — set via rp.Set('WorkType', 'ML Model'), the documented legitimate exception to the no-.Set() rule: the CodeGen’d WorkType union ('Action' | 'Agent' | 'FieldRules' | 'Infer') cannot represent a runtime-registered extension work type. (Same exception PS2-1’s path uses.)
  • Configuration = { modelId, primaryKeyField }, OutputMapping = { fields: { <outputField>: '$.score' | '$.class' } }.
  • Status='Active', ScheduleEnabled=true, CronExpression, Timezone, OnDemandEnabled=true.

Saving that row auto-creates an owned MJ: Scheduled Jobs row via MJRecordProcessEntityServer.Save → reconcileScheduledJob; SchedulingEngine then dispatches it on its cron through RecordProcessScheduledJobDriver → RecordProcessExecutor.RunByID(...) — which resolves the scope, scores each row, and writes the prediction back per the OutputMapping. The schedule-model-scoring.action.ts Action wraps the helper for agent/workflow invocation.

sequenceDiagram
    actor User
    participant Agent as Model Dev Agent
    participant Train as TrainingEngine
    participant Helper as createScheduledModelScoring
    participant RP as MJ: Record Processes
    participant Job as Owned MJ: Scheduled Jobs
    participant Sched as SchedulingEngine
    participant Exec as RecordProcessExecutor
    participant Member as Membership row

    User->>Agent: "predict renewal & re-score monthly"
    Agent->>Train: trainModel(pipeline)
    Train-->>Agent: published MJ: ML Models row
    Agent->>Helper: createScheduledModelScoring({ modelId, targetEntity, outputField, scope, cadence:'Monthly' })
    Helper->>RP: save row (WorkType='ML Model', ScheduleEnabled, cron='0 0 1 * *', OutputMapping)
    RP->>Job: Save → reconcileScheduledJob (auto-creates owned job)
    loop 1st of every month
        Sched->>Exec: RunByID(recordProcessId)
        Exec->>Member: score + write RenewalScore back
    end

Proven live by ps-inproc-scheduled-scoring.ts — one createScheduledModelScoring(...) call, then it verifies the owned Scheduled Job the scheduler would dispatch and runs the Record Process now via RecordProcessExecutor.RunByID (the same path the job driver uses), asserting the write-back landed.

Each Phase 2 capability ships with an MJServer integration script that exercises the real path (no mocked ML). The ps-live-* scripts drive an already-running MJAPI over GraphQL (system API key); the ps-inproc-* scripts bootstrap a real SQLServerDataProvider in-process (used where the path isn’t marshaled over GraphQL yet). All gate the sidecar legs behind PS_INTEGRATION=1 and skip gracefully without it.

CapabilityScriptHarnessProves
PS2-1 — saved Record Process scoringps-live-recordprocess-scoring.tslive / GraphQLa saved WorkType='ML Model' Record Process scores a view + writes columns, via the startup-registered scorer
PS2-2 — per-model Action on Promoteps-live-modelaction-generation.tslive / GraphQLpromoting to Published generates Score with <pipeline> v<n> with ModelID DefaultValue baked in
PS2-3 — scored-query enrichmentps-inproc-scored-query.tsin-processRunQuery + Enrichment appends the model’s prediction column
PS2-6 — scheduled model scoringps-inproc-scheduled-scoring.tsin-processone createScheduledModelScoring(...) call → owned Scheduled Job + a real scored write-back

PS2-5 (catalog awareness for agents) is delivered through PS2-2: every published model is a discoverable Action, which ActionSmith / CodeSmith see for free — no separate code.


Phases 1–2 built the platform and gave the model reach. This is the layer that makes the whole thing usable, visible, and trustworthy for a non-technical business user — the conversational north-star made real, end to end, with every surface generic (binding-driven, entity-agnostic; “members/renewal” is throwaway test data, never a build target). Full plan + commit lineage: plans/predictive-studio-business-experience.md.

16.1 The Predictions surface (the front door)

Section titled “16.1 The Predictions surface (the front door)”

The default Predictive Studio nav item is Predictions (PSPredictionsResourceComponent, in ng-dashboards) — the business front door, modeled on the locked plans/predictive-studio-business-ux/option-b-refined.html:

  • Catalog — every published model reframed as a plain-language prediction card with an outcome badge (TrustVerdicts.DeriveTrustVerdict, a rendering of the gate decision stored on the model’s training run): “Ships — the model beats the flat guess”, “Ships — a simple baseline”, “No signal — not usable” or “Not evaluated yet”. A card that can act shows the headline (“Beats the flat guess by X [lo, hi]”); a prediction that cannot (no signal, or never evaluated under an Evaluation Protocol) shows the stored reason with Open disabled: the trust gate at the catalog level. Cards sort shipped models first, then kept simple baselines, no-signal, and not evaluated.
  • Workspace (open a card) — a trust verdict banner that gates the action bar; a ranked at-risk list (the embedded ps-predictions-grid reads the latest run’s MJ: Process Run Details once, up to PredictiveStudioEngine.MaxRunDetailRows, and hands the parsed rows to the workspace through its RowsLoaded output; highest-risk first, high/medium/low risk tiers for classification, outcome-band tiers for regression) with the features weighted most by the model (AtRiskRules.TopGlobalDrivers over the model’s global feature importance — global, not per-record SHAP; shown only when the model’s Lineage says the importance is coefficients or held-out permutation importance (ImportanceDisplay.GlobalDriversShowable), hidden for split or impurity counts, which models trained before Prediction Units carry until they are retrained); and four trust-gated actions — Review the call list · Save scores (opens the docked Model Dev Agent co-pilot seeded with the request) · Send to a list (creates an MJ List of the highest-risk records at or above AT_RISK_LIST_MIN_RISK_PCT, at most 200, and reports how many of the qualifying records it added) · Share/export (CSV).
  • ”+ New prediction” opens the deterministic-builder agent (§12.4) as a docked mj-conversation-chat-area co-pilot. Embedding the chat correctly requires the conversation lifecycle wiring — bind [isNewConversation] (start true), [conversation]/[conversationId], and handle (conversationCreated) to capture the new conversation + flip out of new-mode (mirror (pendingMessageConsumed) to clear the seed). Without it, the suppressed empty-state input has no conversation to write into and the first send silently no-ops. Both the Predictions co-pilot and the Studio co-pilot use this pattern; copy it for any new embedded PS chat.

Nav structure — three doors. The Predictive Studio app exposes exactly three top-level nav items (metadata/applications/.predictive-studio-application.json), consolidated from an earlier eight-flat-item layout so business users see one door and analysts opt into depth:

  • Predictions (PredictiveStudioPredictionsResource) — the business front door above.
  • Studio (PredictiveStudioStudioResource) — the build/run workbench: an internal left-nav grouped Overview · Define (Prediction Units, Evaluation Protocols) · Build (Training Pipelines, Algorithm Catalog) · Run (Experiments, Compare Runs), plus the shared docked Model Dev Agent co-pilot. The two Define sections list version lineages (mj-ps-versioned-list) and open the latest version of one on its record form.
  • Models (PredictiveStudioModelsResource) — the trained-model lifecycle: an internal left-nav over Model Registry · Models in Production.

The two workbench doors are single BaseResourceComponents that host the same standalone section panels (ps-home, ps-pipelines, …) the old per-section resources hosted — only the wrapping changed. The active section round-trips through a section query param (deep links + back/forward), and the Overview panel’s cross-door links route via RouteHomeNavigate (pure, unit-tested in predictive-studio.nav.ts). The trust translator (@memberjunction/predictive-studio-core) is the single shared rule behind the catalog badges/gate and the agent’s publish gate. Record-form panels. Four panels register with the Studio chunk (record-panels/ps-protocol-panels.ts, BaseFormPanel via @RegisterClassEx) and mount the Studio widgets on the generated record forms: the versions panels on MJ: ML Evaluation Protocols and MJ: ML Prediction Units (the editors save the next version, they never change a row; protocol guard fields follow the protocol’s GuardFieldsEditableBy), the bindings panel on MJ: ML Training Pipelines (bind the pipeline to the unit and protocol versions it trains under), and the evaluation panel on MJ: ML Models, which shows the training run’s protocol results: scores with intervals, the baselines, the improvement over the flat guess, the gate outcome and its reason, calibration and leakage flags, with the winner’s inner score labelled optimistic. For a model trained before Prediction Units the panel reads “This run was not evaluated under an Evaluation Protocol.” (the catalog badge for the same model reads “Not evaluated yet”). The wizard binds the version-1 unit and protocol it creates with a new pipeline; a binary target whose outcome polarity is not declared is refused.

The pure VMs (business-predictions.view-models.ts, predictive-studio.nav.ts), the at-risk rules (AtRiskRules, Core at-risk-rules.ts) and the trust translator are unit-tested. Full design-driven lineage (brief → 3 divergent mockups → locked direction → plan): plans/predictive-studio-business-ux/.

The one-sentence goal: a membership director, in chat, asks “which members are likely to lapse?” — and minutes later every member record shows a renewal-risk score that refreshes monthly, driven and explained conversationally, with the deployed model visible in a control-tower view.

flowchart LR
    User["Business user<br/>(Home → 'Ask the agent')"]
    Agent["Model Development Agent<br/>(collaborate → approve → train → promote)"]
    Offer{{"Offer to operationalize<br/>'score everyone + refresh monthly?'"}}
    Sched["createScheduledModelScoring()<br/>= Record Process (WorkType='ML Model', monthly cron)<br/>+ MJ: ML Model Scoring Binding"]
    Job["Owned MJ: Scheduled Jobs<br/>→ SchedulingEngine (1st of month)"]
    WB["RecordProcessExecutor<br/>score + write prediction back"]
    Rec[("Entity record<br/>(prediction column)")]
    Card["WS1 · Risk card<br/>(generic '*' form panel,<br/>binding-driven)"]
    Prod["WS2 · Models in Production<br/>(bindings + process runs + distribution)"]

    User --> Agent
    Agent --> Offer
    Offer -->|user says yes| Sched
    Sched --> Job
    Job -->|each cadence| WB
    WB --> Rec
    Sched -.binding.-> Card
    Sched -.binding.-> Prod
    Rec --> Card
    Rec --> Prod
    style Sched fill:#ede9fe,stroke:#6d28d9
    style Card fill:#ecfeff,stroke:#0891b2
    style Prod fill:#ecfeff,stroke:#0891b2

The whole loop never leaves the conversation: the agent trains, promotes, proactively offers to operationalize, and on a yes wires up scheduled scoring; the binding it records is what lights up the risk card and the production view.

16.2 The Model Development Agent’s conversational flow

Section titled “16.2 The Model Development Agent’s conversational flow”

The agent (§12) now closes the loop. Its prompt (metadata/prompts/templates/model-development-agent/model-development-agent.template.md, §5 “Operationalize — proactively offer”) teaches it that a trained, promoted model is only useful once its predictions are on the records, kept current. So after a clean promote (or a signed-off leakage flag), it proactively offers — not waiting to be asked — in plain business language tied to the user’s goal:

“This model’s ready to go. Want me to score every active member right now, write each one’s renewal-risk onto their record, and then refresh it every month so it always stays current?”

It offers clear suggestedResponses (Yes, score & schedule monthly / Just score once now / Not yet). On a yes, it calls the Schedule Model Scoring action — now in its MJ: AI Agent Actions toolkit (metadata/agents/.model-development-agent.json) and fully metadata-synced (metadata/actions/predictive-studio/.predictive-studio-actions.json: ModelID, TargetEntityName, OutputField, ScopeFilter, optional Cadence (default Monthly) / PrimaryKeyField / ValueKind; result codes SUCCESS / VALIDATION_ERROR / SCHEDULE_FAILED). It proposes a column name if the user hasn’t named one, confirms the population (ScopeFilter), then reports back in plain English what it scheduled. Keeps it generic — “this works for any entity and any prediction target, not just members or renewal.”

The Action wraps createScheduledModelScoring (§15.6); the helper now also records the scoring binding (§16.6), which is the seam that makes the next two surfaces light up.

16.3 The generic model-prediction form panel (the risk card)

Section titled “16.3 The generic model-prediction form panel (the risk card)”

The payoff renders on any entity a model writes to, with no per-entity code:

  • Component — ModelPredictionPanel (packages/Angular/Explorer/core-entity-forms/src/lib/panels/model-predictions/), a BaseFormPanel registered under the '*' wildcard entity:
    @RegisterClassEx(BaseFormPanel, {
    key: 'model-predictions:model-prediction',
    skipNullKeyWarning: true,
    metadata: { entity: '*', slot: 'after-fields', sortKey: 40 },
    })
  • The '*' slot wildcard — added to the slot host (packages/Angular/Generic/base-forms/src/lib/panel-slot/). entityMatches() now treats '*' as “mount on every entity’s form”; such panels are expected to self-hide when they don’t apply. This is the entity-agnostic-panel contract: one registration, cross-cutting, unobtrusive.
  • Binding-driven — on any record the panel RunViews MJ: ML Model Scoring Bindings filtered to TargetEntityID=<this entity> AND TargetColumn IS NOT NULL. No binding → no cards → the panel self-hides (@if (HasPredictions), i.e. Cards.length > 0). It’s invisible on every form a model doesn’t touch.
  • Entity-neutral band + format — the bound value renders as a meaningful indicator: a low/mid/high band shaded with --mj-brand-primary opacity (no moral “risk” direction baked in — generic by construction); 0–1 values → ”%”, regression → grouped number, classification → class label, missing → ”—”. Plus the top 5 features weighted most by the model (parsed from the model’s FeatureImportance; shown only for coefficient or held-out permutation importance, hidden for split or impurity counts), the model + version (“Pipeline v{n}”), and last scored.

A sibling panel, FeatureValuesPanel (packages/Angular/Explorer/core-entity-forms/src/lib/panels/feature-values/, '*' wildcard, sortKey 41, right after the Model Prediction panel), shows the Feature Values stored for the record in MJ: Feature Values: the latest answer per pipeline and feature with Source, Confidence, pipeline version, Computed, Reasoning and older values, plus a Refresh button. A latest answer that withheld the value (an abstention, or a value a person cleared) reads “no value” and supersedes any older value, which stays in the history. It is read-only and renders nothing when the record has no Feature Values. It is where the output of a feature pipeline with no write target (“Feature value only”) is visible.

16.4 “Models in Production” — the control tower

Section titled “16.4 “Models in Production” — the control tower”

A new Studio section (PSProductionComponent + PSProductionResourceComponent, @RegisterClass(BaseResourceComponent, 'PredictiveStudioProductionResource'); packages/Angular/Explorer/dashboards/src/PredictiveStudio/). For each deployed model it shows: model label + algorithm + problem type + status, what it writes (target entity / column), the mode + schedule phrase (humanizeCron), the last run (the latest MJ: Process Runs row for the binding’s RecordProcessID — status, time, success/error counts) or the binding’s own LastScoredAt / LastRowCount, and a prediction-distribution mini-viz. The distribution (production-distribution.ts, pure + unit-testable) samples up to 5,000 non-null values of the bound column and buckets them entity-neutrally — numeric → three terciles, categorical → top-8-by-frequency + “Other”. Everything reads from MJ: ML Model Scoring Bindings + MJ: Process Runs + the target entity via metadata; zero Member-specific code, multi-provider-correct (RunView.FromMetadataProvider(this.ProviderToUse)).

16.5 The live Studio (de-mocked) + conversational entry

Section titled “16.5 The live Studio (de-mocked) + conversational entry”

The Studio (§18) no longer ships sample arrays on screen — the SAMPLE_* arrays are gone (the only SAMPLE_ token left is the PRODUCTION_SAMPLE_CAP = 5000 constant). Pure, testable view-models (predictive-studio.view-models.ts) feed each panel from live engine data: Registry from engine.Models (metrics / holdout / feature-importance parsed live), Experiments from engine.Sessions / engine.Iterations, Home KPIs + activity feed from engine.Models / engine.Sessions + recent MJ: Process Runs, Compare from the session leaderboard. The previously-dead buttons are wired to Remote Operations behind a ps-confirm-modal:

  • Promote / Mark Validated / Archive → PredictiveStudioPromoteModelOperation ({ modelId, targetStatus, signOff?, reason? } — a leakage sign-off reason is required when the model is flagged).
  • Pause / Resume / Cancel → PredictiveStudioControlExperimentSessionOperation ({ sessionId, action }).

Each mutation force-refreshes the engine (Config(true, …)) and reports success/failure. And Home’s “Ask the agent” CTA opens the docked <mj-conversation-chat-area> pinned to the Model Dev Agent with a seeded starter prompt ([pendingMessage] auto-sends) — one click from Home into a seeded conversation.

16.6 The scoring-binding lineage (the seam that lights it all up)

Section titled “16.6 The scoring-binding lineage (the seam that lights it all up)”

Both §16.3 and §16.4 read MJ: ML Model Scoring Bindings — so operationalizing must record a binding, not just create a scheduled Record Process. createScheduledModelScoring (packages/AI/PredictiveStudio/Engine/src/scheduling/scheduled-model-scoring.ts) now, after saving the Record Process, calls upsertScoringBinding(...) to create/update the binding row with MLModelID, RecordProcessID, TargetEntityID, TargetColumn, and Mode='Scheduled'. That binding is the lineage record — “this model is in production, scoring these records into this column on this schedule” — and the seam that connects the headless scoring loop to the human-facing surfaces. Without it the scheduled job would run silently and both surfaces would stay dark.

  • 'ML Model' WorkType dropdown — migration V202606280215__v5.44.x__RecordProcess_WorkType_AddMLModel.sql adds 'ML Model' to the CK_RecordProcess_WorkType CHECK (drop + re-add the constraint with the five values in one migration). The CHECK is the source of truth — CodeGen derives the value list, ValueListType, and the generated union from it, so after the next CodeGen the union becomes 'Action' | 'Agent' | 'FieldRules' | 'Infer' | 'ML Model' and 'ML Model' is a real dropdown option in the Bulk-Operations / Record-Process studio. Requires CodeGen to regenerate. (This supersedes an earlier two-migration approach — one dropping the CHECK, one hand-seeding EntityFieldValue rows — which was wrong: value lists are CodeGen-derived from the CHECK, never hand-written. See migrations/CLAUDE.md.)
  • Enrichment over GraphQL RunQuery (PS2-3 follow-up) — the RunQuery resolver (packages/MJServer/src/resolvers/QueryResolver.ts) accepts @Arg('Enrichment', () => GraphQLJSONObject, { nullable: true }) and threads it into RunQuery({ …, Enrichment }); the GraphQLDataProvider client carries the same Enrichment?: RunQueryEnrichment ({ EnricherKey, Config }). So browser / Query-Builder callers — not just server-side Skip — can request a scored query via the 'ML Model Score' enricher.
CapabilityKey file(s)Proof
Agent offers to operationalize, schedules monthlymetadata/agents/.model-development-agent.json · metadata/actions/predictive-studio/.predictive-studio-actions.json · …/model-development-agent.template.md (§5)conversational (live agent run); the scheduling path → ps-inproc-scheduled-scoring.ts
Generic risk card ('*' form panel)core-entity-forms/.../panels/model-predictions/ · Generic/base-forms/.../panel-slot/ ('*' wildcard)binding-driven render (lights up via the binding from ps-inproc-scheduled-scoring.ts)
Models in Production sectiondashboards/.../PredictiveStudio/components/ps-production.component.* · resources/ps-production-resource.component.ts · production-distribution.tsreads bindings + MJ: Process Runs (live)
De-mocked Studio + ops wireddashboards/.../PredictiveStudio/predictive-studio.view-models.ts (+ Registry / Experiments components)PredictiveStudioPromoteModelOperation / …ControlExperimentSessionOperation
Scoring-binding on operationalizePredictiveStudio/Engine/src/scheduling/scheduled-model-scoring.ts (upsertScoringBinding)ps-inproc-scheduled-scoring.ts (binding asserted)
'ML Model' WorkType dropdownmigrations/v5/V202606280215__…RecordProcess_WorkType_AddMLModel.sql (CHECK drop+add)(needs CodeGen)
Enrichment over GraphQLMJServer/src/resolvers/QueryResolver.ts · GraphQLDataProviderps-inproc-scored-query.ts

Predictive Studio ships end-to-end. The one explicitly-deferred capability is honest here:

ItemStatusDetail
SP5 — Materialized prediction columnsDeferred, gated on PR #2770The “population-wide, indexed prediction column” optimization runs ML batch-scoring inside the offline materialization refresh and writes predictions as physical columns on a materialized VirtualEntity (Skip/reports/views query them like any table). It depends on PR #2770 landing the MJ: Materialized Results table. The forward-compat seam is already in place: MJ: ML Model Scoring Bindings.MaterializedResultID is a soft reference (not a FK) to that table, and the ScoringBindingMode = 'OnDemand' | 'Scheduled' | 'Materialized' already names the Materialized mode. On-demand + scheduled scoring (§6) ship with zero dependency on #2770 — when the table lands, materialization is additive.

Other open questions carried forward (none block the shipped feature): model serialization native-vs-ONNX for cross-platform portability; reserving algorithm slots (k-NN / Naive Bayes); the precise drift-detection method behind the IDriftDetector seam; sidecar multi-tenancy / per-session resource limits for long agentic searches; and detached fire-and-forget Remote-Op progress (poll-via-status today, full background execution later).


Predictive Studio is an MJ Explorer dashboard built to the ng-dashboards world-class standards (see Dashboard Best Practices) and lazy-loaded (see Lazy Loading Guide). It lives at packages/Angular/Explorer/dashboards/src/PredictiveStudio/.

  • Shell: three BaseResourceComponents — PredictiveStudioPredictionsResource, PredictiveStudioStudioResource and PredictiveStudioModelsResource (§16.1) — each with an internal left-nav, a section query-param round-trip (deep links survive back/forward).

  • Data layer: PredictiveStudioEngine (engine/predictive-studio.engine.ts) — a BaseEngine singleton that caches the PS reference entities via RunView (Algorithms, UseCases, Rankings, Models, Pipelines, TrainingRuns, Experiments, Sessions, Iterations) and exposes domain helpers (BestLevelsForScenarios, RankingsForUseCase, IterationsForSession). No Angular services for data — the canonical MJ pattern.

  • Panels (standalone components in components/, each receiving the engine via @Input()):

    PanelComponentDoor
    OverviewPSHomeComponentStudio — hero band + entry paths + activity timeline
    Prediction Units, Evaluation ProtocolsPSVersionedListComponent (mj-ps-versioned-list)Studio (Define) — version lineages that open on their record forms
    Training PipelinesPSPipelinesComponentStudio (Build) — SVG feature-assembly graph + node inspector + leakage/validation config; Train notifies the stored gate decision
    Algorithm CatalogPSCatalogComponentStudio (Build) — card-gallery + Guide-me scenario picker
    ExperimentsPSExperimentsComponentStudio (Run) — Running/Completed/Pruned columns + leaderboard strip + budget gauges
    Compare RunsPSCompareComponentStudio (Run) — side-by-side / overlay charts / champion-vs-challenger
    Model RegistryPSRegistryComponentModels — master-detail, lifecycle stepper, feature importance, lineage, sign-off gate
    Models in ProductionPSProductionComponentModels — the control tower (§16.4)
  • Embedded copilot: an <mj-conversation-chat-area> (from @memberjunction/ng-conversations, see Conversations UX Stack Guide) docked across all panels — agent picker hidden, pinned to the Model Development Agent, with an appContext carrying the active panel + published-model / running-session counts so the agent can act on the current context.

  • Knowledge Hub side: the Feature Pipelines panel (FeaturePipelinesResourceComponent, §11) lives in the Knowledge Hub dashboard — list / run / author derived-feature pipelines.

  • Lazy load: PredictiveStudioDashboardsModule is exported via the ./predictive-studio-dashboards.module subpath in the dashboards package.json; LoadPSPredictionsResource(), LoadPSStudioResource() and LoadPSModelsResource() are the tree-shaking anchors; the ClassFactory resolves the three resource drivers to the code-split chunk on demand.

The HTML option mockups that informed the pinned designs are at plans/predictive-studio/mockups/.


19. Getting started — train and score a model

Section titled “19. Getting started — train and score a model”
Terminal window
# 1. Install and build from the repo root (Predictive Studio depends on other workspace packages)
pnpm install
pnpm run build
# 2. Set up the bundled Python sidecar environment (once)
pnpm --filter @memberjunction/predictive-studio-sidecar run setup:python
brew install libomp # macOS only (Linux: install libgomp1)

Where Predictive Studio files are stored. One setting decides it: PS_STORAGE_PROVIDER names the MJ: File Storage Providers row (default Local File System, the Local File System Storage driver, rooted at STORAGE_LOCAL_ROOT) that every training run writes its Training Snapshot files and its trained model files to, each recorded as an MJ: Files row with that provider’s ProviderID and the object’s ProviderKey (model files under predictive-studio/model-artifacts/<uuid>/<name>). The provider is selected by name whatever its IsActive flag, and a provider that cannot write here refuses every run before training: one whose driver is not configured on the server (for example Local File System without STORAGE_LOCAL_ROOT), or one that is configured but whose write probe fails (a small object put under predictive-studio/model-artifacts/<uuid>/.probe and then deleted, which catches wrong credentials or an unwritable root; a probe that is written but cannot be deleted is logged as a warning and does not refuse). The refusal happens before any run row or snapshot file is written and before the sidecar is called, whether the run would write a new snapshot or reuse one. A stored file is always read through the provider its own MJ: Files row names, so changing PS_STORAGE_PROVIDER later never strands an existing model. The local driver hands the sidecar file:// URLs and keeps files on one host’s disk, so it works only when MJAPI and the sidecar share one host and disk; a cloud container or multi-host install (more than one MJAPI host, or a remote sidecar) must point PS_STORAGE_PROVIDER at a cloud provider (S3, Azure Blob, Google Cloud Storage…): files on one host’s disk are unreadable from the others and are lost with that disk. Model files trained before 6.3 have an empty ProviderKey; they are still read from PS_ARTIFACT_DIR (default <os tmpdir>/mj-ps-artifacts).

A model is trained from a pipeline bound to a Prediction Unit and an Evaluation Protocol:

  1. Define the unit and the protocol. In the Studio’s Define group (Prediction Units, Evaluation Protocols), or let the pipeline wizard create version 1 of each when it creates a pipeline (it maps them from the target; a binary target whose outcome polarity is not declared is refused). An existing pipeline without them is bound by the “Migrate Pipelines to Prediction Units and Evaluation Protocols” Action (§10.2), or bound by hand in the pipeline’s bindings panel.
  2. Train. Invoke the PredictiveStudio.TrainModel Remote Op (browser/Skip) or the Train ML Model Action (agent). Both delegate to TrainingEngine.trainModel and return the run’s stored gate decision. A pipeline with no unit is refused UNIT_REQUIRED and one with no protocol PROTOCOL_REQUIRED (§10.4).
  3. Read the outcome. The run stores model, keep-simple-baseline or no-signal with its reason; the model form’s evaluation panel shows the protocol’s results (§16.1).

Directly, the shape is:

import { TrainingEngine, TrainingDelegation } from '@memberjunction/predictive-studio';
// productionDeps = await TrainingDelegation.BuildTrainingDeps(provider, contextUser): MetadataEntityFactory,
// RunViewRecordLoader, MJSidecarTrainer (over MLSidecar), the artifact store, and the unit, protocol and
// snapshot stores.
const engine = new TrainingEngine();
const result = await engine.trainModel(
{ pipelineId: retentionPipelineId },
productionDeps,
);
// → result.model: a new immutable MJ: ML Models row (Status='Draft') with the shipped
// artifact, HoldoutMetrics, FeatureImportance and Lineage, or null when the stored gate
// decision ships none (no-signal); result.run: the training run, whose ValidationResults
// holds the ps-run-results/v2 envelope (the gate record, the snapshot and protocol IDs).

A pipeline (MJ: ML Training Pipelines) declares the target entity, target variable, problem type, algorithm, SourceBindings, FeatureSteps DAG, LeakageGuard and ValidationStrategy, and binds the Prediction Unit and Evaluation Protocol versions that decide its rows, decision dates and labels (the unit) and its locked holdout, validation and gate (the protocol). A pipeline without a unit or a protocol is refused; the retired AsOfStrategy column no longer drives anything.

Scoring runs as the 'ML Model' work type. On-demand against a view/list/selection via RecordProcess.RunNow with a scope override (or the PredictiveStudio.ScoreRecordSet op / Score Record Set Action); scheduled via the Schedule trigger (which also upserts an MJ: ML Model Scoring Binding for maintenance, §14). Attach an OutputMapping to write the prediction back as a sortable/filterable column; omit it for ephemeral scores. See the Record Set Processing Guide for the substrate mechanics.

19.4 Run the live integration test (PS_INTEGRATION=1)

Section titled “19.4 Run the live integration test (PS_INTEGRATION=1)”

The engine ships an opt-in end-to-end test that spawns the real managed sidecar and trains a Prediction Unit pipeline through its trainer (Engine/src/__tests__/integration/live-train-score.integration.test.ts):

Terminal window
# Prerequisite: the sidecar venv must exist
pnpm --filter @memberjunction/predictive-studio-sidecar run setup:python
# Run the live integration suite (sets PS_INTEGRATION=1 via vitest.integration.config.ts)
pnpm --filter @memberjunction/predictive-studio run test:integration

It exercises the real unit row reader, AsOfFeatureAssembler, TrainingEngine, MJSidecarTrainer and MLSidecar (managed spawn of the bundled Python), faking only the persistence seams (entity factory, record loader, artifact and snapshot stores; no DB needed). The snapshot files are served to the trainer as file:// URLs, because the trainer opens only http, https and file URLs. It asserts what is in the at-a-glance list: the run completes, the stored gate record counts the 400 holdout rows, and a decision that ships creates the model row from the shipped artifact (no-signal creates none). It skips gracefully with a clear console note when the venv or Python is unavailable, so it is CI-safe by default — the standard npm run test (mocked) always runs; the live suite is explicit opt-in.

19.5 Planted-truth suite (packages/AI/PredictiveStudio/TruthSuite)

Section titled “19.5 Planted-truth suite (packages/AI/PredictiveStudio/TruthSuite)”

The planted-truth suite (@memberjunction/predictive-studio-truth-suite) holds 68 seeded datasets (worlds W1–W9 and scenarios such as C2, R1, T1p–T4) whose ground truth (TrueP, structural coefficients, censoring dates, regime shifts) is known before any model is fit. It loads them into an isolated pstruth SQL Server schema and provides the harness the Predictive Studio containment gates run on. The gates are MJ Tests in the integration test suite: run them with mj test suite "Predictive Studio Gates — Containment". See packages/AI/PredictiveStudio/TruthSuite/README.md for setup.


AreaStatus
13-entity data model + CodeGen (the 10 original entities, plus Prediction Units, Evaluation Protocols and Training Snapshots)✅ built
Algorithm catalog + use cases + 6×7 rankings (metadata)✅ built
MLSidecar self-managing Python sidecar (managed + remote)✅ built
AsOfFeatureAssembler (rows as of each decision date, typed columns) + FeatureAssemblyExecutor (raw/preprocessing split for models trained before Prediction Units, leakage guard)✅ built
TrainingEngine (Prediction Unit + Evaluation Protocol, Training Snapshot, protocol trainer, three-outcome gate, immutable models, lineage)✅ built
MLModelInferenceProcessor (‘ML Model’ RSP work type, ephemeral/write-back)✅ built
ExperimentOrchestrator (waves, leaderboard, pruning, budget-in-runner)✅ built
7 Remote Operations (Train/Score/RunFeaturePipeline/Start+Control Experiment/Promote/CreateScoringProcess) + 6 Actions (the sixth migrates existing pipelines to Prediction Units and Evaluation Protocols)✅ built
Feature Pipelines (category route — FeaturePipelineEngine + KH panel; no new entity)✅ built
Model Development Agent + 3 sub-agents + ML Experiment Results artifact + Angular viewer✅ built
Security model (scope gate, PK field-name guard, UUID gate, promotion state machine, sign-off reason)✅ built
Maintenance (staleness + direction-aware challenger-vs-incumbent, never auto-promotes)✅ built
Multimodal vision-LLM-as-feature ('vision-llm' FeatureStep kind, additive)✅ built
Studio UI (three doors: Predictions, Studio with Define / Build / Run, Models; record-form panels for units, protocols, bindings and run results; engine; embedded copilot; lazy-load)✅ built
Business-user experience (§16) — agent offers + schedules monthly · generic '*' risk-card panel · Models-in-Production section · de-mocked Studio + ops wired · scoring-binding lineage · WorkType value-list + Enrichment-over-GraphQL✅ built
Live integration test (PS_INTEGRATION=1)✅ built
Materialized prediction columns (#2770)⏳ deferred — gated on PR #2770 (§17)

Related guides: Record Set Processing (the scoring + wave + feature-pipeline substrate) · Remote Operations & Transport-Layer Architecture (the invocation surface) · Dashboard Best Practices & Lazy Loading (the Studio UI) · Conversations UX Stack (the embedded copilot) · Agent Memory (the Model Dev Agent’s notes).

Package READMEs: Core (types) · Sidecar (MLSidecar) · Engine (assembly/training/scoring/experiments/ops/actions/maintenance).

For the authoritative, dependency-ordered task list, see plans/predictive-studio.md §14.