# Rubrics Guide

A **rubric** is a named, versioned set of criteria used to score any record in MemberJunction. A person, an
AI prompt, an agent, or a deterministic rule answers the criteria; one function, `RubricScoring`, turns the
answers into a stored score. Rubrics are a core primitive: the testing framework judges test runs with them,
agents check their own output with them, and applications score their own records with them.

**Read this guide before** you author a rubric, score a record, publish a version, bind a rubric to an agent
or a test, or build an application on top of rubrics. It is the map. The package READMEs are the API
reference:

| Layer | Package | What it is |
|---|---|---|
| Math | [`@memberjunction/rubrics-base`](../packages/Rubrics/Base/README.md) | `RubricScoring`, `RubricVersionDiff`, snapshots. Pure functions, browser-safe |
| Engine + evaluators | [`@memberjunction/rubrics`](../packages/Rubrics/Engine/README.md) | `RubricEngine`, the evaluator registry and the LLM / Decision / Agent / Deterministic / Human evaluators, actions, `mj rubric` |
| Server rules | `@memberjunction/core-entities-server` | Publish, submit, supersede, and validation on the entity subclasses |
| Widgets | [`@memberjunction/ng-rubrics`](../packages/Angular/Generic/rubrics/README.md) | Author, answer, result, publish, diff, and comparison widgets |
| Testing | [`@memberjunction/testing-engine`](../packages/TestingFramework/Engine/README.md) | `RubricOracle`, rubric resolution, the calibration test type |

**Contents**

1. [The model in one picture](#1-the-model-in-one-picture)
2. [Quick start: five tasks](#2-quick-start-five-tasks)
3. [Rubrics in tests](#3-rubrics-in-tests)
4. [Rubrics on agents](#4-rubrics-on-agents)
5. [Adopting rubrics in an application](#5-adopting-rubrics-in-an-application)
6. [Scoring reference](#6-scoring-reference)
7. [Versioning reference](#7-versioning-reference)
8. [Evaluators](#8-evaluators)
9. [Consensus and agreement](#9-consensus-and-agreement)
10. [Who can see what](#10-who-can-see-what)
11. [Extending](#11-extending)
12. [CLI and screens](#12-cli-and-screens)
13. [Troubleshooting](#13-troubleshooting)
14. [Worked examples](#14-worked-examples)

---

## 1. The model in one picture

Two halves never mix. The **definition** (rubric → version → criteria) is authored, published once, and then
frozen. The **evaluations** (evaluation → scores) point at exactly one published version, so a score always
means what the version said when it was given.

```mermaid
graph TB
    subgraph DEF["Definition — authored, then frozen at publish"]
        direction LR
        R["Rubric<br/><i>name · category</i>"] --> V["Rubric Version<br/><i>1.2.0 · Draft / Published / Retired<br/>threshold · N/A policy · hashes</i>"]
        V --> C["Criteria tree<br/><i>groups and leaves<br/>weight · gate · rollup</i>"]
        C --> A["Level anchors<br/><i>what 'Partial' means here</i>"]
        V --> B["Bands<br/><i>display labels only</i>"]
        S["Scale<br/><i>levels or numeric</i>"] -.-> C
    end
    subgraph EVAL["Evaluations — immutable once submitted"]
        direction LR
        ANY["Any MJ record<br/><i>test run · agent run · application · bid</i>"]
        E["Evaluation<br/><i>evaluator · score · outcome · passed</i>"] --> ES["Score per criterion<br/><i>level or raw value · rationale · evidence</i>"]
        E -- "subject" --> ANY
    end
    E == "pins exactly one version" ==> V
    style V fill:#0891b2,color:#fff
    style E fill:#7c3aed,color:#fff
```

### The entities

```mermaid
erDiagram
    "MJ: Rubric Categories" ||--o{ "MJ: Rubrics" : groups
    "MJ: Rubrics" ||--o{ "MJ: Rubric Versions" : "has (one Draft at most)"
    "MJ: Rubric Versions" ||--o{ "MJ: Rubric Criteria" : "tree via ParentID"
    "MJ: Rubric Versions" ||--o{ "MJ: Rubric Bands" : "display ranges"
    "MJ: Rubric Scales" ||--o{ "MJ: Rubric Scale Levels" : levels
    "MJ: Rubric Scales" ||--o{ "MJ: Rubric Criteria" : "answered on"
    "MJ: Rubric Criteria" ||--o{ "MJ: Rubric Criterion Levels" : "anchor text per level"
    "MJ: Rubric Versions" ||--o{ "MJ: Rubric Evaluations" : "pinned by"
    "MJ: Rubric Evaluations" ||--o{ "MJ: Rubric Evaluation Scores" : "one per criterion"
    "MJ: Rubric Evaluations" |o--o| "MJ: Rubric Evaluations" : supersedes
    "MJ: Rubrics" ||--o{ "MJ: AI Agent Rubrics" : "bound to agents"
    "MJ: Rubrics" |o--o{ "MJ: Tests" : "Test.RubricID"
    "MJ: Rubrics" |o--o{ "MJ: Test Suites" : "TestSuite.RubricID"
```

| Entity | Holds |
|---|---|
| `MJ: Rubrics` | The stable identity. Name is unique across the install. |
| `MJ: Rubric Versions` | Draft, Published, or Retired. Version numbers, pass threshold, minimum completeness, N/A policy, display range, `ContentHash`, `ScoringHash`, change details. |
| `MJ: Rubric Criteria` | The tree. `NodeType` Group or Criterion, a permanent `Key`, weight, gate, rollup, N/A policy, `EvaluatorConfig`. |
| `MJ: Rubric Criterion Levels` | Anchor text: what a given scale level means for this criterion. |
| `MJ: Rubric Bands` | Display labels over the 0..1 score, for example "Strong" from 0.8. Never part of pass/fail. |
| `MJ: Rubric Scales` / `Scale Levels` | Reusable answer scales: Meets / Partial / Miss, Binary, 1–5 Likert, Compliance, Percentage. |
| `MJ: Rubric Evaluations` | One scoring of one subject: who scored it, the stored score, outcome, completeness, and the cohort columns on its view. |
| `MJ: Rubric Evaluation Scores` | One row per criterion: the answer, rationale, evidence, and the computed contribution. |
| `MJ: AI Agent Rubrics` | Binds a rubric to an agent with a purpose: Evaluation, SelfCheck, or ProductionSampling. |

### The two lifecycles

```mermaid
stateDiagram-v2
    direction LR
    state "Rubric Version" as RV {
        [*] --> Draft
        Draft --> Published: publish computes the number,<br/>the bump, and both hashes
        Published --> Retired
        Retired --> Published
    }
```

```mermaid
stateDiagram-v2
    direction LR
    state "Rubric Evaluation" as EV {
        [*] --> Draft: answers saved as you go
        Draft --> Submitted: submit runs RubricScoring
        Draft --> Failed: evaluator threw
        Submitted --> Superseded: a newer evaluation replaces it
        Submitted --> Withdrawn
    }
```

Published versions and submitted evaluations are **frozen in the database**. Triggers refuse edits with errors
51101–51110 ([Troubleshooting](#13-troubleshooting)). To change a published rubric, publish a new version.
To correct a submitted evaluation, submit a new one that supersedes it.

---

## 2. Quick start: five tasks

### Task 1 — Author and publish a rubric in Explorer

1. Open the **Rubrics** application and create a rubric. It starts with an empty Draft version.
2. In the builder, add criteria. Each leaf needs a scale. Set weights; the builder shows each node's share of
   its group. Mark a criterion as a **gate** when falling short must fail the whole evaluation, and give the
   gate a minimum.
3. Write the anchors: for each level, what that level means for this criterion. Anchors are what an LLM judge
   and a human reviewer both read, so they decide how consistent the scores are.
4. Set the version's pass threshold and, if partial scoring should not count, a minimum completeness.
5. Try sample answers in the preview. It runs the same `RubricScoring` the server runs on submit.
6. **Publish.** The dialog shows the computed bump and the reason for each change. The server assigns the
   number, computes the hashes, and freezes the tree.

To change it later, choose **Start new draft**. The draft is cloned from the latest published version, and
publishing it assigns the next number from how scores would change ([Versioning](#7-versioning-reference)).

### Task 2 — Ship a rubric as metadata

Rubrics that ship with an application or with MJ core live under `metadata/`. A published version cannot be
inserted directly, because its criteria would be refused by the immutability triggers. So a shipped rubric is
authored as a **Draft**, and a separate directory flips it to Published through the server's publish path.

```mermaid
sequenceDiagram
    autonumber
    participant Sync as mj sync push
    participant DB as Database
    participant Srv as Version entity server
    Sync->>DB: rubric-scales/ — scales and levels
    Sync->>DB: rubrics/ — rubric + Draft version + criteria + anchors
    Note over DB: Drafts accept child rows
    Sync->>Srv: rubric-publications/ — { ID, Status: "Published" }
    Srv->>Srv: load the draft tree, validate, diff, hash, number it 1.0.0
    Srv->>DB: save as Published (now frozen)
    Sync->>DB: agents/ — AI Agent Rubrics links by @lookup:MJ: Rubrics.Name
```

Rules that keep this working:

- **Order.** `rubric-scales`, `rubrics`, and `rubric-publications` are pushed before `agents` and `tests` in
  `metadata/.mj-sync.json`. Anything that looks up a rubric by name must come after them.
- **Author Drafts.** Write `"Status": "Draft"` (or omit it) on the version. Never write `MajorVersion`,
  `MinorVersion`, `PatchVersion`, `AppliedBump`, `ContentHash`, `ScoringHash`, or `PublishedAt`. Publish
  computes them.
- **Publish by record.** Each publication record is just `{ "fields": { "Status": "Published" },
  "primaryKey": { "ID": "<version id>" } }`. Pushing it again is a no-op.
- **Keys are permanent.** A criterion `Key` is its identity across versions, consensus, and drift. Use short
  semantic keys (`answers-the-question`), at most 100 characters. Renaming one is a Major change.
- **Fixed IDs.** Every record gets a `uuidgen` primary key, and no `sync` block (see
  [`metadata/CLAUDE.md`](../metadata/CLAUDE.md)).

See `metadata/rubrics/.research-answer.json` and `metadata/rubric-publications/` for the shipped pattern.

### Task 3 — Score a record from server code

`ProviderRubricEngine` returns an engine wired to a provider and a user. `EvaluateRecord` resolves the latest
published version, loads the subject, runs the evaluator, and submits the result in one call.

```typescript
import { ProviderRubricEngine } from '@memberjunction/rubrics';

const engine = ProviderRubricEngine(provider, contextUser);

const result = await engine.EvaluateRecord({
    rubricName: 'Vendor proposal',
    subjectEntityName: 'Vendor Proposals',   // any entity; the record is loaded by ID
    subjectRecordId: proposalId,
    contextEntityName: 'Procurement Rounds',  // optional: what the subject was scored for
    contextRecordId: roundId,
    evaluator: 'LLM',                          // any registered evaluator; LLM when omitted (section 8)
    settings: { ModelID: modelId },            // optional: prompt, model, mode, samples (section 8)
});

result.score;        // 0..1, or null when nothing could be scored
result.outcome;      // 'Passed' | 'BelowThreshold' | 'GateFailed' | 'Incomplete' | ...
result.displayScore; // the score on the version's display range, for example 0..100
result.criteria;     // [{ key, normalizedScore, rationale }]
result.evaluationId; // the stored MJ: Rubric Evaluations row
```

`EvaluateRecord` throws when the rubric has no published version, when the subject is missing or not readable
by `contextUser`, or when the evaluator fails. A failed run is still stored as a `Failed` evaluation with its
`ErrorMessage`. Pass `versionId` to score against one specific version, and `content` when the subject is
already in memory (self-check does this, because the run's payload is not saved yet). Pass `evaluatorConfig`
instead of `evaluator` and `settings` to use a stored evaluator selection as is, the way sampling and self-check do.

From an agent, a workflow, or a low-code builder, use the **Evaluate Record Against Rubric** action. It takes
the same inputs and returns `EvaluationID`, `Score`, `Outcome`, and `Criteria`.

### Task 4 — Have a person score it

A person answers the same criteria in the scoring form (`mj-rubric-scoring-form`). Digits pick a level,
`N` marks a criterion not applicable, required rationale and evidence block submit, and every change is
emitted so the host can autosave the draft. The test review dialog already hosts it for test runs.

To submit a human score without the UI, call the **Submit Human Rubric** action. It writes the evaluation,
its scores, and the move to Submitted in one transaction:

| Input | |
|---|---|
| `RubricVersionID`, `SubjectEntityID`, `SubjectRecordID` | Required. The version and the record being scored. |
| `ContextEntityID`, `ContextRecordID` | Optional context. |
| `SupersedesEvaluationID` | Your earlier evaluation of the same subject, when this one replaces it. |
| `Answers` | JSON array of `{ CriterionId, ScaleLevelId \| RawValue \| IsNotApplicable, Rationale, Evidence }`. |

### Task 5 — Read the result and the consensus

A single evaluation's numbers are columns on `MJ: Rubric Evaluations`: `NormalizedScore`, `Outcome`,
`Passed`, `GateFailed`, `Completeness`, `BandID`. Per-criterion numbers are on `MJ: Rubric Evaluation
Scores`.

Consensus across evaluators is already on the views, so reading it is a plain `RunView`:

```typescript
import { RunView } from '@memberjunction/core';
import { MJRubricEvaluationEntity } from '@memberjunction/core-entities';
import { EscapeSQLString } from '@memberjunction/global';

const rv = RunView.FromMetadataProvider(provider);
const result = await rv.RunView<MJRubricEvaluationEntity>({
    EntityName: 'MJ: Rubric Evaluations',
    ExtraFilter: `SubjectRecordID='${EscapeSQLString(proposalId)}' AND Status='Submitted'`,
    ResultType: 'simple',
    Fields: ['ID', 'EvaluatorType', 'NormalizedScore', 'CohortMeanScore', 'CohortHumanMeanScore', 'CohortAIMeanScore', 'DeviationFromCohortMean'],
}, contextUser);
```

For a mean, median, or trimmed mean with a spread and sample size, use `engine.ConsensusForSubject(...)` or
the **Get Rubric Consensus** action ([Consensus](#9-consensus-and-agreement)).

---

## 3. Rubrics in tests

A test run can be judged by a rubric. The subject is the test run (`MJ: Test Runs`) and the context is the
test, so every run of the same test lands in one cohort.

### Which rubric judges a run

The first source that names a rubric wins:

```mermaid
graph TD
    A{"--rubric on<br/>mj test run / suite?"} -- yes --> USE["Use it"]
    A -- no --> B{"rubric oracle has<br/>its own rubricId?"}
    B -- yes --> USE
    B -- no --> C{"Test.RubricID?"}
    C -- yes --> USE
    C -- no --> D{"TestSuite.RubricID?<br/><i>walking up ParentID</i>"}
    D -- yes --> USE
    D -- no --> E{"Agent's default<br/>Evaluation rubric?"}
    E -- "yes, and the test has<br/>no llm-judge oracle" --> USE
    E -- no --> NONE["No rubric oracle"]
    USE --> PIN["Pin the Published version<br/>at suite start"]
    style USE fill:#16a34a,color:#fff
    style PIN fill:#0891b2,color:#fff
```

- The published version is **pinned when the suite run starts**, so a publish in the middle of a run does not
  split the suite across two versions. A version you name explicitly (`--rubric "Research answer@1.2.0"`) is
  used as given.
- When a rubric resolves and the test has no `rubric` oracle, the driver adds one. It always gates the test's
  status, and it contributes to the score when `scoringWeights` is absent or already names `rubric`.
- When the rubric comes from the agent's default link, the link's `EvaluatorConfig` (its judge, mode and model)
  is used. An explicit `rubric` oracle's own `evaluator` wins over the link's settings.
- A test that already has an `llm-judge` oracle keeps it, and the agent's default rubric is not added beside
  it. Rubrics named on the run, the test, or the suite are still added.

### Moving inline criteria to a rubric

```bash
mj test promote-criteria <test>
```

copies a test's inline judge criteria onto a new Draft rubric and sets `Test.RubricID`. Review the draft,
publish it, and remove the `llm-judge` oracle so the criteria are not scored twice.

### Checking the judge: calibration

An AI judge is only useful if it agrees with people. The **Rubric Judge Calibration** test type scores a gold
set with the AI evaluator and compares it with the submitted human scores on the same rubric major:

```json
{
  "rubricId": "<rubric id>",
  "goldSet": { "subjectEntity": "MJ: Test Runs", "filter": "TestID='...'" },
  "evaluator": { "EvaluatorName": "Decision", "ModelID": "<model id>" }
}
```

`evaluator` is an evaluator selection, the same shape as a link's `EvaluatorConfig`
([Choosing the evaluator for a link](#choosing-the-evaluator-for-a-link)). The older `{ "type": "LLM" }` still
works, with `type` read as the evaluator name. Calibrate each judge you intend to run: an LLM prompt, a
Decision model, and an agent disagree with people in different ways.

The test score is the overall quadratic-weighted kappa, clamped to 0..1. Agreement needs at least 20 subjects
that both a person and the judge scored; with fewer, the statistic is withheld and the sample size is still
reported. Run it after changing the judge prompt, the model, or a rubric's anchors.

---

## 4. Rubrics on agents

An `MJ: AI Agent Rubrics` row binds a rubric to an agent with one of three purposes:

| Purpose | When it runs | Effect | Shipped |
|---|---|---|---|
| **Evaluation** | When a test or a caller asks for the agent's default rubric | None on the run. It is the agent's default judge in tests | Active on 14 agents |
| **SelfCheck** | Every time the agent is about to return Success | Can send a Loop agent back for a retry, or fail it | **Disabled** |
| **ProductionSampling** | Nightly, on a sample of completed runs | Stores evaluations for drift dashboards. Never touches the run | Links Active, **job Disabled** |

### Self-check

```mermaid
sequenceDiagram
    autonumber
    participant Agent as Loop agent
    participant Hook as Self-check hook
    participant Eng as RubricEngine
    Agent->>Hook: about to return Success
    Hook->>Hook: Active SelfCheck link for this agent? (cached)
    alt no link
        Hook-->>Agent: continue — Success
    else link
        Hook->>Eng: EvaluateRecord(content = the answer in memory)
        Eng-->>Hook: passed?, failed criteria and rationales
        alt passed
            Hook-->>Agent: Success
        else failed and retries remain
            Hook-->>Agent: Retry, with the failed criteria as feedback
        else failed, no retries left
            Hook-->>Agent: Failed
        end
    end
    Note over Hook: Each check is recorded as a Validation step on the run.<br/>A self-check that cannot run (an error, missing permission)<br/>is recorded and the run continues.
```

- The judge reads the candidate answer **in memory**. The run's `FinalPayload` is not saved until later.
- `MaxSelfCheckAttempts` is the number of **retries**. 1 means one retry, then fail.
- Flow agents never retry or fail on a self-check. The result is recorded only.
- Every agent success exit goes through the hook, including `finishIf`, sub-agent `terminateAfter`, and
  client-tool `taskComplete`.

**Cost.** Each self-check is one judge LLM call on every successful run, and a retry is another full agent
turn. That is why the five shipped SelfCheck links (Research Agent, ActionSmith, SkillSmith, Infographic,
Duplicate Resolution) are `Disabled`. To turn one on, set the link's `Status` to `Active`, and first confirm
the **Core agent rubrics** test suite passes for that agent.

### Production sampling and drift

The **Evaluate Sampled Agent Runs** scheduled job (daily at 02:00 UTC) walks each Active ProductionSampling
link, keeps a deterministic sample of that agent's completed runs from the last 7 days at the link's `SampleRate`,
skips runs already evaluated for that rubric, and evaluates at most 100 runs per job run with the link's
evaluator. Samples are chosen by hashing the run ID, so the same run is
always in or out, on any database platform. The drift view in the AI dashboard plots per-criterion period
means over Submitted, non-Self evaluations, keyed by criterion `Key` so a new version does not reset a series.

The job ships **Disabled**. Enable it in Scheduled Jobs once you have chosen sample rates you are prepared to
pay for.

### Choosing the evaluator for a link

`EvaluatorConfig` on the link (type `IRubricEvaluatorSelection`) picks the evaluator and its settings:

```json
{ "EvaluatorType": "AIPrompt", "PromptName": "Rubric Judge - Research Agent" }
{ "EvaluatorType": "AIPrompt", "EvaluatorName": "Decision", "ModelID": "<Jev's AI Model ID>" }
```

`EvaluatorName` names any registered evaluator ([section 8](#8-evaluators)), including one your application
registered. Without it, `EvaluatorType` picks a built-in: `AIPrompt` is LLM, `Agent` is Agent, `Deterministic`
is Deterministic. An empty config is LLM SinglePass with the default judge. `PromptID` / `PromptName` (the LLM
judge, or the Decision prompt), `SystemPromptID` / `SystemPromptName`, `CriterionPromptID` /
`CriterionPromptName`, `ModelID`, `ModelSelection`, `AgentID`, `Mode`, `Samples`, and `Extensions` are passed
to the evaluator as its settings. Every core agent's link names a judge written for it; see
[The shipped judges](#the-shipped-judges). Agent eval tests, self-check and the sampling job all honor the whole selection.
`PassThreshold` on the link overrides the version's threshold for that purpose.

---

## 5. Adopting rubrics in an application

Rubrics are built for applications to score their own records: a submissions app scoring abstracts, an ATS
scoring candidates, a procurement app scoring bids. The pattern:

```mermaid
graph LR
    subgraph APP["Your application"]
        REC["Your record<br/><i>Application, Bid, Submission</i>"]
        CTX["Your context<br/><i>Job, Round, Program</i>"]
        UI["Your screens"]
    end
    subgraph CORE["MJ core rubrics"]
        RUB["Published rubric"]
        EV["Evaluations<br/><i>subject = your record<br/>context = your context</i>"]
        W["ng-rubrics widgets"]
    end
    REC -. "SubjectEntityID + SubjectRecordID" .-> EV
    CTX -. "ContextEntityID + ContextRecordID" .-> EV
    EV --> RUB
    UI --> W
    W -- "emits answers" --> UI
    style EV fill:#7c3aed,color:#fff
```

1. **Point evaluations at your records; don't copy scores into your tables.** The subject is any entity and
   record. Your table links to the evaluation (for example a nullable `RubricEvaluationID`) or reads it by
   subject. If you do roll a score up onto your record for list views, copy `NormalizedScore` and `Passed`,
   and treat the evaluation as the source of truth.
2. **Use context for "scored for what".** The same applicant can be scored for two jobs. Context keeps those
   cohorts apart, and consensus compares only evaluations with the same subject, context, rubric, and major.
3. **Use scales that mean something to your reviewers**, and write anchors per criterion. The score is always
   normalized to 0..1; `ScoreDisplayMin` and `ScoreDisplayMax` put it back on your users' range (0..100,
   1..5).
4. **Pass/fail lives in the rubric**, not your code. Use the pass threshold for "good enough" and gates for
   knockouts. Bands are labels only.
5. **App-specific settings go in `EvaluatorConfig.Extensions`** under your app's name, for example
   `{ "Extensions": { "Caliber": { "AppliesTo": ["Interview"] } } }`. Core stores and hashes them (a change is
   a Major bump) and never reads them.
6. **Ship your rubrics as metadata** ([Task 2](#task-2--ship-a-rubric-as-metadata)), with names prefixed by
   your app. Rubric names are unique across the whole install.
7. **Give your subject a content provider** if its useful text is not in plain columns
   ([Extending](#11-extending)). Without one, the judge sees the readable columns of the record.

Things to know:

- The engine loads a subject record by a single-column primary key named `ID`.
- A deterministic rule needs no LLM and is free; use it for checks a rule can decide (a required document is
  present, a value is in range).
- Retiring a version does not invalidate its evaluations. They stay valid for that major; consensus never
  mixes majors.

---

## 6. Scoring reference

Leaves on a levels scale score the level's normalized value, a number from 0 to 1. Numeric leaves map the raw
value across the scale's min and max, and invert that when lower is better. A Percentage scale is a Numeric
scale named Percentage, or any Numeric scale whose minimum is 0 and maximum is 100; the answer is the raw
number, not a level.

A group combines its included children by a weighted mean, a minimum, or a maximum. An advisory criterion is
shown and is not part of the rollup. A gate that falls short fails the evaluation even when the weighted score
is high.

```mermaid
graph TD
    O["Overall 0.75<br/><i>weighted mean</i>"]
    S["Security 0.5<br/><i>group · weight 1 · Minimum</i>"]
    D["Delivery 1.0<br/><i>group · weight 1</i>"]
    E1["Encryption: Compliant → 1.0"]
    E2["Access control: Partial → 0.5<br/><b>gate, minimum 0.5 — met</b>"]
    T["Schedule: Meets → 1.0"]
    N["Notes quality<br/><i>advisory — shown, not counted</i>"]
    O --> S
    O --> D
    O -.-> N
    S --> E1
    S --> E2
    D --> T
```

Not-applicable answers follow the criterion's policy, or the version's policy when the criterion does not set
one:

| Policy | Effect |
|---|---|
| **Exclude and redistribute** | Drops the leaf and spreads its weight across its siblings. The default. |
| **Count as zero** | Keeps the leaf in the rollup at 0. |
| **Fail the evaluation** | Makes the outcome NotApplicableFailure. |
| **Not allowed** | Refuses the answer. |

The outcome is the first match:

| Order | Outcome | When |
|---|---|---|
| 1 | Incomplete | Completeness is below the version's minimum. Both sides are rounded to six places before the comparison. |
| 2 | NotApplicableFailure | A not-applicable answer used Fail the evaluation. |
| 3 | Incomplete | The overall score is null: an unanswered applicable leaf was dropped and nothing included remains to roll up. |
| 4 | GateFailed | A non-advisory gate is unanswered, or its rounded score is below that criterion's gate minimum (0 when not set). |
| 5 | Passed or BelowThreshold | A pass threshold is set. Passed when the rounded score is at least the rounded threshold. |
| 6 | Scored | No pass threshold, and none of the earlier rows matched. |

`Passed` is true only for the Passed outcome. It is null for Scored, and for Incomplete when the version has no
threshold and no gate. Every other outcome sets it false.

Scores are rounded to six decimal places after the arithmetic, and threshold, gate, band, and completeness
comparisons use that rounding. The same answers on the same `ScoringHash` produce the same score. The widgets
do not implement a second copy of this math: the author preview and the server submit both call
`RubricScoring`.

---

## 7. Versioning reference

A rubric has at most one draft. Publishing is what assigns numbers, and the bump says whether scores stay
comparable:

| Bump | When | Scores across the bump |
|---|---|---|
| Initial | The first publish. It becomes 1.0.0. | — |
| **Major** | A non-advisory node was added or removed. Or a non-advisory node's parent, type, weight, scale, gate, gate minimum, N/A policy, rollup, or evaluator config changed. Or `IsAdvisory` flipped. Or the version's N/A policy changed. Or a scale used by a non-advisory node changed its type, range, step, or direction, or a level's value changed. A node that is advisory on both sides does not make those edits Major. | **Not comparable.** Consensus and drift never mix majors. |
| Minor | The pass threshold or minimum completeness changed. A band was added or removed, or its range or tone changed. An advisory node was added, removed, or had a scoring field changed. Evidence or rationale became required or stopped being required. | Comparable; the verdict rules changed. |
| Patch | Wording and order only: names, descriptions, guidance, instructions, anchor text, level labels and descriptions, band labels and wording, sequence, and the display range. A band is matched by label, so a renamed label on the same band is Patch. | Identical scoring. |

- Parent links are compared by the parent's key, so a clone's new IDs are not a change.
- The highest change wins. The author may request a higher bump, never a lower one.
- A draft identical to its base cannot be published. The base is the highest Published or Retired version.
- `ScoringHash` covers only what changes a score. Two versions with the same `ScoringHash` produce the same
  scores for the same answers; a Major bump always changes it.

---

## 8. Evaluators

An evaluator turns a subject into one answer per criterion. It never computes the score: it hands its answers
to `RubricScoring`, so every evaluator scores the same way. Evaluators are plugins. Each one registers under
`BaseRubricEvaluator` by name, and the engine creates the one a call names through the class factory.

```mermaid
graph LR
    CALLER["A call, a link's EvaluatorConfig,<br/>a test, the action, or the CLI"] --> RES["ResolveRubricEvaluatorSelection<br/><i>name + settings</i>"]
    RES --> CF["ClassFactory<br/><i>BaseRubricEvaluator, name</i>"]
    CF --> EV["Evaluator.EvaluateRubric<br/><i>version · content · settings · services</i>"]
    EV --> SC["RubricScoring<br/><i>the only scorer</i>"]
    SC --> ROW["RubricEvaluation<br/><i>EvaluatorType · EvaluatorName · run IDs · Metadata</i>"]
```

### The built-in evaluators

| Name | Stored as | What runs | Settings it reads |
|---|---|---|---|
| `LLM` (default) | AIPrompt | The **Rubric Evaluator** prompt with a **judge** prompt composed into it ([below](#how-the-llm-evaluator-builds-its-prompt)). `SinglePass` asks once for the whole rubric; `PerCriterion` asks once per leaf | `PromptID` or `PromptName` (the judge), `SystemPromptID`/`Name`, `CriterionPromptID`/`Name`, `ModelID`, `ModelSelection`, `Mode`, `Samples` (at most 9) |
| `Decision` | AIPrompt | Typed Score questions on a Decision-type model, all leaves in one call. **Default Decision** binds Jev and LLM Decision. Each question is the criterion as **Rubric Criterion** renders it | `PromptID` or `PromptName` (the decision prompt), `CriterionPromptID`/`Name`, `ModelID` |
| `Agent` | Agent | The **Rubric Evaluation Agent**, a Loop agent that can read the rubric and the subject | `AgentID` |
| `Deterministic` | Deterministic | Each criterion's `EvaluatorConfig.Deterministic` rule. No model call | none |
| `Human` | Human | A person, through the scoring form or the Submit Human Rubric action. The engine never runs it | — |

`AI` is accepted as the old name for `Agent`, and `AIPrompt` as a name for `LLM`. Names are case-insensitive.

Every LLM and Decision call goes through MJ's prompt system, so the prompt's model bindings, failover,
credentials, and cost tracking all apply, and the prompt run is linked from the evaluation's `AIPromptRunID`.
An Agent evaluation links its agent run from `AIAgentRunID`. Both columns point at the run that **produced** the
evaluation, never at the subject.

**LLM or Decision?** A Decision model answers with a calibrated probability for each level, which is cheaper
and steadier than asking a chat model to emit JSON, and the chosen level's probability becomes the answer's
`Confidence`. It returns no rationale and no quotes, so it suits rubrics whose levels are well anchored and whose
criteria do not require evidence. The Decision evaluator refuses a rubric with an evidence-required criterion
before calling anything, writes each rationale as the chosen level and its probability, and leaves a criterion
on a numeric scale unanswered (it is listed in the run metadata as `UnaskedCriteria`). Use LLM when you need
written rationale, quotes, or numeric scales.

The LLM evaluator sends the rubric as the system message and the subject as a separate, delimited user
message, so subject text cannot rewrite the instructions. Its prompts are metadata you can swap, see
[How the LLM evaluator builds its prompt](#how-the-llm-evaluator-builds-its-prompt). A quote cited as evidence
that does not appear in the subject text is dropped, so stored quotes are always real.

A deterministic rule reads a dotted path in the subject content's `data` and maps the comparison to a level:

```json
{
  "Deterministic": {
    "Path": "SecurityCertification",
    "Operator": "in",
    "Values": ["SOC2", "ISO27001"],
    "LevelWhenTrue": "Compliant",
    "LevelWhenFalse": "Non-compliant",
    "NotApplicableWhenMissing": false
  }
}
```

Operators: `equals`, `notEquals`, `in`, `notIn`, `contains`, `exists`, `between`, `gte`, `lte`, `matches`.

### How the LLM evaluator builds its prompt

The LLM evaluator writes no prompt text. The prompts are ordinary rows in `MJ: AI Prompts`, shipped from
`/metadata/prompts` with their templates in `/metadata/prompts/templates/rubrics`. The engine builds **template
data** from the rubric, and the prompts render it. Change a template, push the metadata, and the next
evaluation uses it. No code change, no rebuild.

Three prompts take part, and each has one job:

| Prompt | Role | Shipped as | Swap it with |
|---|---|---|---|
| **Evaluator** (the parent) | The frame: the untrusted-subject rule, the criteria, the scoring rules, and the **JSON reply contract** the engine parses. Has a `{{ judgePrompt }}` slot | `Rubric Evaluator` | `SystemPromptID` / `SystemPromptName` |
| **Judge** (the child) | *How to judge* this kind of subject: what good looks like, what to be strict about, what counts as a serious miss | `Rubric Evaluator - Default Judge`, plus one `Rubric Judge - <Agent>` per core agent | `PromptID` / `PromptName` |
| **Criterion** | Renders one criterion as text: name, key, gate, description, guidance, hints, quote rule, N/A policy, and its levels with their anchors | `Rubric Criterion` | `CriterionPromptID` / `CriterionPromptName` |

This is the same **child-prompt composition** a loop agent uses for its system prompt and agent prompt: the
runner renders the judge, places it in the evaluator's `judgePrompt` slot, and sends **one** system message in
**one** model call, which writes **one** `MJ: AI Prompt Runs` row. The subject never passes through a template.
It travels as its own user message, inside a `<rubric-subject NONCE>` block whose random nonce the subject does
not contain, so subject text cannot close the block or rewrite the instructions.

```mermaid
flowchart LR
    subgraph DATA["Template data the engine builds"]
        RV["Rubric<br/><i>Instructions · N/A policy · PassThreshold</i>"]
        CR["Criteria[]<br/><i>Key · Name · Levels · Anchors · Hints …</i>"]
        MO["Mode<br/><i>SinglePass · PerCriterion</i>"]
    end
    CP["Rubric Criterion<br/><i>rendered once per criterion</i>"]
    JP["Judge prompt<br/><i>Rubric Judge - Sage, or your own</i>"]
    EP["Rubric Evaluator<br/><i>parent · owns the reply JSON</i>"]
    SYS["System message"]
    SUB["User message<br/><i>&lt;rubric-subject NONCE&gt; … &lt;/rubric-subject NONCE&gt;</i>"]
    LLM(("Model<br/><i>one call</i>"))
    PARSE["Engine parses the JSON<br/><i>levels by label · quotes checked</i>"]
    SC["RubricScoring"]

    CR --> CP -->|"Criterion.Text"| EP
    RV --> EP
    MO --> EP
    DATA --> JP -->|"{{ judgePrompt }}"| EP
    EP --> SYS --> LLM
    SUB --> LLM
    LLM --> PARSE --> SC
```

Here is one SinglePass evaluation, call by call. The criterion texts are rendered first, with no model call,
so the Decision evaluator asks exactly the same criterion text an LLM evaluation does.

```mermaid
sequenceDiagram
    autonumber
    participant Eng as LLM evaluator
    participant PS as ProviderPromptService
    participant Run as AIPromptRunner
    participant M as Model
    Eng->>PS: RenderCriteria(Rubric Criterion, one item per leaf)
    PS->>Run: RenderChildPromptTemplates(criterion0 … criterionN)
    Run-->>PS: rendered text per criterion
    PS-->>Eng: Criteria[i].Text
    Eng->>PS: Run(Prompt = Rubric Evaluator, Judge = Rubric Judge - Sage, Data, Subject)
    PS->>Run: ExecutePrompt(parent, childPrompts = [judge → judgePrompt], conversationMessages = [subject])
    Run->>Run: render judge, merge into the parent, pick the model
    Run->>M: system = evaluator + judge + criteria · user = subject
    M-->>Run: {"decisions": [...]}
    Run-->>PS: reply · prompt run ID · cost
    PS-->>Eng: Text, PromptRunID
    Eng->>Eng: parse, drop unknown keys and invented quotes, RubricScoring
```

`PerCriterion` repeats step 5 once per leaf with `Criteria` holding only that leaf, and the evaluator template
asks for `{"chosen", "probabilities", …}` instead of `{"decisions": [...]}`. `Samples` repeats the whole call and
keeps each criterion's median level.

#### Which prompt to change

```mermaid
flowchart TD
    Q{"What do you want<br/>to change?"}
    Q -->|"What good looks like<br/>for this agent or record type"| J["Write a judge prompt<br/>and name it in PromptName"]
    Q -->|"How each criterion<br/>is described to the model"| C["Swap the criterion prompt:<br/>CriterionPromptName"]
    Q -->|"The frame, the rules,<br/>or the reply shape"| S["Swap the evaluator prompt:<br/>SystemPromptName<br/><b>must return the same JSON</b>"]
    Q -->|"Which model runs it"| MD["ModelID pins one.<br/>ModelSelection = Judge lets the<br/>judge's bindings choose"]
    Q -->|"The criteria themselves"| R["Edit the rubric:<br/>a new version, not a prompt"]
```

Almost every customization is a **judge**. The evaluator prompt is a contract with the engine's parser, so
replace it only with a prompt that returns the same JSON. The criterion prompt is shared with the Decision
evaluator; change it when you want every rubric's criteria worded differently.

#### Writing a judge

A judge is a plain prompt with a template and nothing else: no reply format (the evaluator owns that), no
criteria list (the evaluator renders them), no subject (it arrives separately). It describes the subject and
says how to weigh it. The shipped Sage judge is a good model:

```markdown
# Judge: Sage

You are reviewing a turn by **Sage**, the ambient assistant present in every MemberJunction conversation.
The subject is that turn: the person's message, Sage's reply, and any agents or actions it invoked.

- **Right route.** A simple question about MJ should be answered directly. Work that needs a specialist
  should go to that agent, not be improvised.
- **Accurate about MJ.** Names of apps, screens, and features must be real. Inventing one is a serious miss.
- **Says what happened.** When Sage delegated or ran something, the reply says what came back, failures included.
```

Ship it as metadata beside the others:

```json
{
  "fields": {
    "Name": "Rubric Judge - Acme Intake Agent",
    "Description": "How to judge an Acme intake summary.",
    "TemplateText": "@file:templates/rubrics/judges/acme-intake.template.md",
    "CategoryID": "@lookup:MJ: AI Prompt Categories.Name=MJ: System",
    "TypeID": "@lookup:MJ: AI Prompt Types.Name=Chat",
    "Status": "Active",
    "ResponseFormat": "Any"
  },
  "primaryKey": { "ID": "<uuidgen, uppercase>" }
}
```

and name it on the agent's rubric link:

```json
{ "EvaluatorType": "AIPrompt", "PromptName": "Rubric Judge - Acme Intake Agent" }
```

A judge can read the same template data as the evaluator, so it can branch on the rubric
(`{% if Mode == 'PerCriterion' %}`) or mention the subject's entity (`{{ Subject.EntityName }}`). Most don't
need to. Two rules for any rubric template:

- **Use `| safe` on authored text.** MJ renders templates with Nunjucks autoescape on. Without `| safe`, a
  criterion named *Reader's clarity* reaches the model as `Reader&#39;s clarity`.
- **Wrap literal JSON in `{% raw %}…{% endraw %}`.** Otherwise `{{` inside an example is read as a variable.

#### The template data

Every rubric template renders from the same data, built in
[`promptData.ts`](../packages/Rubrics/Engine/src/promptData.ts). Field names are the template variable names.

| Variable | Evaluator & judge | Criterion | What it holds |
|---|---|---|---|
| `Rubric.Instructions` | ✓ | ✓ | The version's instructions, or null |
| `Rubric.NotApplicablePolicy` | ✓ | ✓ | The version's default N/A policy |
| `Rubric.PassThreshold` | ✓ | ✓ | The version's pass threshold, or null |
| `Mode` | ✓ | | `SinglePass` or `PerCriterion` |
| `Criteria[]` | ✓ | | The leaves being asked, in tree order. One leaf in PerCriterion |
| `Criterion` | | ✓ | The one leaf being rendered |
| `Subject.EntityName`, `Subject.RecordID` | ✓ | | Which record is scored. Its **content** is never in the data |
| `judgePrompt` | evaluator only | | The rendered judge |

Each criterion (an item of `Criteria`, or `Criterion`) carries:

| Field | Meaning |
|---|---|
| `Key`, `Name` | The permanent key the reply must use, and the display name |
| `Description`, `Guidance` | The author's description and guidance, or null |
| `Hints` | `EvaluatorConfig.AI.Hints` on the criterion, or null |
| `RequireQuote` | `EvaluatorConfig.AI.RequireQuote`, or the criterion requires evidence |
| `EvidenceRequired`, `RationaleRequired`, `IsGate` | The criterion's flags |
| `NotApplicablePolicy` | The effective policy: the criterion's own, else the version's |
| `Scale` | `{ Type, Min, Max, Step, HigherIsBetter }`, or null |
| `Levels[]` | `{ Label, NormalizedValue, Anchor }`, lowest first. Empty for a numeric scale |
| `Text` | The criterion as the criterion prompt rendered it. Set before the evaluator renders |

#### Previewing the composed prompt

`ProviderPromptService(provider, user).Preview({ Prompt, Judge, Data })` renders the evaluator with its judge
exactly as `Run` would and returns the system message, with no model call. Use it to check a new judge before
you name it on a link, or in a test:

```typescript
import { BuildCriteriaPromptData, PromptData, ProviderPromptService, RenderCriteriaText, RUBRIC_CRITERION_PROMPT, RUBRIC_EVALUATOR_PROMPT } from '@memberjunction/rubrics';

const prompts = ProviderPromptService(provider, contextUser);
const version = await engine.GetRubric({ rubricName: 'Research answer' });
const criteria = await RenderCriteriaText(version, prompts, { Name: RUBRIC_CRITERION_PROMPT });
const system = await prompts.Preview({
    Prompt: { Name: RUBRIC_EVALUATOR_PROMPT },
    Judge: { Name: 'Rubric Judge - Database Research Agent' },
    Data: PromptData(version, 'SinglePass', criteria, { entityName: 'MJ: AI Agent Runs', recordId: runId }),
});
```

#### Choosing the model

By default the **evaluator** prompt's model bindings choose the model, so every LLM evaluation runs on the
models bound to *Rubric Evaluator*. Two settings change that:

- `ModelID` pins one model for the link, whatever the bindings say.
- `ModelSelection: "Judge"` lets the **judge's** bindings choose instead (the runner's `modelSelectionPrompt`).
  Bind a stronger model to one agent's judge and only that agent's evaluations use it.

The prompt run is linked from the evaluation's `AIPromptRunID`, and its row records the composed prompt, the
model, tokens, and cost, like any other prompt run.

#### The shipped judges

Every core agent's Evaluation rubric link names a judge written for that agent, so the shared rubrics
(*Research answer* is shared by eight agents) are judged with each agent's own idea of a good result.

| Judge | Agents | Rubric |
|---|---|---|
| `Rubric Evaluator - Default Judge` | Any link that names none | — |
| `Rubric Judge - Research Agent` | Research Agent | Research answer |
| `Rubric Judge - Database Research Agent` | Database Research Agent | Research answer |
| `Rubric Judge - Web Research Agent` | Web Research Agent | Research answer |
| `Rubric Judge - File Research Agent` | File Research Agent | Research answer |
| `Rubric Judge - Knowledge Base Research Agent` | Knowledge Base Research Agent | Research answer |
| `Rubric Judge - Research Report Writer` | Research Report Writer | Research answer |
| `Rubric Judge - Sage` | Sage | Assistant reply |
| `Rubric Judge - Query Builder` | Query Builder | Query answer |
| `Rubric Judge - Query Strategist` | Query Strategist | Query answer |
| `Rubric Judge - ActionSmith` | ActionSmith | Catalog contract |
| `Rubric Judge - SkillSmith` | SkillSmith | Catalog contract |
| `Rubric Judge - Codesmith Agent` | Codesmith Agent | Generated code |
| `Rubric Judge - Database Designer` | Database Designer | Schema proposal |
| `Rubric Judge - Duplicate Resolution Agent` | Duplicate Resolution Agent | Duplicate decision |
| `Rubric Judge - Infographic Agent` | Infographic Agent | Picture from the data |

The testing framework's `llm-judge` oracle runs through the same prompts: its inline criteria render through
*Rubric Criterion*, and its `judgePrompt` config names the judge (the default judge when omitted).

### What an evaluation records

| Column | Value |
|---|---|
| `EvaluatorType` | The evaluator's type: Human, AIPrompt, Agent, Deterministic, Self, or External |
| `EvaluatorName` | The registered name, for example `Decision` or your own |
| `AIPromptRunID` / `AIAgentRunID` | The run that produced it, when there was exactly one |
| `Metadata.Evaluator` | `Name`, the `Settings` it ran with, and what the evaluator reported: for LLM the `SystemPrompt`, `JudgePrompt` and `CriterionPrompt` it used, every prompt run when there were several, the samples, dropped keys and quotes; for Decision the `Prompt`, `CriterionPrompt`, and `UnaskedCriteria` |

`Self` and `External` are not run by the engine: they are another party's or another system's scores, stored
and shown. **Self is never part of consensus.** A custom evaluator that is neither an AI prompt nor an agent
should report `External`.

## 9. Consensus and agreement

**The cohort** is every Submitted evaluation of the same subject and context, on the same rubric and major
version, except Self. Withdrawn and Superseded evaluations are out.

| Where | What you get |
|---|---|
| `MJ: Rubric Evaluations` view | `CohortMeanScore`, `CohortMinScore`, `CohortMaxScore`, `CohortScoreStdDev`, `CohortEvaluationCount`, `CohortPassedCount`, `CohortHumanMeanScore`, `CohortAIMeanScore`, `SelfAssessmentScore`, and `DeviationFromCohortMean` on every row |
| `MJ: Rubric Evaluation Scores` view | `CriterionCohortMeanScore`, `CriterionCohortHumanMeanScore`, `CriterionCohortAIMeanScore` per criterion |
| `ConsensusForSubject` / Get Rubric Consensus | `Mean` (default), `Median`, or `TrimmedMean` (drops `floor(0.1 × n)` from each end), with `StdDev`, `Range`, and `SampleSize` |

The human columns average Human evaluations. The AI columns average AIPrompt and Agent evaluations.
Deterministic and External are in the overall mean but in neither split. `SelfAssessmentScore` is the
subject's own Self score, shown beside the cohort. `ConsensusForSubject` uses the latest
Published major unless you pass one, and a call without a context matches evaluations whose context is null.

**Agreement** is separate from consensus. Quadratic-weighted Cohen's kappa compares two raters; Krippendorff's
alpha handles two or more. Both are withheld below 20 usable subjects, and the sample size is still returned.

---

## 10. Who can see what

Evaluations are about people's work, so reads are narrow by default:

- **Ordinary users (the UI role)** can create evaluations, and read and update **only their own**: a row-level
  filter matches `EvaluatorUserID` to the current user, on evaluations and on their scores. Nobody in the UI
  role can delete one.
- **The Administer Rubric Evaluations authorization** lifts that filter. Grant it to the roles that run review
  panels, calibration, or reporting.
- **Developer and Integration roles** keep unfiltered access, for agents, jobs, and tests.
- **While a reviewer's own evaluation of a subject is still a Draft**, the evaluation form and the comparison
  matrix hide the cohort columns and peer scores, so a reviewer cannot anchor on others before submitting.

Core has no separate blinding modes. If your process needs something stricter, for example hiding peer scores
until a round closes, express it as entity permissions or row-level filters in your application.

---

## 11. Extending

### Give a subject better content

The judge sees what the content provider for the subject's entity returns: `text`, `data`, and `files`.
Built-in providers cover `MJ: Test Runs`, `MJ: AI Agent Runs`, `MJ: AI Prompt Runs`, and `MJ: Conversations`.
For anything else, the record's readable columns become `data`. Register a provider when your subject's
meaning lives elsewhere, for example in an attached document or child rows:

```typescript
import { RubricContentRegistry } from '@memberjunction/rubrics';

RubricContentRegistry.Instance.Register('Vendor Proposals', record => ({
    text: String(record.ExecutiveSummary ?? ''),
    data: { Price: record.Price, DeliveryWeeks: record.DeliveryWeeks },
}));
```

Register at server startup. A provider receives the row already loaded under the caller's permissions, so it
should not widen what the caller can see.

### App settings on a criterion

Put them in `EvaluatorConfig.Extensions.<YourApp>`. Core never interprets them; your code reads them from the
version snapshot.

### Custom evaluators

Subclass `BaseRubricEvaluator`, register it by name, and every caller can run it: `EvaluateRecord`, a link's
`EvaluatorConfig`, a test's evaluator, the Evaluate Record Against Rubric action, and `mj rubric evaluate`.

```typescript
import { RegisterClass } from '@memberjunction/global';
import { BaseRubricEvaluator, type RubricEvaluatorContext, type RubricEvaluatorRun, type RubricEvaluatorType } from '@memberjunction/rubrics';

@RegisterClass(BaseRubricEvaluator, 'Acme Readability')
export class ReadabilityEvaluator extends BaseRubricEvaluator {
    public get EvaluatorName(): string { return 'Acme Readability'; }
    public get EvaluatorType(): RubricEvaluatorType { return 'External'; }

    public async EvaluateRubric(context: RubricEvaluatorContext): Promise<RubricEvaluatorRun> {
        const target = Number(context.Settings.Extensions?.['Acme Readability'] ?? 60);
        const grade = ReadingEase(context.Content.text ?? '');          // your own logic
        const answers = context.Version.nodes
            .filter(node => node.nodeType === 'Criterion')
            .map(node => ({
                criterionId: node.id,
                scaleLevelId: LevelFor(context.Version, node, grade >= target),
                rationale: `Reading ease ${grade}, target ${target}.`,
                evidence: [],
            }));
        return { ...this.Evaluate(context.Version, answers), metadata: { ReadingEase: grade } };
    }
}
```

- **Score with `this.Evaluate`.** It calls `RubricScoring`, which is what makes your evaluator's scores
  comparable with every other evaluator's.
- **Read settings from `context.Settings`.** Your own go under `Extensions['<your evaluator name>']`, so a link
  can carry them: `{ "EvaluatorName": "Acme Readability", "Extensions": { "Acme Readability": 70 } }`.
- **Use `context.Services` for models.** `Prompts.Run` runs an evaluator prompt with a judge composed into it,
  `Prompts.RenderCriteria` renders criteria through a criterion prompt, `Prompts.Preview` renders without a
  model call, `Decisions` asks Score questions, and `Agent` runs an agent, all through the caller's provider and
  user. Build template data with `BuildCriteriaPromptData` and `PromptData` rather than writing prompt text, so
  your evaluator's prompts stay swappable metadata too. Return the run's ID as `aiPromptRunId` or
  `aiAgentRunId` so the evaluation links to it.
- **Throw to fail.** The engine stores a `Failed` evaluation with your message.
- **The class factory constructs it with no arguments.** Everything a run needs arrives in the context.
- **Replace a built-in** by registering your class under the same name. The highest-priority registration wins.

`ListRubricEvaluators()` reports every registered evaluator with its type and whether the engine can run it.

---

## 12. CLI and screens

None of these commands publish a version.

```bash
mj rubric list
mj rubric show <rubric>[@version]
mj rubric diff <rubric> <version> <version>
mj rubric validate <file>
mj rubric evaluate --rubric <rubric> --entity <name> --record <id> [--evaluator <name>] [--prompt <name>] [--model <id>] [--mode SinglePass|PerCriterion]

mj test run   --rubric <name-or-id>[@version]     # pin a rubric for this run
mj test suite --rubric <name-or-id>[@version]
mj test promote-criteria <test>                   # inline judge criteria → Draft rubric
```

`@memberjunction/ng-rubrics` widgets are presentational: the host loads records and saves what they emit.

| Widget | For |
|---|---|
| `mj-rubric-builder` | Author a draft: criteria, weights and shares, problems that block publish, a live preview |
| `mj-rubric-scoring-form` | Answer a rubric: levels or raw values, N/A, rationale, evidence |
| `mj-rubric-result` | Show one evaluation: score on the display range, band, gates, a bar per criterion |
| `mj-rubric-publish-dialog` | Confirm a publish: the computed bump and each change's reason |
| `mj-rubric-version-diff` | Compare two versions side by side |
| `mj-rubric-comparison-matrix` | Evaluators across, criteria down, disagreement marked |

In Explorer, the **Rubrics** application hosts the catalog, scales, the builder, and the version board. The
evaluation form shows the result and the comparison matrix. The agent form has a **Rubrics** tab for links,
and the Testing dashboards show the rubric result for each run.

---

## 13. Troubleshooting

| Symptom | Cause | What to do |
|---|---|---|
| Error 51101 – 51105 | You changed or deleted a Published or Retired version, its criteria, anchors, or bands | Start a new draft and publish it |
| Error 51106 / 51107 | The scale is used by a published version | Create a new scale. Level descriptions can still be edited |
| Error 51108 – 51110 | You edited or deleted a submitted evaluation or its scores | Submit a new evaluation that supersedes it, or withdraw it |
| `No published version of that rubric.` | The rubric has only a Draft | Publish it, or pass `versionId` |
| `subject not found or not readable` | Wrong ID, or the user cannot read the record | Check the record and the user's permissions |
| A publish is refused as identical | The draft has no change from its base | Make a change, or discard the draft |
| `Lookup failed … 'MJ: Rubrics'` during `mj sync push` | A directory that references rubrics ran before `rubrics` | Keep the rubric directories ahead of `agents` and `tests` in `.mj-sync.json` |
| Outcome `Incomplete` with a high score | Completeness is below the version's minimum | Answer more criteria, or lower the minimum in a new version |
| A user sees only their own evaluations | The UI row filter | Grant the Administer Rubric Evaluations authorization to their role |
| Two evaluations don't show up in one cohort | Different major, context, or one is Self, Draft, Withdrawn, or Superseded | Compare on the same major and context |
| Calibration returns no kappa | Fewer than 20 subjects scored by both a person and the judge | Grow the gold set |
| `The <name> prompt was not found.` from an LLM or Decision evaluation | The link names a judge, evaluator, or criterion prompt this database does not have | Push the prompt metadata (`mj sync push --dir=metadata --include="prompts"`), or fix the name on the link |
| `The criterion prompt rendered N texts for M criteria.` | A swapped criterion prompt failed to render | Preview it; check its template for a Nunjucks error |
| Every answer is dropped as an unknown key | A swapped evaluator prompt changed the reply shape or the key wording | Keep the evaluator's JSON contract, or go back to *Rubric Evaluator* and change the judge instead |
| `&#39;` or `&lt;` in a stored prompt run | A template prints authored text without `\| safe` | Add `\| safe` to that variable |

---

## 14. Worked examples

Each example is a small rubric and the evaluations of it. Scores are the `RubricScoring` result for those
answers. The levels scale is Meets = 1, Partial = 0.5, Miss = 0 unless an example names another. The matching
records ship as **Draft** rubrics under `metadata/rubrics/` so you can publish and try them.

### Agent evaluation

| Key | Weight | Scale | Gate |
|---|---|---|---|
| accuracy | 1 | Meets / Partial / Miss | minimum 0.6 |
| sourcing | 1 | Meets / Partial / Miss | no |
| completeness | 1 | Meets / Partial / Miss | no |

Pass threshold 0.6.

| Evaluation | Answers | Score | Outcome |
|---|---|---|---|
| AI judge | accuracy Partial, sourcing Meets, completeness Meets | 0.833333 | GateFailed |
| Human reviewer | all three Meets | 1 | Passed |

The AI judge's weighted score is above the pass threshold, but accuracy is 0.5, under the gate, so the outcome
is GateFailed. The human review clears the gate and passes.

### Peer review

| Key | Weight | Scale |
|---|---|---|
| argument | 1 | Meets / Partial / Miss |
| evidence | 1 | Meets / Partial / Miss |

| Evaluation | Status | Answers | Score | Outcome |
|---|---|---|---|---|
| Reviewer A | Submitted | both Meets | 1 | Scored |
| Reviewer B | Submitted | argument Meets, evidence Partial | 0.75 | Scored |
| Reviewer C | Withdrawn | both Miss | 0 | Scored, then withdrawn |

The cohort mean is the mean of the submitted scores, `(1 + 0.75) / 2 = 0.875`. Reviewer C's 0 is not in it.

### Awards

| Key | Weight | Scale | Gate |
|---|---|---|---|
| eligible | 1 | Met = 1 / Not met = 0 | minimum 1 |
| craft | 2 | Meets / Partial / Miss | no |
| originality | 1 | Meets / Partial / Miss | no |

Pass threshold 0.6.

| Evaluation | Answers | Score | Outcome |
|---|---|---|---|
| Entry north | eligible Met, craft Meets, originality Partial | 0.875 | Passed |
| Entry south | eligible Not met, craft Meets, originality Meets | 0.75 | GateFailed |

South's weighted score is high, but the eligibility gate is 0, so the entry is out. Ranking inside the
category uses the entries that were scored, not the gated-out row's score.

### Procurement

| Key | Parent | Weight | Scale | Policy |
|---|---|---|---|---|
| security | | 1 | group | |
| delivery | | 1 | group | |
| encryption | security | 1 | Compliant = 1 / Partial = 0.5 / Non-compliant = 0 | NotAllowed |
| schedule | delivery | 1 | Meets / Partial / Miss | Exclude and redistribute |

| Evaluation | Evaluator | Answers | Score | Outcome |
|---|---|---|---|---|
| Vendor packet | Self | encryption Compliant, schedule Meets | 1 | Scored |
| Buyer | Human | encryption Partial, schedule Meets | 0.75 | Scored |

The vendor cannot mark encryption not applicable. The Self score is reported beside the cohort and is not in
the cohort mean. The buyer's 0.75 is the cohort mean when it is the only non-Self submitted score. To make
encryption a knockout, mark it a gate with a minimum: a vendor that is Non-compliant then fails outright,
whatever its weighted score.

### Accreditation

| Key | Parent | Weight | Required |
|---|---|---|---|
| standard-1 | | 1 | group |
| records | standard-1 | 1 | evidence required |
| faculty | standard-1 | 1 | evidence required |

Scale for both leaves: Met = 1 / Not met = 0.

| Evaluation | Evaluator | Answers | Score | Outcome |
|---|---|---|---|---|
| Self-study | Self | both Met, each with a file citation | 1 | Scored |
| Visiting team | Human | records Met, faculty Not met, each with a citation | 0.5 | Scored |

An answer with no citation cannot be submitted. The team's 0.5 is the cohort mean; the self-study stays
visible and is not in it.

### Hiring

| Key | Weight | Scale |
|---|---|---|
| structure | 1 | 1 = 0, 2 = 0.25, 3 = 0.5, 4 = 0.75, 5 = 1, each level anchored |
| evidence | 1 | the same 1–5 anchors |

| Evaluation | Evaluator | Answers | Score | Outcome |
|---|---|---|---|---|
| Interviewer | Human | structure 4, evidence 4 | 0.75 | Scored |
| Transcript | AIPrompt | structure 5, evidence 3 | 0.75 | Scored |

Both are on the same major, so the comparison is 0.75 against 0.75, not a mix with an older rubric. The
anchors are what the interviewer reads while choosing a level, and what the AI judge reads too.

### Shipped agent rubrics

Eight rubrics ship as Published 1.0.0 on Meets / Partial / Miss, with pass threshold 0.7 and a gate minimum of
0.6, so Partial fails the gate: **Research answer, Query answer, Generated code, Schema proposal, Catalog
contract, Picture from the data, Duplicate decision, and Assistant reply.** They are the Evaluation rubrics for
the core agents ([Rubrics on agents](#4-rubrics-on-agents)); *Assistant reply* is Sage's. The **Core agent
rubrics** test suite runs one test per orchestrator against them, Sage included.

Each agent's link names its own judge prompt, so the eight agents that share *Research answer* are still judged
by what a good result means for each of them: the Database Research Agent's judge checks that figures came from
queries it ran, the Web Research Agent's that claims trace to pages it cited. The full list is in
[The shipped judges](#the-shipped-judges).