The working record, the stage contract, and the driver and extractor contracts shared by every content
pipeline stage. Nothing here touches a database. For the end-to-end architecture, ContentAutotagging
relationship, and retrieval integration, see the
Content Pipeline, Training, and Retrieval Guide.
One structure passes from stage to stage: an identity, a fixed set of well-known fields that map onto real columns, and an in-memory extension space.
record.Propose('Title', parsed.heading, 6, 'Extract.Html'); // persisted, competes on confidence
record.SetExtension('Extract.Html', 'headingDepth', 2); // in memory, for a later stage this run
record.MarkComplete(); // nothing further needs to happen
Stages read and write the record, while the hydrator and committer (in @memberjunction/content-pipeline)
persist the working record's fields and child items. Stages that perform direct side effects beyond the
working record (durable file copies, tag/attribute rows, embedding/vector store writes, and dependent
deletion marking) suppress those writes when StageContext.IsTest is true so the same stage code can
run live or as a dry-run test.
Confidence, not order. Each well-known field carries an integer confidence and the stage (SetBy) that
set it. Across different stages (or different SetBy sub-sources in the same run), a proposal wins only if
its confidence is strictly higher; when a stage re-executes on a hydrated field it previously wrote, it can
refresh its own prior value at any confidence, and repeated proposals from the exact same SetBy within one
run win at equal-or-higher confidence (>=). The baseline scores live in ConfidenceScale, can be
overridden per source type, per source, or per run under Configuration.Confidence via ResolveConfidence,
and are recorded per field in the FieldConfidence column:
{ "Title": { "Score": 4, "SetBy": "Discover.RSS" } }
Every stage resolves its driver keys (ExtractorKey, SegmenterKey, ClassifierKey, AccessDriverKey)
and confidence scale (Confidence) through one precedence order, from most specific to least specific:
ExtractorKeyOverride on an item)Configuration (StageContext.Configuration, populated from RecordProcessRun.Configuration.Options)ContentSource.Configuration and per-extension ContentSource.Configuration.Extractors)ContentSourceType.Configuration / ContentType.Configuration)ContentFileType.ExtractorKey, capability detection, or the built-in default)A stage extends BasePipelineStage and registers under its name; adding one is writing a class, never
editing a list.
@RegisterClass(BasePipelineStage, 'Extract')
export class ExtractStage extends BasePipelineStage {
public readonly Name = 'Extract';
public readonly Entity: WorkingRecordEntity = 'Content Item';
public readonly StatusField = 'ExtractionStatus';
public async Run(record: WorkingRecord, context: StageContext): Promise<StageOutcome> { ... }
}
Outcome.Complete / Skipped / Fatal / Retry, or throw TransientStageError /
FatalStageError. Any other throw is treated as fatal.context.Signal. It fires when the stage's time budget runs out.Finalize is an optional batch step for work far cheaper in bulk (Embed). It runs when the stage is
last in the run's list.The contract's names (Signal, Log, transient vs fatal) follow MJ's Work Queue handler contract, so a
future Record Set Processing → Work Queue bridge is an adapter, not a change to any stage.
access/BaseAccessDriver.ts): turn a Content Source's Configuration.AccessCredentials
(resolved from MJ: Credentials) and driver configuration into an in-memory AccessArtifact (headers,
query params, cookies, or a token), cached for the run and never persisted on a Content Item.discover/SourceDriver.ts): the request/response types a Content Source Type's
driver (ContentSourceType.DriverClass) implements for Discover (Discover, with incremental Since
filtering, SupportsExhaustiveWalk for orphan sync, custom Columns, and per-item Request recipes)
and Extract (Fetch).extract/BaseContentExtractor.ts): turn bytes into one or more text blocks, registered
by key and chosen by the extractor cascade.tag/BaseContentClassifier.ts): turn extracted text and an active taxonomy into tag
observations and summary proposals, registered by key (ManagedPrompt by default).graph/):
GraphOverlay.types.ts — ContentSourceGraphOverlayConfig, ResolvedGraphOverlayConfig, ResolveGraphOverlayConfig, GraphTypeDescriptor (MJ: Graph Types with IsCanon & Rank), GraphConceptTypeDescriptor (MJ: Graph Concept Types with Rank), DefaultConceptTypeRanks, GetConceptTypeRank, AuthorityTierPrecedence, GetAuthorityTierRank, SemanticGraphType, GraphConceptType, ConceptTaxonomyMode, and GraphGroundingMode.ConceptSeedClustering.ts — deterministic pre-clustering (ClusterContentIntoConceptSeeds, ExtractLeafTopicFromHeading, ScoreConceptNameSimilarity, InferConceptTypeFromSignals) that groups segmented chunks, heading breadcrumbs, and tags across one or more Content Sources by topic and AuthorityTier, resolves ranked GraphConceptType IDs (Policy [10], Metric [20], Process [30], Role [40], Domain [50], Term [60]), aligns with SchemaGroundingTargets (MJ: Entities, MJ: Queries, MJ: Entity Alternate Base Views, MJ: AI Skills, MJ: Actions), and flags numeric/currency threshold conflicts before invoking a synthesizer.BaseConceptSynthesizer.ts — pluggable synthesizer contract (SynthesizeCluster(request)) producing SynthesizedGraphConcepts and SynthesizedGraphEdges (SourcedFrom, Mentions, Broader Than, Describes, Defines, Measures, Implements, Contradicts). See the Canon-First Methodology & Semantic Graphs Guide.extract/): byte signatures and file-type resolution over rules loaded from
MJ: Content File Types, plus the fail-closed object-key template for durable copies.PipelineTuning): operational knobs a run may set under Options.Tuning.