Interface IContentSourceClassificationConfiguration

How chunk text + vectors are stored for this source. Default 'alwaysChunk'.

• 'alwaysChunk' (default): every content item gets at least one ContentItemChunk row holding its text — even items small enough to fit in a single chunk — and ContentItem.VectorRecordID is never set. The ContentItemChunk table is always the single source of truth for vectors.
• 'mixed': items that fit in a single chunk keep their text and vector id on the ContentItem (no chunk row); only larger items are split into ContentItemChunk rows.

Options passed to the cleaning strategy named by CleanerKey. Selector rules are per-source because the right selector is a property of the site's template.

Enable vectorization for this source. Default true

Maximum cumulative cost (USD) the run may incur before pausing. NULL/unset = unlimited.

Maximum number of content items the autotagger may PROCESS (hand to the LLM) per run before the run is paused via the existing CancellationRequested machinery. Does not include items skipped by change-detection — those are free. NULL/unset = unlimited.

Most intuitive "do at most N this run, do the rest next time" knob. When checking budgets after a batch, this is evaluated FIRST (before tag / token / cost caps) because it is the most user-facing and not tied to a specific model's pricing or tokenization.

Pause is graceful — the next invocation re-crawls, change-detection skips the items already tagged in DB, and the remaining items get processed.

Maximum number of new tags the autotagger may auto-create for a single ContentItem. Once reached, further free-text tags from that item are routed to MJ:Tag Suggestions with Reason='MaxItemTagsExceeded' instead of being created. NULL/unset = unlimited.

Maximum number of new tags the autotagger may auto-create across an entire run before the run is paused via the existing CancellationRequested machinery. NULL/unset = unlimited. Pause is graceful — the run resumes from LastProcessedOffset when restarted.

Maximum cumulative LLM tokens (prompt + completion) the run may consume before pausing. Reads from ContentProcessRunDetail.TotalTokensUsed rollup. NULL/unset = unlimited.

Options passed to the segmentation strategy named by SegmenterKey.

Sizing note: TargetTokens should be driven by the shape of your QUERIES, not by the embedding model's context window. The window is an upper bound; a good chunk is about as much content as a good answer, so that a matching chunk is mostly signal.

Whether to share existing tag taxonomy with the LLM during autotagging. Default true

Source-type-specific configuration values. The keys here correspond to the RequiredFields[].Key values defined on the parent ContentSourceType's Configuration.

Examples:

• Entity type: { EntityID: "uuid", EntityDocumentID: "uuid" }
• RSS Feed: { URL: "https://example.com/feed.xml" }
• Cloud Storage: { FileStorageProviderKey: "Azure Blob Storage", PathPrefix: "/documents" }
• Local File System: { Path: "/var/data/documents" }
• Website: { URL: "https://example.com" } — see Website sub-object below for crawl knobs

Lower confidence band (0.0-1.0) that routes a semantic match into the human-in-the-loop MJ:Tag Suggestions queue instead of auto-applying or auto-creating. A score s is routed as: s >= TagMatchThreshold → apply; SuggestThreshold <= s < TagMatchThreshold → enqueue suggestion (Reason='BelowThreshold'); s < SuggestThreshold → fall through to handleNoMatch (governed by TagTaxonomyMode). When unset, defaults to TagMatchThreshold - 0.05 at runtime.

Similarity threshold (0.0-1.0) for matching ContentItemTags to formal Tags. Default 0.9

Root Tag ID for constrained/auto-grow modes — limits taxonomy operations to this subtree

Tag taxonomy matching mode: constrained (only match within subtree), auto-grow (match or create within subtree), free-flow (match or create anywhere)

The MJ entity this source's VECTORS resolve to, for search attribution. Optional; when unset, attribution falls back to the index's Entity Documents exactly as before.

Why this exists. SearchEngine groups results by EntityName and evaluates THAT entity's CanRead/RLS, so a match it cannot name is dropped rather than shown unlabelled. A source whose vectors carry minimal metadata (VectorMetadata.FieldStrategy: 'explicit') has no Entity key to read, and a source populated outside MJ's pipeline has no Entity Document either — this is how such a source states the answer once instead of paying for it per vector.

Distinct from the EntityID column, which is the entity an Entity-type source pulls records from and is null for file/RSS/website sources. This is the entity its vectors are.

Set it to an ISA extension rather than the base entity when that is where row-level security lives: attribution decides which entity's RLS is applied, so naming the base entity of an extension evaluates the wrong rules.

One declaration names one entity, so it is only meaningful when a source's vectors are all at the same level. Under ChunkTextStorage: 'mixed' a source emits ContentItem-level vectors for single-chunk items and ContentItemChunk-level vectors for the rest — two different entities — so leave this unset there and let per-vector Entity metadata carry it.

Setting this also changes what gets WRITTEN — with FieldStrategy: 'explicit', new vectors omit Entity and carry ContentSourceID instead, so the entity name lives in one place rather than on every vector. That additionally requires ChunkTextStorage: 'alwaysChunk', VectorIDStrategy: 'recordId', and a name here resolving to MJ: Content Item Chunks or a subtype. Anything unmet and Entity is written anyway, because a match search cannot attribute is dropped by the permission filter rather than returned unlabelled — 'mixed' would need two names, 'hash' leaves no recoverable record id, and a declaration naming the ITEM entity would point search at a table holding none of these ids.

Note that last one when row-level security is your reason for declaring an extension: under 'alwaysChunk' the vectors are chunk rows, so the extension has to extend the CHUNK entity. Existing vectors are untouched and keep resolving through their stored key.

Vector-database record-id strategy for this source's chunks. Default 'recordId'.

• 'recordId' (default, recommended): each ContentItemChunk's unique RecordID is used as its vector-DB record id. Safe with the soft-delete + PurgeDeletedChunks flow — a re-chunk mints new rows with new ids, so a superseded (soft-deleted) chunk and its replacement never share a vector id, and purging the old one can't orphan the live chunk's vector.
• 'hash': a deterministic hash of the parent content item id (5.49 EntityDocument parity). NOT safe with re-chunking + purge — a replacement chunk reuses the superseded chunk's id, so purging the old chunk would delete the live chunk's vector. Use only for sources that are never re-chunked or purged.

Controls what goes into each vector's metadata. Vector-store metadata has real storage + performance cost, so this lets a source keep it minimal. Falls back to the ContentType's default, then 'default'.

Website-crawler settings — only meaningful for content sources whose ContentSourceType is "Website". Replaces the legacy per-key ContentSourceParam rows; AutotagWebsite reads from this sub-object first and falls back to ContentSourceParam rows for sources configured before this field existed.

In the future, source-type-specific knobs like these may move to a pluggable per-source-type sub-interface scheme (one named property per source type). This is the first opt-in implementation; other source types will follow the same pattern as their knobs grow.