The registration key for this segmenter. Must match the key passed to
@RegisterClass so metadata-driven resolution round-trips.
Modalities this segmenter can produce. Used to validate configuration.
ProtectedresolveFill in defaults for any option the caller left unset.
Optionaloptions: SegmentationOptionsSegment content into embeddable units.
Never throws for content-shaped problems — inspect Success/ErrorMessage.
ProtectedSegmentProduce the raw, un-normalized segments for this content.
Implementations should focus purely on where the boundaries are; the base
class enforces the token ceiling afterwards, so returning a segment that is
too large is acceptable (it will be split, preserving Title and offsets).
ProtectedtokensStaticResolveResolve a registered segmenter by key via the MJ class factory. Returns null when no segmenter is registered under that key.
Uses TryCreateInstance rather than CreateInstance deliberately: the latter never returns
null for an unregistered key — it falls back to new BaseSegmenter(), a hollow object whose
abstract Key/SegmentCore are undefined. That failure stays invisible until something calls
it, so an unresolvable key must be reported as such here.
Segments prose by asking an LLM where the topics change.
Structure-aware segmentation only works when the author left structure behind. Transcripts, scanned reports, and long-form articles frequently have none — the topic shifts, but no heading marks it. This segmenter finds those latent boundaries and names them, producing titled sections that behave like headings the document never had.
Cost posture
The LLM pass is the expensive part of ingestion, so this class is written to avoid it whenever it wouldn't pay off: short documents short-circuit to structural segmentation, blocks are truncated to a preview before being shown to the model, and any failure degrades to
StructuralTextrather than failing the ingestion run. The model is asked to classify block indices, never character offsets — models are unreliable at arithmetic over long strings, and a wrong offset would silently corrupt chunk provenance.Because it runs through
AIPromptRunner, every pass is a trackedMJ: AI Prompt Runwith full token and cost attribution, and the prompt itself is versioned metadata rather than a string literal in code.