The registration key for this segmenter. Must match the key passed to
@RegisterClass so metadata-driven resolution round-trips.
Modalities this segmenter can produce. Used to validate configuration.
ProtectedresolveFill in defaults for any option the caller left unset.
Optionaloptions: SegmentationOptionsSegment content into embeddable units.
Never throws for content-shaped problems — inspect Success/ErrorMessage.
ProtectedSegmentProduce the raw, un-normalized segments for this content.
Implementations should focus purely on where the boundaries are; the base
class enforces the token ceiling afterwards, so returning a segment that is
too large is acceptable (it will be split, preserving Title and offsets).
ProtectedtokensStaticResolveResolve a registered segmenter by key via the MJ class factory. Returns null when no segmenter is registered under that key.
Uses TryCreateInstance rather than CreateInstance deliberately: the latter never returns
null for an unregistered key — it falls back to new BaseSegmenter(), a hollow object whose
abstract Key/SegmentCore are undefined. That failure stays invisible until something calls
it, so an unresolvable key must be reported as such here.
Segments text along its own document structure (markdown
#headings or HTML<h1>-<h6>), emitting one segment per section and preserving the heading hierarchy as parent/child links.This is the recommended default for documents, PDFs-converted-to-text, wiki pages, and knowledge-base articles. Compared with splitting purely on a token budget, sections are coherent units of meaning, so each vector represents one topic instead of an arbitrary window that may straddle two.
When the content has no headings it degrades gracefully to paragraph segments, which is still better than fixed windows because paragraph breaks are authored boundaries. In both cases BaseSegmenter enforces the token ceiling.