The registration key for this segmenter. Must match the key passed to
@RegisterClass so metadata-driven resolution round-trips.
Modalities this segmenter can produce. Used to validate configuration.
Estimated tokens for a string — exposed for callers reasoning about sizing.
ProtectedresolveFill in defaults for any option the caller left unset.
Optionaloptions: SegmentationOptionsSegment content into embeddable units.
Never throws for content-shaped problems — inspect Success/ErrorMessage.
ProtectedSegmentProduce the raw, un-normalized segments for this content.
Implementations should focus purely on where the boundaries are; the base
class enforces the token ceiling afterwards, so returning a segment that is
too large is acceptable (it will be split, preserving Title and offsets).
ProtectedtokensStaticResolveResolve a registered segmenter by key via the MJ class factory. Returns null when no segmenter is registered under that key.
Uses TryCreateInstance rather than CreateInstance deliberately: the latter never returns
null for an unregistered key — it falls back to new BaseSegmenter(), a hollow object whose
abstract Key/SegmentCore are undefined. That failure stays invisible until something calls
it, so an unresolvable key must be reported as such here.
Splits text toward a target size, closing on the best available natural boundary near that target rather than cutting at a fixed offset.
Why this beats a fixed window
A fixed window cuts wherever the budget runs out, which routinely lands mid-paragraph — the chunk then straddles two ideas and matches neither query well. This segmenter treats the target as a goal with a tolerance band and escalates through boundary quality as it goes:
UndershootPercentof target, close on a paragraph break.OvershootPercent, accept a word break.MaxSegmentTokensceiling.Segment sizes therefore vary — deliberately. A slightly short segment that ends at a paragraph is worth more at retrieval time than an exactly-sized one that ends mid-clause.
It also declines to split at all when the whole text is only modestly over target (
NoSplitPercent), which avoids the common pathology of one full-size chunk followed by a runt carrying two sentences and no context.This is the recommended default for prose when document structure isn't available; prefer
StructuralTextwhen the content has headings, since an authored boundary beats an inferred one.