OptionalMaxHard ceiling on tokens per text segment. The base class splits any
oversized segment via TextChunker so no segmenter can exceed it.
Default: 512.
OptionalMinSegments whose text estimates below this many tokens are merged forward into the next segment, preventing a spray of near-empty vectors. Default: 0 (off).
OptionalNoIf the whole text is within this percent above target, emit it as ONE segment rather than splitting it into a large piece plus a small remainder. Default: 40.
OptionalOverlapOverlap tokens applied when an oversized segment must be split. Default: 10% of max.
OptionalOvershootHow far above target (percent) it keeps looking for a sentence or word boundary before giving up and cutting at the hard ceiling. Default: 20.
OptionalTargetDesired segment size in tokens.
Size this to your queries, not to your embedding model. The model's context window is an upper bound, not a target — a chunk should be about as much content as a good answer to a typical query, so that a matching chunk is mostly signal. If queries are short paraphrases, smaller chunks retrieve better; if downstream summarization wants context, larger ones do. Default: 512.
OptionalUndershootHow far below target (percent) the segmenter may close on a good boundary. Entering this band is what makes segment sizes vary in service of clean breaks. Default: 20.
Options specific to AdaptiveBoundarySegmenter.