Class BaseSegmenterAbstract

Base class for content segmentation strategies.

Why this exists

Chunking used to be a private helper inside whichever pipeline needed it, which meant every new strategy (structure-aware, LLM topic boundaries, audio chapters) would have been another bespoke branch. BaseSegmenter turns "how do I split this content" into a registered, swappable strategy selected from metadata — the same pattern BaseEmbeddings and VectorDBBase already use for their providers.

Adding a new strategy

Implement SegmentCore and register the class. That is the whole contract — the base class handles validation, the token ceiling, small-segment merging, sequence numbering, parent/child remapping, and provenance stamping:

@RegisterClass(BaseSegmenter, 'MyStrategy')
export class MySegmenter extends BaseSegmenter {
    public get Key(): string { return 'MyStrategy'; }
    public get SupportedModalities(): ContentModality[] { return ['text']; }

    protected async SegmentCore(params: SegmentationParams): Promise<RawSegment[]> {
        return myBoundaries(params.Text ?? '').map(b => ({
            Modality: 'text', Text: b.Text, StartOffset: b.Start, EndOffset: b.End
        }));
    }
}
Copy

Subclasses emit RawSegments and never worry about numbering: ParentIndex refers to positions in the array they just returned, and the base class remaps it to real Sequence values after any oversized segment has been split.