AbstractAbstractKeyRegistration key; must match the key passed to @RegisterClass.
ProtectedapplyWhitespace normalization and truncation, shared by every cleaner.
Optionaloptions: ContentCleaningOptionsOptionalwarnings: string[]Clean content ahead of segmentation.
Never throws for content-shaped problems — inspect Success/ErrorMessage. On
failure the ORIGINAL content is returned rather than an empty string, so a bad
selector degrades to "not cleaned" instead of silently discarding the document.
Protected AbstractCleanPerform the cleaning. The base class handles validation, whitespace, and truncation.
ProtectednormalizeCollapse horizontal whitespace and runs of blank lines, while preserving the single blank line that marks a paragraph break — segmenters rely on it as a boundary signal.
StaticResolveResolve a registered cleaner by key.
Uses TryCreateInstance because CreateInstance never returns null for an unknown
key — it silently yields a hollow base instance whose abstract members are undefined.
Base class for content cleaning strategies.
Cleaning is deliberately a separate stage from segmentation, and a separate plug-in point. The two answer different questions — cleaning asks which text is actually content, segmentation asks where that content divides — and they change for different reasons: a new CMS template needs new selectors, not a new chunking strategy. Splitting them also means the cleaning rules apply once and benefit every downstream consumer (embedding chunks, tagging chunks, full-text indexing) instead of being reimplemented per pipeline.
Garbage that survives this stage is expensive: it gets embedded, stored, retrieved, and eventually shown to a user or an agent. Navigation chrome repeated across a thousand pages produces a thousand near-identical vectors that crowd out real answers.