Registration key; must match the key passed to @RegisterClass.
ProtectedapplyWhitespace normalization and truncation, shared by every cleaner.
Optionaloptions: ContentCleaningOptionsOptionalwarnings: string[]Clean content ahead of segmentation.
Never throws for content-shaped problems — inspect Success/ErrorMessage. On
failure the ORIGINAL content is returned rather than an empty string, so a bad
selector degrades to "not cleaned" instead of silently discarding the document.
ProtectedCleanPerform the cleaning. The base class handles validation, whitespace, and truncation.
ProtectednormalizeCollapse horizontal whitespace and runs of blank lines, while preserving the single blank line that marks a paragraph break — segmenters rely on it as a boundary signal.
StaticResolveResolve a registered cleaner by key.
Uses TryCreateInstance because CreateInstance never returns null for an unknown
key — it silently yields a hollow base instance whose abstract members are undefined.
Extracts readable text from HTML using CSS selectors.
Real-world source pages are mostly not content: navigation, sidebars, cookie banners, share widgets, related-article rails, and advertising typically outweigh the article itself. Stripping tags alone keeps all of that text, and because chrome repeats across every page of a site it produces many near-identical chunks that crowd out real answers at retrieval time.
The high-leverage control is
IncludeSelectors— naming the one element that holds the content (.article-body,main,#post) discards everything else without having to enumerate what to drop.ExcludeSelectorsthen handles whatever survives inside it.Both are per-source configuration because the right selector is a property of the site's template, not of MemberJunction. A sensible default exclusion list handles sources that haven't been tuned yet.