` cells are separated by spaces (not tabs or delimiters). If you need structured table data, extract it with a dedicated HTML parser before passing through `ExtractFromPlainText`.
**Image alt text:** Alt text within `![description]() ` is not extracted. If alt text is important for your use case, extract it separately before calling `ExtractFromHTML`.
**Pre-formatted code blocks:** Content within `` tags has its internal whitespace normalized by the whitespace collapsing step. If exact formatting of code blocks matters, extract `` content separately before processing.
**Uncommon named entities:** Only the 13 most common entities are in the lookup table. Less common entities like `«`, `»`, or `€` will not be decoded. If your content uses these, consider a pre-processing step with a more comprehensive entity decoder.
### Recommendation for Complex HTML
For production HTML with complex structure (rich email templates, CMS content, web scraping output), consider a two-step approach:
1. Use a proper HTML parser (e.g., `cheerio`, `jsdom`) to extract the semantic content you care about
2. Pass the extracted text through `TextExtractor.ExtractFromPlainText()` for final normalization
`ExtractFromHTML` is intentionally dependency-light for server-side use without browser dependencies. It handles 90%+ of real-world HTML correctly.
---
## Integration with the Vectorization Pipeline
The vectorization pipeline (`@memberjunction/ai-vector-sync`) uses `TextChunker` and `TextExtractor` to prepare entity record content for embedding.
### Typical Flow
```
Entity Record
|
v
TextExtractor.ExtractByMimeType() -- Clean raw content (HTML, plain text)
|
v
TextExtractor.TruncateToTokenLimit() -- (optional) Cap total text length
|
v
TextChunker.ChunkText() -- Split into embedding-sized chunks
|
v
Embedding Model -- Generate vector for each chunk
|
v
Vector Database -- Upsert vectors with record metadata
```
### How Entity Content Maps to Text
MemberJunction entities store content in various formats depending on the entity type and field:
| Content Source | MIME Type | Extraction Method |
|---|---|---|
| Rich text fields (HTML) | `text/html` | `ExtractFromHTML` |
| Plain text fields | `text/plain` | `ExtractFromPlainText` |
| Description/notes fields | Usually plain text | `ExtractFromPlainText` |
| Concatenated field values | Plain text | `ExtractFromPlainText` |
The vectorization pipeline constructs a text representation of each record by concatenating relevant fields, then runs it through extraction and chunking.
### Chunk-to-Vector Mapping
Each `TextChunk` becomes a separate vector in the database. The chunk's `Index`, `StartOffset`, and `EndOffset` are stored as metadata alongside the vector, enabling:
- **Source tracing:** Given a search result, identify exactly which portion of the original record matched
- **Re-chunking detection:** When chunk settings change, compare offsets to determine which vectors need regeneration
- **Deduplication:** Identical chunks from the same record (due to overlap) can be detected via offset comparison
### Pipeline Code Example
```typescript
import { TextExtractor, TextChunker } from '@memberjunction/ai-vectors';
// 1. Extract clean text from the record's content
const rawText = TextExtractor.ExtractFromHTML(htmlContent);
// 2. Chunk into embedder-friendly pieces
const chunks = TextChunker.ChunkText({
Text: rawText,
MaxChunkTokens: 512,
Strategy: 'sentence',
});
// 3. Embed each chunk (via EntityVectorSyncer or directly)
for (const chunk of chunks) {
const embedding = await embeddingModel.EmbedTexts({
texts: [chunk.Text],
model: null
});
// Store embedding with chunk.Index, chunk.StartOffset, chunk.EndOffset as metadata
}
```
---
## Integration with the Autotagging Pipeline
The autotagging system uses the same text processing utilities to prepare content for classification and tag extraction by LLMs.
### Autotagging Flow
```
Entity Record
|
v
TextExtractor -- Clean raw content
|
v
TextChunker -- Split into manageable pieces (if content exceeds model limits)
|
v
LLM / Classification Model -- Extract tags, categories, or labels from each chunk
|
v
Tag Aggregation -- Merge and deduplicate tags across all chunks
|
v
Tag Assignment -- Write tags back to entity metadata
```
### Key Differences from Vectorization
| Aspect | Vectorization | Autotagging |
|---|---|---|
| Chunk size | 256-512 tokens (embedding model limit) | 1024-4096 tokens (LLM context window) |
| Strategy | Sentence (semantic precision) | Paragraph or fixed (broader context) |
| Overlap | 10% (search continuity) | 0-5% (tag extraction is less boundary-sensitive) |
| Output per chunk | One embedding vector | One or more tags/labels |
| Aggregation | All chunk vectors stored independently | Tags merged and deduplicated across chunks |
### Chunking for Autotagging
When autotagging, larger chunks generally produce better results because the classification model has more context to determine relevant tags. Use the paragraph strategy with a higher `MaxChunkTokens`:
```typescript
const cleanText = TextExtractor.ExtractFromHTML(record.Content);
const chunks = TextChunker.ChunkText({
Text: cleanText,
MaxChunkTokens: 2048,
OverlapTokens: 0, // Tags are less sensitive to boundary context
Strategy: 'paragraph'
});
// Send each chunk to the LLM for tag extraction, then merge results
const allTags: Set = new Set();
for (const chunk of chunks) {
const tags = await extractTagsFromChunk(chunk.Text);
tags.forEach(tag => allTags.add(tag));
}
```
---
## Performance Considerations
### Chunk Size and Total Count
The relationship between `MaxChunkTokens` and total chunk count is roughly inverse-linear. For a 10,000-token document:
| MaxChunkTokens | OverlapTokens | Approximate Chunks |
|---|---|---|
| 256 | 25 | ~43 |
| 512 | 51 | ~22 |
| 1024 | 102 | ~11 |
| 2048 | 0 | ~5 |
Each chunk requires one embedding API call, so doubling the chunk size roughly halves the number of API calls and stored vectors. Choose the largest chunk size that still provides good retrieval quality for your use case.
### Memory Usage
`TextChunker` processes text in a single pass and does not build an AST or DOM tree. Memory usage is proportional to the input text size plus the output chunk array. For a 1MB text document:
- Input string: ~1MB
- Sentence/paragraph split array: ~1MB (references into the input)
- Output chunks: ~1MB + overlap duplication
- **Total peak: ~3-4MB**
This is well within Node.js default heap limits for any reasonable document size. Extremely large documents (100MB+) should be pre-split into sections before chunking.
### Processing Speed
All three strategies are single-pass with O(n) complexity relative to input length:
| Strategy | Relative Speed | Why |
|---|---|---|
| Fixed | Fastest | Simple character-offset arithmetic |
| Paragraph | Fast | Single regex split on `\n\n` |
| Sentence | Slightly slower | Regex-based sentence boundary detection |
For typical documents (10-100KB), all strategies complete in under 1ms. The embedding API call dominates total pipeline time by orders of magnitude -- optimizing chunk strategy for speed is rarely worthwhile.
### TextExtractor Performance
HTML extraction performance depends on the complexity of the input:
| Input Size | Entity Decoding | Tag Stripping | Total |
|---|---|---|---|
| 10KB HTML | <1ms | <1ms | <1ms |
| 100KB HTML | ~1ms | ~1ms | ~2ms |
| 1MB HTML | ~5ms | ~10ms | ~15ms |
For batch processing thousands of records, consider running extraction in parallel with a concurrency limit to avoid overwhelming the event loop:
```typescript
// Process 10 records concurrently
const concurrency = 10;
for (let i = 0; i < records.length; i += concurrency) {
const batch = records.slice(i, i + concurrency);
await Promise.all(batch.map(async (record) => {
const text = TextExtractor.ExtractFromHTML(record.Content);
const chunks = TextChunker.ChunkText({ Text: text, MaxChunkTokens: 512 });
// ... embed chunks
}));
}
```
---
## Real-World Content Type Examples
### Customer Support Tickets
```typescript
import { TextExtractor, TextChunker } from '@memberjunction/ai-vectors';
// Support tickets often contain HTML from rich text editors
const ticketBody = record.Description; // HTML content
const cleanText = TextExtractor.ExtractFromHTML(ticketBody);
const chunks = TextChunker.ChunkText({
Text: cleanText,
MaxChunkTokens: 256, // Small chunks for precise similarity matching
Strategy: 'sentence', // Preserve individual customer statements
OverlapTokens: 25
});
```
### Knowledge Base Articles
```typescript
// Long-form articles with section headers
const articleContent = TextExtractor.ExtractFromHTML(record.Content);
const chunks = TextChunker.ChunkText({
Text: articleContent,
MaxChunkTokens: 512, // Balance between precision and context
Strategy: 'paragraph', // Respect the article's section structure
OverlapTokens: 50
});
```
### Product Descriptions
```typescript
// Short text that may not need chunking at all
const description = TextExtractor.ExtractFromPlainText(record.Description);
const tokens = TextChunker.EstimateTokenCount(description);
if (tokens <= 512) {
// Small enough for a single embedding -- skip chunking entirely
// Pass description directly to the embedding model
} else {
const chunks = TextChunker.ChunkText({
Text: description,
MaxChunkTokens: 512,
Strategy: 'sentence'
});
}
```
### Server Logs and Audit Trails
```typescript
// Structured, line-oriented content
const logText = TextExtractor.ExtractFromPlainText(logContent);
const chunks = TextChunker.ChunkText({
Text: logText,
MaxChunkTokens: 256,
Strategy: 'fixed', // No sentence structure to preserve
OverlapTokens: 0 // Each chunk is independent
});
```
### Email Content
```typescript
// Emails can be HTML or plain text
const emailBody = TextExtractor.ExtractByMimeType(
record.Body,
record.ContentType // 'text/html' or 'text/plain'
);
// Truncate very long email threads before chunking
const truncated = TextExtractor.TruncateToTokenLimit(emailBody, 8192);
const chunks = TextChunker.ChunkText({
Text: truncated,
MaxChunkTokens: 512,
Strategy: 'paragraph', // Email threads have natural paragraph breaks
OverlapTokens: 50
});
```
### Concatenated Entity Fields
```typescript
// Build a text representation from multiple entity fields
const parts = [
`Name: ${record.Name}`,
`Category: ${record.Category}`,
`Description: ${TextExtractor.ExtractFromPlainText(record.Description)}`,
`Notes: ${TextExtractor.ExtractFromHTML(record.Notes)}`
].filter(p => p && !p.endsWith(': '));
const fullText = parts.join('\n\n');
const chunks = TextChunker.ChunkText({
Text: fullText,
MaxChunkTokens: 512,
Strategy: 'paragraph' // Each field becomes a natural paragraph
});
```
### Binary Document Content (PDF, DOCX)
```typescript
// Binary formats need a dedicated parser first
import pdfParse from 'pdf-parse';
const pdfBuffer = await fs.readFile(filePath);
const pdfResult = await pdfParse(pdfBuffer);
// Then normalize and chunk the extracted text
const cleanText = TextExtractor.ExtractFromPlainText(pdfResult.text);
const chunks = TextChunker.ChunkText({
Text: cleanText,
MaxChunkTokens: 512,
Strategy: 'paragraph', // PDF text often preserves paragraph breaks
OverlapTokens: 50
});
```
---
## Summary
| Decision | Recommendation |
|---|---|
| **Strategy for prose** | `sentence` |
| **Strategy for structured docs** | `paragraph` |
| **Strategy for logs/code** | `fixed` |
| **Chunk size for search** | 256-512 tokens |
| **Chunk size for RAG** | 512-1024 tokens |
| **Chunk size for autotagging** | 1024-4096 tokens |
| **Overlap for search** | 10% (default) |
| **Overlap for RAG** | 15-25% |
| **Overlap for autotagging** | 0-5% |
| **Token estimation** | Use `EstimateTokenCount` for chunking; use `tiktoken` for billing/exact limits |
| **HTML with complex structure** | Use a DOM parser first, then `ExtractFromPlainText` |
| **Binary formats (PDF, DOCX)** | Use dedicated parser first, then `ExtractFromPlainText` |
|