Token-aware text chunker that respects natural boundaries.
Supports three strategies:
.
!
?
\n\n
Static
Split text into chunks that fit within the token limit.
Estimate token count using whitespace splitting. This is a fast approximation; for production accuracy, use tiktoken.
Token-aware text chunker that respects natural boundaries.
Supports three strategies:
.,!,?), never mid-sentence. Best for prose and natural language text.\n\n). Best for structured documents.