Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SplitParagraphIntoChunks ¶
SplitParagraphIntoChunks takes a paragraph and a maxChunkSize as input, and returns a slice of strings where each string is a chunk of the paragraph that is at most maxChunkSize long, ensuring that words are not split. Words longer than maxChunkSize are split into smaller chunks. For overlap and other options, use SplitParagraphIntoChunksWithOptions.
func SplitParagraphIntoChunksWithOptions ¶ added in v0.5.3
SplitParagraphIntoChunksWithOptions splits a paragraph into chunks according to opts. Chunks are word-boundary aligned; consecutive chunks may overlap by opts.Overlap characters (word-aligned). Words longer than opts.MaxSize are split into smaller chunks when opts.SplitLongWords is true.
Types ¶
type Options ¶ added in v0.5.3
type Options struct {
// MaxSize is the maximum characters per chunk (required, must be > 0).
MaxSize int
// Overlap is the overlap in characters between consecutive chunks, word-aligned (0 = no overlap).
// Must be < MaxSize; values >= MaxSize are clamped to MaxSize-1.
Overlap int
// SplitLongWords, when true, splits words longer than MaxSize into smaller chunks so no chunk exceeds MaxSize (default true).
SplitLongWords bool
}
Options configures paragraph chunking.
Source Files
Click to show internal directories.
Click to hide internal directories.