Documentation
¶
Overview ¶
Package textsplitters provides utilities for splitting text into chunks.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type RecursiveCharacterTextSplitter ¶
type RecursiveCharacterTextSplitter struct {
// ChunkSize is the maximum size of each chunk in characters.
ChunkSize int
// ChunkOverlap is the number of overlapping characters between chunks.
ChunkOverlap int
// Separators is the list of separators to try, in order.
Separators []string
// LengthFunction computes the length of a string. Defaults to len().
LengthFunction func(string) int
}
RecursiveCharacterTextSplitter splits text by recursively trying different separators until chunks are small enough.
func NewRecursiveCharacterTextSplitter ¶
func NewRecursiveCharacterTextSplitter(chunkSize, chunkOverlap int) *RecursiveCharacterTextSplitter
NewRecursiveCharacterTextSplitter creates a splitter with default settings.
func (*RecursiveCharacterTextSplitter) SplitDocuments ¶
func (s *RecursiveCharacterTextSplitter) SplitDocuments(documents []*core.Document) []*core.Document
SplitDocuments splits multiple documents into smaller documents.
func (*RecursiveCharacterTextSplitter) SplitText ¶
func (s *RecursiveCharacterTextSplitter) SplitText(text string) []string
SplitText splits a text string into chunks.
func (*RecursiveCharacterTextSplitter) WithSeparators ¶
func (s *RecursiveCharacterTextSplitter) WithSeparators(seps []string) *RecursiveCharacterTextSplitter
WithSeparators sets custom separators.
Click to show internal directories.
Click to hide internal directories.