textsplitters

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 12, 2026 License: MIT Imports: 2 Imported by: 0

Documentation

Overview

Package textsplitters provides utilities for splitting text into chunks.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type RecursiveCharacterTextSplitter

type RecursiveCharacterTextSplitter struct {
	// ChunkSize is the maximum size of each chunk in characters.
	ChunkSize int

	// ChunkOverlap is the number of overlapping characters between chunks.
	ChunkOverlap int

	// Separators is the list of separators to try, in order.
	Separators []string

	// LengthFunction computes the length of a string. Defaults to len().
	LengthFunction func(string) int
}

RecursiveCharacterTextSplitter splits text by recursively trying different separators until chunks are small enough.

func NewRecursiveCharacterTextSplitter

func NewRecursiveCharacterTextSplitter(chunkSize, chunkOverlap int) *RecursiveCharacterTextSplitter

NewRecursiveCharacterTextSplitter creates a splitter with default settings.

func (*RecursiveCharacterTextSplitter) SplitDocuments

func (s *RecursiveCharacterTextSplitter) SplitDocuments(documents []*core.Document) []*core.Document

SplitDocuments splits multiple documents into smaller documents.

func (*RecursiveCharacterTextSplitter) SplitText

func (s *RecursiveCharacterTextSplitter) SplitText(text string) []string

SplitText splits a text string into chunks.

func (*RecursiveCharacterTextSplitter) WithSeparators

WithSeparators sets custom separators.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL