README
¶
recursive splitter
Recursive splitter is a splitter that splits the text into chunks recursively. Useful for splitting long text into chunks.
OverlapSize in config can set the overlap content length from last chunk, this may help to keep the context of last chunk.
Usage
example at: examples/main.go
run example: cd examples && go run main.go
import (
"context"
"fmt"
"os"
"github.com/cloudwego/eino-ext/components/document/transformer/splitter/recursive"
)
func main() {
ctx := context.Background()
splitter, err := recursive.NewSplitter(ctx, &recursive.Config{
ChunkSize: 1500,
OverlapSize: 300,
})
docs, err := splitter.Transform(ctx, []*schema.Document{
{Content: "test content"},
})
}
Examples
See the following examples for more usage:
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewSplitter ¶
NewSplitter create a recursive splitter.
Types ¶
type Config ¶
type Config struct {
ChunkSize int
// OverlapSize is the maximum allowed overlapping length between chunks. Overlapping can mitigate loss of information when context is divided.
OverlapSize int
// Separators are sequentially used to split text.
// When the current separator cannot split the text into a size smaller than ChunkSize, the next separator will be used to attempt to split until the chunk size is smaller than ChunkSize or there are no separator available.
// ["\n", ".", "?", "!"] by default.
Separators []string
// LenFunc is used to calculate string length. Use builtin function len() by default.
LenFunc func(string) int
// KeepType specifies if separator will be kept in split chunks. Discard separator by default.
KeepType KeepType
// IDGenerator is an optional function to generate new IDs for split chunks.
// If nil, the original document ID will be used for all splits.
IDGenerator IDGenerator
}
type IDGenerator ¶
IDGenerator generates new IDs for split chunks
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.