chunk

package

v1.9.18 Latest Latest Go to latest Published: Nov 20, 2025 License: Apache-2.0 Imports: 7 Imported by: 0

Details

This section is empty.

This section is empty.

This section is empty.

type Chunk struct {
	Index   int
	Content string
}

Chunk represents a piece of text from a document

type Processor struct{}

Processor handles document processing

func New() *Processor

New creates a new document processor

func (p *Processor) ChunkText(text string, size, overlap int, respectWordBoundaries bool) []Chunk

ChunkText splits text into overlapping chunks

func (p *Processor) CollectFiles(paths []string) ([]string, error)

CollectFiles recursively collects all files from given paths Skips paths that don't exist instead of returning an error

func (p *Processor) FileHash(path string) (string, error)

FileHash calculates SHA256 hash of a file

func (p *Processor) ProcessFile(path string, chunkSize, overlap int, respectWordBoundaries bool) ([]Chunk, error)

ProcessFile reads a file and splits it into chunks