chunk

package

v1.16.0 Latest Latest Go to latest Published: Dec 30, 2025 License: Apache-2.0 Imports: 6 Imported by: 0

Details

This section is empty.

This section is empty.

func FileHash(path string) (string, error)

FileHash calculates SHA256 hash of a file

type Chunk struct {
	Index    int
	Content  string
	Metadata map[string]string
}

Chunk represents a piece of text from a document

func ProcessFile(dp DocumentProcessor, path string) ([]Chunk, error)

ProcessFile reads a file and processes it using the given document processor

type DocumentProcessor interface {
	Process(path string, content []byte) ([]Chunk, error)
}

DocumentProcessor takes file content and returns chunks. Config (size, overlap, etc.) is set at construction time.

type TextDocumentProcessor struct {
	// contains filtered or unexported fields
}

TextDocumentProcessor is the default text-based chunker

func NewTextDocumentProcessor(size, overlap int, respectWordBoundaries bool) *TextDocumentProcessor

NewTextDocumentProcessor creates a text-based document processor

func (t *TextDocumentProcessor) Process(_ string, content []byte) ([]Chunk, error)

Process implements DocumentProcessor for text-based chunking