chunker

package

v0.5.0 Latest Latest Go to latest Published: Apr 5, 2026 License: MIT Imports: 3 Imported by: 0

Details

This section is empty.

This section is empty.

This section is empty.

type BreakpointChunker struct {
	TargetSize int // target chunk size in runes
	MinSize    int // minimum chunk size before emitting
}

BreakpointChunker splits sections using break-point scoring with distance decay.

func NewBreakpointChunker(targetSize int) *BreakpointChunker

func (c *BreakpointChunker) Chunk(doc *parser.Document) []Chunk

type Chunk struct {
	Seq         int
	Text        string
	HeadingPath string
	Ordinal     int // byte offset in original document
}

Chunk is a unit of text to be indexed and embedded.

type Chunker interface {
	Chunk(doc *parser.Document) []Chunk
}

Chunker splits a parsed document into indexable chunks.