chunker

package

v0.19.3 Latest Latest Go to latest Published: Oct 3, 2025 License: MIT Imports: 6 Imported by: 0

Documentation ¶

Rendered for

This section is empty.

This section is empty.

This section is empty.

type Chunk struct {
	StartByte int
	EndByte   int
	Length    int
	StartLine int
	Content   string
	FilePath  string
	Language  string
}

Chunk represents a code chunk

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker wraps the C chunker

Chunker must have Close called to release memory.

func NewChunkerSize(maxChunkSize, chunkOverlap int) (*Chunker, error)

func NewChunkerSplitCode(maxChunkSize int) (*Chunker, error)

func NewChunkerSplitCodeSimple(maxChunkSize int) (*Chunker, error)

func (c *Chunker) AddFile(filePath, sourceCode string)

func (c *Chunker) ChunkFiles() error

func (c *Chunker) Chunks() iter.Seq[Chunk]

Chunks returns an iterator of all chunks found.

func (c *Chunker) Clear()

Clear removes all files and chunks but keeps allocated memory for reuse

func (c *Chunker) Close()

Close frees all chunker memory including files and chunks