Documentation
¶
Overview ¶
Package chunk splits extracted Markdown into retrieval-sized pieces.
This is deliberately not delegated to xberg's own chunker: chunk metadata (HeadingPath, CharStart/CharEnd) needs to round-trip into a citation UI, and Markdown-aware recursive splitting is a small, well-understood algorithm — not a place outsourcing saves meaningful effort, unlike OCR/table-reconstruction.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chunk ¶
type Chunk struct {
Index int
Content string
CharStart int
CharEnd int
HeadingPath []string // e.g. {"Chapter 2", "Section 2.1"}, for citations
}
Chunk is one piece of a document, ready to embed.
type Chunker ¶
type Chunker struct {
// contains filtered or unexported fields
}
Chunker splits text into overlapping, size-bounded chunks.
func (*Chunker) SplitMarkdown ¶
SplitMarkdown splits Markdown text heading-aware: each top-level unit is a heading's content, further split by SplitText if still too large. The heading trail becomes each resulting chunk's HeadingPath.
type Config ¶
type Config struct {
// Size is the target maximum chunk length, in bytes.
Size int
// Overlap is how much trailing content of one chunk is repeated at the
// start of the next, to avoid losing context at a chunk boundary.
Overlap int
}
Config controls chunk size.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig matches the values validated in the reference RAG implementation this package's algorithm is modeled on.
Source Files