Documentation
¶
Overview ¶
Package chunk turns a ParsedDoc into deterministic, line-anchored chunks. Markdown chunks by heading, code by declaration, and everything else by paragraph/line windows. All chunkers share a token budget (target/overlap) and a pluggable TokenCounter so chunk sizing stays config-driven and model-agnostic in V0.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Chunker ¶
type Chunker interface {
// Chunk returns the chunks for doc. Ordinals are 0-based and contiguous.
Chunk(doc model.ParsedDoc) []model.Chunk
}
Chunker splits a parsed document into ordered chunks. Implementations must be deterministic: the same ParsedDoc always yields identical chunks, which keeps re-ingest idempotent and golden tests stable.
type CodeChunker ¶
type CodeChunker struct {
// contains filtered or unexported fields
}
CodeChunker emits one chunk per top-level declaration. The Go parser produces one Section per func/method/type (with its doc comment) and exact line spans; when parsing failed upstream the parser falls back to a single whole-file Section, which wholeOrWindows splits by lines.
type Config ¶
Config carries the token budget that bounds chunk sizing. Sections estimated above TargetTokens are split into windows of roughly TargetTokens with OverlapTokens of carry-over; smaller sections stay whole.
func ConfigFrom ¶
ConfigFrom builds a Config from target and overlap token counts. It is the single place configuration values become a chunk.Config, so callers (ingest, capture, move, eval) stop re-assembling the struct field by field and the chunk package owns the constructor for its own type.
type MarkdownChunker ¶
type MarkdownChunker struct {
// contains filtered or unexported fields
}
MarkdownChunker emits one chunk per heading section, splitting oversized sections into overlapping line windows. Line ranges are preserved exactly so citations can point at the source lines.
func (MarkdownChunker) Chunk ¶
func (m MarkdownChunker) Chunk(doc model.ParsedDoc) []model.Chunk
Chunk implements Chunker. Sections within the token budget stay whole; larger sections are split into windows of ~TargetTokens with OverlapTokens overlap, each window carrying the precise 1-based line range it covers.
type TextChunker ¶
type TextChunker struct {
// contains filtered or unexported fields
}
TextChunker splits plain text into paragraph-aware, token-budgeted windows. Paragraphs (blank-line separated) are packed into windows of ~TargetTokens; a single oversized paragraph is itself line-windowed. Line ranges stay exact.
func (TextChunker) Chunk ¶
func (t TextChunker) Chunk(doc model.ParsedDoc) []model.Chunk
Chunk implements Chunker. Paragraph packing reduces to line windowing: lines already include the blank separators, so wholeOrWindows keeps coherent paragraph groups while never splitting mid-line, preserving exact ranges.
type TokenCounter ¶
type TokenCounter interface {
// Count returns the estimated number of tokens in text.
Count(text string) int
}
TokenCounter estimates the token length of a string. It is the seam that lets a real model tokenizer drop in at Phase 4 without touching the chunkers.
type WordEstimator ¶
type WordEstimator struct{}
WordEstimator is the default offline TokenCounter. It approximates token count as ceil(words * 1.3), where words are whitespace-separated fields. The 1.3 factor reflects that sub-word tokenizers (BPE/wordpiece) typically split natural-language words into slightly more than one token on average. It needs no vocabulary file and is fully deterministic, satisfying the local-first constraint; a model-matched tokenizer replaces it at Phase 4.
func (WordEstimator) Count ¶
func (WordEstimator) Count(text string) int
Count implements TokenCounter.
Source Files