Documentation
¶
Overview ¶
Package chunk turns ([]parse.SymbolSpan, source) into ([]types.Chunk) — the records the embedder + vector store actually persist.
Strategy:
- Each symbol span (function/method/type/...) becomes one chunk.
- Spans whose Text exceeds MaxInputTokens are split into sub-chunks (function bodies) or truncated at the head so the signature stays intact. Note that the mock embedder has infinite max input, so this only triggers with the real ONNX embedder.
- A file_header chunk captures the first N lines of the file — package decl + imports + top-level const/var. This lets queries like "what package owns the metrics client" hit the right file even when nothing function-level matches.
Index ¶
Constants ¶
const DefaultFileHeaderLines = 50
DefaultFileHeaderLines is the number of leading lines of each file captured as a "file_header" chunk when Options.FileHeaderLines is unset. 50 is enough to cover the package decl + a typical import block + top-level consts. Per-project ckv.yaml can override this via chunking.file_header_lines.
const FileHeaderLines = DefaultFileHeaderLines
FileHeaderLines is the legacy export retained for callers that reference the constant directly. New code should set Options.FileHeaderLines and read Chunker.opts.
Variables ¶
This section is empty.
Functions ¶
func BuildEmbedText ¶
BuildEmbedText returns the embedder input for c with a rule-based contextual prefix prepended. Cheap, deterministic, no LLM call.
Format: one descriptive line + blank line + raw chunk text.
File header chunks:
"language: go. file: server.go. file header.\n\n<text>"
Doc section chunks (markdown):
"language: markdown. file: docs/x.md. section: why-sqlite-vec.\n\n<text>"
Symbol chunks:
"language: go. file: server.go. symbol: Server.Listen (Method).\n\n<text>"
Design notes:
- The prefix is regenerated on every build/reindex. It is NOT persisted in c.Text — chunk IDs hash the raw text only, so re-running with the prefix on/off does not invalidate IDs.
- The query intent is NOT prefixed. The asymmetry is intentional (Anthropic Contextual Retrieval pattern): prefixed chunks become easier to retrieve from natural-language queries because the embedding now carries location/type signal in addition to body.
- We pick natural-language phrasing over bracketed tags so the embedder (typically trained on English prose) parses it cleanly.
func RawEmbedText ¶
RawEmbedText returns c.Text unchanged. Pass it as embedTextFn to disable contextual prefixing — useful for A/B measurement and for the existing baseline test corpus.
Types ¶
type Chunker ¶
type Chunker struct {
// contains filtered or unexported fields
}
Chunker turns parsed spans into Chunks.
type Input ¶
type Input struct {
File string // repo-relative path
Language string // "go" | "typescript" | "solidity" | "markdown"
CommitHash string // built-time git HEAD
Source []byte // full file contents
Spans []parse.SymbolSpan
}
Input is everything the chunker needs about one file.
type Options ¶
type Options struct {
MaxInputTokens int // hard upper bound on chunk Text size; 0 → no cap
IncludeFileHeader bool // emit a file_header chunk per file (default: true)
// FileHeaderLines overrides DefaultFileHeaderLines (50). 0 keeps
// the default. Sourced from project ckv.yaml.chunking.file_header_lines.
FileHeaderLines int
// IncludeFileFull emits an additional coarse chunk covering the whole file
// (Phase B multi-granularity, roadmap §3.1). Off by default — opt-in for
// measurement. Unlike file_header (first N lines) this spans the entire
// file, giving file/module-level queries a coarse target alongside the fine
// symbol chunks. Skipped for markdown (heading sections are already coarse).
IncludeFileFull bool
}
Options configure chunking. Zero value uses documented defaults.
Source Files