Documentation
¶
Index ¶
- func BackfillEmbeddings(ctx context.Context, st backfillStore, embedder embed.Embedder, ...) (int, error)
- func ChunkSymbol(filePath, language string, sym store.Symbol) []store.SymbolChunk
- func ChunkingEnabled() bool
- func EmbeddingText(filePath, language string, sym store.Symbol) string
- func EmbeddingTextWithNeighbors(filePath, language string, sym store.Symbol, neighbors []string) string
- func IsTestFile(name string) bool
- func NormalizeDocstring(doc string) string
- func SkipDir(name string) bool
- func Watch(ctx context.Context, root string, reindex func(context.Context) error, ...) error
- type ExtractorFactory
- type FakeWalker
- type FileRecord
- type FileWalker
- type Indexer
- type LanguageExtractor
- type Parser
- type Stats
- type Walker
- type WalkerConfig
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BackfillEmbeddings ¶
func BackfillEmbeddings(ctx context.Context, st backfillStore, embedder embed.Embedder, log *slog.Logger) (int, error)
BackfillEmbeddings embeds every symbol that has no stored vector (the state a --fast structure-only index leaves behind) and upserts the results in chunks, so an interrupted run resumes where it stopped. Returns the number of symbols embedded.
func ChunkSymbol ¶
func ChunkSymbol(filePath, language string, sym store.Symbol) []store.SymbolChunk
ChunkSymbol splits a capped symbol's lossless body into embeddable windows. Each chunk is prefixed with the same natural-language header the per-symbol embedding leads with, because a bare window of code carries no clue about which symbol it belongs to. Returns nil when the symbol is not capped.
func ChunkingEnabled ¶
func ChunkingEnabled() bool
ChunkingEnabled reports whether chunk embeddings should be built. It reads the same switch the retrieval channel uses, so the two cannot disagree: an index full of chunks no query will ever read is pure waste, and a weight pointing at chunks that were never built is a silent no-op.
func EmbeddingText ¶
EmbeddingText builds the index-time text embedded for a symbol. DECISION(2026-06): v2 profile — a natural-language line (name words, kind, path words) leads the text, followed by docstring, so paraphrastic queries can match symbols that have no doc comment. ASSUMES: the embedder weights early tokens more and the symbol name carries the core semantics. REVISIT IF: paraphrastic slice on the gen corpus does not improve vs v1.
func EmbeddingTextWithNeighbors ¶
func EmbeddingTextWithNeighbors(filePath, language string, sym store.Symbol, neighbors []string) string
EmbeddingTextWithNeighbors adds the names of the symbol's graph neighbours.
DECISION(2026-09): the callers and callees of a symbol are free vocabulary that the symbol's own text does not carry. `resize()` in image.js has nothing to do with the word "avatar" until you notice that `uploadAvatar` calls it — which is exactly the gap that makes an issue report fail to find the code it describes. Measured on SWE-Explore, 14.3% of gold files sit in the index and are never retrieved even at rank 200, and every attempt to reach them by ranking, pool size or graph traversal failed; this attacks the same gap from the indexing side, at no query cost and with no model.
Only the backfill path can supply neighbours, because during the first index pass the edges do not exist yet — embeddings are computed before the call graph is extracted. ASSUMES: neighbour names are more signal than noise at this cap. REVISIT IF: hub symbols (high degree) get worse rather than better.
func IsTestFile ¶
DECISION: test file patterns are filtered at the file level, not directory level, because some projects keep legitimate non-test code in directories named "test/". Patterns cover Go (_test.go), TS/JS (.test.*, .spec.*) and Python (test_*.py, *_test.py). IsTestFile reports whether a filename follows a test naming convention. Exported so ranking can ask the same question the walker asks: two copies of this rule silently disagreeing is how a knob ends up dead on arrival (the retrieval package's own isTestFile covers Go and TypeScript only).
func NormalizeDocstring ¶
NormalizeDocstring keeps the prose of a doc comment and drops the machinery around it: annotation-only lines, tool directives and markup noise.
DECISION(2026-08): a docstring is not free text to us — it is the largest semantic field in the embedding text, the reranker document and the agent payload. In three ecosystems the MAJORITY of what extractors captured carried no meaning at all: PHP 70% of documented symbols had only `@param Type $x` lines, C++ 59% (plus 22 symbols whose entire "doc" was a NOLINT suppression), TypeScript 61% were bare `/** @internal */`. Embedding those shapes the vector by annotation syntax instead of purpose, and a symbol left with nothing is better served by its name, signature and body. REVISIT IF: a language arrives whose tags carry the only description (then extract the tag's text rather than dropping the line).
func SkipDir ¶
SkipDir reports whether a directory (by base name) should be excluded from both indexing and watching: known build/vendor/VCS dirs and any dotfile dir.
func Watch ¶
func Watch(ctx context.Context, root string, reindex func(context.Context) error, log *slog.Logger, debounce time.Duration) error
Watch monitors root for source-file changes and calls reindex, debounced by `debounce`. reindex is expected to be an incremental pass (hash-skip), so re-running it on any change re-embeds only what actually changed. Calls are single-flighted: changes arriving while a reindex runs set a dirty flag that schedules exactly one more pass afterward. Watch blocks until ctx is done.
Types ¶
type ExtractorFactory ¶
type ExtractorFactory func(language string) (LanguageExtractor, bool)
ExtractorFactory returns a LanguageExtractor for the given language name. The bool indicates whether the language is supported.
type FakeWalker ¶
type FakeWalker struct {
// contains filtered or unexported fields
}
FakeWalker serves pre-supplied channels for use in tests.
func NewFakeWalker ¶
func NewFakeWalker(records <-chan FileRecord, errs <-chan error) *FakeWalker
func (*FakeWalker) Walk ¶
func (f *FakeWalker) Walk(_ context.Context, _ string) (<-chan FileRecord, <-chan error)
type FileRecord ¶
type FileWalker ¶
type FileWalker interface {
Walk(ctx context.Context, root string) (<-chan FileRecord, <-chan error)
}
FileWalker is the walker interface consumed by Indexer.
type Indexer ¶
type Indexer struct {
// contains filtered or unexported fields
}
func NewIndexer ¶
func NewIndexer(s store.Store, p Parser, e embed.Embedder, w FileWalker, ef ExtractorFactory, log *slog.Logger) *Indexer
func (*Indexer) Index ¶
DECISION: file processing is sequential (single goroutine consuming the walker channel). SQLite has a single-writer constraint, so parallelism here would require a work queue + serialized writes anyway. Add bounded parallelism for parse+extract if profiling shows it matters.
func (*Indexer) SetDocOverlay ¶
SetDocOverlay installs synthetic docstrings merged into symbols during indexing. DECISION(2026-06): purpose summaries close the vocabulary gap between intent-style queries ("why") and code text ("how") — the dominant paraphrastic failure on the gen corpus. The summary is prepended to any existing docstring so it leads the embedding text and the reranker document. REVISIT IF: paraphrastic slice does not improve after overlay reindex.
func (*Indexer) SetForceReindex ¶
SetForceReindex disables the unchanged-file hash skip so every file is re-parsed and re-embedded. Needed after the doc overlay changes: file hashes stay the same but embedding texts do not. Per-symbol embedding reuse still applies, so only symbols whose text actually changed hit the embedder.
type LanguageExtractor ¶
type LanguageExtractor interface {
Symbols(tree *tree_sitter.Tree, source []byte) []store.Symbol
Edges(tree *tree_sitter.Tree, source []byte, nameToID map[string]int64) []store.Edge
}
LanguageExtractor extracts symbols and edges from a parsed syntax tree. SymbolID/FileID are NOT set on returned symbols — caller assigns after persistence. DECISION: intra-file resolution only — cross-file is fuzzy match later.
type Parser ¶
type Parser interface {
Parse(ctx context.Context, source []byte, language string) (*tree_sitter.Tree, error)
LanguageNames() []string
Close() error
}
func NewTreeSitterParser ¶
type Walker ¶
type Walker struct {
// contains filtered or unexported fields
}
func NewWalkerWithConfig ¶
func NewWalkerWithConfig(log *slog.Logger, cfg WalkerConfig) *Walker
type WalkerConfig ¶
type WalkerConfig struct {
IncludeTests bool
}
Source Files
Directories