Documentation
¶
Overview ¶
Package index provides semantic search over a recall graph. It embeds each node's text via the shared embedder (internal/embedder, any OpenAI-compatible endpoint) and answers queries with exact cosine similarity.
At ~10^3-10^4 nodes a brute-force scan is fast (sub-millisecond) and exact, so no ANN index is used; swap in HNSW (github.com/fogfish/hnsw, as the snapshot codec already does) if the node count grows past ~10^5.
Index ¶
- func NewEmbedder(ctx context.Context, endpoint, model string, dims, maxInputChars int, ...) (*embedder.Embedder, error)
- type AttrFilter
- type AttrVocabulary
- type EmbedStatter
- type Embedder
- type Hit
- type Index
- func (ix *Index) AttrVocabulary(kinds map[string]bool, repoFilter string, sel Select, key string) AttrVocabulary
- func (ix *Index) Build(ctx context.Context) error
- func (ix *Index) CacheSnapshot() map[string][]float32
- func (ix *Index) CountMatching(kinds map[string]bool, repoFilter string, sel Select) int
- func (ix *Index) DumpHNSW() map[string][]byte
- func (ix *Index) EmbedMissing(ctx context.Context) error
- func (ix *Index) LoadCache(c map[string][]float32)
- func (ix *Index) LoadHNSW(dump map[string][]byte) bool
- func (ix *Index) NoEmbed() bool
- func (ix *Index) PruneCache() int
- func (ix *Index) Search(ctx context.Context, query string, kinds map[string]bool, repoFilter string, ...) ([]Hit, error)
- func (ix *Index) SetBruteForce(b bool)
- func (ix *Index) SetNoEmbed(b bool)
- func (ix *Index) SetPendingHNSW(dump map[string][]byte)
- func (ix *Index) Stats() *SearchStats
- func (ix *Index) TitledNodes(kinds map[string]bool, repoFilter string, sel Select, name string, max int) []*graph.Node
- func (ix *Index) UseStats(s *SearchStats)
- type LatencySnapshot
- type Order
- type SearchStats
- type SearchStatsSnapshot
- type Select
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewEmbedder ¶
func NewEmbedder(ctx context.Context, endpoint, model string, dims, maxInputChars int, logger zerolog.Logger) (*embedder.Embedder, error)
NewEmbedder builds the shared embedder pointed at a llama.cpp-compatible /v1/embeddings endpoint (e.g. http://localhost:8090), truncated to dims. maxInputChars hard-caps a node's text before the embed call: set it below the model's token window × ~3.5 so oversized nodes are TRUNCATED here rather than REJECTED by the model (nomic 2048-tok → 8192; bge/MiniLM 512-tok → ~1800).
Types ¶
type AttrFilter ¶
type AttrFilter struct{ Key, Value string }
AttrFilter narrows a search to nodes carrying an exact attribute value (NodeInfo.Attrs, set by catalog entries). The zero value matches everything.
Deliberately ONE exact key=value with no operators, ranges or booleans: search is a single cheap call with no query language, and a caller that needs more than this should ask a better question rather than grow a filter DSL here.
func ParseAttrFilter ¶
func ParseAttrFilter(s string) (AttrFilter, error)
ParseAttrFilter reads a "key=value" pair. An empty string yields the zero filter; a string with no "=" is an error rather than a silent no-op, so a malformed filter cannot quietly widen a search the caller meant to narrow.
func (AttrFilter) Empty ¶
func (f AttrFilter) Empty() bool
Empty reports whether the filter narrows anything.
type AttrVocabulary ¶
type AttrVocabulary struct {
// Nodes is the size of the scope. Zero means the note has nothing to say
// about attributes (an unknown kind is already reported as such).
Nodes int
// Keys maps each attribute key to how many scope nodes carry it.
Keys map[string]int
// Values maps each distinct value of the key asked about to how many scope
// nodes carry it; empty when no key was asked or none carries it.
Values map[string]int
}
AttrVocabulary is what a scope's nodes say about themselves: which attribute keys they carry and, for one key the caller asked about, which values. It exists so a reply can NAME a vocabulary mismatch instead of silently ranking around it: the team map was asked for kind=Student order=experience:desc, no Student carries "experience", orderByAttr put every node in its "no attribute, sorts last" bucket and the set came back in cosine order — the reply looked healthy and the caller read an 8-year student as the most experienced when three 30-year students sat further down. A second call, attr=experience=30, returned 0 with no note at all.
type EmbedStatter ¶
type EmbedStatter interface {
Metrics() embedder.EmbedMetrics
}
EmbedStatter is an optional capability an Embedder may implement to report its own query-embedding call/cache-hit counters. *embedder.Embedder does (it is the only place that actually knows whether a given embed was a cache hit or a real API round-trip — see its embed method). Callers that want that number on a metrics surface type-assert for this rather than re-deriving or approximating it: SearchStats deliberately does not track its own embedCalls/embedCacheHits, since the embedder is shared across every map's Index (all of them route query embeds through the same *embedder.Embedder instance) and duplicating the counters here would either drift from the real ones or double-count.
type Embedder ¶
type Embedder interface {
EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
EmbedQuery(ctx context.Context, text string) ([]float32, error)
}
Embedder is the subset of embedder.Embedder this package needs.
type Hit ¶
type Hit struct {
QN string `json:"qn"`
Repo string `json:"repo,omitempty"`
Kind string `json:"kind"`
Title string `json:"title"`
File string `json:"file"`
Line int `json:"line"`
Score float32 `json:"score"`
Preview string `json:"preview"`
Language string `json:"language"`
// Parent is the qualified name of the node that contains this one, set
// only for a child node (a catalog entry's part, a doc's section). A search
// over a catalog whose entries carry parts returns the parts — the team
// map answered "most experienced student" with five Answer parts — and a
// brief hit gave no way back to the entity that holds the facts short of a
// second call per hit.
Parent string `json:"parent,omitempty"`
// Fused is the fusion score this hit was ORDERED by, and the only quantity a
// caller may re-sort on. Score is a cosine — it says how similar a hit is,
// not why it was selected — so sorting by it reorders a list its own numbers
// did not choose, burying keyword-promoted hits. Not serialized: a
// within-query ordinal means nothing to a reader.
Fused float64 `json:"-"`
}
Hit is one semantic search result.
func CutAtLargestGap ¶
CutAtLargestGap drops the tail of a ranked list at its steepest score drop.
Hits are scored by cosine, and on a corpus where documents resemble each other every node scores SOMETHING against every query: a query with one right answer still fills k with near-ties a few points below it. The signal is not the absolute score — that floor moves with the query — but where the list falls off a cliff. Keeping everything above the largest single drop adapts to how many answers the query actually has, which a fixed threshold cannot: measured over an 86-case suite it keeps ~2 hits for a one-answer query and ~9 for an eight-answer one, lifting precision on single-answer queries from 0.05 to 0.76.
It trades recall for precision (1.00 to 0.84 on those same queries), so it is opt-in: a caller that wants the full ranked list still gets it.
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index embeds graph nodes and serves semantic searches. It caches embeddings by node text so an incremental rebuild only embeds genuinely new/changed text — unchanged nodes (which become fresh *Node objects on each rebuild) reuse their cached vector instead of hitting the embedder again.
Search runs over an HNSW approximate-nearest-neighbour index (O(log N) per query) so it scales to many-repo node counts; a brute-force exact fallback is retained (useBruteForce) for correctness A/B testing.
func (*Index) AttrVocabulary ¶
func (ix *Index) AttrVocabulary(kinds map[string]bool, repoFilter string, sel Select, key string) AttrVocabulary
AttrVocabulary scans the scope an order or attr would be applied to — the tool's kinds, the repo filter and the selection's KINDS, deliberately not its Attr, since the question is what that scope holds — and reports the keys its nodes carry and the values of one key. Membership goes through wants with the selection's attr blanked, so the scope can never diverge from the set the search itself ranks. A plain scan over the graph; it runs only when a reply needs a note, never on the search path.
func (*Index) Build ¶
Build embeds every node, reusing cached vectors for unchanged text and only calling the embedder for cache misses. Safe to call repeatedly after a graph rebuild — it becomes an incremental embed.
func (*Index) CacheSnapshot ¶
CacheSnapshot returns a copy of the text→vector cache, for persisting a graph snapshot. The copy decouples the on-disk payload from later mutation of the live cache.
func (*Index) CountMatching ¶
CountMatching reports how many nodes the predicate selects, independent of any query or reply cap. A caller that names a set needs to know the set's real size: truncating a ranked list drops a less-relevant tail, but truncating a selection changes the answer, and doing that silently is indistinguishable from the set being that small.
func (*Index) DumpHNSW ¶
DumpHNSW serializes the current ANN index to a flat map suitable for gob encoding into a snapshot body. Returns nil if no ANN index is built yet.
func (*Index) EmbedMissing ¶
EmbedMissing fills in vectors for any node currently lacking one, reusing the text cache. Used after an incremental graph rebuild — equivalent to Build now that Build is cache-aware, kept as an intent-revealing name.
func (*Index) LoadCache ¶
LoadCache pre-populates the text→vector cache from a snapshot before Build, so a snapshot-backed graph embeds nothing: every node's text is already a cache hit and Build only (re)builds the HNSW index. Used by the read-only snapshot server.
func (*Index) LoadHNSW ¶
LoadHNSW reconstructs the ANN index from a previously dumped map, skipping the O(N·logN) rebuild. Returns true on success; on any error the caller should fall back to rebuildANNLocked. The HNSW is created with the SAME construction params as rebuildANNLocked so Read overwrites a compatible shell.
func (*Index) PruneCache ¶
PruneCache drops every cached vector whose text no current node has, and returns how many it dropped. The cache is keyed by text and only ever grows: a node whose body changed leaves its old text behind. Written into a snapshot, that dead weight is carried into every rebuild seeded from it, so a snapshot seeded from its predecessor prunes before it is written.
func (*Index) Search ¶
func (ix *Index) Search(ctx context.Context, query string, kinds map[string]bool, repoFilter string, sel Select, k int) ([]Hit, error)
Search returns the top-k nodes most similar to query. kinds restricts results to those node kinds (nil/empty = any); repoFilter restricts to one repository.
Default path is HNSW (O(log N) traversal of the ANN index); a kind/repo filter is applied AFTER the ANN fetch, so the fetch is widened to compensate for hits dropped by the filter. SetBruteForce switches to the exact O(N·D) scan for A/B comparison of recall and latency.
func (*Index) SetBruteForce ¶
SetBruteForce toggles the exact brute-force scan instead of HNSW — used to A/B test recall/latency of the ANN index against the exact baseline.
func (*Index) SetNoEmbed ¶
SetNoEmbed marks the index as never calling the embedder for a node with no cached vector (see the noEmbed field). Call before Build.
func (*Index) SetPendingHNSW ¶
SetPendingHNSW stages a serialized ANN graph (from a snapshot body) for Build to deserialize instead of rebuilding. Pass nil/empty to keep the rebuild path.
func (*Index) Stats ¶
func (ix *Index) Stats() *SearchStats
Stats returns the Index's current SearchStats — the shared, injected instance if UseStats was called, otherwise the private one New created.
func (*Index) TitledNodes ¶
func (ix *Index) TitledNodes(kinds map[string]bool, repoFilter string, sel Select, name string, max int) []*graph.Node
TitledNodes returns up to max nodes of the same scope AttrVocabulary scans whose title, or the last segment of whose qn, IS name — case-insensitive, and a leading '@' on either side is ignored. An attr filter whose KEY is unknown often carries a value that names a node rather than an attribute: the team map was asked four times for attr=github=octocat42, "github" is no attribute, and octocat42 is a Student's title (its qn ends ::octocat42) — 0 results, no note, four times. Plain scan, note path only.
func (*Index) UseStats ¶
func (ix *Index) UseStats(s *SearchStats)
UseStats swaps in a shared SearchStats instance, so its counts survive this *Index being replaced (e.g. evict/reload in Mode B) as long as every replacement is given the same instance. Must be called before the Index is shared with any concurrent caller — see the field's doc comment. A nil s is a no-op (keeps the private instance New created).
type LatencySnapshot ¶
type LatencySnapshot struct {
Count int64 `json:"count"`
AvgTotalUs int64 `json:"avgTotalUs"`
MaxTotalUs int64 `json:"maxTotalUs"`
AvgEmbedUs int64 `json:"avgEmbedUs"`
MaxEmbedUs int64 `json:"maxEmbedUs"`
AvgIndexUs int64 `json:"avgIndexUs"`
MaxIndexUs int64 `json:"maxIndexUs"`
}
LatencySnapshot is a point-in-time readout of search latency, in microseconds. Averages are derived from Count (0 when Count is 0).
type Order ¶
Order is an attribute to sort a selection by, and the direction. The zero value orders nothing.
func ParseOrder ¶
ParseOrder reads "key", "key:asc" or "key:desc". An empty string yields the zero order; an unknown direction is an error rather than a silent default, so a typo cannot quietly invert the answer to a superlative.
type SearchStats ¶
type SearchStats struct {
// contains filtered or unexported fields
}
SearchStats accumulates process-lifetime metrics for the hybrid (semantic + lexical) search path, so retrieval quality can be judged from live traffic instead of only from offline ground-truth runs (docs/todo/corpus-map-router.md's calibration work covers the latter; this is the former).
One instance is meant to be SHARED across every *Index that searches the same logical map over the server's whole lifetime. In Mode B a project's *Index is torn down and rebuilt on every evict/reload cycle (engine.Registry.load creates a fresh *Indexer, hence a fresh *Index, each time a project is revived from its snapshot) — counters stored ONLY on the *Index struct itself would silently reset to zero on every eviction, which is not "process-lifetime cumulative". engine.Registry owns exactly one SearchStats for its whole process lifetime and injects it into every *Index it builds via Index.UseStats, called once, before the Index is published to any concurrent reader. An Index that never receives an injected instance keeps the private one New creates for it — still correct, just not shared (this is what Mode A's single long-lived Index gets, which is fine: nothing recreates it mid-process there).
Every simple counter is an atomic — increment-only on the hot path, no allocation. The latency fields need a compound update (sum + count + max together) so they are guarded by a small mutex instead; the critical section is a handful of integer ops, not a source of contention at this server's request rate.
func NewSearchStats ¶
func NewSearchStats() *SearchStats
NewSearchStats returns a fresh, zeroed SearchStats.
func (*SearchStats) Snapshot ¶
func (s *SearchStats) Snapshot() SearchStatsSnapshot
Snapshot reads every counter at one point in time. Individual fields are not read atomically WITH RESPECT TO EACH OTHER (there is no global lock), so under concurrent writers this is a consistent-enough readout for a metrics endpoint, not a transactional one.
type SearchStatsSnapshot ¶
type SearchStatsSnapshot struct {
Searches int64 `json:"searches"`
SemanticCalls int64 `json:"semanticCalls"`
LexicalAbstentions int64 `json:"lexicalAbstentions"`
CoverageRejections int64 `json:"coverageRejections"`
KeywordPromotions int64 `json:"keywordPromotions"`
SemanticOnlyHits int64 `json:"semanticOnlyHits"`
BothChannelHits int64 `json:"bothChannelHits"`
Latency LatencySnapshot `json:"latency"`
}
SearchStatsSnapshot is a point-in-time, JSON-friendly readout of SearchStats.
type Select ¶
type Select struct {
// Kinds, when non-empty, selects nodes of those kinds. A map's kinds are
// its own vocabulary (Student, Service, Doc), reported by graph.Stats.
Kinds map[string]bool
// Attr, when set, selects nodes carrying that exact key/value.
Attr AttrFilter
// Order, when set, sorts the selected set by an attribute instead of by
// similarity to the query. A superlative — "most experienced student",
// "oldest incident" — is a sort, not a similarity, and no similarity rank
// can answer it: the team map ranked a kind=Student selection by cosine
// and put an 11-year student first, so answering needed all 37 entries read
// back and compared by the caller. Naming an order selects the whole scope
// (Select.Empty is false), because ordering only means something over a set
// the caller chose, never over the few nodes a query happened to retrieve.
Order Order
}
Select is the predicate the CALLER named, as distinct from the kinds a calling tool scopes itself to. The two are different requests: a tool's scope says which part of a map it is willing to answer from, while a caller naming a set means the members ARE the answer. So a Select is resolved against the graph and ranked within, rather than the query ranking everything and non-members being dropped afterwards — which returns a member only when the query happened to retrieve it.
Source Files