embeddings

package
v1.31.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 15, 2026 License: Apache-2.0 Imports: 39 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// DefaultNearDupK is the exported form of defaultNearDupK for callers in
	// other packages (e.g. semhealth) that need to pass k to FindNearDuplicates.
	DefaultNearDupK int = defaultNearDupK
)

Variables

This section is empty.

Functions

func AutoIndex 

func AutoIndex(pipeline *Pipeline, dirs []string, keyFn RepoKeyFunc, opts AutoIndexOpts)

AutoIndex scans directories for Git repositories and indexes them with a bounded worker pool plus per-repo retry-with-backoff on transient errors.

Each immediate subdirectory containing a .git folder is treated as a repo. keyFn should be codegraph.GraphNameFor (passed from caller to avoid an import cycle).

Rollback to byte-identical legacy behavior: opts.Concurrency=1, RetryMax=0.

Model-fingerprint guard: before indexing each repo, AutoIndex calls pipeline.InvalidateIfModelChanged. When the stored embed_model differs from the active model (e.g. jina-code-v2 → code-rank-embed), all stale vectors for that repo are purged atomically so IncrementalSync sees no prior SHA and runs a full re-embed with the new model. This prevents mixed-space garbage from old + new vectors being returned by semantic_search.

func ExtractQueryKeywords 

func ExtractQueryKeywords(query string) []string

ExtractQueryKeywords splits a natural language query into meaningful search terms by removing stopwords and short tokens. Returns lowercase terms >= 3 chars. Delegates to lextoken.KeywordTokenize — the canonical implementation.

func NewPipelineCache 

func NewPipelineCache() *kitcache.Cache

NewPipelineCache constructs a *kitcache.Cache pre-tuned for the embed pipeline (15-minute L1 TTL, modest item ceiling). Lives here rather than in register.go so wiring code stays free of cache-policy details.

Prometheus metrics are published via go-kit cache.WithMetrics using prometheus.DefaultRegisterer and the "embed_pipeline" cache name.

func PublishRRFWeights 

func PublishRRFWeights(w RRFWeights)

PublishRRFWeights records the deployed RRF weights to Prometheus. Idempotent (set, not increment); safe to call multiple times if config reloads land. Sparse, Graph, Hotspot, and Recency default to 0.0 (dark-launched). Flipping RRF_WEIGHT_* after A/B gate will be visible here at next startup.

func RecordGraphCandidates added in v1.21.0 

func RecordGraphCandidates(nPageRank, nCalls2Hop, nCommunity, nTotal int, durationSec float64)

RecordGraphCandidates increments the per-stage counters and the latency histogram for one GraphCandidates call. Called by graph_candidates.go.

func RecordIndexCancelled added in v1.21.0 

func RecordIndexCancelled(tool, phase string)

RecordIndexCancelled increments the gocode_index_cancelled_total counter for the given tool and phase. Call at any point where ctx.Err() != nil aborts an index write path.

func RecordIndexPartialAbort added in v1.21.0 

func RecordIndexPartialAbort(repo string)

RecordIndexPartialAbort increments gocode_index_partial_abort_total{repo} when a background index run has committed ≥1 chunk (rowsWritten > 0) but is aborting before writing all chunks and before advancing the repo_state SHA. This is the "0→100→0→100" churn signature: partial rows survive but SHA stays frozen.

func SanitizeAndFormatSparseVector added in v1.21.0 

func SanitizeAndFormatSparseVector(v sparse.SparseVector, dim int) string

SanitizeAndFormatSparseVector sanitizes v and formats it as a pgvector sparsevec text literal: `{i1:w1,i2:w2,…}/dim`.

Sanitization (applied before formatting, never mutates the caller's slice):

  • drops entries with zero weight (pgvector rejects them)
  • deduplicates indices (keeps the last occurrence, matching SPLADE convention)
  • drops entries with index ≥ dim (out-of-range for the column's sparsevec(dim))
  • prunes to the top sparseMaxTerms (256) highest-weight entries (pgvector HNSW on sparsevec_ip_ops hard-caps at 1000 nonzero terms; SPLADE doc-side can exceed)

After sanitization an empty result is formatted as "" (not `{}/dim`) so callers can detect a fully-degenerate vector and bind NULL instead. Non-empty results are always index-ascending (required by pgvector).

func SetEmbeddingsCoverageRows added in v1.22.0 

func SetEmbeddingsCoverageRows(repoKey string, n int)

SetEmbeddingsCoverageRows sets the gocode_index_embeddings_coverage_rows gauge. Called after a successful full-walk orphan reconciliation.

func SetEmbeddingsPresentGauge added in v1.21.0 

func SetEmbeddingsPresentGauge(repo string, count int)

SetEmbeddingsPresentGauge sets gocode_repo_embeddings_present{repo} to 1 when count > 0, and to 0 otherwise. Called after each successful IndexRepo or IncrementalSync pass to keep the gauge current.

func SetOrphanRepoKeysGauge added in v1.21.0 

func SetOrphanRepoKeysGauge(n float64)

SetOrphanRepoKeysGauge sets the gocode_orphan_repo_keys gauge. Called by the orphan_sweep MCP tool after a sweep to report the post-sweep count.

func SetRepoInfoGauge added in v1.21.0 

func SetRepoInfoGauge(repo, path string)

SetRepoInfoGauge records the (repo key → path) mapping in the gocode_repo_info gauge. The gauge value is always 1; presence of the series is the signal.

Call at the entry of any code path that receives a (repoKey, root) pair — IncrementalSync and IndexRepo cover all autoindex and on-demand flows. Idempotent: repeated calls with the same labels are no-ops on the Prometheus side (GaugeVec.WithLabelValues reuses the existing metric).

func StaleDemoteEnabled added in v1.21.0 

func StaleDemoteEnabled() bool

StaleDemoteEnabled reports whether the stale-demote safety-net is active. Callers outside this package (e.g. tool_semantic_search.go) read this flag rather than duplicating the env check.

func WarmRepoStateAdvancedZeroEmbeddings added in v1.22.2 

func WarmRepoStateAdvancedZeroEmbeddings(repoKeys []string)

WarmRepoStateAdvancedZeroEmbeddings pre-touches gocode_repo_state_advanced_with_zero_embeddings_total{repo} for every repoKey supplied, so the series exists before the first real event.

Why this matters: Prometheus increase() treats a counter series's first sample as having nothing to subtract from — a label combination that comes into existence AT the moment of a bad event yields increase()==0 for that exact event. A repo desyncing for the first time after a process restart would therefore be invisible to GocodeRepoStateAdvancedZeroEmbeddings until its SECOND desync in the same process lifetime. Pre-touching at boot with the known repo_key list (code_repo_state — repos indexed at least once) closes that window for every repo already on record.

Repos indexed for the first time after boot are not covered by this pre-touch, but are lower risk: SetRepoState establishes their code_repo_state row (and, transitively, their eligibility for this call on the NEXT boot) as part of the same successful-index path that would need to desync to trigger the counter at all.

Types

type AutoIndexOpts 

type AutoIndexOpts struct {
	// Concurrency is the maximum number of repos indexed in parallel.
	// Must be >=1; values <1 are normalized to 1 (serial fallback).
	Concurrency int
	// RetryMax is the maximum number of retries per repo on transient
	// failures. RetryMax=0 disables retry (single attempt only).
	RetryMax int
	// RetryBase is the initial backoff duration; doubles on each retry.
	RetryBase time.Duration
}

AutoIndexOpts controls the bounded worker pool and retry policy used by AutoIndex. A zero/invalid value falls back to DefaultAutoIndexOpts.

func DefaultAutoIndexOpts 

func DefaultAutoIndexOpts() AutoIndexOpts

DefaultAutoIndexOpts returns sane defaults: concurrency=1, retry_max=3, retry_base=5s (exponential backoff: 5s, 10s, 20s).

type BackfillOpts added in v1.21.0 

type BackfillOpts struct {
	// RepoKey scopes the backfill to one repo (empty = all repos).
	RepoKey string
	// RepoRootLookup resolves a repoKey to its absolute disk root.
	// Returns ("", false) when the repo is not available on disk.
	// Callers build this from AUTO_INDEX_DIRS + codegraph.GraphNameFor.
	// Injected rather than computed internally to avoid an import cycle
	// between internal/embeddings and internal/codegraph.
	RepoRootLookup func(repoKey string) (root string, ok bool)
	// WriteSparsesBatch is the batch UPDATE writer. Defaults to
	// store.UpdateSparseEmbeddingsBatch. Injectable for testing.
	// A single batch covers all surviving candidates in one page (one round-trip
	// instead of one per row). Non-fatal contract: failure leaves those rows NULL
	// and they are retried on the next backfill run via the IS NULL cursor.
	WriteSparsesBatch func(ctx context.Context, rows []SparseUpdate) error
}

BackfillOpts controls the behaviour of BackfillSparse.

type BackfillResult added in v1.21.0 

type BackfillResult struct {
	Backfilled   int
	SkippedDrift int
	SkippedMiss  int
	EmbedFailed  int
	WriteFailed  int // batch DB write timed out or errored; rows stay NULL for retry
	Total        int // rows examined
}

BackfillResult reports the outcome of a backfill run.

type BackfillRow added in v1.21.0 

type BackfillRow struct {
	RepoKey    string
	FilePath   string // relative path (stored as-is in code_embeddings)
	SymbolName string
	SymbolKind string
	Language   string
	StartLine  int
	BodyHash   uint64 // stored hash (FNV-64a of buildEmbedText at index time)
}

BackfillRow is a single NULL-sparse row selected for backfill.

type EmbeddingRecord 

type EmbeddingRecord struct {
	RepoKey         string
	FilePath        string
	SymbolName      string
	SymbolKind      string // function, method, class, etc.
	Language        string
	StartLine       int
	BodyHash        uint64              // for change detection
	EmbedModel      string              // embedding model name (e.g. "code-rank-embed"); written to code_embeddings.embed_model
	Embedding       []float32           // dense code-rank-embed vector (768-dim)
	SparseEmbedding sparse.SparseVector // SPLADE sparse vector (30522-dim); zero value → NULL in DB
}

EmbeddingRecord holds a single symbol embedding for storage.

type ExactDupPair added in v1.21.0 

type ExactDupPair struct {
	A, B SymbolRef
	// BodyHash is the scanned BIGINT column value. The ingest-side hash is uint64
	// (FNV-64a), cast to int64 on write; values with the high bit set appear as
	// negative here. The sign is irrelevant for equality joins — this is the only
	// use of the field.
	BodyHash int64
}

ExactDupPair is two distinct symbols sharing an identical body_hash. Exact (textual) clones are found via a fast indexed equality scan on the (repo_key, body_hash) partial index — no vector distance computation required.

type Expander 

type Expander struct {
	// contains filtered or unexported fields
}

Expander enriches semantic search results with 1-hop CALLS neighbors from Apache AGE.

func NewExpander 

func NewExpander(pool *pgxpool.Pool) *Expander

NewExpander creates an Expander backed by the given connection pool.

func (*Expander) Expand 

func (e *Expander) Expand(ctx context.Context, graphName string, results []SearchResult, maxExtra int) []SearchResult

Expand queries the AGE graph for 1-hop CALLS neighbors of the symbols in results. Returns at most maxExtra additional SearchResult entries (source="graph") not already present in the input. If the graph does not exist or any query fails, returns nil gracefully.

func (*Expander) GraphCandidates added in v1.21.0 

func (e *Expander) GraphCandidates(
	ctx context.Context,
	graphName string,
	queryTerms []string,
	seeds []SearchResult,
	prSignals []graphx.Signal,
	opts *GraphCandidatesOpts,
) []GraphHit

GraphCandidates generates graph-arm candidates from three sub-generators and returns them as []GraphHit ready for MergeRRF's graph arm.

graphName is the AGE graph name (codegraph.GraphNameFor(repoKey)). queryTerms is the list of keywords extracted from the user query. seeds is the ordered dense search results (top-N used for sub-arms b+c). prSignals is used only by the caller (annotateWithPageRank) and is no longer consumed by sub-arm (a) — pass nil or the fetched batch; both are safe. opts controls hop depth, fan-out, seed count, and topK.

Returns nil on any AGE error — the caller falls back to 3-arm behavior. Performance contract: 3 Cypher round-trips maximum (sub-arms a, b, and c). Sub-arm (a) issues one AGE query: symbols whose name/file matches query keywords, ordered by pagerank DESC — a single round-trip that replaces the broken in-memory filter over the global top-200 batch (which yielded 0 hits because the global top-200 are generic infrastructure symbols: Write, Error, Close, etc.).

func (*Expander) PairsConnectedByCalls added in v1.21.0 

func (e *Expander) PairsConnectedByCalls(ctx context.Context, graphName string, pairs []PairKey) (map[PairKey]bool, error)

PairsConnectedByCalls returns the subset of pairs where the two endpoints have a CALLS edge between them in either direction within the AGE graph.

The method issues a single Cypher query that matches both (a)-[:CALLS]->(b) and (b)-[:CALLS]->(a) orderings by filtering on the combined name set of all pair endpoints and checking membership in the input set after reconstructing the canonical PairKey from each returned edge.

Graph-missing or any query error → returns an empty map and nil error. The caller must not treat graph-down as fatal; graceful degradation is a correctness requirement (a graph hiccup must not hide real duplicates by silently removing nothing).

func (*Expander) PairsSharingInterface added in v1.21.0 

func (e *Expander) PairsSharingInterface(ctx context.Context, graphName string, pairs []PairKey) (map[PairKey]bool, error)

PairsSharingInterface returns the subset of pairs that are interface-impl siblings rather than refactor-worthy duplicates.

Interface-sibling pairs are the largest false-positive class: multiple concrete types implementing the same interface (e.g. *GitHubForge.FetchREADME vs *GitLabForge.FetchREADME, or four Search methods) look semantically identical but are distinct correct implementations, not duplicates.

Two discriminator paths, selected per call:

  • EXACT (preferred): when the graph has real (type)-[:IMPLEMENTS]->(interface) edges (a Go repo reindexed with the go/types satisfaction pass), a pair is a sibling iff both methods' receiver types implement a COMMON interface. This is the durable 2-hop fix and closes the residual false-negative the heuristic could not: two exported same-name methods on DISTINCT receivers that share NO interface are now correctly REPORTED, not suppressed.

  • HEURISTIC (fallback): when the graph has ZERO IMPLEMENTS edges (non-Go repos, or a Go repo not yet reindexed after the indexer change), it falls back to the #218 signature-receiver discriminator. Degradation is graceful — never worse than #218.

Same graceful-degradation contract as PairsConnectedByCalls: any graph error yields an empty map and nil error (a hiccup must not hide real duplicates).

type FileIndexResult added in v1.20.0 

type FileIndexResult struct {
	Embedded int   // newly-embedded symbol count (new or body-changed)
	Skipped  int   // hash-matched symbols — no embed call issued
	Deleted  int64 // symbols removed from DB (file shrank or was deleted)
}

FileIndexResult summarizes the outcome of a single-file incremental index.

type FileLineHit 

type FileLineHit struct {
	FilePath string
	Line     int
}

FileLineHit is a file path + line number from keyword search.

type GraphCandidatesOpts added in v1.21.0 

type GraphCandidatesOpts struct {
	// MaxHops is the hop depth for the CALLS traversal (sub-arm b). Default 2.
	MaxHops int
	// FanOutCap is the per-seed CALLS result cap (sub-arm b). Default 10.
	FanOutCap int
	// TopSeeds is the number of dense seeds to use for sub-arms (b) and (c). Default 3.
	TopSeeds int
	// TopK is the maximum total candidates returned. Default 20.
	TopK int
}

GraphCandidatesOpts controls the graph-candidate generation. Zero values use the package defaults above.

type GraphHit added in v1.21.0 

type GraphHit struct {
	FilePath   string
	SymbolName string
	SymbolKind string
	Line       int // start_line from the index row; 0 when unknown
}

GraphHit is a single graph-candidate or signal result. The same shape is used by the graph, hotspot, and recency RRF arms because all three are keyed by FilePath:SymbolName and carry the same symbol metadata. Source is set per-arm before fusion.

type HotspotHit added in v1.27.0 

type HotspotHit = GraphHit

HotspotHit is a ranked list entry for the hotspot RRF arm (churn x complexity). It is a type alias for GraphHit so the two arms remain distinct in signatures.

type HybridResult 

type HybridResult struct {
	SearchResult
	Source   string  // "semantic", "keyword", "sparse", "graph", "hotspot", "recency", or "hybrid"
	RRFScore float64 // combined reciprocal rank fusion score
}

HybridResult extends SearchResult with source attribution.

func MergeRRF 

func MergeRRF(
	semantic []SearchResult, keyword []KeywordHit, sparse []SparseHit,
	graph, hotspot, recency []GraphHit, topK int, weights RRFWeights,
) []HybridResult

MergeRRF combines semantic, keyword, sparse, graph, hotspot, and recency results using Weighted Reciprocal Rank Fusion. Backed by go-kit/rerank.WeightedRRF (Cormack-Clarke 2009 with per-list weights, k=60). Key = FilePath + ":" + SymbolName for dedup; results in multiple lists are attributed "hybrid". Returns at most topK results.

weights are env-driven (RRF_WEIGHT_SEMANTIC / RRF_WEIGHT_KEYWORD / RRF_WEIGHT_SPARSE / RRF_WEIGHT_GRAPH / RRF_WEIGHT_HOTSPOT / RRF_WEIGHT_RECENCY) and surfaced via Prometheus gauge gocode_rrf_weights{retriever}.

Dark-launch guarantee: when a weight is 0 OR its list is empty, that arm contributes nothing to ranking. A 0-weight arm contributes 1/(k+rank)*0 = 0 to every doc, so the existing ordering is preserved. This is verified for Sparse and Graph by existing tests and applies to Hotspot/Recency as well.

type IncrementalSyncMode added in v1.20.0 

type IncrementalSyncMode string

IncrementalSyncMode is the typed label for which code path IncrementalSync took. Values are stable wire strings used in Prometheus labels — do not rename. 

const (
	// IncrementalSyncIncremental is the normal git-diff path.
	IncrementalSyncIncremental IncrementalSyncMode = "incremental"
	// IncrementalSyncSkipSHAMatch is the fast-path when SHA is unchanged.
	IncrementalSyncSkipSHAMatch IncrementalSyncMode = "skip-sha-match"
	// IncrementalSyncFullFallbackBootstrap is triggered when no prior SHA exists in DB.
	IncrementalSyncFullFallbackBootstrap IncrementalSyncMode = "full-fallback-bootstrap"
	// IncrementalSyncFullFallbackNoGit is triggered when the path is not a git repo.
	IncrementalSyncFullFallbackNoGit IncrementalSyncMode = "full-fallback-no-git"
	// IncrementalSyncFullFallbackDiffError is triggered when git diff exec fails.
	IncrementalSyncFullFallbackDiffError IncrementalSyncMode = "full-fallback-diff-error"
)

type IncrementalSyncResult added in v1.20.0 

type IncrementalSyncResult struct {
	// Mode describes which code path was taken.
	Mode IncrementalSyncMode

	PrevSHA    string
	CurrentSHA string

	// FilesChanged is the count of files returned by git diff (0 on full-fallback paths).
	FilesChanged int
	// FilesEmbedded is the sum of FileIndexResult.Embedded across processed files
	// (or IndexResult.Indexed on full-fallback paths).
	FilesEmbedded int
	// FilesSkipped is the sum of FileIndexResult.Skipped (hash-matched symbols).
	FilesSkipped int
	// FilesDeleted is the sum of FileIndexResult.Deleted (tombstoned symbols).
	FilesDeleted int64

	// Errors collects per-file errors. When non-empty, SHA is NOT advanced so the
	// next boot retries the affected files (hash-skip from IndexFile makes retries cheap).
	Errors []error
}

IncrementalSyncResult is the return value of Pipeline.IncrementalSync.

type IndexResult 

type IndexResult struct {
	Indexed int
	Skipped int
	Total   int
}

IndexResult summarizes the outcome of an indexing run.

type KeywordHit 

type KeywordHit struct {
	FilePath   string
	SymbolName string
	Line       int
}

KeywordHit represents a grep/code_search match mapped to a symbol.

type NearDupResult added in v1.21.0 

type NearDupResult struct {
	// Pairs is the deduplicated set of near-duplicate candidates.
	Pairs []SimilarPair
	// SearchErrors is the number of per-symbol Store.Search calls that failed.
	// Non-zero means the result is INCOMPLETE — some symbols were skipped.
	// A SQLSTATE 57014 (statement_timeout) on an individual Search increments
	// this counter and is reflected here.
	SearchErrors int
}

NearDupResult is returned by FindNearDuplicates and carries both the deduplicated pair slice and a count of per-symbol Search errors so the caller can distinguish a complete run from a partial one.

type PairKey added in v1.21.0 

type PairKey struct {
	// A and B are "file_path:symbol_name" endpoint strings, A <= B lexicographically.
	A, B string
}

PairKey identifies an unordered symbol pair by "file_path:symbol_name" endpoints, canonicalized so A <= B (matching the store's unique-pair ordering convention used by FindSimilarPairs and FindExactDuplicates).

func NewPairKey added in v1.21.0 

func NewPairKey(file1, sym1, file2, sym2 string) PairKey

NewPairKey builds a canonical PairKey from two (file, symbol) endpoints. The endpoints are combined as "file:symbol" and sorted so A <= B, ensuring that (file1, sym1, file2, sym2) and (file2, sym2, file1, sym1) produce the same key — matching the store's "(a.file:a.sym) < (b.file:b.sym)" ordering.

type Pipeline 

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline orchestrates embedding indexing for repository symbols.

func NewPipeline 

func NewPipeline(client *embed.Client, store *Store, model string, opts ...PipelineOpt) *Pipeline

NewPipeline creates a Pipeline backed by the given client and store.

Pass a non-nil fileCache via WithFileCache to enable per-file symbol-entry caching keyed on (repoKey, file.RelPath) and validated by file modTime+size. When fileCache is nil, behavior is byte-identical to the v0.32.0 baseline.

model is the active embedding model name (e.g. "code-rank-embed"). It is stored alongside head_sha so that a model switch on next startup triggers a full reindex. Pass "" to retain legacy behaviour (no model tracking).

func (*Pipeline) EmbedModel added in v1.21.0 

func (p *Pipeline) EmbedModel() string

EmbedModel returns the active embedding model name configured for this pipeline. Returns "" for legacy pipelines created without model tracking. Used by semantic_search to detect stale-space hits: a stored model that does not match EmbedModel() means the indexed vectors are in the wrong space.

func (*Pipeline) IncrementalSync added in v1.20.0 

func (p *Pipeline) IncrementalSync(ctx context.Context, repoKey, root string) (*IncrementalSyncResult, error)

IncrementalSync reconciles a repo's embedding index via git-diff against the last-indexed SHA, calling Pipeline.IndexFile per changed file. Bumps SetRepoState only on full success (Errors empty).

Falls back to full IndexRepo when:

  • currentSHA == "" (non-git path)
  • prevSHA == "" (never indexed — bootstrap)
  • git diff exec failure (corrupted state — safer to full re-scan)

Same-SHA path (prevSHA == currentSHA) bumps indexed_at timestamp via SetRepoState but does no file work. Returns mode="skip-sha-match".

Returns non-nil result describing the path taken. Returns error only on catastrophic failures (DB connection lost). Per-file errors are collected in Result.Errors, not returned — caller decides retry policy.

func (*Pipeline) IndexFile added in v1.20.0 

func (p *Pipeline) IndexFile(ctx context.Context, repoKey, root, relPath string) (*FileIndexResult, error)

IndexFile incrementally indexes one file: parses it, extracts symbols, computes the diff against the currently-indexed set (Store.GetSymbolsForFile), embeds only NEW or CHANGED symbols (body_hash differs), and DELETEs symbols that disappeared from this file in the new parse.

root is the repo's absolute path on the host (for symbol path resolution); relPath is the file path relative to root (used as code_embeddings.file_path). repoKey is the canonical repo identifier (see codegraph.GraphNameFor).

Returns counts of newly-embedded, unchanged (skipped), and deleted symbols.

Side effects: Postgres writes; HTTP call to embed-server ONLY for new/changed symbols (zero HTTP calls if file unchanged from last index).

IndexFile does NOT touch repo_state / repoMainBranchSHA — that is repo-level fingerprinting owned by IndexRepo. IndexFile is file-level only.

func (*Pipeline) IndexProgress 

func (p *Pipeline) IndexProgress(repoKey string) (done, total int, running bool)

IndexProgress returns (done, total, running) for the given repo.

func (*Pipeline) IndexRepo 

func (p *Pipeline) IndexRepo(ctx context.Context, repoKey, root string) (*IndexResult, error)

IndexRepo indexes all functions and methods in a repository for semantic search.

func (*Pipeline) IndexRepoAsync 

func (p *Pipeline) IndexRepoAsync(repoKey, root string) bool

IndexRepoAsync starts background indexing if not already running. Returns true if indexing was started, false if already in progress. The background goroutine uses context.Background() (not the caller's context) so that client disconnects do not abort the indexing — Bug #2 fix.

This is a backward-compatible wrapper around IndexRepoAsyncWithTool that passes tool="autoindex". Callers that know their triggering MCP tool should use IndexRepoAsyncWithTool directly so the cancel counter is properly attributed.

func (*Pipeline) IndexRepoAsyncWithTool added in v1.21.0 

func (p *Pipeline) IndexRepoAsyncWithTool(tool, repoKey, root string) bool

IndexRepoAsyncWithTool starts background indexing if not already running, with tool attribution for observability. Returns true if indexing was started, false if already in progress.

The background goroutine runs under a bounded context (p.indexBudget, default 30m) derived from context.Background() — decoupled from any request context (Bug #2 fix) but bounded so a goroutine waiting on a permanently-unreachable embed server does not leak indefinitely.

tool is the MCP tool name that triggered this index (e.g. "semantic_search", "code_research"). It labels the gocode_index_cancelled_total counter so cancellations are attributable. Callers that do not know the tool should pass "autoindex".

Concurrency: the check-and-claim is atomic (LoadOrStore); only one goroutine per repoKey runs at a time.

func (*Pipeline) InvalidateIfModelChanged added in v1.21.0 

func (p *Pipeline) InvalidateIfModelChanged(ctx context.Context, repoKey string) bool

InvalidateIfModelChanged purges code_embeddings for repoKey when the stored embed_model differs from the active model (p.embedModel). This ensures stale vectors from a previous model are not mixed with new query vectors after a model upgrade (they live in different embedding spaces even when dimension is equal). When a purge occurs, the next IncrementalSync call treats the repo as unindexed and runs a full re-embed.

Called by autoindex.go per-repo before triggering IncrementalSync. No-op when embedModel is "" (legacy / test pipelines without model tracking).

func (*Pipeline) IsIndexing 

func (p *Pipeline) IsIndexing(repoKey string) bool

IsIndexing returns true if background indexing is running for the given repo.

type PipelineOpt 

type PipelineOpt func(*Pipeline)

PipelineOpt configures a Pipeline at construction time.

func WithFileCache 

func WithFileCache(c *kitcache.Cache) PipelineOpt

WithFileCache wires a *kitcache.Cache to memoize per-file symbol entries. Validator (modTime+size) ensures stale entries are evicted on the next indexRepo pass after a file is touched. Pass nil to disable explicitly.

func WithIndexBudget added in v1.21.0 

func WithIndexBudget(d time.Duration) PipelineOpt

WithIndexBudget sets the per-goroutine deadline for IndexRepoAsyncWithTool. When the budget expires the background goroutine's context is cancelled, which propagates to embedAndUpsert and terminates the stuck goroutine. d=0 uses the defaultIndexBudget (30m). Set via INDEX_BUDGET env var in the cmd layer (e.g. "INDEX_BUDGET=45m"); test code passes short durations directly.

func WithSparseEmbedder added in v1.21.0 

func WithSparseEmbedder(e sparse.SparseEmbedder) PipelineOpt

WithSparseEmbedder wires a SparseEmbedder into the indexing pipeline. When set, each batch of symbols also gets a SPLADE sparse vector, written to the sparse_embedding column (Phase P2). When nil (default), the pipeline is byte-identical to the dense-only path — no sparse calls, no sparse writes.

Sparse embedding is strictly additive: the dense vector is always persisted regardless of sparse outcome. Failures in the sparse leg (embed call or DB write) log at WARN, bump gocode_sparse_embed_failures_total, and leave that row's sparse_embedding NULL — never blocking dense persistence or SHA advance.

VocabSize guard: if e.VocabSize() != 30522 (the sparsevec column dimension), sparse indexing is refused (logs WARN, pipeline stays dense-only). This prevents out-of-range index corruption when a non-standard SPLADE head is wired.

func WithSparseMaxBatch added in v1.21.0 

func WithSparseMaxBatch(n int) PipelineOpt

WithSparseMaxBatch overrides the per-request input cap for the sparse server (EMBED_MAX_INPUT_ARRAY on the embed-server side). The default is sparseServerMaxDocs (32). Override when the server cap changes without a go-code redeploy (via SPARSE_EMBED_MAX_ARRAY → Config.SparseEmbedMaxArray).

type QueryEmbedder added in v1.21.0 

type QueryEmbedder interface {
	EmbedQuery(ctx context.Context, text string) ([]float32, error)
}

QueryEmbedder is the subset of embed.Client that query-path callers need. Exported so cmd/go-code can use it as the SemanticDeps.QueryClient field type. Satisfied by *embed.Client and by *codeRankQueryClient.

func NewQueryClient added in v1.21.0 

func NewQueryClient(client QueryEmbedder, model string) QueryEmbedder

NewQueryClient wraps client so that EmbedQuery calls apply the model-correct retrieval prefix. When model != codeRankEmbedModel the client is returned unwrapped (zero-overhead, identical to prior behaviour).

Document embedding (Embed calls on the returned *embed.Client) is never affected — this wrapper intercepts ONLY EmbedQuery.

type RRFWeights 

type RRFWeights struct {
	Semantic float64
	Keyword  float64
	// Sparse is the per-list weight for the SPLADE sparse-retrieval arm.
	// Default env value: 0.0 (dark-launched — plumbed but inert until A/B
	// in Phase 6 clears the gate). Post-A/B recommended value: 0.2–0.4
	// (below dense per research note in 2026-06-01 SPLADE landscape report).
	// Tune via RRF_WEIGHT_SPARSE env. Must be ≥ 0.
	Sparse float64
	// Graph is the per-list weight for the graph-candidate arm (Phase 1 graph-first
	// retrieval plan). Default env value: 0.0 (dark-launched — byte-identical until
	// A/B clears the gate). See RRF_WEIGHT_GRAPH in cmd/go-code/config.go.
	// Post-A/B recommended band: 0.2–0.4 (below dense per plan ADR).
	// Tune via RRF_WEIGHT_GRAPH env. Must be ≥ 0.
	Graph float64
	// Hotspot is the per-list weight for the churn x complexity signal arm.
	// Default env value: 0.0 (dark-launched — plumbed but inert until A/B
	// validates the quality gain). Post-A/B recommended band: 0.1–0.2.
	// Tune via RRF_WEIGHT_HOTSPOT env. Must be ≥ 0.
	Hotspot float64
	// Recency is the per-list weight for the last-modified time signal arm.
	// Default env value: 0.0 (dark-launched — plumbed but inert until A/B
	// validates the quality gain). Post-A/B recommended band: 0.05–0.15.
	// Tune via RRF_WEIGHT_RECENCY env. Must be ≥ 0.
	Recency float64
}

RRFWeights are per-retriever weights for the WeightedRRF fusion in MergeRRF. All fields must be ≥ 0 (negative values panic in go-kit/rerank.WeightedRRF — programmer error, not runtime). Weights == (1.0, 1.0, 1.0, 1.0, 1.0, 1.0) is the math identity used by tests; deployed defaults are controlled by cmd/go-code/config.go and keep Sparse/Hotspot/Recency dark-launched when their env weights are unset. Weights == (1.0, 1.0) in the 2-arm sense is mathematically identical to plain RRF (Cormack-Clarke 2009).

func DefaultRRFWeights 

func DefaultRRFWeights() RRFWeights

DefaultRRFWeights returns the (1.0, 1.0, 1.0, 1.0, 1.0, 1.0) math identity used by tests that don't thread per-deployment config. The deployed env defaults for Sparse, Graph, Hotspot, and Recency are 0.0 (dark-launch); DefaultRRFWeights() is the math identity, not the rollout policy. See defaultRRFWeight* in cmd/go-code/config.go.

type RecencyHit added in v1.27.0 

type RecencyHit = GraphHit

RecencyHit is a ranked list entry for the recency RRF arm (last-modified time). It is a type alias for GraphHit so the two arms remain distinct in signatures.

type RepoKeyFunc 

type RepoKeyFunc func(root string) string

RepoKeyFunc generates a graph key from a repo root path.

type SearchOpts 

type SearchOpts struct {
	RepoKey     string  // optional filter
	Language    string  // optional filter
	TopK        int     // default 20, max 100
	MaxDistance float32 // 0 = no filter; cosine distance threshold (0.0-1.0)
}

SearchOpts controls search filtering and result count.

type SearchResult 

type SearchResult struct {
	RepoKey    string
	FilePath   string
	SymbolName string
	SymbolKind string
	Language   string
	StartLine  int
	Distance   float32   // cosine distance (lower = more similar)
	Source     string    // "semantic", "keyword", "hybrid", "graph" — set by caller
	PageRank   float32   // structural importance from graph analysis (0 if not available)
	UpdatedAt  time.Time // last upsert time — used by stale-demote safety-net
}

SearchResult is a single semantic search hit.

func ApplyStaleDemote added in v1.21.0 

func ApplyStaleDemote(results []SearchResult, generation time.Time, enabled bool) []SearchResult

ApplyStaleDemote partitions results into fresh-then-stale using a binary criterion: a result is stale when its UpdatedAt is older than (generation - staleDemoteEpsilon).

Guarantees:

  • When generation.IsZero() or !enabled: returns results unchanged (byte-identical — no allocation, same slice header).
  • When all results are fresh: returns results unchanged (byte-identical).
  • Stable within each partition (fresh order preserved, stale order preserved).
  • Each demoted result increments gocode_semantic_stale_demoted_total.

The function does NOT filter results out; it demotes them to the bottom. This is a safety-net: a missed orphan surfaces at rank N+1 or later rather than at rank 1-5, and the counter fires so the operator knows to sweep.

type SimilarPair 

type SimilarPair struct {
	SymbolA    string
	FileA      string
	LineA      int
	KindA      string // symbol_kind of A (e.g. "function", "method", "class")
	SymbolB    string
	FileB      string
	LineB      int
	KindB      string // symbol_kind of B
	Similarity float32
}

SimilarPair represents two semantically similar symbols within the same repo.

type SimilarPairOpts 

type SimilarPairOpts struct {
	RepoKey   string
	Threshold float32 // minimum cosine similarity (default 0.92)
	Limit     int     // max pairs returned (default 50)
}

SimilarPairOpts controls the similarity search parameters.

type SparseHit added in v1.21.0 

type SparseHit struct {
	FilePath   string
	SymbolName string
	Line       int // start_line from the index row
}

SparseHit is a single SPLADE sparse-retrieval result. Shape is symmetric with KeywordHit so P4 can fuse all three arms via the same FilePath:SymbolName key already used by MergeRRF. Source is always "sparse" before fusion.

type SparseUpdate added in v1.21.0 

type SparseUpdate struct {
	RepoKey    string
	FilePath   string
	SymbolName string
	// Literal is the pre-formatted sparsevec text literal as returned by
	// SanitizeAndFormatSparseVector. Must be non-empty; callers skip empty ones.
	Literal string
}

SparseUpdate holds the identity and literal vector for one batch sparse write.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store manages vector embeddings in PostgreSQL with pgvector.

func NewStore 

func NewStore(pool *pgxpool.Pool) *Store

NewStore creates a Store backed by the given connection pool.

func (*Store) BM25Search added in v1.21.0 

func (s *Store) BM25Search(ctx context.Context, repoKey, queryText, language string, topK int) ([]KeywordHit, error)

BM25Search retrieves the top-K symbols most relevant to queryText via BM25F scoring over a trigram-prefiltered candidate set.

Protocol:

  1. Tokenize queryText (lextoken.Tokenize — identifier-aware, no stopword filter). Empty result → return (nil, nil) with no DB I/O (mirrors SearchSparse gate).
  2. Fetch candidates: reuse SearchBySymbolName (trigram similarity SQL, ILIKE fallback) to pull candidate rows whose symbol_name/file_path match the terms. Candidate set is bounded by the existing SQL LIMIT (not the full table).
  3. Build []ranking.Document from candidates: Path = file_path (BM25F Path field, ×3 weight) Symbols = [symbol_name] + lextoken.SplitIdentifier(symbol_name) (×5 weight) Docs = nil (no body column in code_embeddings; see plan §strategy-a)
  4. Score each candidate via ranking.NewBM25F + ScoreTerms; sort DESC; take topK.
  5. Map to []KeywordHit{FilePath, SymbolName, Line} — the Keyword-slot contract consumed by MergeRRF (rrf.go:81).

Corpus note: BM25F's IDF and avgDL are computed over the candidate set, not the global corpus (local-IDF approximation — standard for candidate-reranking pipelines fused via RRF). See plan §strategy-a for the trade-off rationale.

Gating: empty terms or candidate-fetch failure → (nil, nil) non-fatal; caller falls back to grep (Phase 4 wiring). DB errors bump gocode_bm25_query_failures_total.

func (*Store) BackfillSparse added in v1.21.0 

func (s *Store) BackfillSparse(
	ctx context.Context,
	sparseClient sparse.SparseEmbedder,
	opts BackfillOpts,
) (*BackfillResult, error)

BackfillSparse populates sparse_embedding for all rows where it is NULL.

Design invariants (plan P5):

  • IS NULL cursor: re-running picks up exactly the rows still NULL (idempotent).
  • Hash drift guard: if the freshly-computed textHash diverges from body_hash the row is skipped (counted skipped_drift); we never embed stale text against the stored dense vector — that would produce a semantically inconsistent pair. These rows self-heal on the next incremental index, which re-embeds both.
  • Per-item error isolation (lesson_per_item_error_freezes_batch_gate): a skipped_drift or skipped_missing row NEVER aborts the page. The IS NULL cursor advances because at least some rows are written; purely-bad pages (all drift/missing) terminate safely via the "no candidates after filtering" early-exit, which returns a zero-progress result that the caller can detect.
  • Batch-by-32: sparse embed calls respect sparseServerMaxDocs (P2 plan §P2).
  • Metrics-first: 4 outcome counters + 1 gauge ship with the feature so a stalled or partial backfill is always observable.

func (*Store) CountEmbeddings added in v1.21.0 

func (s *Store) CountEmbeddings(ctx context.Context, repoKey string) (int, error)

CountEmbeddings returns the number of code_embeddings rows for a given repoKey. Used by the same-SHA index gate to detect the frozen-empty state: when indexed_sha == HEAD but COUNT == 0 the repo needs recovery re-indexing.

The query uses the (repo_key) index so it is cheap (index scan + aggregate). It runs only on the same-SHA branch so it never adds latency to populated repos.

On error: returns (0, err). All three callers treat a non-nil error as fail-open — they fall through to re-index rather than skipping. This is the correct behaviour: a transient COUNT failure should not freeze a repo forever (fail-open: a frozen repo recovers even if COUNT transiently fails).

func (*Store) CountOrphanRepoKeys added in v1.21.0 

func (s *Store) CountOrphanRepoKeys(ctx context.Context) (int64, error)

CountOrphanRepoKeys returns the number of distinct repo_keys present in code_embeddings but absent from code_repo_state. Used for the gocode_orphan_repo_keys gauge.

func (*Store) DeleteExplicitOrphans added in v1.22.0 

func (s *Store) DeleteExplicitOrphans(ctx context.Context, repoKey string, orphanKeys []string) (int64, error)

DeleteExplicitOrphans removes code_embeddings rows for repoKey whose (file_path, symbol_name) appears in the supplied orphanKeys slice.

orphanKeys is the EXPLICIT set of keys to delete -- i.e. (DB keys) minus (freshly-parsed keys). The caller is responsible for computing this set. Deletion targets only the rows in orphanKeys; no other rows are touched.

When orphanKeys is empty, no rows are deleted (no-op).

Deletion is chunked by intraKeyOrphanChunkSize to bound per-statement size and avoid statement_timeout on large repos. Each chunk issues one DELETE WHERE (file_path,symbol_name) IN (VALUES ...) -- a positive IN-list. Chunking a positive IN is correct: each chunk only targets its own slice of orphans, so no valid row is ever collateral.

Returns the total rows deleted across all chunks.

func (*Store) DeleteOrphanRepoKeys added in v1.21.0 

func (s *Store) DeleteOrphanRepoKeys(ctx context.Context) (int64, error)

DeleteOrphanRepoKeys removes all code_embeddings rows whose repo_key has no corresponding row in code_repo_state. These orphans accumulate when:

  • a worktree checkout creates a new repo_key (GraphNameFor hashes the root path) but the worktree is later removed without running DeleteRepo;
  • a repo was bulk-indexed and then deregistered without cleanup.

Safety direction: delete embeddings-keys-not-in-state, NOT the reverse. code_repo_state is the canonical set of "repos we actively manage"; deleting embeddings for absent-state rows is safe because the next indexRepo run will re-populate them if the repo is re-registered. The inverse (deleting state rows whose embeddings are absent) would silently wipe valid index-freshness records.

This function does NOT delete code_repo_state rows. The operator controls state lifecycle (register/deregister via indexRepo or DeleteRepo).

Returns the number of orphan embedding rows deleted.

func (*Store) DeleteRepo 

func (s *Store) DeleteRepo(ctx context.Context, repoKey string) error

DeleteRepo removes all embeddings for a given repo_key.

func (*Store) DeleteSymbolsForFile added in v1.20.0 

func (s *Store) DeleteSymbolsForFile(ctx context.Context, repoKey, filePath string, keepSymbolNames []string) (int64, error)

DeleteSymbolsForFile removes index rows for symbols in (repoKey, filePath) EXCEPT those whose names appear in keepSymbolNames. Used to reconcile a file after re-parsing: symbols missing from the new parse (deleted / renamed) must be evicted. Returns rows-affected count.

If keepSymbolNames is empty, all symbols for the file are deleted (use this for IndexFile when the file is removed from the source tree).

func (*Store) EnsureSchema 

func (s *Store) EnsureSchema(ctx context.Context) error

EnsureSchema creates the pgvector extension and embeddings table if needed. After creating tables it attempts to transfer ownership to CURRENT_USER (best-effort: warns instead of failing if the role is not the table owner).

func (*Store) FindExactDuplicates added in v1.21.0 

func (s *Store) FindExactDuplicates(ctx context.Context, repoKey string) ([]ExactDupPair, error)

FindExactDuplicates returns pairs of distinct symbols within repoKey that share an identical non-zero body_hash. A non-zero body_hash means the symbol body was hashed at index time; DEFAULT 0 means "no hash" and is excluded to avoid false collisions between unhashed symbols.

The result is ordered deterministically by (a.body_hash, a.file_path, a.symbol_name, b.file_path, b.symbol_name) and capped at maxExactDupPairs. Because the scan uses an indexed equality join on the (repo_key, body_hash) partial index (WHERE body_hash <> 0) it does not need the O(n²) statement_timeout guard used by FindSimilarPairs.

func (*Store) FindNearDuplicates added in v1.21.0 

func (s *Store) FindNearDuplicates(ctx context.Context, repoKey string, k int, maxDist float32) (NearDupResult, error)

FindNearDuplicates finds near-duplicate symbol pairs in repoKey using per-symbol HNSW k-NN queries — N × O(log N) instead of the O(N²) all-pairs self-join used by FindSimilarPairs.

For each symbol it calls Store.Search with the symbol's own embedding as the constant query vector, requesting k+1 neighbours (k+1 because the symbol itself is always among the results at distance 0 and is dropped). Pairs whose cosine distance exceeds maxDist are discarded. The result is deduplicated using nearDupPairKey so each unordered pair appears exactly once regardless of which endpoint discovered it.

Per-symbol resilience: if an individual Search fails (including SQLSTATE 57014 statement_timeout), the error is logged, SearchErrors is incremented, and processing continues with the remaining symbols. The returned NearDupResult carries SearchErrors > 0 when any query failed, signalling an incomplete run to the caller. A non-nil error is reserved for a fatal failure of the bulk symbol load (the entire result set would be wrong).

Semantic note: this returns each symbol's top-k nearest neighbours, not all pairs under maxDist. A symbol with more than k near-duplicates will surface only its k closest. k=defaultNearDupK (5) covers the common case; the operator can pass a larger k for exhaustive analysis.

func (*Store) FindSimilarPairs 

func (s *Store) FindSimilarPairs(ctx context.Context, opts SimilarPairOpts) ([]SimilarPair, error)

FindSimilarPairs finds semantically similar function pairs within a repo using pgvector cosine distance self-join. Results are ordered by similarity descending.

The self-join is wrapped in a short transaction with SET LOCAL statement_timeout to prevent it from pinning a CPU core for minutes on large repos. SQLSTATE 57014 (query_canceled) is treated as "no pairs found" and logged at Debug level.

func (*Store) GetEmbedModelForRepo added in v1.21.0 

func (s *Store) GetEmbedModelForRepo(ctx context.Context, repoKey string) string

GetEmbedModelForRepo returns the embed_model stored in code_embeddings for repoKey (reading from any row for that repo_key), or "" when no rows exist or on error. Used as a fallback by the semantic_search stale-space guard when code_repo_state has no entry for the repo (e.g. orphan vectors left behind by a removed checkout). The per-row embed_model added in 2026-06-13 closes the blind spot: even without a state row the guard can detect that the stored vectors are in the wrong embedding space.

func (*Store) GetHashes 

func (s *Store) GetHashes(ctx context.Context, repoKey string) (map[string]uint64, error)

GetHashes returns a map of symbol_name -> body_hash for all embeddings in a repo.

func (*Store) GetIndexedAt added in v1.21.0 

func (s *Store) GetIndexedAt(ctx context.Context, repoKey string) time.Time

GetIndexedAt returns the indexed_at timestamp for a repo, or zero time when no row exists (repo never indexed). Used by stale-demote to determine the current index generation: any embedding row with updated_at < indexed_at was not re-touched this generation and is a candidate orphan.

Returns zero time on any error (schema init failure, no rows) so callers can treat zero as "generation unknown → skip demote" safely.

func (*Store) GetRepoState 

func (s *Store) GetRepoState(ctx context.Context, repoKey string) (string, error)

GetRepoState returns the last-indexed sha for a repo, or "" when no row exists.

func (*Store) GetStoredModel added in v1.21.0 

func (s *Store) GetStoredModel(ctx context.Context, repoKey string) string

GetStoredModel returns the embed_model stored in code_repo_state for repoKey, or "" when no row exists or on any error. Used by semantic_search to detect stale-space hits at query time: results returned from a repo whose stored model differs from the active model are in the wrong embedding space and must be discarded, triggering a full reindex.

This is a cheap single-row SELECT; the common case (model matches) adds one round-trip to the search path, which is negligible next to the vector scan.

func (*Store) GetSymbolsForFile added in v1.20.0 

func (s *Store) GetSymbolsForFile(ctx context.Context, repoKey, filePath string) ([]SymbolIdentity, error)

GetSymbolsForFile returns the symbol identity rows currently indexed for a specific file in a repo. Used by incremental update paths (Pipeline.IndexFile) to compute the diff: symbols present in DB but no longer in the file source must be DELETEd; symbols whose BodyHash changed must be re-embedded.

Returns rows sorted by symbol_name for determinism.

func (*Store) InvalidateRepoIfModelChanged added in v1.21.0 

func (s *Store) InvalidateRepoIfModelChanged(ctx context.Context, repoKey, activeModel string) (bool, error)

InvalidateRepoIfModelChanged purges code_embeddings for repoKey and resets its head_sha to "" when the stored embed_model differs from activeModel. This forces a full reindex on next query, producing vectors in the new embedding space. No-ops when the stored model matches.

The purge is atomic per repo: DELETE code_embeddings + DELETE code_repo_state row both run inside a single transaction so semantic_search never sees a half-purged state (either all old vectors remain or none do).

Returns (true, nil) when a purge was performed, (false, nil) when not needed.

func (*Store) ListRepoKeys added in v1.22.2 

func (s *Store) ListRepoKeys(ctx context.Context) ([]string, error)

ListRepoKeys returns every repo_key present in code_repo_state — the set of repos go-code has indexed at least once. Used at boot to pre-touch gocode_repo_state_advanced_with_zero_embeddings_total{repo} for known repos (see WarmRepoStateAdvancedZeroEmbeddings) so the counter series exists before any new bad event, not just after it.

func (*Store) MatchKeywordHits 

func (s *Store) MatchKeywordHits(ctx context.Context, repoKey string, hits []FileLineHit) ([]KeywordHit, error)

MatchKeywordHits maps keyword search hits to the nearest indexed symbol. For each hit, finds the symbol in the same file with start_line <= hit.Line (i.e., the function containing that line). Returns deduplicated KeywordHit results.

func (*Store) Search 

func (s *Store) Search(ctx context.Context, query []float32, opts SearchOpts) ([]SearchResult, error)

Search finds the top-K most similar symbols to the query embedding using cosine distance.

func (*Store) SearchBySymbolName 

func (s *Store) SearchBySymbolName(
	ctx context.Context,
	repoKey string,
	keywords []string,
	language string,
	limit int,
) ([]SearchResult, error)

SearchBySymbolName searches the embedding index by symbol name and file path using pg_trgm trigram similarity — correctly handles code abbreviations like "init" matching "initializes", "llm" matching "llms" etc. Returns results sorted by combined trigram similarity score DESC.

Rune token secondary symbols are stored as "$state:L<line>" (e.g. "$state:L7"). The trigram similarity between "$state" and "$state:L7" is ≈0.57 (well above the 0.1 threshold), so rune token queries work without special-casing.

func (*Store) SearchSparse added in v1.21.0 

func (s *Store) SearchSparse(
	ctx context.Context,
	queryText string,
	sparseClient sparse.SparseEmbedder,
	opts SearchOpts,
) ([]SparseHit, error)

SearchSparse retrieves the top-K symbols most relevant to the query via the SPLADE sparse_embedding column using negative inner product (<#> operator).

Protocol:

  1. Embed the query text to a SparseVector via sparseClient.EmbedSparseQuery.
  2. Sanitize and prune the query vector (SanitizeAndFormatSparseVector, same helper used at index time) to guard against fat query vectors that would violate pgvector's HNSW 1000-nonzero cap.
  3. Issue: SELECT … sparse_embedding <#> $1::sparsevec AS neg_ip … ORDER BY neg_ip LIMIT $3 — <#> returns negative inner product; ORDER BY ASC puts the most-similar rows first.
  4. Filter WHERE sparse_embedding IS NOT NULL to skip rows not yet backfilled (P5).

Gating: returns (nil, nil) when sparseClient is nil (SPARSE_EMBED_URL not set) or when the query text produces an empty sparse vector (all-stopword query). In both cases zero DB I/O is issued.

Failures: embed-fail bumps gocode_sparse_query_failures_total{stage="embed"} and returns empty; DB-fail bumps {stage="query"} and returns empty. P4 treats an empty sparse arm as dense+keyword only — byte-identical to today.

func (*Store) SetRepoState 

func (s *Store) SetRepoState(ctx context.Context, repoKey, sha, model string) error

SetRepoState records the sha and the active embedding model that was successfully indexed. model is stored alongside head_sha so that a model switch triggers a full reindex on next startup even when the repo's git SHA has not changed.

func (*Store) Stats 

func (s *Store) Stats(ctx context.Context) (map[string]int, error)

Stats returns embedding counts per repo.

func (*Store) UpdateSparseEmbeddingsBatch added in v1.21.0 

func (s *Store) UpdateSparseEmbeddingsBatch(ctx context.Context, rows []SparseUpdate) error

UpdateSparseEmbeddingsBatch writes sparse_embedding for a slice of rows in a single multi-row UPDATE statement, reducing round-trips from O(N) to O(1) per batch.

The UPDATE shape: 

UPDATE public.code_embeddings AS c
   SET sparse_embedding = v.vec::sparsevec, updated_at = NOW()
  FROM (VALUES ($1,$2,$3,$4::text), ...) AS v(repo_key,file_path,symbol_name,vec)
 WHERE c.repo_key=v.repo_key AND c.file_path=v.file_path AND c.symbol_name=v.symbol_name

Batches are capped at sparseBatchSize (100) rows; callers may pass any number and this method splits internally. The cap is data-size-bound (sparsevec literal text ~1.5 KB/row × 100 = ~150 KB/statement), not param-count-bound. Non-fatal contract: failure leaves those rows NULL; the caller is responsible for logging+counting; no row is written on error.

func (*Store) Upsert 

func (s *Store) Upsert(ctx context.Context, records []EmbeddingRecord) error

Upsert stores embeddings for symbols using multi-row INSERT with ON CONFLICT.

type SymbolIdentity added in v1.20.0 

type SymbolIdentity struct {
	SymbolName string
	BodyHash   uint64 // matches EmbeddingRecord.BodyHash (BIGINT in DB)
	StartLine  int
}

SymbolIdentity holds the identity fields for a single indexed symbol. Used by incremental update paths (Pipeline.IndexFile) to compute diffs: symbols present in DB but absent from new parse → DELETE; symbols whose BodyHash changed → re-embed.

type SymbolRef added in v1.21.0 

type SymbolRef struct {
	FilePath   string
	SymbolName string
	SymbolKind string
	StartLine  int
}

SymbolRef identifies one symbol in the code_embeddings table.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.