rag

package
v1.2.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 2, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package rag provides the RemembrancesService that bundles KB, Events, and Code indexing.

Package rag provides RAG (Retrieval-Augmented Generation) storage and search on top of the existing Pando SQLite database.

It supports:

  • Chunked document storage with JSON metadata
  • Vector similarity search (top-k ANN) via sqlite-vec / vec0 virtual tables
  • Full-text search (BM25) via SQLite FTS5
  • Hybrid search combining both with Reciprocal Rank Fusion

Prerequisites:

  • The sqlite-vec extension must be loaded before any Store is used. This is handled automatically by internal/db.Connect(), which calls sqlite_vec.Auto().
  • Call Store.Init after obtaining a *sql.DB to create the vec0 virtual table.

Index

Constants

View Source 
const (
	SourceCode   = "code"
	SourceKB     = "kb"
	SourceEvents = "events"
	SourceMemory = "memory"
)

Candidate sources reported to a RelevanceFilter.

View Source 
const DefaultEmbeddingDim = 1536

DefaultEmbeddingDim is the default vector dimension (OpenAI text-embedding-ada-002 / -3-small).

Variables

This section is empty.

Functions

func BuildMemoryBlock added in v0.416.3 

func BuildMemoryBlock(ctx context.Context, kbStore *kb.KBStore, query string, cfg config.RemembrancesConfig) string

BuildMemoryBlock retrieves relevant memories and formats them as a <memories> XML block for injection into the system prompt. Returns an empty string when no memories are found or the KB store is unavailable.

After building the block it fires goroutines to increment hit counters for each returned memory — these are best-effort and non-blocking.

func RelevanceFilterDisabled added in v1.2.7 

func RelevanceFilterDisabled(ctx context.Context) bool

RelevanceFilterDisabled reports whether ctx came from WithoutRelevanceFilter.

func WithoutRelevanceFilter added in v1.2.7 

func WithoutRelevanceFilter(ctx context.Context) context.Context

WithoutRelevanceFilter returns a context under which the enrichment and memory pipelines skip the relevance filter and keep every candidate. The agent uses it for turns that are not eligible for filtering (subagents, the context-enricher agent, resumed delegations).

Types

type Chunk 

type Chunk struct {
	// ID is the database row ID (set after insert, zero for new chunks).
	ID int64
	// Collection groups related chunks together (e.g. project path, session ID).
	Collection string
	// Source identifies the origin document (file path, URL, …).
	Source string
	// Content is the raw text of this chunk.
	Content string
	// ChunkIndex is the zero-based position of this chunk inside its source document.
	ChunkIndex int
	// Metadata is an arbitrary JSON string stored alongside the chunk.
	Metadata  string
	CreatedAt time.Time
	UpdatedAt time.Time
}

Chunk is a unit of text stored in the RAG system.

type ContextEnricher added in v0.236.1 

type ContextEnricher struct {
	// contains filtered or unexported fields
}

ContextEnricher performs pre-prompt KB, events and code searches in parallel and formats only relevant results (above minScore) as context to prepend to the user's message. A QueryPlanner derives per-source queries from the raw prompt before search.

func NewContextEnricher added in v0.236.1 

func NewContextEnricher(svc *RemembrancesService, cfg EnricherConfig) *ContextEnricher

NewContextEnricher creates a ContextEnricher from the given RemembrancesService and config values. Returns nil when the service is nil.

func (*ContextEnricher) EnrichContext added in v0.236.1 

func (e *ContextEnricher) EnrichContext(ctx context.Context, query string) string

EnrichContext searches KB, events, and code index in parallel using queries derived from the raw user prompt via the QueryPlanner, filters results below minScore, and returns a formatted context block. Sections with no results above the threshold are omitted entirely. Returns an empty string when nothing relevant is found or enrichment is disabled.

func (*ContextEnricher) EnrichContextWithResult added in v1.2.7 

func (e *ContextEnricher) EnrichContextWithResult(ctx context.Context, query string) (string, FilterResult)

EnrichContextWithResult is EnrichContext that also reports what the relevance filter did. The result is the zero FilterResult when no filter is installed or nothing was retrieved.

func (*ContextEnricher) IsEnabled added in v0.403.0 

func (e *ContextEnricher) IsEnabled() bool

IsEnabled reports whether context enrichment is currently active.

func (*ContextEnricher) PlannerMode added in v0.403.0 

func (e *ContextEnricher) PlannerMode() string

PlannerMode returns "llm" when an LLMPlanner is active, "heuristic" otherwise.

func (*ContextEnricher) SetEnabled added in v0.403.0 

func (e *ContextEnricher) SetEnabled(v bool)

SetEnabled enables or disables enrichment at runtime without removing the enricher from the agent. When disabled, EnrichContext returns an empty string immediately.

func (*ContextEnricher) SetPlanner added in v0.403.0 

func (e *ContextEnricher) SetPlanner(p QueryPlanner)

SetPlanner replaces the query planner used to derive per-source queries. Pass nil to revert to the default HeuristicPlanner.

func (*ContextEnricher) SetRelevanceFilter added in v1.2.7 

func (e *ContextEnricher) SetRelevanceFilter(f RelevanceFilter)

SetRelevanceFilter installs the decision-model relevance filter applied once over all retrieved candidates before they are formatted. Pass nil to turn it off. Safe to call concurrently with EnrichContext.

type EnricherConfig added in v0.402.0 

type EnricherConfig struct {
	KBResults      int
	CodeResults    int
	CodeProject    string
	EventsResults  int
	EventsSubject  string
	EventsLastDays int
	MinScore       float64

	// Char budgets per section (0 = no per-section limit).
	KBMaxChars     int
	CodeMaxChars   int
	EventsMaxChars int
	// TotalMaxChars caps the combined output across all sections (0 = no cap).
	TotalMaxChars int
}

EnricherConfig holds all tunable parameters for ContextEnricher.

type EnrichmentPlan added in v0.402.0 

type EnrichmentPlan struct {
	// SemanticQuery is a cleaned query suitable for KB semantic search.
	SemanticQuery string
	// CodeQuery is a keyword-focused query for code hybrid search.
	CodeQuery string
	// EventsQuery is a brief topical query for past-event search.
	EventsQuery string
	// KBResults overrides the default KB result count (0 = use default).
	KBResults int
	// CodeResults overrides the default code result count (0 = use default).
	CodeResults int
	// EventsResults overrides the default events result count (0 = use default).
	EventsResults int
}

EnrichmentPlan holds the per-source queries and result limits derived from a raw user prompt. Instead of sending the raw prompt to every backend, each source receives a targeted query.

type FilterResult added in v1.2.7 

type FilterResult struct {
	// Applied is true when the decision model judged at least one candidate.
	Applied bool
	// Kept and Dropped count the candidates after filtering.
	Kept, Dropped int
	// BySource holds {kept, dropped} per candidate source.
	BySource map[string][2]int
	// Latency is the wall-clock time spent in the filter.
	Latency time.Duration
	// Reason is empty on full success, otherwise the error class
	// (unreachable, timeout, ...) or "no_router", "hosted_provider", "partial".
	Reason string
	// Probabilities maps candidate ID to p(useful) for the judged candidates.
	Probabilities map[string]float64
}

FilterResult describes what a RelevanceFilter did on one call.

func BuildMemoryBlockWithResult added in v1.2.7 

func BuildMemoryBlockWithResult(ctx context.Context, kbStore *kb.KBStore, query string, cfg config.RemembrancesConfig, filter RelevanceFilter) (string, FilterResult)

BuildMemoryBlockWithResult is BuildMemoryBlock with an optional relevance filter (nil = off, byte-identical output). Memories from a pinned scope (cfg.MemoryPinnedScopes) are never dropped, and hit counters are only incremented for memories that are actually injected.

func NewFilterResult added in v1.2.7 

func NewFilterResult(cands []RelevanceCandidate, keep []bool) FilterResult

NewFilterResult builds a FilterResult with per-source counts from keep. It is a helper for filter implementations and for the callers of Filter.

type HeuristicPlanner added in v0.402.0 

type HeuristicPlanner struct{}

HeuristicPlanner derives queries without calling an LLM. It applies a series of text-cleaning rules to remove imperative/meta language and preserve technical terms.

func (*HeuristicPlanner) Plan added in v0.402.0 

func (p *HeuristicPlanner) Plan(_ context.Context, prompt string) (*EnrichmentPlan, error)

Plan implements QueryPlanner using purely local heuristics.

type HybridSearchOptions added in v0.210.0 

type HybridSearchOptions struct {
	Query           string
	Limit           int
	ProjectIDs      []string
	IncludeKB       bool
	IncludeSessions bool
	IncludeCode     bool
}

type HybridSearchResult added in v0.210.0 

type HybridSearchResult struct {
	Source     string                 `json:"source"`
	Title      string                 `json:"title,omitempty"`
	Path       string                 `json:"path,omitempty"`
	Content    string                 `json:"content"`
	Score      float64                `json:"score"`
	Rank       int                    `json:"rank"`
	Metadata   map[string]interface{} `json:"metadata,omitempty"`
	SessionID  string                 `json:"session_id,omitempty"`
	ProjectID  string                 `json:"project_id,omitempty"`
	FilePath   string                 `json:"file_path,omitempty"`
	SymbolName string                 `json:"symbol_name,omitempty"`
}

type LLMPlanner added in v0.403.0 

type LLMPlanner struct {
	// contains filtered or unexported fields
}

LLMPlanner uses a dedicated LLM provider to produce an EnrichmentPlan from the raw user prompt. If the primary provider fails it falls back to an optional secondary provider, and if that also fails it falls back to the HeuristicPlanner so that context enrichment continues to work even when the model is unavailable.

func NewLLMPlanner added in v0.403.0 

func NewLLMPlanner(primary, secondary PlannerProvider, systemPromptFn func(string) string) *LLMPlanner

NewLLMPlanner creates an LLMPlanner. systemPromptFn builds the system prompt for the agent given the provider name. secondary may be nil (no coder fallback).

func (*LLMPlanner) Plan added in v0.403.0 

func (p *LLMPlanner) Plan(ctx context.Context, userPrompt string) (*EnrichmentPlan, error)

Plan implements QueryPlanner. It asks the LLM for an optimised retrieval plan. On any error it degrades gracefully through secondary → heuristic.

type PlannerProvider added in v0.403.0 

type PlannerProvider interface {
	// SendRequest sends a simple system+user message and returns the text response.
	SendRequest(ctx context.Context, systemPrompt, userPrompt string) (string, error)
	// ProviderName returns the model provider name (used to select the system prompt variant).
	ProviderName() string
}

PlannerProvider is a minimal interface for LLM providers used by LLMPlanner. It avoids importing llm/provider (which would create an import cycle via llm/tools→rag). Implementations live in app/ where both sides can be imported safely.

type QueryPlanner added in v0.402.0 

type QueryPlanner interface {
	Plan(ctx context.Context, prompt string) (*EnrichmentPlan, error)
}

QueryPlanner derives an EnrichmentPlan from a raw user prompt.

type RelevanceCandidate added in v1.2.7 

type RelevanceCandidate struct {
	// Source is "code", "kb", "events" or "memory".
	Source string
	// ID identifies the item inside its source (path, symbol, event id, memory key).
	ID string
	// Text is the excerpt that would be injected, used to judge relevance.
	Text string
	// Score is the retrieval (embedding/hybrid) score.
	Score float64
	// Pinned marks candidates that must never be dropped (pinned memory scopes).
	// Filters may still look at them but the caller keeps them regardless.
	Pinned bool
}

RelevanceCandidate is one retrieved item that may be injected into the prompt.

type RelevanceFilter added in v1.2.7 

type RelevanceFilter interface {
	Filter(ctx context.Context, prompt string, cands []RelevanceCandidate) (keep []bool, res FilterResult)
}

RelevanceFilter decides which retrieved candidates are useful for a prompt. Implementations must be fail-open: on any problem they keep everything and report Applied=false. keep has the same length as cands.

type RemembrancesService added in v0.7.0 

type RemembrancesService struct {
	KB     *kb.KBStore
	Events *events.EventStore
	Code   *code.CodeIndexer
	// contains filtered or unexported fields
}

RemembrancesService groups KB, Events, and Code indexing stores. All components share the same SQLite database connection and use provider-configured embedders.

func NewRemembrancesService added in v0.7.0 

func NewRemembrancesService(db *sql.DB, cfg *config.RemembrancesConfig) (*RemembrancesService, error)

NewRemembrancesService creates a RemembrancesService from the app configuration and an existing SQLite connection. Returns nil (no error) when remembrances is disabled.

func NewRemembrancesServiceWithProxy added in v0.326.0 

func NewRemembrancesServiceWithProxy(db *sql.DB, cfg *config.RemembrancesConfig, proxy *dbproxy.DBProxy) (*RemembrancesService, error)

NewRemembrancesServiceWithProxy creates a RemembrancesService from the app configuration and an existing SQLite connection. When proxy is non-nil, mutating KB/events/code-indexing writes are redirected through the primary instance while reads still use the provided DB connection.

func (*RemembrancesService) DocumentEmbedder added in v0.330.0 

func (s *RemembrancesService) DocumentEmbedder() embeddings.Embedder

func (*RemembrancesService) HybridSearch added in v0.210.0 

func (s *RemembrancesService) HybridSearch(ctx context.Context, opts HybridSearchOptions) ([]HybridSearchResult, error)

func (*RemembrancesService) SetDocumentEmbedder added in v0.715.6 

func (s *RemembrancesService) SetDocumentEmbedder(e embeddings.Embedder)

SetDocumentEmbedder overrides the document embedder DocumentEmbedder() returns. docEmbedder is otherwise only set by NewRemembrancesServiceWithProxy; this exists so tests that construct a RemembrancesService directly (a bare &RemembrancesService{KB: store} struct literal, common across internal/api and internal/llm/tools tests) can exercise callers of DocumentEmbedder() — e.g. the embedding-staleness check on GET /api/v1/remembrances/enrichment (PANDO-US-0029) — without going through the full provider-config wiring.

type SearchOptions 

type SearchOptions struct {
	// Collection restricts results to a single collection. Empty means all collections.
	Collection string
	// TopK is the maximum number of results to return. Defaults to 5.
	TopK int
	// MinScore excludes results with Score below this value (0 means no filter).
	MinScore float64
}

SearchOptions controls search behaviour.

type SearchResult 

type SearchResult struct {
	Chunk
	// Distance is the L2 (or cosine) distance from the query vector (lower = more similar).
	// Populated only by SearchVector and SearchHybrid.
	Distance float64
	// Score is a normalised relevance score in [0, 1] where higher is better.
	// For vector search:  1 / (1 + Distance).
	// For FTS search:     normalised BM25 (−bm25 / max(−bm25) across the result set).
	// For hybrid search:  Reciprocal Rank Fusion score.
	Score float64
	// Rank is the 1-based position in the result list.
	Rank int
}

SearchResult is a ranked entry returned from vector, FTS, or hybrid search.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store manages RAG chunks, embeddings, and full-text search on a SQLite database.

Vector embeddings are stored as little-endian float32 BLOBs in the rag_chunks table and similarity search is performed in Go – no SQLite extension is required. Full-text search uses SQLite's built-in FTS5 engine.

The Store does NOT own the *sql.DB – callers are responsible for opening and closing the database via internal/db.Connect().

Usage: 

db, err := db.Connect()
store := rag.New(db, rag.StoreOptions{EmbeddingDim: 1536})
if err := store.Init(ctx); err != nil { … }

func New 

func New(db *sql.DB, opts StoreOptions) *Store

New creates a Store backed by db. Call Init before any other method.

func (*Store) CountChunks 

func (s *Store) CountChunks(ctx context.Context, collection string) (int64, error)

CountChunks returns the total number of chunks (all when collection="").

func (*Store) DeleteChunk 

func (s *Store) DeleteChunk(ctx context.Context, id int64) error

DeleteChunk removes a chunk and its FTS entry. It is a no-op when the chunk does not exist.

func (*Store) DeleteCollection 

func (s *Store) DeleteCollection(ctx context.Context, collection string) error

DeleteCollection removes all chunks (and their FTS entries) that belong to the given collection.

func (*Store) Dim 

func (s *Store) Dim() int

Dim returns the configured embedding dimension.

func (*Store) GetChunk 

func (s *Store) GetChunk(ctx context.Context, id int64) (*Chunk, error)

GetChunk retrieves a chunk by ID. Returns (nil, nil) when not found.

func (*Store) GetChunkEmbedding 

func (s *Store) GetChunkEmbedding(ctx context.Context, id int64) ([]float32, error)

GetChunkEmbedding retrieves the stored embedding for a chunk. Returns (nil, nil) when the chunk has no embedding.

func (*Store) Init 

func (s *Store) Init(ctx context.Context) error

Init validates the configured embedding dimension against the stored one and persists it on first use. It is idempotent.

func (*Store) InsertChunk 

func (s *Store) InsertChunk(ctx context.Context, chunk Chunk, embedding []float32) (int64, error)

InsertChunk inserts a new chunk and optionally its embedding vector.

Pass a nil or empty embedding to store a text-only chunk (participates in FTS search but not in vector search).

Returns the auto-assigned chunk ID.

func (*Store) ListChunks 

func (s *Store) ListChunks(ctx context.Context, collection string, limit, offset int) ([]Chunk, error)

ListChunks returns paginated chunks for a collection (all when collection="").

func (*Store) RebuildFTS 

func (s *Store) RebuildFTS(ctx context.Context) error

RebuildFTS rebuilds the FTS5 index from the rag_chunks content table. Use this to recover from index corruption or after bulk inserts that bypassed the normal InsertChunk / UpdateChunk path.

func (*Store) SearchFTS 

func (s *Store) SearchFTS(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)

SearchFTS performs a full-text search using SQLite FTS5 (BM25 ranking).

query follows FTS5 syntax: plain terms, phrase queries ("…"), prefix queries (term*), column filters (content:term), boolean operators (AND / OR / NOT).

Results are ordered by descending relevance with scores normalised to [0, 1].

func (*Store) SearchHybrid 

func (s *Store) SearchHybrid(ctx context.Context, query string, embedding []float32, opts SearchOptions) ([]SearchResult, error)

SearchHybrid combines vector and full-text search using Reciprocal Rank Fusion (RRF). Both searches run concurrently.

Pass a nil embedding to skip vector search (equivalent to SearchFTS).

RRF formula: score(d) = Σ 1 / (rrfK + rank(d, list)) where rrfK = 60.

func (*Store) SearchVector 

func (s *Store) SearchVector(ctx context.Context, embedding []float32, opts SearchOptions) ([]SearchResult, error)

SearchVector performs an exact k-NN search by loading all embeddings from the collection and computing cosine similarity in Go.

This approach requires no SQLite extension and is suitable for typical RAG workloads (up to ~100k chunks). Chunks without embeddings are skipped.

Results are ordered by descending cosine similarity (most similar first).

func (*Store) UpdateChunk 

func (s *Store) UpdateChunk(ctx context.Context, id int64, chunk Chunk, embedding []float32) error

UpdateChunk replaces the text, metadata, and optionally the embedding of an existing chunk. Pass nil embedding to leave the stored vector unchanged.

type StoreOptions 

type StoreOptions struct {
	// EmbeddingDim is the dimensionality of the embedding vectors.
	// Must match the dimension used when the vec0 table was first created.
	// Defaults to DefaultEmbeddingDim (1536).
	EmbeddingDim int
}

StoreOptions configures a new Store.

Source Files

View all Source files

Directories

Path Synopsis
Package code provides tree-sitter based code indexing with semantic search.
 Package code provides tree-sitter based code indexing with semantic search.
Package embeddings provides a unified interface for generating embeddings from multiple providers (OpenAI, Google, Ollama, Anthropic/Voyage).
 Package embeddings provides a unified interface for generating embeddings from multiple providers (OpenAI, Google, Ollama, Anthropic/Voyage).
Package events provides temporal event storage with semantic search capabilities.
 Package events provides temporal event storage with semantic search capabilities.
Package kb provides a knowledge base system for storing and searching documents.
 Package kb provides a knowledge base system for storing and searching documents.
Package proxy provides the server-side IPC dispatcher for Remembrances write operations (KB, Events, Code indexing).
 Package proxy provides the server-side IPC dispatcher for Remembrances write operations (KB, Events, Code indexing).
Package treesitter provides AST walking and symbol extraction utilities.
 Package treesitter provides AST walking and symbol extraction utilities.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.