Documentation
¶
Overview ¶
Package index implements the SQLite FTS5 shadow index that memory.fetch_context searches and propose_update updates incrementally.
Production version of spike S4 (spikes/s4-fts5-incremental). The two PRAGMAs identified there (journal_mode=WAL, synchronous=NORMAL) are non-optional: without them per-COMMIT fsync dominates and per-update cost blows past the <10ms budget.
See docs/patterns/sqlite-fts5-shadow-index.md and design doc v0.4.1 §20.
Index ¶
- Constants
- Variables
- type FileDoc
- type Index
- func (i *Index) Close() error
- func (i *Index) CountFiles(ctx context.Context) (int, error)
- func (i *Index) CountSections(ctx context.Context) (int, error)
- func (i *Index) DeleteFile(ctx context.Context, file string) error
- func (i *Index) DeleteSections(ctx context.Context, file string, sectionIDs []string) error
- func (i *Index) GetFile(ctx context.Context, file string) (FileDoc, error)
- func (i *Index) GetSection(ctx context.Context, file, sectionID string) (SectionInfo, error)
- func (i *Index) IndexFile(ctx context.Context, store, baseDir, relPath string, cat schema.Category, ...) error
- func (i *Index) Init(ctx context.Context) error
- func (i *Index) ListFiles(ctx context.Context, categoryFilter string) ([]FileDoc, error)
- func (i *Index) ListSections(ctx context.Context, file string) ([]SectionInfo, error)
- func (i *Index) Path() string
- func (i *Index) RebuildAll(ctx context.Context, memDir string, sch *schema.Schema, opts RebuildOpts) error
- func (i *Index) Search(ctx context.Context, query string, limit int) ([]SearchResult, error)
- func (i *Index) SearchPerStore(ctx context.Context, query string, kPerStore int, stores []string) ([]SearchResult, error)
- func (i *Index) UpsertFile(ctx context.Context, doc FileDoc) error
- func (i *Index) UpsertSections(ctx context.Context, sections []SectionDoc) error
- func (i *Index) Version(ctx context.Context) (int, error)
- type RankingContext
- type RebuildOpts
- type SearchResult
- type SectionDoc
- type SectionInfo
Constants ¶
const ( // ScopeBoost is applied to results whose file matches any of the // caller-supplied scope strings (substring match). ScopeBoost = 2.0 // FreshBoost is applied to results in files marked fresh in the // FreshFiles map of RankingContext. FreshBoost = 1.5 // ArchivePenalty is applied to results whose file lives under the // archive path prefix. ArchivePenalty = 0.4 // StalePenalty is applied to results in files marked stale. StalePenalty = 0.6 // ActiveBranchBoost is applied to results whose section content // references the active branch name (e.g., a decision scoped to a // feature branch). Suppressed for generic integration branches. ActiveBranchBoost = 1.3 // ChangedRefBoost is applied to decisions/pitfalls whose content // references a file with uncommitted changes — surface the prior art // for whatever you're touching right now. ChangedRefBoost = 1.4 // LowConfidencePenalty is applied to sections that declare a low-trust // confidence (inferred / stale / unknown). LowConfidencePenalty = 0.8 )
Multiplier constants from design doc v0.4.1 §20.4. BM25 scores in SQLite FTS5 are NEGATIVE (more negative = better match), so multiplication acts in the natural direction:
- boost (multiplier > 1) → score becomes MORE negative → ranks higher
- penalty (multiplier < 1) → score becomes LESS negative → ranks lower
Example: a result with raw BM25 score -10.0
×2.0 boost → -20.0 (better) ×0.4 penalty → -4.0 (worse)
const LocalStore = "local"
LocalStore is the store name for the consuming repo's own .agent-memory/ content — every row that existed before federation. All the pre-PR4 query methods scope to this store, so an index with no cached external stores behaves exactly as it did before (the federation opt-in invariant). Cached external stores are indexed under their manifest name. Kept in sync with the reserved name rejected by config.validateStores.
const Schema = `` /* 990-byte string literal not displayed */
Schema mirrors design doc v0.4.1 §20.2, extended with the federation `store` dimension (docs/design/federated-memory.md, PR4). Three tables:
memory_search — FTS5 virtual table; the search hot path.
memory_sections — byte offsets + content hash per (store, file, section_id).
The fetch path reads this to splice section bytes out
of the source Markdown file.
memory_docs — per-file metadata for ranking signals (freshness,
archived, local_state, etc.).
`store` names which memory the row came from: LocalStore for the consuming repo's own .agent-memory/, or a manifest store name for a cached external "landscape" store (meta/cache/stores/<name>/). It is UNINDEXED in the FTS5 table — stored and filterable with `store = ?`, but never tokenized, so a store name can't pollute MATCH relevance. It is the LAST FTS5 column so the existing positional snippet() index (content = column 4) is unchanged.
const SchemaVersion = 2
SchemaVersion is the value Init stores in PRAGMA user_version. Bumping it triggers a rebuild-on-version-bump: Init drops the old tables and recreates the current schema (the index is a rebuildable cache, and the FTS5 column set plus the composite primary keys can't be migrated in place). The caller then repopulates via RebuildAll — every index-opening path guards on an empty index, so a migrated index self-heals on first use.
v1 → v2 (PR4): added the `store` dimension to all three tables.
Variables ¶
var ErrNotFound = errors.New("index: row not found")
ErrNotFound is returned when GetSection / GetFile can't find the row.
Functions ¶
This section is empty.
Types ¶
type FileDoc ¶
type FileDoc struct {
Store string // memory this file belongs to; empty defaults to LocalStore
File string
Category string
Freshness string
Confidence string
LastModified string // RFC3339; empty if unknown
Committed bool
LocalState bool
Archived bool
SizeBytes int
Checksum string
}
FileDoc bundles per-file metadata for ranking + status reporting.
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index is an open SQLite shadow index. Not safe for concurrent calls from multiple goroutines (the underlying *sql.DB is configured with SetMaxOpenConns(1) to match the single-writer concurrency model in v0.4.1 §11).
func Open ¶
Open opens (or creates) the index at path. WAL + synchronous=NORMAL are applied via the connection URI so every connection database/sql hands out has them set.
func (*Index) Close ¶
Close closes the underlying database. Idempotent — calling on a half-closed Index returns nil.
func (*Index) CountFiles ¶
CountFiles returns the total number of rows in memory_docs.
func (*Index) CountSections ¶
CountSections returns the total number of rows in memory_sections. Used by tests and status reporting.
func (*Index) DeleteFile ¶
DeleteFile removes ALL rows for the given local file across the three tables. Used when a memory file is removed from the layout (e.g., archive flow). Scoped to LocalStore for the same reason as DeleteSections.
func (*Index) DeleteSections ¶
DeleteSections removes the listed (file, section_id) rows from both memory_search and memory_sections in a single transaction. An empty sectionIDs slice is a no-op.
Scoped to LocalStore: the incremental apply path only ever removes the consuming repo's own sections. Cached external stores are only ever rebuilt wholesale by sync → RebuildAll, never edited section-by-section.
func (*Index) GetFile ¶
GetFile returns the memory_docs row for file. Returns ErrNotFound when missing.
func (*Index) GetSection ¶
GetSection returns the memory_sections row for a (file, section_id) pair. Returns ErrNotFound if the row doesn't exist.
func (*Index) IndexFile ¶
func (i *Index) IndexFile(ctx context.Context, store, baseDir, relPath string, cat schema.Category, opts RebuildOpts) error
IndexFile reads, optionally assigns missing anchor IDs, parses, and indexes a single file under the given store. The full path on disk is baseDir/relPath (relPath in forward-slash form).
AssignMissingIDs runs only for the local store AND only when the category requires section IDs: cached external stores are read-only copies we never mutate. When it runs, modified bytes are written back atomically before parsing; files that don't need assignments are not touched.
func (*Index) Init ¶
Init applies the schema (idempotent via IF NOT EXISTS) and writes the SchemaVersion to PRAGMA user_version.
Rebuild-on-version-bump: if the on-disk index was written by an older schema version, Init drops the stale tables before recreating them, leaving an empty index for the caller to repopulate via RebuildAll. We never ALTER in place — the FTS5 column set and the composite primary keys changed in v2, neither of which SQLite can migrate incrementally. A fresh database (user_version 0) and an already-current one are both handled by the plain CREATE IF NOT EXISTS.
func (*Index) ListFiles ¶
ListFiles returns the local store's memory_docs entries; if categoryFilter is non-empty, only rows whose category matches are returned. Cached external stores are excluded (status/index generation are local-only in PR4).
func (*Index) ListSections ¶
ListSections returns every section for a local file in document order (ascending byte_start).
func (*Index) RebuildAll ¶
func (i *Index) RebuildAll(ctx context.Context, memDir string, sch *schema.Schema, opts RebuildOpts) error
RebuildAll wipes the index and rebuilds it from disk: first the local store (the consuming repo's own .agent-memory/, rooted at memDir), then every cached external "landscape" store under meta/cache/stores/<name>/ (federation, PR4). memDir is the absolute path to .agent-memory/.
This is the canonical "make everything consistent" operation; the incremental path (per-file upserts after propose_update) keeps the local store fresh between rebuilds, and sync rebuilds the cached stores. With no cached stores the cache dir is absent and only the local store is indexed — identical to pre-federation behavior (the opt-in invariant).
func (*Index) Search ¶
Search runs an FTS5 MATCH query and returns hits sorted by ascending BM25 score (FTS5 convention: lower = better). limit ≤ 0 falls back to 50.
The raw query is treated as natural language, NOT as FTS5 query syntax: sanitizeFTSMatch tokenizes it and quotes each term, so punctuation, hyphens (`auto-apply`), and reserved words (AND/OR/NEAR) are matched literally instead of crashing the parser with a syntax/"no such column" error. Terms are OR-ed (match any) and ranked by BM25, so a multi-word natural-language query retrieves the best partial matches. A query with no alphanumeric tokens returns no results without error — callers (the fetch pipeline) handle the empty case by returning the bootstrap pack.
func (*Index) SearchPerStore ¶ added in v0.5.0
func (i *Index) SearchPerStore(ctx context.Context, query string, kPerStore int, stores []string) ([]SearchResult, error)
SearchPerStore runs the FTS query independently within each named store and returns up to kPerStore hits per store (per-store-fair retrieval: a single noisy store can't crowd out the others in a global top-N). Every result carries its Store; ordering within a store is ascending BM25. The caller (PR5 fetch) merges and re-ranks across stores, applying each store's priority multiplier. An empty stores slice, or a query with no alphanumeric tokens, returns nil. Unknown store names simply contribute no rows.
func (*Index) UpsertFile ¶
UpsertFile inserts or updates the memory_docs row for (store, file).
func (*Index) UpsertSections ¶
func (i *Index) UpsertSections(ctx context.Context, sections []SectionDoc) error
UpsertSections replaces (file, section_id) rows for every entry in `sections`. Each row is handled in one DB transaction:
DELETE FROM memory_search WHERE file=? AND section_id=? INSERT INTO memory_search (...) VALUES (...) INSERT INTO memory_sections (...) VALUES (...) ON CONFLICT DO UPDATE
FTS5 has no ON CONFLICT, so memory_search uses DELETE-then-INSERT. memory_sections uses standard UPSERT.
An empty slice is a no-op (no transaction opened).
type RankingContext ¶
type RankingContext struct {
// Scope: each entry is checked as a substring of the result's file
// path. A hit applies ScopeBoost. Multiple scope entries → boost
// applies if ANY matches (boost is applied at most once per result).
Scope []string
// ArchivePathPrefix marks the archive directory. Defaults to
// "archive/" if empty. Results whose File starts with this prefix
// receive ArchivePenalty.
ArchivePathPrefix string
// FreshFiles → results in these files get FreshBoost. Lookup map.
FreshFiles map[string]bool
// StaleFiles → results in these files get StalePenalty. Lookup map.
StaleFiles map[string]bool
// ActiveBranch is the current git branch name. A result whose section
// content references it earns ActiveBranchBoost. Empty / generic
// integration branches (main, master, …) apply no boost.
ActiveBranch string
// ChangedFiles are repo-relative paths with uncommitted changes. A
// decision/pitfall section referencing any of them earns ChangedRefBoost.
ChangedFiles []string
}
RankingContext bundles the inputs ApplyRankingSignals needs. All fields are optional; an empty RankingContext leaves BM25 ordering unchanged.
File-level signals (FreshFiles / StaleFiles, scope, archive) are keyed by file path ALONE. That is correct while the fetch path is local-only (PR4): every ranked result comes from the local store. PR5 (multi-store fetch) MUST re-key these by (store, file) before merging results — otherwise a file path present in more than one store (e.g. "decisions.md" in both local and a landscape store) would collect another store's freshness boost. SearchResult already carries Store for exactly this.
type RebuildOpts ¶
type RebuildOpts struct {
// AssignMissingIDs runs internal/markdown.AssignMissingIDs on every
// file whose category has SectionIDRequired=true before indexing.
// Mutates the file on disk via atomic write; only modified files are
// touched. Idempotent — running rebuild twice produces the same
// on-disk state.
AssignMissingIDs bool
}
RebuildOpts configures RebuildAll.
type SearchResult ¶
type SearchResult struct {
Store string // memory the hit came from (LocalStore or a cached store name)
File string
SectionID string
Title string
Headings string
Snippet string // FTS5 snippet() output, marked with [..]
Score float64 // bm25() — lower is better in FTS5
Content string // full indexed section body; feeds content-based ranking signals
}
SearchResult is one hit from a Search query, ranked by BM25.
func ApplyRankingSignals ¶
func ApplyRankingSignals(results []SearchResult, rctx RankingContext) []SearchResult
ApplyRankingSignals modifies the Score field of each result per the multipliers documented in design doc §20.4, then sorts results by ascending Score (FTS5 convention: lower = better). Returns the same slice for caller convenience.
File-level signals (scope, freshness, archive, stale) key off the result's path; content-level signals (active-branch reference, changed-file reference, low confidence) inspect the indexed section body (SearchResult.Content).
Multipliers compose: a result that's in scope AND stale gets score × ScopeBoost × StalePenalty.
The order in which multipliers are applied does not affect the final score (multiplication is commutative); the iteration order in this implementation is fixed for predictability when debugging.
type SectionDoc ¶
type SectionDoc struct {
// Store names the memory this section belongs to. Empty defaults to
// LocalStore (the consuming repo) — the incremental propose/apply path
// only ever touches the local store, so its callers can leave it unset.
Store string
File string
SectionID string
Heading string
HeadingLevel int
Title string // FTS5: typically equals Heading
Headings string // FTS5: breadcrumb of ancestor headings (future use)
Content string // FTS5: section body
Tags string // FTS5: space-separated tag tokens (future use)
ByteStart int
ByteEnd int
ContentHash string
}
SectionDoc bundles the per-section data the index needs. Field names mirror the FTS5 + memory_sections columns; the index doesn't try to validate them further (e.g., that ByteEnd > ByteStart) — that's the caller's job.
type SectionInfo ¶
type SectionInfo struct {
File string
SectionID string
Heading string
HeadingLevel int
ByteStart int
ByteEnd int
ContentHash string
}
SectionInfo mirrors a memory_sections row. Returned by GetSection / ListSections; used by the fetch path to splice section bytes out of the source Markdown file.
Source Files