Documentation
¶
Overview ¶
Package index implements the SQLite FTS5 shadow index that memory.fetch_context searches and propose_update updates incrementally.
Production version of spike S4 (spikes/s4-fts5-incremental). The two PRAGMAs identified there (journal_mode=WAL, synchronous=NORMAL) are non-optional: without them per-COMMIT fsync dominates and per-update cost blows past the <10ms budget.
See docs/patterns/sqlite-fts5-shadow-index.md and design doc v0.4.1 §20.
Index ¶
- Constants
- Variables
- type FileDoc
- type Index
- func (i *Index) Close() error
- func (i *Index) CountFiles(ctx context.Context) (int, error)
- func (i *Index) CountSections(ctx context.Context) (int, error)
- func (i *Index) DeleteFile(ctx context.Context, file string) error
- func (i *Index) DeleteSections(ctx context.Context, file string, sectionIDs []string) error
- func (i *Index) GetFile(ctx context.Context, file string) (FileDoc, error)
- func (i *Index) GetSection(ctx context.Context, file, sectionID string) (SectionInfo, error)
- func (i *Index) IndexFile(ctx context.Context, memDir, relPath string, cat schema.Category, ...) error
- func (i *Index) Init(ctx context.Context) error
- func (i *Index) ListFiles(ctx context.Context, categoryFilter string) ([]FileDoc, error)
- func (i *Index) ListSections(ctx context.Context, file string) ([]SectionInfo, error)
- func (i *Index) Path() string
- func (i *Index) RebuildAll(ctx context.Context, memDir string, sch *schema.Schema, opts RebuildOpts) error
- func (i *Index) Search(ctx context.Context, query string, limit int) ([]SearchResult, error)
- func (i *Index) UpsertFile(ctx context.Context, doc FileDoc) error
- func (i *Index) UpsertSections(ctx context.Context, sections []SectionDoc) error
- func (i *Index) Version(ctx context.Context) (int, error)
- type RankingContext
- type RebuildOpts
- type SearchResult
- type SectionDoc
- type SectionInfo
Constants ¶
const ( // ScopeBoost is applied to results whose file matches any of the // caller-supplied scope strings (substring match). ScopeBoost = 2.0 // FreshBoost is applied to results in files marked fresh in the // FreshFiles map of RankingContext. FreshBoost = 1.5 // ArchivePenalty is applied to results whose file lives under the // archive path prefix. ArchivePenalty = 0.4 // StalePenalty is applied to results in files marked stale. StalePenalty = 0.6 // ActiveBranchBoost is applied to results whose section content // references the active branch name (e.g., a decision scoped to a // feature branch). Suppressed for generic integration branches. ActiveBranchBoost = 1.3 // ChangedRefBoost is applied to decisions/pitfalls whose content // references a file with uncommitted changes — surface the prior art // for whatever you're touching right now. ChangedRefBoost = 1.4 // LowConfidencePenalty is applied to sections that declare a low-trust // confidence (inferred / stale / unknown). LowConfidencePenalty = 0.8 )
Multiplier constants from design doc v0.4.1 §20.4. BM25 scores in SQLite FTS5 are NEGATIVE (more negative = better match), so multiplication acts in the natural direction:
- boost (multiplier > 1) → score becomes MORE negative → ranks higher
- penalty (multiplier < 1) → score becomes LESS negative → ranks lower
Example: a result with raw BM25 score -10.0
×2.0 boost → -20.0 (better) ×0.4 penalty → -4.0 (worse)
const Schema = `` /* 864-byte string literal not displayed */
Schema mirrors design doc v0.4.1 §20.2. Three tables:
memory_search — FTS5 virtual table; the search hot path.
memory_sections — byte offsets + content hash per (file, section_id).
The fetch path reads this to splice section bytes out
of the source Markdown file.
memory_docs — per-file metadata for ranking signals (freshness,
archived, local_state, etc.).
const SchemaVersion = 1
SchemaVersion is the value Init stores in PRAGMA user_version. Bumping it signals a schema migration; the migrator (deferred to v0.5) compares the stored value against this constant on open.
Variables ¶
var ErrNotFound = errors.New("index: row not found")
ErrNotFound is returned when GetSection / GetFile can't find the row.
Functions ¶
This section is empty.
Types ¶
type FileDoc ¶
type FileDoc struct {
File string
Category string
Freshness string
Confidence string
LastModified string // RFC3339; empty if unknown
Committed bool
LocalState bool
Archived bool
SizeBytes int
Checksum string
}
FileDoc bundles per-file metadata for ranking + status reporting.
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index is an open SQLite shadow index. Not safe for concurrent calls from multiple goroutines (the underlying *sql.DB is configured with SetMaxOpenConns(1) to match the single-writer concurrency model in v0.4.1 §11).
func Open ¶
Open opens (or creates) the index at path. WAL + synchronous=NORMAL are applied via the connection URI so every connection database/sql hands out has them set.
func (*Index) Close ¶
Close closes the underlying database. Idempotent — calling on a half-closed Index returns nil.
func (*Index) CountFiles ¶
CountFiles returns the total number of rows in memory_docs.
func (*Index) CountSections ¶
CountSections returns the total number of rows in memory_sections. Used by tests and status reporting.
func (*Index) DeleteFile ¶
DeleteFile removes ALL rows for the given file across the three tables. Used when a memory file is removed from the layout (e.g., archive flow).
func (*Index) DeleteSections ¶
DeleteSections removes the listed (file, section_id) rows from both memory_search and memory_sections in a single transaction. An empty sectionIDs slice is a no-op.
func (*Index) GetFile ¶
GetFile returns the memory_docs row for file. Returns ErrNotFound when missing.
func (*Index) GetSection ¶
GetSection returns the memory_sections row for a (file, section_id) pair. Returns ErrNotFound if the row doesn't exist.
func (*Index) IndexFile ¶
func (i *Index) IndexFile(ctx context.Context, memDir, relPath string, cat schema.Category, opts RebuildOpts) error
IndexFile reads, optionally assigns missing anchor IDs, parses, and indexes a single file. The full path on disk is memDir/relPath (relPath in forward-slash form).
If opts.AssignMissingIDs is true AND the category requires section IDs, AssignMissingIDs runs first; modified bytes are written back atomically before parsing. Files that don't need assignments are not touched.
func (*Index) Init ¶
Init applies the schema (idempotent via IF NOT EXISTS) and writes the SchemaVersion to PRAGMA user_version.
func (*Index) ListFiles ¶
ListFiles returns memory_docs entries; if categoryFilter is non-empty, only rows whose category matches are returned.
func (*Index) ListSections ¶
ListSections returns every section for a file in document order (ascending byte_start).
func (*Index) RebuildAll ¶
func (i *Index) RebuildAll(ctx context.Context, memDir string, sch *schema.Schema, opts RebuildOpts) error
RebuildAll wipes the index and re-walks memDir, indexing every Markdown file that maps to a schema category. memDir is the absolute path to .agent-memory/.
The two pre-existing rows for any file present in the wipe survive — we fully reset the three index tables and rebuild from disk. This is the canonical "make everything consistent" operation; M3's incremental path (per-file upserts after propose_update) keeps the index fresh between rebuilds.
func (*Index) Search ¶
Search runs an FTS5 MATCH query and returns hits sorted by ascending BM25 score (FTS5 convention: lower = better). limit ≤ 0 falls back to 50.
The raw query is treated as natural language, NOT as FTS5 query syntax: sanitizeFTSMatch tokenizes it and quotes each term, so punctuation, hyphens (`auto-apply`), and reserved words (AND/OR/NEAR) are matched literally instead of crashing the parser with a syntax/"no such column" error. Terms are OR-ed (match any) and ranked by BM25, so a multi-word natural-language query retrieves the best partial matches. A query with no alphanumeric tokens returns no results without error — callers (the fetch pipeline) handle the empty case by returning the bootstrap pack.
func (*Index) UpsertFile ¶
UpsertFile inserts or updates the memory_docs row for file.
func (*Index) UpsertSections ¶
func (i *Index) UpsertSections(ctx context.Context, sections []SectionDoc) error
UpsertSections replaces (file, section_id) rows for every entry in `sections`. Each row is handled in one DB transaction:
DELETE FROM memory_search WHERE file=? AND section_id=? INSERT INTO memory_search (...) VALUES (...) INSERT INTO memory_sections (...) VALUES (...) ON CONFLICT DO UPDATE
FTS5 has no ON CONFLICT, so memory_search uses DELETE-then-INSERT. memory_sections uses standard UPSERT.
An empty slice is a no-op (no transaction opened).
type RankingContext ¶
type RankingContext struct {
// Scope: each entry is checked as a substring of the result's file
// path. A hit applies ScopeBoost. Multiple scope entries → boost
// applies if ANY matches (boost is applied at most once per result).
Scope []string
// ArchivePathPrefix marks the archive directory. Defaults to
// "archive/" if empty. Results whose File starts with this prefix
// receive ArchivePenalty.
ArchivePathPrefix string
// FreshFiles → results in these files get FreshBoost. Lookup map.
FreshFiles map[string]bool
// StaleFiles → results in these files get StalePenalty. Lookup map.
StaleFiles map[string]bool
// ActiveBranch is the current git branch name. A result whose section
// content references it earns ActiveBranchBoost. Empty / generic
// integration branches (main, master, …) apply no boost.
ActiveBranch string
// ChangedFiles are repo-relative paths with uncommitted changes. A
// decision/pitfall section referencing any of them earns ChangedRefBoost.
ChangedFiles []string
}
RankingContext bundles the inputs ApplyRankingSignals needs. All fields are optional; an empty RankingContext leaves BM25 ordering unchanged.
type RebuildOpts ¶
type RebuildOpts struct {
// AssignMissingIDs runs internal/markdown.AssignMissingIDs on every
// file whose category has SectionIDRequired=true before indexing.
// Mutates the file on disk via atomic write; only modified files are
// touched. Idempotent — running rebuild twice produces the same
// on-disk state.
AssignMissingIDs bool
}
RebuildOpts configures RebuildAll.
type SearchResult ¶
type SearchResult struct {
File string
SectionID string
Title string
Headings string
Snippet string // FTS5 snippet() output, marked with [..]
Score float64 // bm25() — lower is better in FTS5
Content string // full indexed section body; feeds content-based ranking signals
}
SearchResult is one hit from a Search query, ranked by BM25.
func ApplyRankingSignals ¶
func ApplyRankingSignals(results []SearchResult, rctx RankingContext) []SearchResult
ApplyRankingSignals modifies the Score field of each result per the multipliers documented in design doc §20.4, then sorts results by ascending Score (FTS5 convention: lower = better). Returns the same slice for caller convenience.
File-level signals (scope, freshness, archive, stale) key off the result's path; content-level signals (active-branch reference, changed-file reference, low confidence) inspect the indexed section body (SearchResult.Content).
Multipliers compose: a result that's in scope AND stale gets score × ScopeBoost × StalePenalty.
The order in which multipliers are applied does not affect the final score (multiplication is commutative); the iteration order in this implementation is fixed for predictability when debugging.
type SectionDoc ¶
type SectionDoc struct {
File string
SectionID string
Heading string
HeadingLevel int
Title string // FTS5: typically equals Heading
Headings string // FTS5: breadcrumb of ancestor headings (future use)
Content string // FTS5: section body
Tags string // FTS5: space-separated tag tokens (future use)
ByteStart int
ByteEnd int
ContentHash string
}
SectionDoc bundles the per-section data the index needs. Field names mirror the FTS5 + memory_sections columns; the index doesn't try to validate them further (e.g., that ByteEnd > ByteStart) — that's the caller's job.
type SectionInfo ¶
type SectionInfo struct {
File string
SectionID string
Heading string
HeadingLevel int
ByteStart int
ByteEnd int
ContentHash string
}
SectionInfo mirrors a memory_sections row. Returned by GetSection / ListSections; used by the fetch path to splice section bytes out of the source Markdown file.
Source Files