Help us shape the future of Go! Take the 2026 Go Developer Survey

index

package
v0.10.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package index provides the SQLite-backed derived index for VaultMind.

Index

Constants

View Source 
const (
	CallerAgent         = "agent"
	CallerAgentNeighbor = "agent-neighbor"
	CallerHook          = "hook"
)

Caller* constants name the provenance of an access event. The string values land in the note_accesses.caller column and let `vaultmind self` filter "what I engaged with" from "what the harness pre-loaded."

CallerAgent is the default for explicit agent reads (note get, the target of an Ask). CallerAgentNeighbor is set when an Ask's context-pack pulls a neighbor in alongside the target — still real engagement, but lower-intent than a direct read. CallerHook is set when a Claude Code hook (SessionStart persona load, UserPromptSubmit pointer fanout, etc.) fires the access; these accesses populate the activation log but `self` filters them out by default so its hot list reflects deliberate engagement rather than ambient harness traffic.

Set via the VAULTMIND_CALLER env var or passed explicitly to RecordNoteAccessAs. RecordNoteAccess (no caller arg) reads the env and falls back to CallerAgent.

View Source 
const (

	// SparseFormatCurrent is the one stamp the embed path reads (Principle 7).
	// It is not the goose schema version, and it is never inferred from which
	// token ids a stored map contains. 2 means CLS, SEP, PAD, and UNK are
	// absent. Missing or lower means those tokens are still in the stored maps.
	SparseFormatCurrent = 2

	// SparseFormatStaleNotice and SparseFormatStaleRemedy are the words doctor,
	// ask, and a plain index share. One sentence, one command.
	SparseFormatStaleNotice = "sparse vectors are an older format; hybrid ranking still counts special tokens"
	SparseFormatStaleRemedy = "vaultmind index --embed"
)
View Source 
const StatusDraft = "draft"

StatusDraft is the frontmatter status of a note that has not been accepted. Accepting a draft is flipping this value. Other status values are unchanged.

View Source 
const TypeGap = "gap"

TypeGap is the note type of a question the vault did not answer. A gap is not knowledge. vault.GapType is the same string: the two packages cannot import each other, and a vault test locks them.

Variables

This section is empty.

Functions

func ComputeAliasMentions 

func ComputeAliasMentions(db *DB, minAliasLen int) (int, error)

ComputeAliasMentions scans every note body for occurrences of aliases and domain-note titles, then writes alias_mention edges into the links table. It returns the number of new edges inserted. Edges shorter than minAliasLen characters are skipped. Calling this function clears any previous alias_mention edges before computing fresh ones.

func ComputeTagOverlap 

func ComputeTagOverlap(db *DB, threshold float64) (int, error)

ComputeTagOverlap scans the tags table for notes sharing common tags and writes tag_overlap edges into the links table weighted by TF-IDF-style tag specificity. Only pairs whose combined score meets threshold are inserted. Calling this function clears any previous tag_overlap edges before computing fresh ones.

func CountFTS 

func CountFTS(d *DB, query string, filters ...SearchFilters) (int, error)

CountFTS returns the total number of documents matching the query and filters, independent of any limit/offset. Used for pagination totals.

func DecodeColBERTEmbedding 

func DecodeColBERTEmbedding(data []byte, _ int) ([][]float32, error)

DecodeColBERTEmbedding deserializes a ColBERT BLOB back to a per-token matrix. The dims are read from the 4-byte header; the dims parameter is ignored (kept for API compat).

func DecodeEmbedding 

func DecodeEmbedding(data []byte) ([]float32, error)

DecodeEmbedding deserializes raw little-endian bytes back to a float32 slice.

func DecodeSparseEmbedding 

func DecodeSparseEmbedding(data []byte) (map[int32]float32, error)

DecodeSparseEmbedding deserializes packed (int32, float32) pairs back to a sparse map. Returns an empty map (not nil) when data is empty.

func DeleteNoteByPath 

func DeleteNoteByPath(d *DB, path string) error

DeleteNoteByPath removes a note and all its dependent rows from every table within a single transaction. It is used by the incremental indexer to clean up notes whose source files no longer exist on disk.

func DetectEmbeddingDims 

func DetectEmbeddingDims(d *DB) (int, error)

DetectEmbeddingDims returns the dimensionality of stored embeddings, or 0 if none exist. Uses a single-row query — does not load all embeddings.

In mixed-state vaults (some notes embedded with MiniLM, others with BGE-M3 after a model upgrade) this returns whichever row SQLite happens to scan first, so the result is not authoritative for "what model is this vault using." Use DetectEmbeddingDimsCounts when you need the full picture.

func EmbeddedModel added in v0.9.10 

func EmbeddedModel(dbPath string) (string, error)

EmbeddedModel names the model the vault's embeddings were made with: BGE-M3 when any note has sparse or ColBERT vectors, MiniLM when only dense vectors exist, and "" for a vault that was never embedded. A write embeds its note with this model, so one index never mixes models.

func EncodeColBERTEmbedding 

func EncodeColBERTEmbedding(colbert [][]float32) []byte

EncodeColBERTEmbedding serializes a per-token embedding matrix with a 4-byte dims header. Format: [uint32 dims][float32 data...] where data is tokens*dims floats.

func EncodeEmbedding 

func EncodeEmbedding(vec []float32) []byte

EncodeEmbedding serializes a float32 slice to raw little-endian bytes for BLOB storage.

func EncodeSparseEmbedding 

func EncodeSparseEmbedding(sparse map[int32]float32) []byte

EncodeSparseEmbedding serializes a sparse vector as packed (int32 token_id, float32 weight) pairs.

func GapClosures added in v0.10.13 

func GapClosures(db RowQuery) (map[string]GapCredit, error)

GapClosures loads, in one query, the notes that closed a gap. A note is gap-closing when a closed gap stores closed_by as that note id. The values live in frontmatter_kv, JSON-encoded, the same place gap.Measure reads them. Files are not parsed here.

Two closed gaps naming the same note collapse to one entry: the earlier opened_at wins, and an equal opened_at keeps the lower gap id (the query is ordered by gap id). A row that cannot be parsed is skipped. A nil db yields an empty map.

func HasColBERTEmbeddings 

func HasColBERTEmbeddings(d *DB) (bool, error)

HasColBERTEmbeddings returns true if any note has a stored ColBERT embedding.

func HasEmbeddings 

func HasEmbeddings(d *DB) (bool, error)

HasEmbeddings returns true if any note in the index has a stored embedding.

func HasSparseEmbeddings 

func HasSparseEmbeddings(d *DB) (bool, error)

HasSparseEmbeddings returns true if any note has a stored sparse embedding.

func HookCaller added in v0.10.13 

func HookCaller() bool

HookCaller reports whether VAULTMIND_CALLER names a hook. The rule is hookCallerName, the same one resolveCaller uses to record the access as CallerHook, so a new hook script is classified without a second list.

func LoadEmbedding 

func LoadEmbedding(d *DB, noteID string) ([]float32, error)

LoadEmbedding reads the embedding for a single note. Returns nil, nil if no embedding stored.

func PathPrefixLike added in v0.10.1 

func PathPrefixLike(prefix string) string

PathPrefixLike is the LIKE pattern for a vault-path prefix: %, _ and \ in the prefix match literally, and the pattern ends in %. tree --path and ask --path both use it.

func PendingEmbeddings added in v0.9.10 

func PendingEmbeddings(dbPath, model string) (int, error)

PendingEmbeddings counts the notes that have no vectors for model yet — what an incremental embed pass would embed.

func PurgeEmbeddings added in v0.2.2 

func PurgeEmbeddings(dbPath string) (int64, error)

PurgeEmbeddings clears every stored embedding (dense, sparse, colbert) in the vault's index, returning the number of rows cleared. It is what --full does before re-embedding: switching embedding models requires discarding the old vectors, since a mixed-dimension index silently breaks retrieval. All three columns are cleared in ONE UPDATE so the bgem3 modality-parity trigger (which aborts a bge-m3 dense left without sparse+colbert) never fires mid-flight.

func ReadSparseFormat added in v0.10.13 

func ReadSparseFormat(db *DB) (int, error)

ReadSparseFormat returns the stamp. A missing row is 0, the old format.

func RecordNoteAccess 

func RecordNoteAccess(d *DB, noteID string) error

RecordNoteAccess records a note access with the default caller (read from VAULTMIND_CALLER env var, falling back to CallerAgent). Backwards compatible with pre-2026-05-01 callers: the call signature is unchanged, so existing call sites don't have to be rewritten.

Use RecordNoteAccessAs when the caller is known structurally (e.g. query.Ask passes CallerAgent for the target and CallerAgentNeighbor for context-pack neighbors). Use RecordNoteAccess when the caller is determined by the runtime context (e.g. a shell hook setting VAULTMIND_CALLER).

func RecordNoteAccessAs 

func RecordNoteAccessAs(d *DB, noteID, caller string) error

RecordNoteAccessAs records a note access with an explicit caller label. Two side effects:

  1. Inserts into note_accesses with (note_id, caller, accessed_at) — the per-event log that `self` and future ACT-R retrieval scoring read from.
  2. Updates the scalar (notes.access_count, notes.last_accessed_at) — kept for backward compatibility and fast lookup on hot paths.

Best-effort: each per-note tracking miss is the caller's responsibility to log at debug; never fail the user query over optional bookkeeping.

It records no delivery information. Prefer RecordNoteAccessDelivered wherever the answer is known: a nil delivery is stored as NULL, which ListDeliveredNotes reads as "fall back to what caller meant at the time" rather than as a recorded false. That fallback is right for rows written before delivery was tracked and wrong as a habit for new ones.

Both recorders share one write path so the two can never drift into inserting different columns for the same event.

func RecordNoteAccessDelivered added in v0.7.0 

func RecordNoteAccessDelivered(d *DB, noteID, caller string, bodyDelivered bool) error

RecordNoteAccessDelivered records an access and whether it DELIVERED note text, rather than leaving that to be inferred from the caller.

Caller answers "who initiated this" and is still the right axis for deliberate-vs-ambient. It was only ever a stand-in for "what arrived", and that stand-in held solely because the hook and neighbour callers never delivered anything. Once they did, every consumer reading caller as delivery silently inverted — `vaultmind self` began hiding real reads, and hid more of them the better delivery worked.

The bool comes from the same AskResult.DeliveredTo value the experiment ledger records, so both ledgers answer the delivery question from one source instead of two disagreeing heuristics. 

func ResolveLinks(db *DB) (int, error)

ResolveLinks updates unresolved links by matching dst_raw against note IDs, titles, and aliases. Sets dst_note_id and resolved=TRUE for matches.

func SparseFormatStale added in v0.10.13 

func SparseFormatStale(db *DB) (bool, error)

SparseFormatStale reports that stored sparse vectors predate SparseFormatCurrent. A vault with no sparse vectors is not stale: there is nothing old to mix with a query. The decision is the stamp, not token id 0.

func SparseFormatStaleFile added in v0.10.13 

func SparseFormatStaleFile(dbPath string) (bool, error)

SparseFormatStaleFile opens dbPath and reports SparseFormatStale.

func StoreColBERTEmbedding 

func StoreColBERTEmbedding(d *DB, noteID string, colbert [][]float32) error

StoreColBERTEmbedding writes a ColBERT embedding BLOB for a note.

func StoreEmbedding 

func StoreEmbedding(d *DB, noteID string, vec []float32) error

StoreEmbedding writes an embedding BLOB for a note that already exists in the index.

func StoreNote 

func StoreNote(d *DB, rec NoteRecord) error

StoreNote deletes all existing rows for the note, then inserts fresh rows into every table within a single transaction (delete-before-reinsert). StoreNote stores a note within its own transaction.

func StoreNoteInTx 

func StoreNoteInTx(tx *sql.Tx, rec NoteRecord) error

StoreNoteInTx stores a note within an existing transaction. Used by Rebuild for batch transactions.

func StoreSparseEmbedding 

func StoreSparseEmbedding(d *DB, noteID string, sparse map[int32]float32) error

StoreSparseEmbedding writes a sparse embedding BLOB for a note.

func StripForAliasMatch 

func StripForAliasMatch(body string) string

StripForAliasMatch removes markup that should be excluded from alias detection: code fences, inline code, wikilinks (keeping aliased display text), and HTML comments.

func WithEmptyBodyWarn added in v0.10.13 

func WithEmptyBodyWarn(ctx context.Context, allow func(id, path string) bool) context.Context

WithEmptyBodyWarn limits the empty-body warning to notes allow returns true for. A nil allow, and a context without this value, warns about every empty note.

func WriteSparseFormat added in v0.10.13 

func WriteSparseFormat(db *DB, format int) error

WriteSparseFormat stores the stamp. The caller writes it only after every note in the migration has been processed. A model or IO error does not stamp.

The stamp is one row, and an older binary does not know it. That binary can write old-format sparse vectors on another machine and leave the stamp where it is, so this binary will not rewrite them on a plain `index --embed`. Upgrade every machine that writes the vault, then run `vaultmind index --full --embed`.

Types

type BlockRecord 

type BlockRecord struct {
	BlockID   string
	Heading   string
	StartLine int
	EndLine   int
}

BlockRecord represents a block ID anchor for storage.

type BlockRow 

type BlockRow struct {
	BlockID string `json:"block_id"`
	Heading string `json:"heading,omitempty"`
	Line    int    `json:"line"`
}

BlockRow represents a block ID in query results.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB wraps *sql.DB with schema initialization and VaultMind-specific helpers.

func Open 

func Open(dbPath string) (*DB, error)

Open opens (or creates) a SQLite database at dbPath, creates the parent directory if needed, applies the full VaultMind schema, and configures pragmas (WAL mode, foreign key enforcement).

func (*DB) AllNoteTitles 

func (d *DB) AllNoteTitles() ([]NoteTitle, error)

AllNoteTitles returns every note's ID and title from the index. Titles that are empty in the notes table (shouldn't happen post-index but guard anyway) are returned as-is — callers filter if they care.

func (*DB) Begin 

func (d *DB) Begin() (*sql.Tx, error)

Begin starts a transaction.

func (*DB) Close 

func (d *DB) Close() error

Close closes the underlying database connection.

func (*DB) Exec 

func (d *DB) Exec(query string, args ...interface{}) (sql.Result, error)

Exec executes a query that doesn't return rows.

func (*DB) NoteCount added in v0.3.0 

func (d *DB) NoteCount() (int, error)

NoteCount returns how many notes the index holds. Callers use it to decide whether corpus-scale judgements (e.g. noise-floor relevance) are meaningful for this vault at all, so it counts rows rather than loading them.

func (*DB) NoteHashes 

func (d *DB) NoteHashes() (map[string]NoteHashInfo, error)

NoteHashes returns a map of note path → NoteHashInfo for all notes in the database. Used by the incremental indexer to detect changed and deleted notes.

func (*DB) PathOwningID added in v0.4.2 

func (d *DB) PathOwningID(id string) (string, error)

PathOwningID returns the path currently indexed under id, or "" if no row holds it. The incremental indexer needs this because notes.id is UNIQUE and upsertNote resolves a conflict by reassigning path — so without asking first, a second file claiming an existing id takes the row silently.

func (*DB) Query 

func (d *DB) Query(query string, args ...interface{}) (*sql.Rows, error)

Query executes a query that returns rows.

func (*DB) QueryFullNote 

func (d *DB) QueryFullNote(id string) (*FullNote, error)

QueryFullNote returns complete note data including body, headings, blocks, aliases, tags. Uses GROUP_CONCAT subqueries to fold aliases and tags into the main note query, reducing the number of DB round-trips from 6 to 4.

func (*DB) QueryNoteByID 

func (d *DB) QueryNoteByID(id string) (*NoteRow, error)

QueryNoteByID returns the note with the given ID, or nil if not found.

func (*DB) QueryNoteByPath 

func (d *DB) QueryNoteByPath(path string) (*NoteRow, error)

QueryNoteByPath returns the note at the given vault-relative path, or nil.

func (*DB) QueryNotesByAlias 

func (d *DB) QueryNotesByAlias(alias string, normalized bool) ([]NoteRow, error)

QueryNotesByAlias returns notes whose aliases match the given string. If normalized is true, compares against alias_normalized (lowercase, whitespace-collapsed).

func (*DB) QueryNotesByNormalized 

func (d *DB) QueryNotesByNormalized(normalized string) ([]NoteRow, error)

QueryNotesByNormalized searches for notes whose title or alias, when hyphens and underscores are replaced with spaces and lowercased, matches the given normalized input.

func (*DB) QueryNotesByTitle 

func (d *DB) QueryNotesByTitle(title string, caseInsensitive bool) ([]NoteRow, error)

QueryNotesByTitle returns notes matching the given title. If caseInsensitive is true, uses LOWER() comparison.

func (*DB) QueryRow 

func (d *DB) QueryRow(query string, args ...interface{}) *sql.Row

QueryRow executes a query that returns at most one row.

func (*DB) UpdateMTime 

func (d *DB) UpdateMTime(path string, mtime int64) error

UpdateMTime updates the mtime column for the note at the given path. Used when a file's content hash is unchanged but its mtime has changed.

type DraftSet added in v0.10.13 

type DraftSet = NonKnowledge

DraftSet is the non-knowledge set.

func DraftIDs added in v0.10.13 

func DraftIDs(db RowQuery) (DraftSet, error)

DraftIDs loads the non-knowledge set: status draft, or type gap. The set is never nil. A nil db yields an empty set: there is nothing to hide.

func (DraftSet) Contains added in v0.10.13 

func (s DraftSet) Contains(id string) bool

Contains reports whether id is a draft or a gap. A nil set contains nothing, so a caller that forgot to load the set must not treat that as "hide everyone".

func (DraftSet) Visible added in v0.10.13 

func (s DraftSet) Visible(id string, include bool) bool

Visible reports whether id may be delivered as a neighbor, a context item, or a recited note. include is the caller's draft opt-in (app.retrieval.include_drafts). It never reveals a gap. An id the caller named is not passed here: asking for that note is a direct read.

type DuplicateID added in v0.4.2 

type DuplicateID struct {
	ID      string `json:"id"`
	Kept    string `json:"kept"`    // path that holds the id
	Skipped string `json:"skipped"` // path that was not indexed
}

DuplicateID names one id claimed by two files on disk, and which file kept it. The DuplicateIDs counter says how many collisions occurred; without the paths the operator cannot act, and the losing file is NOT in the index at all — a note absent from memory with only a number to show for it.

type EmbedResult 

type EmbedResult struct {
	Embedded    int    `json:"embedded"`
	Skipped     int    `json:"skipped"`
	Errors      int    `json:"errors"`
	EmptyOutput int    `json:"empty_output,omitempty"`
	Model       string `json:"model,omitempty"`
	// SparseReembedded is how many sparse vectors a format upgrade rewrote.
	// Dense and ColBERT bytes are left as they were. Absent when nothing was rewritten.
	SparseReembedded int `json:"sparse_reembedded,omitempty"`
}

EmbedResult holds the outcome of an embedding pass.

EmptyOutput counts notes whose embedder returned without error but with empty Sparse and/or ColBERT outputs — the heads produced no usable tokens. These notes are NOT counted as Embedded (their sparse_embedding / colbert_embedding columns would be NULL); they remain pending for the next run. See vaultmind#22 for the silent-failure pattern this surfaces.

type EmbedderSource added in v0.10.13 

type EmbedderSource interface {
	Borrow(ctx context.Context, model string) (embedding.Embedder, bool)
}

EmbedderSource lends an embedder for a model token. RunEmbed does not Close a borrowed embedder: the source owns the session lifetime. ok is false when the source will not supply that model, and RunEmbed builds one and closes it.

type EmbeddingDimsCount 

type EmbeddingDimsCount struct {
	Dims  int // 384 = MiniLM, 1024 = BGE-M3, 0 = no dense
	Count int
}

EmbeddingDimsCount is one (dimensions, count) pair from a vault.

func DetectEmbeddingDimsCounts 

func DetectEmbeddingDimsCounts(d *DB) ([]EmbeddingDimsCount, error)

DetectEmbeddingDimsCounts returns the count of notes per dense-embedding dimensionality. A consistent vault has exactly one entry; a mixed-state vault (mid-upgrade from MiniLM to BGE-M3, or partial-rebuild) returns multiple. Used by `doctor` to surface mixed state explicitly instead of claiming a single model name. See vaultmind#22 dig.

type FTSResult 

type FTSResult struct {
	ID       string  `json:"id"`
	Type     string  `json:"type"`
	Title    string  `json:"title"`
	Path     string  `json:"path"`
	Snippet  string  `json:"snippet"`
	Score    float64 `json:"score"`
	IsDomain bool    `json:"is_domain_note"`
}

FTSResult represents a single full-text search hit per SRS-09.

func SearchFTS 

func SearchFTS(d *DB, query string, limit, offset int, filters ...SearchFilters) ([]FTSResult, error)

SearchFTS performs a full-text search against the fts_notes table. Returns results ordered by relevance (rank), limited and offset as specified.

type FullNote 

type FullNote struct {
	ID          string                 `json:"id"`
	Type        string                 `json:"type"`
	Path        string                 `json:"path"`
	Title       string                 `json:"title"`
	Frontmatter map[string]interface{} `json:"frontmatter"`
	// Body is the text an agent receives. VaultMind bookkeeping comments are
	// removed. An author HTML comment stays. RawBody is the indexed text
	// before that removal.
	Body string `json:"body,omitempty"`
	// RawBody is the stored body, markers included. note get prints this
	// because it is an explicit raw read. Delivery paths use Body.
	RawBody string `json:"-"`
	// BodyRawUnfilled is true when body_raw IS NULL. An empty string is a
	// stored empty body; only NULL can still be filled by a re-index.
	BodyRawUnfilled bool         `json:"-"`
	Headings        []HeadingRow `json:"headings,omitempty"`
	Blocks          []BlockRow   `json:"blocks,omitempty"`
	IsDomain        bool         `json:"is_domain_note"`
	Aliases         []string     `json:"-"`
	Tags            []string     `json:"-"`
}

FullNote contains all data for a single note.

type GapCredit added in v0.10.13 

type GapCredit struct {
	Question string    `json:"question"`
	OpenedAt time.Time `json:"opened_at"`
	ClosedAt time.Time `json:"closed_at"`
}

GapCredit is the question a note answered, and when that question was open. The map key is the knowledge note id (closed_by), not the gap id.

type HeadingRecord 

type HeadingRecord struct {
	Slug  string
	Level int
	Title string
}

HeadingRecord represents a heading for storage.

type HeadingRow 

type HeadingRow struct {
	Level int    `json:"level"`
	Title string `json:"title"`
	Slug  string `json:"slug"`
}

HeadingRow represents a heading in query results.

type IndexAndEmbedResult 

type IndexAndEmbedResult struct {
	Index *IndexResult `json:"index"`
	Embed *EmbedResult `json:"embed,omitempty"`
}

IndexAndEmbedResult combines index and optional embed results for command output.

type IndexError 

type IndexError struct {
	Path  string `json:"path"`
	Kind  string `json:"kind"` // "read" | "parse" | "store" | "delete"
	Error string `json:"error"`
}

IndexError names a specific per-file failure during Rebuild or Incremental. The counter in IndexResult.Errors tells you *how many* files failed; ErrorDetails tells you WHICH files and WHY — without it, a partial-index failure is an unactionable number (manifesto #3).

type IndexResult 

type IndexResult struct {
	DBPath            string `json:"db_path"`
	Indexed           int    `json:"indexed"`
	DomainNotes       int    `json:"domain_notes"`
	UnstructuredNotes int    `json:"unstructured_notes"`
	Errors            int    `json:"errors"`
	Skipped           int    `json:"skipped"`
	DuplicateIDs      int    `json:"duplicate_ids"`
	Added             int    `json:"added"`
	Updated           int    `json:"updated"`
	Deleted           int    `json:"deleted"`
	// BodiesBackfilled counts unchanged notes whose delivery body (body_raw)
	// was filled by this run. They are also counted in Skipped, but the
	// index did change, so the summary must not call it already current.
	BodiesBackfilled  int                `json:"bodies_backfilled,omitempty"`
	FullRebuild       bool               `json:"full_rebuild"`
	DurationMs        int64              `json:"duration_ms"`
	CompletedAt       string             `json:"completed_at"`
	ErrorDetails      []IndexError       `json:"error_details,omitempty"`
	DuplicateDetails  []DuplicateID      `json:"duplicate_details,omitempty"`
	SkippedSymlinks   []string           `json:"skipped_symlinks,omitempty"`
	PostIndexWarnings []PostIndexWarning `json:"post_index_warnings,omitempty"`
	// SparseFormatStale is set by a plain index (no --embed) when stored sparse
	// vectors predate the current format. The index does not rewrite them and
	// does not load a model. Absent when the stamp is current or there is no
	// sparse column.
	SparseFormatStale bool `json:"sparse_format_stale,omitempty"`
}

IndexResult holds the outcome of an index rebuild.

type Indexer 

type Indexer struct {

	// Progress receives the sparse-format re-embed line. Nil writes to stderr.
	Progress io.Writer
	// contains filtered or unexported fields
}

Indexer orchestrates vault scanning, parsing, and SQLite storage.

func NewIndexer 

func NewIndexer(vaultRoot, dbPath string, cfg *vault.Config) *Indexer

NewIndexer creates an Indexer for the given vault.

func (*Indexer) EmbedNotes 

func (idx *Indexer) EmbedNotes(ctx context.Context, dbPath string, embedder embedding.Embedder) (*EmbedResult, error)

EmbedNotes computes and stores embeddings for all notes that don't have one yet. It opens its own DB connection (like Rebuild/Incremental) so it can be called after the indexer has closed its connection.

func (*Indexer) Incremental 

func (idx *Indexer) Incremental() (*IndexResult, error)

func (*Indexer) IndexFile 

func (idx *Indexer) IndexFile(relPath string) error

IndexFile re-indexes a single file by its vault-relative path.

The path is confined here rather than trusted from the caller. The previous //nolint comment asserted "relPath is vault-relative" — an invariant nothing enforced, true only because every current caller happens to pass a validated path. That is the state H3/H4 were in right up until someone added a caller who did not.

func (*Indexer) Rebuild 

func (idx *Indexer) Rebuild() (*IndexResult, error)

Rebuild performs a full rebuild: scan all .md files, parse, and store.

func (*Indexer) ReembedStaleSparse added in v0.10.13 

func (idx *Indexer) ReembedStaleSparse(ctx context.Context, db *DB, embedder embedding.Embedder, progress io.Writer) (int, error)

ReembedStaleSparse rewrites sparse_embedding for every note that has one when the stamp is older than SparseFormatCurrent. Dense and ColBERT bytes are not touched. The stamp is written once every note has been processed. A model or IO error aborts without stamping. A fresh stamp returns without calling the embedder.

An empty new map is stored as a zero-length blob. That is the new-format value (no positive weights). Migration 006 rejects NULL, which is what EncodeSparseEmbedding emits for an empty map, and accepts the zero-length blob. The sparse retriever scores that blob as no overlap.

func (*Indexer) RunEmbed 

func (idx *Indexer) RunEmbed(ctx context.Context, dbPath, model string, full bool) (*EmbedResult, error)

RunEmbed runs an embedding pass against the index DB. The embedder is constructed lazily — only when there's pending work to do.

Why lazy: the BGE-M3 model is ~2.2GB on disk and CGO+ORT session creation pegs a CPU core for ~1s every time it's invoked. Running `vaultmind index --embed --model bge-m3` against a fully-embedded vault (which happens whenever the user re-runs after editing zero notes — hooks, scripts, retries, doctor checks) used to pay that load cost unconditionally. Heat without work. Counting pending notes first lets us skip the model load entirely when there's nothing to do.

func (*Indexer) SetEmbedderSource added in v0.10.13 

func (idx *Indexer) SetEmbedderSource(src EmbedderSource)

SetEmbedderSource installs the process embedder for this pass. The indexer does not close what Borrow returns.

type LinkRecord 

type LinkRecord struct {
	DstNoteID  string
	DstRaw     string
	EdgeType   string
	TargetKind string
	Heading    string
	BlockID    string
	Resolved   bool
	Confidence string
	Origin     string
	Weight     float64
}

LinkRecord represents a single outbound edge for storage.

type NonKnowledge added in v0.10.13 

type NonKnowledge struct {
	// contains filtered or unexported fields
}

NonKnowledge is every note that must not be delivered as knowledge. Drafts are hidden until the caller opts in. Gaps are always hidden. One set, so ranking and graph delivery cannot disagree (Principle 7).

DraftSet is the historical name. Call sites and the ADR-102 tests keep it.

type NoteAccessStats 

type NoteAccessStats struct {
	NoteID         string
	AccessCount    int
	LastAccessedAt string // RFC3339Nano UTC, empty when never accessed
	Title          string
	NoteType       string
}

NoteAccessStats reports the access counters for a single note. Useful for doctor / debugging / verifying that RecordNoteAccess is firing on the paths it's supposed to. Title and NoteType are populated by ListAccessedNotes so the self-rendering layer can produce human-readable output without a separate join. LookupNoteAccess leaves them empty (single-id callers don't need them).

func ListAccessedNotes 

func ListAccessedNotes(d *DB) ([]NoteAccessStats, error)

ListAccessedNotes returns access stats across all notes with at least one recorded access, sorted newest-first by last access timestamp. Backs `vaultmind self` and any caller that wants "everything that's been touched." For the agent-only filtered view (excluding hook accesses), use ListAccessedNotesByCaller.

Pre-2026-05-01 this read from the scalar columns. Post-migration-007 it reads from the events table so callers see consistent data with the caller-filtered variant.

func ListAccessedNotesByCaller 

func ListAccessedNotesByCaller(d *DB, caller string) ([]NoteAccessStats, error)

ListAccessedNotesByCaller returns access stats restricted to events fired by the given caller. Used by `vaultmind self` to filter out hook fan-outs from the proprioceptive view: the SessionStart hook and per-turn pointer-recall fire RecordNoteAccess across many notes before the agent does any deliberate work, and showing them in the "hot" list pollutes the engagement signal `self` is supposed to surface. Pass an empty string to include all callers (matches ListAccessedNotes behaviour).

The "exclude" semantic — "show all callers EXCEPT X" — is provided by ListAccessedNotesExcludingCaller, which is the shape `self` actually wants ("agent + agent-neighbor, not hook").

func ListAccessedNotesExcludingCaller 

func ListAccessedNotesExcludingCaller(d *DB, excludedCaller string) ([]NoteAccessStats, error)

ListAccessedNotesExcludingCaller returns access stats from all callers EXCEPT the one named. Single-caller exclusion form; see ListAccessedNotesExcludingCallers for the multi-caller form.

func ListAccessedNotesExcludingCallers 

func ListAccessedNotesExcludingCallers(d *DB, excludedCallers []string) ([]NoteAccessStats, error)

ListAccessedNotesExcludingCallers returns access stats from all callers EXCEPT the ones named. Used by `vaultmind self` to filter out *both* hook fan-outs (CallerHook) and Ask context-pack neighbors (CallerAgentNeighbor) so the proprioceptive view reflects only deliberate-target accesses (CallerAgent — Ask top-hit + note get).

Round-1 review caught hook pollution; round-2 review caught the next-louder source: a single Ask fires N+1 access events (target + N neighbors) and an off-target nonsense query's neighbor fan-out dominates the hot list. Both pollutions close at the same caller- dimension layer the schema already provides.

Empty list returns the unfiltered view (matches ListAccessedNotes).

func ListDeliveredNotes added in v0.7.0 

func ListDeliveredNotes(d *DB) ([]NoteAccessStats, error)

ListDeliveredNotes returns access stats counting only accesses that delivered note text — the honest basis for "what have I actually engaged with".

Rows written before delivery was tracked carry NULL, which is NOT the same as false. For those the caller heuristic was correct at the time: an `agent` access really was a deliberate read, a `hook` access really did deliver nothing. So NULL falls back to that original meaning rather than being retroactively asserted in either direction — writing 0 would claim a measurement nobody took, and writing 1 would fabricate deliveries.

func LookupNoteAccess 

func LookupNoteAccess(d *DB, noteID string) (NoteAccessStats, error)

LookupNoteAccess returns the access stats for a single note, or (zero-stats, nil) when the note doesn't exist (deliberately mirrors QueryFullNote's "not found" semantics — caller checks whether NoteID came back populated). Reads from the scalar columns; for caller-aware lookups use the event-table queries directly.

type NoteColBERTEmbedding 

type NoteColBERTEmbedding struct {
	NoteID   string
	ColBERT  [][]float32
	Type     string
	Title    string
	Path     string
	IsDomain bool
}

NoteColBERTEmbedding pairs a note ID with its ColBERT matrix and metadata.

func LoadAllColBERTEmbeddings 

func LoadAllColBERTEmbeddings(d *DB, dims int) ([]NoteColBERTEmbedding, error)

LoadAllColBERTEmbeddings returns all notes that have stored ColBERT embeddings. dims is the embedding dimensionality for decoding.

func LoadColBERTEmbeddingsFor added in v0.9.16 

func LoadColBERTEmbeddingsFor(d *DB, dims int, ids []string) ([]NoteColBERTEmbedding, error)

LoadColBERTEmbeddingsFor returns the ColBERT embeddings of the named notes only. ColBERT is nearly all of a BGE-M3 index (921 of 926 MB on a 416-note vault), so reading just the candidates is most of what narrowing saves. Unknown ids and notes without ColBERT are skipped.

type NoteEmbedding 

type NoteEmbedding struct {
	NoteID    string
	Embedding []float32
	Type      string
	Title     string
	Path      string
	IsDomain  bool
}

NoteEmbedding pairs a note ID with its embedding vector and metadata.

func LoadAllEmbeddings 

func LoadAllEmbeddings(d *DB) ([]NoteEmbedding, error)

LoadAllEmbeddings returns all notes that have stored embeddings, including metadata. This is a single query that avoids N+1 lookups when scoring and filtering results.

type NoteHashInfo 

type NoteHashInfo struct {
	Hash       string
	MTime      int64
	HasRawBody bool
}

NoteHashInfo holds the content hash and modification time for a note.

type NoteRecord 

type NoteRecord struct {
	ID       string
	Path     string
	Title    string
	Type     string
	Status   string
	Created  string
	Updated  string
	BodyText string // Search-normalized text: FTS and embedding input.
	BodyRaw  string // Parsed body as written: delivery and display.
	Hash     string
	MTime    int64
	IsDomain bool
	Aliases  []string
	Tags     []string
	ExtraKV  map[string]interface{}
	Links    []LinkRecord
	Headings []HeadingRecord
	Blocks   []BlockRecord
}

NoteRecord is the storage-ready representation of a parsed note. The indexer builds this from parser.ParsedNote + file metadata.

type NoteRow 

type NoteRow struct {
	ID       string
	Type     string
	Title    string
	Path     string
	Status   string
	IsDomain bool
}

NoteRow is a lightweight note record returned by query methods.

type NoteSparseEmbedding 

type NoteSparseEmbedding struct {
	NoteID   string
	Sparse   map[int32]float32
	Type     string
	Title    string
	Path     string
	IsDomain bool
}

NoteSparseEmbedding pairs a note ID with its sparse vector and metadata.

func LoadAllSparseEmbeddings 

func LoadAllSparseEmbeddings(d *DB) ([]NoteSparseEmbedding, error)

LoadAllSparseEmbeddings returns all notes that have stored sparse embeddings.

type NoteTitle 

type NoteTitle struct {
	ID    string
	Title string
}

NoteTitle pairs a note's ID with its display title. Used by callers that need to list notes by title without loading full frontmatter/body (e.g. the ask command's fuzzy-title fallback on zero hits).

type PostIndexWarning 

type PostIndexWarning struct {
	Step  string `json:"step"`
	Error string `json:"error"`
}

PostIndexWarning reports failure of a post-store pass (link resolution, alias detection, tag overlap). These run after the note-store transaction commits; their failure leaves a partially-connected graph the operator can't distinguish from a successful run without this surface.

Conventional Step values: "orphan_sweep", "link_resolution", "alias_mention", "tag_overlap".

type RowQuery added in v0.10.13 

type RowQuery interface {
	Query(query string, args ...interface{}) (*sql.Rows, error)
}

RowQuery is the read surface DraftIDs needs. *DB satisfies it, and so does any other querier with the same Query method.

type SearchFilters 

type SearchFilters struct {
	Type          string // Filter by note type (empty = no filter)
	Tag           string // Filter by tag (empty = no filter)
	PathPrefix    string // Only notes whose vault path starts with this prefix (empty = no filter)
	IncludeDrafts bool   // Rank notes whose status is draft. Off by default.
}

SearchFilters holds optional filters for FTS search.

func (SearchFilters) ExcludesDrafts added in v0.10.13 

func (f SearchFilters) ExcludesDrafts() bool

ExcludesDrafts reports whether a search drops status:draft notes. Every retrieval lane calls this, so CLI, MCP, serve and federation agree.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.