Documentation
¶
Overview ¶
Package index is the local index: a SQLite store of markdown files and their embedded chunks, plus the incremental (re)indexer that keeps it in sync with a directory tree. The driver is modernc.org/sqlite (pure Go — no cgo for the storage side; cgo is only ever pulled in by internal/embed for ONNX inference).
Index ¶
- func DefaultDBPath(vault string) string
- type ChunkRow
- type Embedder
- type FileMeta
- type LinkRow
- type Progress
- type Report
- type Stats
- type Store
- func (s *Store) AllChunks(pathPrefix string) ([]ChunkRow, error)
- func (s *Store) AllFiles() ([]string, error)
- func (s *Store) AllLinks() ([]LinkRow, error)
- func (s *Store) Close() error
- func (s *Store) EmbedStamp() (string, error)
- func (s *Store) FileHeadings() (map[string]map[string]bool, error)
- func (s *Store) RebuildReason(canEmbed bool) (string, error)
- func (s *Store) Reindex(root string, embedFn Embedder, force bool) (*Report, error)
- func (s *Store) SetProgress(fn Progress)
- func (s *Store) Stats() (Stats, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultDBPath ¶
DefaultDBPath is where the index lives for a given vault root: <vault>/.semantic/index.db. Per-vault, gitignorable, travels with the tree.
Types ¶
type ChunkRow ¶
type ChunkRow struct {
RelPath string
Key string
Heading string
Variant string
Text string
Line int
Vec embed.Vec
}
ChunkRow is one chunk joined to its file, as returned to the search layer.
type Embedder ¶
Embedder turns chunk text into a vector. Injected so the indexer is testable without the ONNX runtime; production passes embed.Get. A nil Embedder opts a Reindex call into skip-embed mode: chunks are stored with a zero-length placeholder vec instead, and a later Reindex with a real Embedder detects and heals them regardless of whether the file's stat or content hash otherwise looks unchanged.
type FileMeta ¶
FileMeta is the on-disk identity of a markdown file: its path relative to the vault root (forward-slashed) plus the stat + content-hash used to decide whether it needs re-embedding.
type LinkRow ¶
LinkRow is one stored outbound link joined to its source file's rel-path, as returned to the graph layer.
type Progress ¶ added in v0.1.3
Progress is called once per file as Reindex walks the tree, with the file's vault-relative path and how many files have been reached so far. There is no total: the walk discovers the tree as it goes, and counting it first would mean a second full traversal to make a denominator nobody needs to see a command is alive.
type Report ¶
type Report struct {
Added int // files newly indexed
Updated int // files whose content changed and were re-embedded
Unchanged int // files skipped (stat match, or hash match after a touch)
Relinked int // files whose links were re-extracted without re-embedding
Deleted int // files removed from the index (gone from disk)
Files int // total files in the index afterward
Chunks int // total chunks in the index afterward
Duration time.Duration
// Rebuild explains why this run redid work the content hashes said was
// unnecessary — an upgrade that changed the chunker, the link extractor, or
// the embedding model. Empty on an ordinary incremental run. Callers should
// surface it: an automatic rebuild is a long pause on a command that is
// normally instant, and silence makes it look like a hang.
Rebuild string
}
Report summarizes a Reindex run.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store wraps the SQLite index database.
func Open ¶
Open opens (creating if needed) the index database at path, creating the parent directory and applying the schema. Foreign keys are enabled so a file delete cascades to its chunks; a busy timeout absorbs brief lock contention. A single open connection serializes access, which is all a single-process CLI needs and sidesteps SQLite "database is locked".
func (*Store) AllChunks ¶
AllChunks returns every chunk in the index, optionally restricted to files whose rel_path begins with pathPrefix (empty = no filter). The search layer cosine-ranks these in memory.
func (*Store) AllFiles ¶
AllFiles returns every indexed file's rel-path, sorted. The graph layer uses this as the universe of link targets to resolve against.
func (*Store) AllLinks ¶
AllLinks returns every stored link edge (source file → raw target). Targets are unresolved — the graph layer maps them onto AllFiles.
func (*Store) EmbedStamp ¶ added in v0.1.3
EmbedStamp returns the representation the index's stored vectors were built with, or "" when nothing has ever embedded into it.
Reindex compares this itself and heals a mismatch. It is exported for the one caller that cannot rely on that: --no-reindex skips the healing, and ranking a query vector against vectors from another model returns confident nonsense rather than an error. A caller about to do that needs to be able to ask first.
func (*Store) FileHeadings ¶
FileHeadings returns, per indexed file rel-path, the set of section-anchor slugs that file defines — the valid targets for a link's #fragment. Slugs come from the stored heading breadcrumbs via the same slugifier the chunker uses for keys, so no re-parse is needed. Files with no headings are absent from the map (any #anchor into them is unresolvable).
func (*Store) RebuildReason ¶
RebuildReason reports why the next Reindex will redo work that the content hashes alone would skip, or "" when it will not. Reindex reports the same string afterwards; this exists so a caller can say so *first*. A rebuild turns an ordinarily instant command into a multi-minute one, and an unexplained pause is indistinguishable from a hang.
canEmbed must match the embedder the caller will pass to Reindex, or the answer describes a different run than the one about to happen.
func (*Store) Reindex ¶
Reindex walks root, incrementally syncing the index to the indexable files under it (markdown and source code — see chunkers): new/changed files are chunked and embedded, touched-but-identical files are cheaply restatted, and files gone from disk are removed. embedFn supplies vectors (embed.Get in production); pass nil to skip embedding entirely (see Embedder).
The change test is two-tier: a (mtime, size) match skips the file without reading it; otherwise the file is read and its sha256 compared, so a touch that didn't change bytes costs a read but no embedding. force bypasses both checks, re-chunking, re-embedding, and re-extracting links for every file regardless of whether its content changed — the escape hatch for when the chunker or link extractor itself changes behavior, since content-hash diffing has no way to know an unchanged file would now produce different chunks or links.
func (*Store) SetProgress ¶ added in v0.1.3
SetProgress installs the hook Reindex reports through, or nil for none. Embedding is the slow part of indexing, and on a large tree it is minutes during which the process otherwise looks hung.
Source Files