Documentation
¶
Overview ¶
Package store is graphmaster's SQLite layer.
It owns two database files inside a repository's .graphmaster directory:
- graph.db: the derived code graph (files, nodes, edges, imports, FTS5 symbol search) plus small steering/hook state tables. It is versioned with PRAGMA user_version; a mismatch, unreadable file or failed quick_check moves it aside and recreates an empty schema (no migrations; the indexer rebuilds it from source).
- cache.db: content-addressed parse cache that survives graph.db rebuilds (see Cache).
Writes go through a single Writer (one connection, batched transactions, per-file atomicity) while the process holds the cross-process write lock (see Lock). Readers use DB() and the typed accessors concurrently thanks to WAL mode.
Index ¶
- Constants
- Variables
- func IsBusy(err error) bool
- func MatchExpr(query string, prefix bool) string
- func SymbolKey(path, qname string, kind model.NodeKind) []byte
- func TokenizeIdent(name string) string
- type Cache
- func (c *Cache) Close() error
- func (c *Cache) GC(ctx context.Context, opt GCOptions) (CacheStats, error)
- func (c *Cache) Get(ctx context.Context, blob []byte, lang model.Lang, xver int) ([]byte, bool, error)
- func (c *Cache) Put(ctx context.Context, blob []byte, lang model.Lang, xver int, facts []byte) error
- func (c *Cache) PutBatch(ctx context.Context, entries []CacheEntry) error
- func (c *Cache) Stats(ctx context.Context) (CacheStats, error)
- type CacheEntry
- type CacheStats
- type Counts
- type Dir
- type Edge
- type File
- type GCOptions
- type Import
- type Lock
- type Node
- type Options
- type Package
- type Store
- func (s *Store) AddDirtyHints(ctx context.Context, paths []string, ts int64) error
- func (s *Store) Close() error
- func (s *Store) Counts(ctx context.Context) (Counts, error)
- func (s *Store) DB() *sql.DB
- func (s *Store) DeleteMeta(ctx context.Context, key string) error
- func (s *Store) Dir() string
- func (s *Store) FileByPath(ctx context.Context, path string) (*File, error)
- func (s *Store) FinishBulk(ctx context.Context) error
- func (s *Store) ForEachFile(ctx context.Context, fn func(*File) error) error
- func (s *Store) LoadDirs(ctx context.Context) ([]Dir, error)
- func (s *Store) MatchSymbols(ctx context.Context, match string, limit int) ([]int64, error)
- func (s *Store) Meta(ctx context.Context, key string) (value string, ok bool, err error)
- func (s *Store) Name(ctx context.Context, id int64) (string, error)
- func (s *Store) NameID(ctx context.Context, name string) (id int64, ok bool, err error)
- func (s *Store) NewWriter(ctx context.Context, opt WriterOptions) (*Writer, error)
- func (s *Store) NodeByID(ctx context.Context, id int64) (*Node, error)
- func (s *Store) Path() string
- func (s *Store) PendingNames(ctx context.Context, limit int) ([]int64, error)
- func (s *Store) RebuildReason() string
- func (s *Store) Rebuilt() bool
- func (s *Store) Replaced() bool
- func (s *Store) SearchSubstring(ctx context.Context, sub string, limit int) ([]int64, error)
- func (s *Store) SearchSymbols(ctx context.Context, query string, limit int) ([]int64, error)
- func (s *Store) SetMeta(ctx context.Context, key, value string) error
- func (s *Store) TakeDirtyHints(ctx context.Context) ([]string, error)
- type Writer
- func (w *Writer) Abort() error
- func (w *Writer) ClearDirs() error
- func (w *Writer) ClearPending(nameIDs ...int64) error
- func (w *Writer) Close() error
- func (w *Writer) DeleteDir(path string) error
- func (w *Writer) DeleteFile(fileID int64) error
- func (w *Writer) DeleteFileGraph(fileID int64) error
- func (w *Writer) DeleteNode(fileID, nodeID int64) error
- func (w *Writer) EnqueuePending(nameIDs ...int64) error
- func (w *Writer) Exec(query string, args ...any) (sql.Result, error)
- func (w *Writer) FileDone() error
- func (w *Writer) Flush() error
- func (w *Writer) InsertEdge(e *Edge) (int64, error)
- func (w *Writer) InsertImport(im *Import) error
- func (w *Writer) InsertNode(n *Node) (int64, error)
- func (w *Writer) Intern(name string) (int64, error)
- func (w *Writer) Query(query string, args ...any) (*sql.Rows, error)
- func (w *Writer) QueryRow(query string, args ...any) *sql.Row
- func (w *Writer) SetMeta(key, value string) error
- func (w *Writer) UpsertDir(d *Dir) error
- func (w *Writer) UpsertFile(f *File) (int64, error)
- func (w *Writer) UpsertPackage(p *Package) (int64, error)
- type WriterOptions
Constants ¶
const ( MetaSchemaVer = "schema_ver" MetaState = "state" // StateBuilding | StateReady MetaProgress = "progress" // free-form, e.g. "37" MetaLastScanNS = "last_scan_ns" MetaGitIndexStat = "gitindex_stat" MetaBulk = "bulk" // "1" while a bulk build defers indexes/FTS MetaExtractorVerPf = "extractor_ver:" // + model.Lang.String() )
Well-known meta keys.
const ( StateBuilding = "building" StateReady = "ready" )
Values for MetaState.
const ( GraphDBName = "graph.db" CacheDBName = "cache.db" LockName = "write.lock" )
File names inside the .graphmaster directory.
const ( DefaultBatchFiles = 500 DefaultBatchDelay = 200 * time.Millisecond )
Default batch thresholds (PLAN: ~500 files / ~200ms per commit).
const CacheSchemaVersion = 1
CacheSchemaVersion is cache.db's PRAGMA user_version. A mismatch discards the cache (it is only an accelerator).
const (
DefaultCacheMaxAge = 30 * 24 * time.Hour
)
Cache GC defaults.
const SchemaVersion = 2
SchemaVersion is stored in graph.db's PRAGMA user_version. Bump it on any change to schema.sql or indexes.sql; older databases are rebuilt.
Variables ¶
var ( // ErrNotFound is returned by accessors when a row does not exist. ErrNotFound = errors.New("store: not found") // ErrLocked is returned by Open when graph.db must be rebuilt but the // write lock could not be acquired within Options.LockBudget. ErrLocked = errors.New("store: write lock busy") )
Sentinel errors.
Functions ¶
func MatchExpr ¶
MatchExpr turns free-form user input into an FTS5 expression for the sym_fts tokens column. Each word matches either as a whole token or as the conjunction of its identifier parts; words are ANDed. With prefix set every term is a prefix query. Returns "" when the input has no searchable words.
func SymbolKey ¶
SymbolKey returns the 8-byte nodes.skey for a definition: FNV-64a of (path, qname, kind). It is stable across reindexing as long as the symbol keeps its file, qualified name and kind, so it can be shown to agents as a compact hex handle.
func TokenizeIdent ¶
TokenizeIdent expands an identifier into the text indexed by sym_fts: the identifier itself followed by its lowercase camel/snake/kebab parts, e.g. "getUserByID" -> "getUserByID get user by id". Identifiers that consist of a single part are returned unchanged.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache is cache.db: encoded parse facts keyed by (git blob SHA-1, language, extractor version). It survives graph.db rebuilds. Safe for concurrent use.
func OpenCache ¶
OpenCache opens or creates cache.db at path. A corrupt file or a schema version mismatch deletes and recreates it.
func (*Cache) GC ¶
GC evicts entries older than MaxAge, then trims the cache to twice the live size, oldest first.
func (*Cache) Get ¶
func (c *Cache) Get(ctx context.Context, blob []byte, lang model.Lang, xver int) ([]byte, bool, error)
Get returns cached facts. Hits refresh last_used at most once a day.
func (*Cache) Put ¶
func (c *Cache) Put(ctx context.Context, blob []byte, lang model.Lang, xver int, facts []byte) error
Put stores (or replaces) one entry.
type CacheEntry ¶
CacheEntry is one parse_cache row for PutBatch.
type CacheStats ¶
type CacheStats struct {
Entries int64
Bytes int64 // sum of facts sizes
Deleted int64 // entries removed by the GC run that produced the stats
}
CacheStats describes cache.db contents.
type Counts ¶
type Counts struct {
Files, Nodes, Edges, Unresolved int64
DBBytes int64 // graph.db + WAL on disk
}
Counts summarizes the graph for status output.
type Dir ¶
Dir is a row of the dirs table: the persisted untracked-cache entry for one directory (see repo.DirRow, which has the same shape). Path "" is the repository root. Names is the encoded directory listing; it is opaque to the store.
type Edge ¶
type Edge struct {
ID int64
FileID int64 // file whose facts produced the edge; Src lives in it
Src int64
Dst int64
Kind model.EdgeKind
NameID int64 // target name; InsertEdge fills it from Name when 0
Name string // optional, used only to intern NameID
Qual string
Line uint32
Col uint32
Conf uint8 // 0..100
}
Edge is a row of the edges table. Dst 0 means unresolved (NULL).
type File ¶
type File struct {
ID int64
Path string // repo-relative, forward slashes
Lang model.Lang
Size int64
MtimeNS int64
Ino uint64
Mode uint32
Blob []byte // git blob SHA-1 (20 bytes) or nil
Tracked bool
Status model.FileStatus
LOC int
PkgID int64 // 0 = none
}
File is a row of the files table. Zero ID means "not yet stored".
type GCOptions ¶
type GCOptions struct {
Now time.Time // default time.Now()
MaxAge time.Duration // entries unused for longer are dropped (default 30 days)
// LiveBytes is the facts size referenced by the current graph. When
// positive, least recently used entries are evicted until the cache
// holds at most 2x LiveBytes.
LiveBytes int64
}
GCOptions configure Cache.GC.
type Import ¶
type Import struct {
FileID int64
Spec string
Alias string
Names []string
TargetFile int64 // 0 = none
TargetPkg int64 // 0 = none
Line uint32
}
Import is a row of the imports table.
type Lock ¶
type Lock struct {
// contains filtered or unexported fields
}
Lock is the cross-process single-writer lock (.graphmaster/write.lock). It is an advisory flock released automatically when the process dies. A Lock is not safe for concurrent use by multiple goroutines.
type Node ¶
type Node struct {
ID int64
FileID int64
ParentID int64 // 0 = top level
Kind model.NodeKind
NameID int64 // interned name; InsertNode fills it from Name when 0
Name string // plain name; filled by NodeByID
QName string
// Lines are 1-based, columns 0-based, bytes 0-based with exclusive end.
StartLine, StartCol, EndLine uint32
StartByte, EndByte uint32
Sig string
Flags model.DefFlags
SKey []byte // 8 bytes, see SymbolKey
}
Node is a row of the nodes table (a definition).
type Options ¶
type Options struct {
// QuickCheck runs PRAGMA quick_check on open; a failure triggers a
// rebuild. It reads the whole file, so use it after crashes, not on
// every open.
QuickCheck bool
// Lock, if non-nil and held by the caller, is used for a rebuild
// instead of acquiring a new one. Open never releases it.
Lock *Lock
// LockBudget bounds the wait for the write lock during a rebuild
// (default 5s). Ignored when Lock is held.
LockBudget time.Duration
}
Options configure Open.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an open graph.db. It is safe for concurrent use; writes must go through a Writer.
func Open ¶
Open opens (creating if needed) dir/graph.db, where dir is the repository's .graphmaster directory. A database with a different schema version, an unreadable file or (with QuickCheck) a failed integrity check is moved aside to graph.db.old and recreated empty under the write lock; Rebuilt then reports true and the caller must reindex.
func (*Store) AddDirtyHints ¶
AddDirtyHints records paths reported as edited by hooks (ts in unix ns). Autocommit; safe to call from short-lived hook processes.
func (*Store) DB ¶
DB exposes the underlying pool for read queries. Do not write through it while a Writer is active; use the Writer's Exec instead.
func (*Store) DeleteMeta ¶
DeleteMeta removes a meta key.
func (*Store) FileByPath ¶
FileByPath returns the files row for a repo-relative path, or ErrNotFound.
func (*Store) FinishBulk ¶
FinishBulk ends a bulk build: it creates the secondary indexes, rebuilds sym_fts/sym_tri from nodes, clears the bulk flag and checkpoints the WAL. Call it after the bulk Writer is closed and before the resolution pass. It is idempotent and may also be used to repair the search index.
func (*Store) ForEachFile ¶
ForEachFile streams every files row in path order (for the stat sweep). Iteration stops at the first error returned by fn.
func (*Store) MatchSymbols ¶
MatchSymbols runs a raw FTS5 MATCH expression against sym_fts (columns tokens and path) and returns node ids ordered by rank.
func (*Store) NameID ¶
NameID returns the id of an interned name. ok is false if the name was never interned.
func (*Store) PendingNames ¶
PendingNames returns queued name ids in ascending order; limit <= 0 returns all.
func (*Store) RebuildReason ¶
RebuildReason explains why Open recreated the database, or "".
func (*Store) Rebuilt ¶
Rebuilt reports whether Open recreated the database (the graph is empty and must be fully reindexed).
func (*Store) Replaced ¶
Replaced reports whether graph.db on disk is no longer the file this Store opened (another process rebuilt it). Long-lived readers should reopen.
func (*Store) SearchSubstring ¶
SearchSubstring finds symbols whose name contains sub (case-insensitive) using the trigram index. Inputs shorter than three characters return nil; callers should fall back to SearchSymbols.
func (*Store) SearchSymbols ¶
SearchSymbols runs a prefix token search (see MatchExpr) over symbol names and returns node ids, best bm25 match first.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is the single batched writer for graph.db. It owns one connection and keeps an IMMEDIATE transaction open across several files, committing only at file boundaries (FileDone), so every file's changes are atomic.
A Writer is not safe for concurrent use; drive it from one goroutine. The caller must hold the cross-process write Lock for the Writer's lifetime. The context passed to NewWriter is used for every operation.
func (*Writer) ClearDirs ¶
ClearDirs drops the whole directory cache (e.g. after config excludes or core.excludesFile change) in the current batch.
func (*Writer) ClearPending ¶
ClearPending removes name ids from the resolution queue.
func (*Writer) DeleteFile ¶
DeleteFile removes a file completely: its graph (see DeleteFileGraph), the files row, and import targets pointing at it.
func (*Writer) DeleteFileGraph ¶
DeleteFileGraph removes a file's nodes, out-edges, imports, dependency attributions (dep_owners) and FTS rows in the current batch. Edges from other files into it are kept with dst = NULL and their names (plus the file's old definition names) are queued in pending_names. The files row itself is kept so the id stays stable for reinsertion.
func (*Writer) DeleteNode ¶
DeleteNode removes one definition of file fileID whose other definitions stay (an in-place rewrite of the file), with its FTS rows. Like DeleteFileGraph, edges from other files into it are kept with dst = NULL and their names queued in pending_names; unlike it, the definition's own name is not queued, and the file's own edges are left to the caller (children and edges of the file must be replaced by it).
func (*Writer) EnqueuePending ¶
EnqueuePending adds name ids to the resolution queue.
func (*Writer) Exec ¶
Exec runs an arbitrary write inside the current batch, for operations not covered by typed helpers (e.g. resolver edge updates).
func (*Writer) FileDone ¶
FileDone marks the end of one file's changes and commits when the batch thresholds are reached.
func (*Writer) Flush ¶
Flush commits the open transaction, if any. Call it at file boundaries only (e.g. when the producer is idle).
func (*Writer) InsertEdge ¶
InsertEdge stores an edge, interning e.Name when e.NameID is 0 and Name is set. Sets e.ID.
func (*Writer) InsertImport ¶
InsertImport stores an import row.
func (*Writer) InsertNode ¶
InsertNode stores a definition, interning n.Name when n.NameID is 0, and indexes it for search (unless the writer is in bulk mode). Sets n.ID.
func (*Writer) Query ¶
Query reads through the writer's connection, observing uncommitted batch state. It does not start a transaction.
func (*Writer) UpsertFile ¶
UpsertFile inserts or updates the row for f.Path, keeping its id stable, and sets f.ID.
type WriterOptions ¶
type WriterOptions struct {
// MaxFiles commits once this many files are finished (default 500).
MaxFiles int
// MaxDelay commits at the next file boundary once the open
// transaction is older than this (default 200ms).
MaxDelay time.Duration
// Bulk defers secondary indexes and FTS population for a fresh full
// build. Call Store.FinishBulk after the writer is closed.
Bulk bool
// OnCommit, if set, is called after every commit that included at
// least one finished file, with the total files committed so far.
OnCommit func(filesCommitted int)
}
WriterOptions configure a Writer.