Help us shape the future of Go! Take the 2026 Go Developer Survey

store

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 18, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package store is graphmaster's SQLite layer.

It owns two database files inside a repository's .graphmaster directory:

  • graph.db: the derived code graph (files, nodes, edges, imports, FTS5 symbol search) plus small steering/hook state tables. It is versioned with PRAGMA user_version; a mismatch, unreadable file or failed quick_check moves it aside and recreates an empty schema (no migrations; the indexer rebuilds it from source).
  • cache.db: content-addressed parse cache that survives graph.db rebuilds (see Cache).

Writes go through a single Writer (one connection, batched transactions, per-file atomicity) while the process holds the cross-process write lock (see Lock). Readers use DB() and the typed accessors concurrently thanks to WAL mode.

Index

Constants

View Source
const (
	MetaSchemaVer      = "schema_ver"
	MetaState          = "state"    // StateBuilding | StateReady
	MetaProgress       = "progress" // free-form, e.g. "37"
	MetaLastScanNS     = "last_scan_ns"
	MetaGitIndexStat   = "gitindex_stat"
	MetaBulk           = "bulk"           // "1" while a bulk build defers indexes/FTS
	MetaExtractorVerPf = "extractor_ver:" // + model.Lang.String()
)

Well-known meta keys.

View Source
const (
	StateBuilding = "building"
	StateReady    = "ready"
)

Values for MetaState.

View Source
const (
	GraphDBName = "graph.db"
	CacheDBName = "cache.db"
	LockName    = "write.lock"
)

File names inside the .graphmaster directory.

View Source
const (
	DefaultBatchFiles = 500
	DefaultBatchDelay = 200 * time.Millisecond
)

Default batch thresholds (PLAN: ~500 files / ~200ms per commit).

View Source
const CacheSchemaVersion = 1

CacheSchemaVersion is cache.db's PRAGMA user_version. A mismatch discards the cache (it is only an accelerator).

View Source
const (
	DefaultCacheMaxAge = 30 * 24 * time.Hour
)

Cache GC defaults.

View Source
const SchemaVersion = 2

SchemaVersion is stored in graph.db's PRAGMA user_version. Bump it on any change to schema.sql or indexes.sql; older databases are rebuilt.

Variables

View Source
var (
	// ErrNotFound is returned by accessors when a row does not exist.
	ErrNotFound = errors.New("store: not found")
	// ErrLocked is returned by Open when graph.db must be rebuilt but the
	// write lock could not be acquired within Options.LockBudget.
	ErrLocked = errors.New("store: write lock busy")
)

Sentinel errors.

Functions

func MatchExpr

func MatchExpr(query string, prefix bool) string

MatchExpr turns free-form user input into an FTS5 expression for the sym_fts tokens column. Each word matches either as a whole token or as the conjunction of its identifier parts; words are ANDed. With prefix set every term is a prefix query. Returns "" when the input has no searchable words.

func SymbolKey

func SymbolKey(path, qname string, kind model.NodeKind) []byte

SymbolKey returns the 8-byte nodes.skey for a definition: FNV-64a of (path, qname, kind). It is stable across reindexing as long as the symbol keeps its file, qualified name and kind, so it can be shown to agents as a compact hex handle.

func TokenizeIdent

func TokenizeIdent(name string) string

TokenizeIdent expands an identifier into the text indexed by sym_fts: the identifier itself followed by its lowercase camel/snake/kebab parts, e.g. "getUserByID" -> "getUserByID get user by id". Identifiers that consist of a single part are returned unchanged.

Types

type Cache

type Cache struct {
	// contains filtered or unexported fields
}

Cache is cache.db: encoded parse facts keyed by (git blob SHA-1, language, extractor version). It survives graph.db rebuilds. Safe for concurrent use.

func OpenCache

func OpenCache(ctx context.Context, path string) (*Cache, error)

OpenCache opens or creates cache.db at path. A corrupt file or a schema version mismatch deletes and recreates it.

func (*Cache) Close

func (c *Cache) Close() error

Close closes cache.db.

func (*Cache) GC

func (c *Cache) GC(ctx context.Context, opt GCOptions) (CacheStats, error)

GC evicts entries older than MaxAge, then trims the cache to twice the live size, oldest first.

func (*Cache) Get

func (c *Cache) Get(ctx context.Context, blob []byte, lang model.Lang, xver int) ([]byte, bool, error)

Get returns cached facts. Hits refresh last_used at most once a day.

func (*Cache) Put

func (c *Cache) Put(ctx context.Context, blob []byte, lang model.Lang, xver int, facts []byte) error

Put stores (or replaces) one entry.

func (*Cache) PutBatch

func (c *Cache) PutBatch(ctx context.Context, entries []CacheEntry) error

PutBatch stores entries in one transaction.

func (*Cache) Stats

func (c *Cache) Stats(ctx context.Context) (CacheStats, error)

Stats returns entry count and payload size.

type CacheEntry

type CacheEntry struct {
	Blob  []byte
	Lang  model.Lang
	XVer  int
	Facts []byte
}

CacheEntry is one parse_cache row for PutBatch.

type CacheStats

type CacheStats struct {
	Entries int64
	Bytes   int64 // sum of facts sizes
	Deleted int64 // entries removed by the GC run that produced the stats
}

CacheStats describes cache.db contents.

type Counts

type Counts struct {
	Files, Nodes, Edges, Unresolved int64
	DBBytes                         int64 // graph.db + WAL on disk
}

Counts summarizes the graph for status output.

type Dir

type Dir struct {
	Path       string
	MtimeNS    int64
	IgnoreStat string
	Names      []byte
}

Dir is a row of the dirs table: the persisted untracked-cache entry for one directory (see repo.DirRow, which has the same shape). Path "" is the repository root. Names is the encoded directory listing; it is opaque to the store.

type Edge

type Edge struct {
	ID     int64
	FileID int64 // file whose facts produced the edge; Src lives in it
	Src    int64
	Dst    int64
	Kind   model.EdgeKind
	NameID int64  // target name; InsertEdge fills it from Name when 0
	Name   string // optional, used only to intern NameID
	Qual   string
	Line   uint32
	Col    uint32
	Conf   uint8 // 0..100
}

Edge is a row of the edges table. Dst 0 means unresolved (NULL).

type File

type File struct {
	ID      int64
	Path    string // repo-relative, forward slashes
	Lang    model.Lang
	Size    int64
	MtimeNS int64
	Ino     uint64
	Mode    uint32
	Blob    []byte // git blob SHA-1 (20 bytes) or nil
	Tracked bool
	Status  model.FileStatus
	LOC     int
	PkgID   int64 // 0 = none
}

File is a row of the files table. Zero ID means "not yet stored".

type GCOptions

type GCOptions struct {
	Now    time.Time     // default time.Now()
	MaxAge time.Duration // entries unused for longer are dropped (default 30 days)
	// LiveBytes is the facts size referenced by the current graph. When
	// positive, least recently used entries are evicted until the cache
	// holds at most 2x LiveBytes.
	LiveBytes int64
}

GCOptions configure Cache.GC.

type Import

type Import struct {
	FileID     int64
	Spec       string
	Alias      string
	Names      []string
	TargetFile int64 // 0 = none
	TargetPkg  int64 // 0 = none
	Line       uint32
}

Import is a row of the imports table.

type Lock

type Lock struct {
	// contains filtered or unexported fields
}

Lock is the cross-process single-writer lock (.graphmaster/write.lock). It is an advisory flock released automatically when the process dies. A Lock is not safe for concurrent use by multiple goroutines.

func NewLock

func NewLock(dir string) *Lock

NewLock returns the write lock for the .graphmaster directory dir.

func (*Lock) Held

func (l *Lock) Held() bool

Held reports whether this Lock currently holds the lock.

func (*Lock) Path

func (l *Lock) Path() string

Path returns the lock file path.

func (*Lock) TryLock

func (l *Lock) TryLock(ctx context.Context, budget time.Duration) (bool, error)

TryLock acquires the lock, waiting at most budget. A budget <= 0 tries exactly once. It returns (false, nil) when the budget runs out and an error only for I/O failures or when ctx itself is done.

func (*Lock) Unlock

func (l *Lock) Unlock() error

Unlock releases the lock. It is a no-op when not held.

type Node

type Node struct {
	ID       int64
	FileID   int64
	ParentID int64 // 0 = top level
	Kind     model.NodeKind
	NameID   int64  // interned name; InsertNode fills it from Name when 0
	Name     string // plain name; filled by NodeByID
	QName    string
	// Lines are 1-based, columns 0-based, bytes 0-based with exclusive end.
	StartLine, StartCol, EndLine uint32
	StartByte, EndByte           uint32
	Sig                          string
	Flags                        model.DefFlags
	SKey                         []byte // 8 bytes, see SymbolKey
}

Node is a row of the nodes table (a definition).

type Options

type Options struct {
	// QuickCheck runs PRAGMA quick_check on open; a failure triggers a
	// rebuild. It reads the whole file, so use it after crashes, not on
	// every open.
	QuickCheck bool
	// Lock, if non-nil and held by the caller, is used for a rebuild
	// instead of acquiring a new one. Open never releases it.
	Lock *Lock
	// LockBudget bounds the wait for the write lock during a rebuild
	// (default 5s). Ignored when Lock is held.
	LockBudget time.Duration
}

Options configure Open.

type Package

type Package struct {
	ID   int64
	Lang model.Lang
	Name string
	Dir  string
	Kind uint8
}

Package is a row of the packages table, unique per (Lang, Name).

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is an open graph.db. It is safe for concurrent use; writes must go through a Writer.

func Open

func Open(ctx context.Context, dir string, opts Options) (*Store, error)

Open opens (creating if needed) dir/graph.db, where dir is the repository's .graphmaster directory. A database with a different schema version, an unreadable file or (with QuickCheck) a failed integrity check is moved aside to graph.db.old and recreated empty under the write lock; Rebuilt then reports true and the caller must reindex.

func (*Store) AddDirtyHints

func (s *Store) AddDirtyHints(ctx context.Context, paths []string, ts int64) error

AddDirtyHints records paths reported as edited by hooks (ts in unix ns). Autocommit; safe to call from short-lived hook processes.

func (*Store) Close

func (s *Store) Close() error

Close closes the database.

func (*Store) Counts

func (s *Store) Counts(ctx context.Context) (Counts, error)

Counts returns row counts and on-disk size.

func (*Store) DB

func (s *Store) DB() *sql.DB

DB exposes the underlying pool for read queries. Do not write through it while a Writer is active; use the Writer's Exec instead.

func (*Store) DeleteMeta

func (s *Store) DeleteMeta(ctx context.Context, key string) error

DeleteMeta removes a meta key.

func (*Store) Dir

func (s *Store) Dir() string

Dir returns the .graphmaster directory.

func (*Store) FileByPath

func (s *Store) FileByPath(ctx context.Context, path string) (*File, error)

FileByPath returns the files row for a repo-relative path, or ErrNotFound.

func (*Store) FinishBulk

func (s *Store) FinishBulk(ctx context.Context) error

FinishBulk ends a bulk build: it creates the secondary indexes, rebuilds sym_fts/sym_tri from nodes, clears the bulk flag and checkpoints the WAL. Call it after the bulk Writer is closed and before the resolution pass. It is idempotent and may also be used to repair the search index.

func (*Store) ForEachFile

func (s *Store) ForEachFile(ctx context.Context, fn func(*File) error) error

ForEachFile streams every files row in path order (for the stat sweep). Iteration stops at the first error returned by fn.

func (*Store) LoadDirs

func (s *Store) LoadDirs(ctx context.Context) ([]Dir, error)

LoadDirs returns every cached directory in path order.

func (*Store) MatchSymbols

func (s *Store) MatchSymbols(ctx context.Context, match string, limit int) ([]int64, error)

MatchSymbols runs a raw FTS5 MATCH expression against sym_fts (columns tokens and path) and returns node ids ordered by rank.

func (*Store) Meta

func (s *Store) Meta(ctx context.Context, key string) (value string, ok bool, err error)

Meta returns a meta value; ok is false when the key is absent.

func (*Store) Name

func (s *Store) Name(ctx context.Context, id int64) (string, error)

Name returns the name for an id, or ErrNotFound.

func (*Store) NameID

func (s *Store) NameID(ctx context.Context, name string) (id int64, ok bool, err error)

NameID returns the id of an interned name. ok is false if the name was never interned.

func (*Store) NewWriter

func (s *Store) NewWriter(ctx context.Context, opt WriterOptions) (*Writer, error)

NewWriter starts a writer on a dedicated connection.

func (*Store) NodeByID

func (s *Store) NodeByID(ctx context.Context, id int64) (*Node, error)

NodeByID returns a node with its Name filled in, or ErrNotFound.

func (*Store) Path

func (s *Store) Path() string

Path returns the graph.db path.

func (*Store) PendingNames

func (s *Store) PendingNames(ctx context.Context, limit int) ([]int64, error)

PendingNames returns queued name ids in ascending order; limit <= 0 returns all.

func (*Store) RebuildReason

func (s *Store) RebuildReason() string

RebuildReason explains why Open recreated the database, or "".

func (*Store) Rebuilt

func (s *Store) Rebuilt() bool

Rebuilt reports whether Open recreated the database (the graph is empty and must be fully reindexed).

func (*Store) Replaced

func (s *Store) Replaced() bool

Replaced reports whether graph.db on disk is no longer the file this Store opened (another process rebuilt it). Long-lived readers should reopen.

func (*Store) SearchSubstring

func (s *Store) SearchSubstring(ctx context.Context, sub string, limit int) ([]int64, error)

SearchSubstring finds symbols whose name contains sub (case-insensitive) using the trigram index. Inputs shorter than three characters return nil; callers should fall back to SearchSymbols.

func (*Store) SearchSymbols

func (s *Store) SearchSymbols(ctx context.Context, query string, limit int) ([]int64, error)

SearchSymbols runs a prefix token search (see MatchExpr) over symbol names and returns node ids, best bm25 match first.

func (*Store) SetMeta

func (s *Store) SetMeta(ctx context.Context, key, value string) error

SetMeta upserts a meta value outside any batch (autocommit).

func (*Store) TakeDirtyHints

func (s *Store) TakeDirtyHints(ctx context.Context) ([]string, error)

TakeDirtyHints atomically returns and clears all dirty hints.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is the single batched writer for graph.db. It owns one connection and keeps an IMMEDIATE transaction open across several files, committing only at file boundaries (FileDone), so every file's changes are atomic.

A Writer is not safe for concurrent use; drive it from one goroutine. The caller must hold the cross-process write Lock for the Writer's lifetime. The context passed to NewWriter is used for every operation.

func (*Writer) Abort

func (w *Writer) Abort() error

Abort rolls back everything since the last commit. The writer stays usable.

func (*Writer) ClearDirs

func (w *Writer) ClearDirs() error

ClearDirs drops the whole directory cache (e.g. after config excludes or core.excludesFile change) in the current batch.

func (*Writer) ClearPending

func (w *Writer) ClearPending(nameIDs ...int64) error

ClearPending removes name ids from the resolution queue.

func (*Writer) Close

func (w *Writer) Close() error

Close commits pending work and releases the connection.

func (*Writer) DeleteDir

func (w *Writer) DeleteDir(path string) error

DeleteDir removes a cached directory in the current batch.

func (*Writer) DeleteFile

func (w *Writer) DeleteFile(fileID int64) error

DeleteFile removes a file completely: its graph (see DeleteFileGraph), the files row, and import targets pointing at it.

func (*Writer) DeleteFileGraph

func (w *Writer) DeleteFileGraph(fileID int64) error

DeleteFileGraph removes a file's nodes, out-edges, imports, dependency attributions (dep_owners) and FTS rows in the current batch. Edges from other files into it are kept with dst = NULL and their names (plus the file's old definition names) are queued in pending_names. The files row itself is kept so the id stays stable for reinsertion.

func (*Writer) DeleteNode

func (w *Writer) DeleteNode(fileID, nodeID int64) error

DeleteNode removes one definition of file fileID whose other definitions stay (an in-place rewrite of the file), with its FTS rows. Like DeleteFileGraph, edges from other files into it are kept with dst = NULL and their names queued in pending_names; unlike it, the definition's own name is not queued, and the file's own edges are left to the caller (children and edges of the file must be replaced by it).

func (*Writer) EnqueuePending

func (w *Writer) EnqueuePending(nameIDs ...int64) error

EnqueuePending adds name ids to the resolution queue.

func (*Writer) Exec

func (w *Writer) Exec(query string, args ...any) (sql.Result, error)

Exec runs an arbitrary write inside the current batch, for operations not covered by typed helpers (e.g. resolver edge updates).

func (*Writer) FileDone

func (w *Writer) FileDone() error

FileDone marks the end of one file's changes and commits when the batch thresholds are reached.

func (*Writer) Flush

func (w *Writer) Flush() error

Flush commits the open transaction, if any. Call it at file boundaries only (e.g. when the producer is idle).

func (*Writer) InsertEdge

func (w *Writer) InsertEdge(e *Edge) (int64, error)

InsertEdge stores an edge, interning e.Name when e.NameID is 0 and Name is set. Sets e.ID.

func (*Writer) InsertImport

func (w *Writer) InsertImport(im *Import) error

InsertImport stores an import row.

func (*Writer) InsertNode

func (w *Writer) InsertNode(n *Node) (int64, error)

InsertNode stores a definition, interning n.Name when n.NameID is 0, and indexes it for search (unless the writer is in bulk mode). Sets n.ID.

func (*Writer) Intern

func (w *Writer) Intern(name string) (int64, error)

Intern returns the id of name, inserting it if needed.

func (*Writer) Query

func (w *Writer) Query(query string, args ...any) (*sql.Rows, error)

Query reads through the writer's connection, observing uncommitted batch state. It does not start a transaction.

func (*Writer) QueryRow

func (w *Writer) QueryRow(query string, args ...any) *sql.Row

QueryRow is the single-row form of Query.

func (*Writer) SetMeta

func (w *Writer) SetMeta(key, value string) error

SetMeta upserts a meta value inside the current batch.

func (*Writer) UpsertDir

func (w *Writer) UpsertDir(d *Dir) error

UpsertDir inserts or replaces a cached directory in the current batch.

func (*Writer) UpsertFile

func (w *Writer) UpsertFile(f *File) (int64, error)

UpsertFile inserts or updates the row for f.Path, keeping its id stable, and sets f.ID.

func (*Writer) UpsertPackage

func (w *Writer) UpsertPackage(p *Package) (int64, error)

UpsertPackage inserts or updates the package (Lang, Name) and sets p.ID.

type WriterOptions

type WriterOptions struct {
	// MaxFiles commits once this many files are finished (default 500).
	MaxFiles int
	// MaxDelay commits at the next file boundary once the open
	// transaction is older than this (default 200ms).
	MaxDelay time.Duration
	// Bulk defers secondary indexes and FTS population for a fresh full
	// build. Call Store.FinishBulk after the writer is closed.
	Bulk bool
	// OnCommit, if set, is called after every commit that included at
	// least one finished file, with the total files committed so far.
	OnCommit func(filesCommitted int)
}

WriterOptions configure a Writer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL