storage

package
v0.3.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 1, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

CANARY: REQ=ENG-4319; FEATURE="ContextManagement"; ASPECT=Engine; STATUS=IMPL; UPDATED=2025-10-18

CANARY: REQ=ENG-4312; FEATURE="DatabaseMigrations"; ASPECT=Storage; STATUS=IMPL; OWNER=canary; UPDATED=2025-10-16

CANARY: REQ=ENG-4317; FEATURE="GapRepository"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-17

CANARY: REQ=ENG-4319; FEATURE="DatabaseModes"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-18

CANARY: REQ=ENG-4319; FEATURE="ProjectRegistry"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-18

CANARY: REQ=ENG-4315; FEATURE="DocDatabaseSchema"; ASPECT=Storage; STATUS=IMPL; OWNER=canary; UPDATED=2026-08-30

CANARY: REQ=ENG-4306; FEATURE="TokenStorage"; ASPECT=Storage; STATUS=IMPL; OWNER=canary; UPDATED=2025-10-16

Index

Constants

View Source
const (
	DBDriver        = "sqlite"
	DBMigrationPath = "migrations"
	DBSourceName    = "iofs"
	DBURLProtocol   = "sqlite://"
	MigrateAll      = "all"
	LatestVersion   = 10 // Update this when adding new migrations
)
View Source
const DefaultProjectID = "default"

DefaultProjectID is the identity a token carries when nothing configured one. Migration 000007 backfills it onto every pre-scoping row, so it is the value that makes an unconfigured database's rows reachable by a scoped query.

View Source
const DefaultSearchLimit = 25

DefaultSearchLimit caps keyword searches to protect agent context. Deliberately small; callers raise it explicitly (--limit / limit param) when they need more.

Variables

View Source
var ErrDatabaseNotPopulated = errors.New("database not migrated")
View Source
var ErrInvalidOrderBy = errors.New("INVALID_ORDER_BY")

ErrInvalidOrderBy is returned when a caller asks for an ordering that is not in the allowlist below.

View Source
var ErrProjectRequired = errors.New("PROJECT_REQUIRED")

ErrProjectRequired is returned when an unscoped query matches rows in more than one project. Guessing which project the caller meant would answer a question they did not ask, so the ambiguity is reported instead.

View Source
var ErrReadOnlyReservation = errors.New("cannot reserve an id through a read-only database handle")

ErrReadOnlyReservation is returned when a reservation is attempted through a read-only handle. Allocating an id is a write; asking for one from a reader is a caller mistake, not a transient failure.

View Source
var ErrSchemaOutOfDate = errors.New("index schema is out of date; run 'canary index'")

ErrSchemaOutOfDate is what a read-only open returns when the database on disk predates the current schema. It is a state problem with one fix, and it is deliberately NOT solved by migrating: a read command that rewrote the schema behind the caller would be the very side effect OpenRO exists to prevent, and it would do so while another process may be reading. The message is the whole remedy, so commands surface it verbatim.

Functions

func AutoMigrate

func AutoMigrate(dbPath string) error

AutoMigrate automatically migrates the database if needed.

Progress banners go to stderr, never stdout: several commands emit a machine-readable line on stdout, and a schema notice interleaved with it would corrupt output the caller is parsing. AutoMigrate creates the database when it is missing, so only a writer may call it.

func DatabasePopulated

func DatabasePopulated(db *sqlx.DB, targetVersion int) (bool, error)

DatabasePopulated checks if the database is fully migrated and populated We only return an error here if we're getting database issues. Bool return should reflect the state of the database.

func InitDB

func InitDB(dbPath string) (*sqlx.DB, error)

InitDB initializes the database connection

func MigrateDB

func MigrateDB(dbPath string, steps string) error

MigrateDB applies the database migrations stored in migrations/*.sql It takes a single argument which is either "all" to migrate to the latest version or an integer to migrate by that many steps.

func NeedsMigration

func NeedsMigration(dbPath string) (bool, int, error)

NeedsMigration checks if the database exists and needs migration

func OrderKeys added in v0.3.3

func OrderKeys() []string

OrderKeys lists the order keys a caller may name, sorted, excluding the empty default. It is the single source for the allowlist in help text and in the INVALID_ORDER_BY contract.

func SchemaDDL added in v0.3.3

func SchemaDDL() (string, error)

SchemaDDL returns the concatenated forward (`.up.sql`) migrations, in migration order, each preceded by a header naming its file. It is the single source of truth behind both `canary db schema` and docs/DB_SCHEMA.md, so the documented schema can never drift from the embedded migrations that actually build the database.

func TeardownDB

func TeardownDB(dbPath string, steps string) error

TeardownDB is the negative inverse of MigrateDB, rolling back migrations It takes a single argument which is either "all" to roll back all migrations or an integer to roll back by that many steps.

Types

type Checkpoint

type Checkpoint struct {
	ID           int
	Name         string
	Description  string
	CommitHash   string
	CreatedAt    string
	TotalTokens  int
	StubCount    int
	ImplCount    int
	TestedCount  int
	BenchedCount int
	SnapshotJSON string
}

Checkpoint represents a state snapshot

type ContextManager

type ContextManager struct {
	// contains filtered or unexported fields
}

ContextManager manages the current project context

func NewContextManager

func NewContextManager(manager *DatabaseManager) *ContextManager

NewContextManager creates a new context manager

func (*ContextManager) DetectProject

func (cm *ContextManager) DetectProject() (*Project, error)

DetectProject attempts to detect the current project from the working directory

func (*ContextManager) GetCurrent

func (cm *ContextManager) GetCurrent() (*Project, error)

GetCurrent returns the currently active project

func (*ContextManager) SwitchTo

func (cm *ContextManager) SwitchTo(projectID string) error

SwitchTo switches the current project context to the specified project ID

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB wraps the SQLite database connection. Construct it with OpenRW (the writer, which may create and migrate) or OpenRO (the reader, which never creates anything).

func OpenRO added in v0.3.3

func OpenRO(dbPath string) (*DB, error)

OpenRO opens dbPath read-only. A missing database is reported as fs.ErrNotExist and NOTHING is created -- not the file, not its parent directory. Read commands use this so running `canary list` in a repository that was never indexed leaves the repository exactly as it found it.

A database that exists but predates the current schema is refused with ErrSchemaOutOfDate. This is the single choke point for that check because every read command reaches the index through here: without it a v6 database answered `canary list` with a raw "no such column: content_hash" from deep inside the query layer, which names neither the problem nor the fix.

The connection is opened with SQLite's own mode=ro and query_only, so a write attempted through it fails at the engine rather than relying on every caller to behave.

func OpenRW added in v0.3.3

func OpenRW(dbPath string) (*DB, error)

OpenRW opens dbPath for reading and writing, creating and migrating it when necessary. Only a command that may legitimately modify the index calls this: it is the one entry point allowed to bring a database into existence.

The connection is configured with a busy timeout (so a concurrent writer yields a wait rather than an immediate SQLITE_BUSY), WAL journalling (so a reader is never blocked by the indexer), and immediate transactions (so `canary index` takes its write lock at BEGIN instead of discovering a conflict halfway through the rebuild).

func (*DB) Close

func (db *DB) Close() error

Close closes the database connection

func (*DB) CreateCheckpoint

func (db *DB) CreateCheckpoint(projectID, name, description, commitHash, snapshotJSON string) error

CreateCheckpoint creates a state snapshot of projectID's tokens ("" counts every project).

func (*DB) DeleteAllTokens added in v0.3.0

func (db *DB) DeleteAllTokens(projectID string) error

DeleteAllTokens clears one project's token rows. `canary index` treats the token index as fully derived state: each run rebuilds it from a whole-tree scan, so rows for tokens that no longer exist on disk (renamed or remapped REQ IDs, deleted files) must not survive a re-index. The projectID scope keeps one project's rebuild from wiping a sibling project's rows out of a shared database.

`canary index` does not call this directly -- it uses ReplaceIndex, which performs the same delete inside the rebuild transaction. CANARY: REQ=CP-285; FEATURE="IndexRebuild"; ASPECT=Storage; STATUS=TESTED; TEST=devnw.dev/canary/pkg/storage:TestCANARY_CP_285_IndexRebuildPrunes; UPDATED=2026-08-31

func (*DB) GetAllTokens

func (db *DB) GetAllTokens() ([]*Token, error)

GetAllTokens retrieves all tokens across all projects. It is deliberately unscoped: its callers (drift detection) reason about the whole database.

func (*DB) GetCheckpoints

func (db *DB) GetCheckpoints() ([]*Checkpoint, error)

GetCheckpoints retrieves all checkpoints

func (*DB) GetFilesByReqID

func (db *DB) GetFilesByReqID(projectID, reqID string, excludeSpecs bool) (map[string][]*Token, error)

CANARY: REQ=CBIN-CLI-001; FEATURE="QueryAbstraction"; ASPECT=Storage; STATUS=TESTED; TEST=TestCANARY_CBIN_CLI_001_Storage_GetFilesByReqID; UPDATED=2026-08-29 GetFilesByReqID groups tokens by file path for a requirement within projectID ("" spans every project, subject to GetTokensByReqID's ambiguity rule).

func (*DB) GetIndexMeta added in v0.3.3

func (db *DB) GetIndexMeta(projectID string) (*IndexMeta, error)

GetIndexMeta returns the recorded index metadata for one project, or (nil, nil) when that project's index has never been built. A missing row is an answer, not an error. An empty projectID defaults to DefaultProjectID.

func (*DB) GetRefsByKind added in v0.3.0

func (db *DB) GetRefsByKind(projectID, kind string, limit int) ([]*Ref, error)

GetRefsByKind returns refs of the given kind across all requirements, ordered by file then line. limit<=0 defaults to a 100-row cap. projectID scopes the query; "" spans every project, mirroring ListTokens.

func (*DB) GetRefsByReqID

func (db *DB) GetRefsByReqID(projectID, reqID string) ([]*Ref, error)

GetRefsByReqID returns refs for one requirement, ordered by file then line. projectID scopes the query; "" spans every project, mirroring ListTokens.

func (*DB) GetTokensByProject

func (db *DB) GetTokensByProject(projectID string) ([]*Token, error)

CANARY: REQ=ENG-4319; FEATURE="TokenNamespacing"; ASPECT=Storage; STATUS=TESTED; TEST=TestTokenIsolationBetweenProjects; UPDATED=2026-08-31 GetTokensByProject retrieves all tokens for a specific project.

func (*DB) GetTokensByReqID

func (db *DB) GetTokensByReqID(projectID, reqID string) ([]*Token, error)

CANARY: REQ=ENG-4319; FEATURE="TokenNamespacing"; ASPECT=Storage; STATUS=TESTED; TEST=TestAuditF08,TestAuditF08SingleProjectUnscoped; UPDATED=2026-08-31 GetTokensByReqID retrieves every token for a requirement within projectID.

An empty projectID means "whichever project this database holds": if the matching rows all belong to one project they are returned, and if they span more than one the call fails with ErrProjectRequired rather than mixing two projects' answers into one.

func (*DB) ListTokens

func (db *DB) ListTokens(projectID string, filters map[string]any, idPattern string, orderKey string, limit int) ([]*Token, error)

CANARY: REQ=ENG-4318; FEATURE="PriorityFiltering"; ASPECT=Storage; STATUS=TESTED; TEST=TestAuditF07,TestAuditF07AllowedKeys; UPDATED=2026-08-30 ListTokens retrieves tokens with filters and ordering.

projectID scopes the query; "" spans every project in the database. idPattern is a Go regexp applied to req_id after the query runs. orderKey must be one of OrderKeys() or "" (the default ordering); anything else is refused with ErrInvalidOrderBy and no query is issued.

func (*DB) Path added in v0.3.3

func (db *DB) Path() string

Path returns the database file this handle was opened from.

func (*DB) PutIndexMeta added in v0.3.3

func (db *DB) PutIndexMeta(m IndexMeta) error

PutIndexMeta writes one project's index_meta row, replacing whatever was there for that project. It is exported for callers that record metadata outside a rebuild; `canary index` uses the transactional form so metadata and rows commit together or not at all. An empty m.ProjectID defaults to DefaultProjectID, mirroring upsertArgs's token.ProjectID default.

func (*DB) ReadOnly added in v0.3.3

func (db *DB) ReadOnly() bool

ReadOnly reports whether this handle was opened read-only.

func (*DB) ReplaceIndex added in v0.3.3

func (db *DB) ReplaceIndex(projectID string, tokens []*Token, refs map[string][]Ref, meta IndexMeta) error

ReplaceIndex rebuilds one project's slice of the index in a single transaction: the project's existing token rows are deleted, every supplied token is inserted, the supplied reference kinds are replaced, and the metadata row is written. Any failure rolls the whole thing back, so a run that dies partway leaves the previous index exactly as it was rather than an emptied or half-populated table.

refs is keyed by ref kind; a kind present with an empty slice clears that kind. Kinds absent from the map are left untouched.

func (*DB) ReplaceRefs

func (db *DB) ReplaceRefs(projectID, kind string, refs []Ref) error

ReplaceRefs atomically replaces all refs of the given kind within one project. An empty projectID defaults to DefaultProjectID, mirroring upsertArgs's token.ProjectID default -- so a caller that never configured project scoping still lands its refs somewhere reachable rather than under an empty, unqueryable identity.

func (*DB) ReserveID added in v0.3.3

func (db *DB) ReserveID(projectID, prefix string) (string, error)

ReserveID allocates the next identifier in a series and records the allocation, returning e.g. "BUG-API-007".

The allocation is a single immediate transaction: the write lock is taken at BEGIN (OpenRW sets _txlock=immediate), the highest number in the series is read, and the successor is INSERTed. The table's primary key is the actual guarantee -- if a concurrent writer commits the same number first, this transaction's INSERT is rejected and the whole read-then-write is retried against the new maximum. No caller can be handed an id that another caller already holds.

The series is seeded from the token index the first time it is used, so a database that predates this table does not start re-issuing ids its own rows already carry.

projectID is required. Numbering is per project, so an unscoped reservation would allocate out of whatever mixture of projects the database happens to hold and hand two projects the same id.

func (*DB) SearchTokens

func (db *DB) SearchTokens(projectID, keywords string, limit int) ([]*Token, error)

CANARY: REQ=ENG-4323; FEATURE="ContextCaps"; ASPECT=Storage; STATUS=TESTED; TEST=TestCANARY_CBIN_205_SearchTokensLimit; UPDATED=2026-08-28 SearchTokens searches by keywords across keyword tags, feature names, requirement IDs, file paths, test names, and bench names, bounded by limit (or DefaultSearchLimit when limit <= 0). projectID scopes the search; "" spans every project in the database.

func (*DB) UpdatePriority

func (db *DB) UpdatePriority(projectID, reqID, feature string, priority int) error

CANARY: REQ=CBIN-308; FEATURE="ScopedWrites"; ASPECT=Storage; STATUS=TESTED; TEST=TestUpdatePriorityRejectsUnscoped,TestUpdateSpecStatusRejectsUnscoped,TestPrioritizeWritesOneProject; UPDATED=2026-08-30 UpdatePriority updates the priority of a token within projectID.

projectID is required. The unscoped default a *read* enjoys has no writer equivalent: an unscoped UPDATE does not answer a broader question, it rewrites every project's rows in a shared database. Refusing here is the same rule DeleteAllTokens and ReplaceIndex already enforce.

func (*DB) UpdateSpecStatus

func (db *DB) UpdateSpecStatus(projectID, reqID, specStatus string) error

UpdateSpecStatus updates the spec status within projectID. projectID is required, for the reason spelled out on UpdatePriority.

func (*DB) UpsertToken

func (db *DB) UpsertToken(token *Token) error

UpsertToken inserts or updates a token

type DatabaseManager

type DatabaseManager struct {
	// contains filtered or unexported fields
}

DatabaseManager manages both global and local database connections

func NewDatabaseManager

func NewDatabaseManager() *DatabaseManager

NewDatabaseManager creates a new database manager

func (*DatabaseManager) Close

func (dm *DatabaseManager) Close() error

Close closes the database connection

func (*DatabaseManager) DB

func (dm *DatabaseManager) DB() *sql.DB

DB returns the underlying database connection

func (*DatabaseManager) Discover

func (dm *DatabaseManager) Discover() error

Discover attempts to find an existing database, with local taking precedence over global

func (*DatabaseManager) Initialize

func (dm *DatabaseManager) Initialize(mode DatabaseMode) error

Initialize initializes the database in the specified mode

func (*DatabaseManager) Location

func (dm *DatabaseManager) Location() string

Location returns the database file path

func (*DatabaseManager) Mode

func (dm *DatabaseManager) Mode() DatabaseMode

Mode returns the current database mode

type DatabaseMode

type DatabaseMode int

DatabaseMode represents the initialization mode for the database

const (
	GlobalMode DatabaseMode = iota
	LocalMode
)

func (DatabaseMode) String

func (dm DatabaseMode) String() string

String returns the string representation of DatabaseMode

type GapCategory

type GapCategory struct {
	ID          int
	Name        string
	Description string
	CreatedAt   time.Time
}

GapCategory represents a gap category

type GapConfig

type GapConfig struct {
	ID                  int
	MaxGapInjection     int
	MinHelpfulThreshold int
	RankingStrategy     string
	CreatedAt           time.Time
	UpdatedAt           time.Time
}

GapConfig represents gap analysis configuration

type GapEntry

type GapEntry struct {
	ID               int
	GapID            string
	ReqID            string
	Feature          string
	Aspect           string
	Category         string
	Description      string
	CorrectiveAction string
	CreatedAt        time.Time
	CreatedBy        string
	HelpfulCount     int
	UnhelpfulCount   int
}

GapEntry represents a gap analysis entry

type GapQueryFilter

type GapQueryFilter struct {
	ReqID    string
	Feature  string
	Aspect   string
	Category string
	Limit    int
}

GapQueryFilter represents query filters for gap entries

type GapRepository

type GapRepository struct {
	// contains filtered or unexported fields
}

GapRepository handles gap analysis database operations

func NewGapRepository

func NewGapRepository(db *DB) *GapRepository

NewGapRepository creates a new gap repository

func (*GapRepository) CreateEntry

func (r *GapRepository) CreateEntry(entry *GapEntry) error

CreateEntry creates a new gap analysis entry

func (*GapRepository) GenerateGapReport

func (r *GapRepository) GenerateGapReport(reqID string) (string, error)

GenerateGapReport generates a formatted gap analysis report

func (*GapRepository) GetCategories

func (r *GapRepository) GetCategories() ([]*GapCategory, error)

GetCategories retrieves all gap categories

func (*GapRepository) GetConfig

func (r *GapRepository) GetConfig() (*GapConfig, error)

GetConfig retrieves the gap analysis configuration

func (*GapRepository) GetEntriesByReqID

func (r *GapRepository) GetEntriesByReqID(reqID string) ([]*GapEntry, error)

GetEntriesByReqID retrieves all gap entries for a requirement

func (*GapRepository) GetEntryByGapID

func (r *GapRepository) GetEntryByGapID(gapID string) (*GapEntry, error)

GetEntryByGapID retrieves a gap entry by its gap ID

func (*GapRepository) GetTopGaps

func (r *GapRepository) GetTopGaps(reqID string, config *GapConfig) ([]*GapEntry, error)

GetTopGaps retrieves top gaps for a requirement based on configuration

func (*GapRepository) MarkHelpful

func (r *GapRepository) MarkHelpful(gapID string) error

MarkHelpful increments the helpful count for a gap entry

func (*GapRepository) MarkUnhelpful

func (r *GapRepository) MarkUnhelpful(gapID string) error

MarkUnhelpful increments the unhelpful count for a gap entry

func (*GapRepository) QueryEntries

func (r *GapRepository) QueryEntries(filter GapQueryFilter) ([]*GapEntry, error)

QueryEntries queries gap entries with filters

func (*GapRepository) UpdateConfig

func (r *GapRepository) UpdateConfig(config *GapConfig) error

UpdateConfig updates the gap analysis configuration

type IndexMeta added in v0.3.3

type IndexMeta struct {
	// Root is the directory the index was built from.
	Root string
	// ProjectID is the project the indexed tokens were written under.
	ProjectID string
	// CommitSHA is the git HEAD at index time, or "" when the tree is not a
	// git repository. It is never fabricated.
	CommitSHA string
	// ParserSchema is canaryscan.ParserSchemaVersion at index time, so an
	// index built by an older grammar can be recognised as stale.
	ParserSchema int
	// ScanDigest is the digest of the scanned file set (see index command).
	ScanDigest string
	// IndexedAt is the RFC3339 UTC timestamp of the run.
	IndexedAt string
}

IndexMeta records what an index was built from. Without it a reader can only see rows, not whether those rows still describe the tree in front of it -- which is how `canary next` came to announce "all requirements completed" over a database that was empty because it had never been built.

type Project

type Project struct {
	ID        string
	Name      string
	Path      string
	Active    bool
	CreatedAt string
	Metadata  string // JSON metadata
}

Project represents a registered project in the canary system

type ProjectRegistry

type ProjectRegistry struct {
	// contains filtered or unexported fields
}

ProjectRegistry manages project registration and queries

func NewProjectRegistry

func NewProjectRegistry(manager *DatabaseManager) *ProjectRegistry

NewProjectRegistry creates a new project registry

func (*ProjectRegistry) GetByID

func (pr *ProjectRegistry) GetByID(id string) (*Project, error)

GetByID retrieves a project by its ID

func (*ProjectRegistry) GetByPath

func (pr *ProjectRegistry) GetByPath(path string) (*Project, error)

GetByPath retrieves a project by its path

func (*ProjectRegistry) List

func (pr *ProjectRegistry) List() ([]*Project, error)

List returns all registered projects

func (*ProjectRegistry) Register

func (pr *ProjectRegistry) Register(project *Project) error

Register adds a new project to the registry

func (*ProjectRegistry) Remove

func (pr *ProjectRegistry) Remove(id string) error

Remove deletes a project from the registry

type Ref

type Ref struct {
	ProjectID  string `db:"project_id" json:"project_id"`
	ReqID      string `db:"req_id" json:"req_id"`
	Kind       string `db:"kind" json:"kind"`
	FilePath   string `db:"file_path" json:"file_path"`
	LineNumber int    `db:"line_number" json:"line_number"`
	Context    string `db:"context" json:"context,omitempty"`
}

Ref is a requirement reference found outside CANARY tokens (diagrams, docs).

type Token

type Token struct {
	ID          int
	ReqID       string
	Feature     string
	Aspect      string
	Status      string
	FilePath    string
	LineNumber  int
	Test        string
	Bench       string
	Owner       string
	Priority    int
	Phase       string
	Keywords    string
	SpecStatus  string
	CreatedAt   string
	UpdatedAt   string
	StartedAt   string
	CompletedAt string
	CommitHash  string
	Branch      string
	DependsOn   string
	Blocks      string
	RelatedTo   string
	RawToken    string
	IndexedAt   string

	// CANARY: REQ=ENG-4315; FEATURE="DocDatabaseSchema"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-16
	// Documentation tracking fields
	DocPath      string // Comma-separated doc file paths (e.g., "user:docs/user.md,api:docs/api.md")
	DocHash      string // Comma-separated SHA256 hashes (abbreviated, first 16 chars)
	DocType      string // Documentation type (user, technical, feature, api, architecture)
	DocCheckedAt string // ISO 8601 timestamp of last staleness check
	DocStatus    string // DOC_CURRENT, DOC_STALE, DOC_MISSING, DOC_UNHASHED

	// CANARY: REQ=ENG-4319; FEATURE="TokenNamespacing"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-18
	// Multi-project support
	ProjectID string // Project identifier for token isolation

	// ContentHash is the hex SHA-256 of the file this token was read from at
	// index time, so a row can be checked against disk without re-scanning.
	ContentHash string
}

Token represents a parsed CANARY token with extended metadata

Directories

Path Synopsis
CANARY: REQ=ENG-4319; FEATURE="TestInfrastructure"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-18
CANARY: REQ=ENG-4319; FEATURE="TestInfrastructure"; ASPECT=Storage; STATUS=IMPL; UPDATED=2025-10-18

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL