canaryscan

package
v0.3.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

CANARY: REQ=CP-268; FEATURE="MermaidRefs"; ASPECT=Engine; STATUS=TESTED; TEST=TestCANARY_CBIN_202_ExtractDiagramRefs,TestCANARY_CBIN_202_ScanDiagramRefsSkipsOversizedFile; UPDATED=2026-08-31

CANARY: REQ=ENG-4325; FEATURE="MigrateNotes"; ASPECT=Engine; STATUS=TESTED; TEST=TestCANARY_CBIN_301_ExtractMigrateNotes; UPDATED=2026-08-31

CANARY: REQ=CP-277; FEATURE="StalenessConfig"; ASPECT=Engine; STATUS=TESTED; TEST=TestCANARY_CBIN_304_StaleDaysFromConfig,TestCANARY_CBIN_304_UpdateStaleReportsEvidenceCurrency,TestCANARY_CBIN_304_UpdateStaleMutatesNothing; UPDATED=2026-08-30

Index

Constants

View Source 
const (
	MaxFileBytes  = 16 << 20 // 16 MiB per file
	MaxLineBytes  = 1 << 20  // 1 MiB per logical line
	MaxFieldBytes = 1024     // per token field value, UTF-8 bytes
	// ParserSchemaVersion identifies the token grammar/semantics. Bumped to
	// 2 when STATUS became a pure declaration (no lexical promotion).
	ParserSchemaVersion = 2
	// TokenSchemaVersion seeds SnapshotSeed (records.go): bump it whenever
	// the snapshot's parse semantics change, so a stored ScanDigest computed
	// under the old schema reads as stale even if project.yaml, .canaryignore,
	// and every token file are byte-identical.
	TokenSchemaVersion = 2
)

Scanner and token-grammar limits. These bound what a single scan will read and what a single token field may hold, so a hostile or accidental input (a multi-gigabyte artifact, a minified one-line bundle) cannot exhaust memory or wedge a scan.

View Source 
const (
	IssueBinary         = "binary"
	IssueFileTooLarge   = "file_too_large"
	IssueLineTooLarge   = "line_too_large"
	IssueReadError      = "read_error"
	IssueParseError     = "parse_error"
	IssueStatusConflict = "status_conflict"
	IssueSubscanError   = "subscan_error"
)

Scan issue reasons.

View Source 
const DefaultStaleDays = config.DefaultStaleDays

DefaultStaleDays is the fallback staleness window (in days) when neither Config.StaleDays nor .canary/project.yaml's verification.staleness_days is set. It is pkg/config's constant under canaryscan's historical name.

View Source 
const EvidenceFile = ".canary/evidence.json"

EvidenceFile is the evidence store's path relative to a project root.

Variables

View Source 
var ErrTokenMoved = errors.New("token not found at recorded location; run 'canary index'")

ErrTokenMoved reports that the recorded file:line no longer holds the token identity expects -- the source moved, changed, or was deleted since the caller's row was last read. The caller has no safe edit to make; it must tell the operator to reindex.

View Source 
var UndeclaredKey = evidence.FeatureKey{Feature: "*", Aspect: "*"}

UndeclaredKey is the stand-in feature/aspect used for a claimed requirement that declares no tokens at all. Such a claim can never be satisfied by evidence — there is nothing for evidence to attest to — so it is reported as one missing entry rather than passing vacuously.

Functions

func AnnotateSources 

func AnnotateSources(rep *Report, reg *sources.Registry)

AnnotateSources stamps each requirement with its source name and ticket URL.

func Claims added in v0.3.3 

func Claims(gapPath string, reg *sources.Registry) ([]string, error)

Claims returns the requirement IDs claimed in gapPath, in first-seen order with duplicates removed. The claim grammar is reg.ClaimPattern(): "✅ <KEY>-<digits>"; a nil registry means the default (CBIN).

func DeclaredFeatures added in v0.3.3 

func DeclaredFeatures(rep Report, reqID string) []evidence.FeatureKey

DeclaredFeatures returns rep's declared feature/aspect keys for reqID, deduplicated. Two tokens differing only in OWNER or UPDATED describe one thing to prove, so the key is required once, not twice. An id with no declarations returns nil.

func DefaultSkipRegex 

func DefaultSkipRegex() *regexp.Regexp

DefaultSkipRegex returns the default skip path regex.

func HeadCommit added in v0.3.3 

func HeadCommit(root string) (string, error)

HeadCommit returns root's current commit SHA. Evidence binds to a commit, so a repository whose HEAD cannot be read cannot be verified.

func LoadCanaryIgnore 

func LoadCanaryIgnore(root string) (*ignore.GitIgnore, error)

LoadCanaryIgnore loads .canaryignore from root. Returns nil if file missing.

func RefTimeFromEnv 

func RefTimeFromEnv() time.Time

RefTimeFromEnv returns time from CANARY_TEST_TIMESTAMP (RFC3339) for tests; zero if unset or invalid.

func ReportEvidenceCurrency added in v0.3.3 

func ReportEvidenceCurrency(rep Report, staleDiags []string, recs []evidence.Record, projectID, commit string) []string

ReportEvidenceCurrency answers, for every requirement named by staleDiags, whether it has passing evidence at the current commit.

This replaces the old --update-stale behavior, which rewrote UPDATED= dates in source. Rewriting a date made a stale claim *look* fresh without any new proof — the exact failure mode evidence-backed verification exists to prevent. Nothing is mutated here: the report is the whole output, one line per requirement, sorted by ID: 

CANARY_UPDATE_STALE req=<id> evidence=current
CANARY_UPDATE_STALE req=<id> evidence=missing

"current" means every feature/aspect the requirement declares has a PASS record for this project at this commit; anything less is "missing".

func RequiredFeatures added in v0.3.3 

func RequiredFeatures(rep Report, claimed []string) map[string][]evidence.FeatureKey

RequiredFeatures maps each claimed requirement to the feature/aspect pairs it declares in rep. A claimed requirement rep knows nothing about maps to a single UndeclaredKey entry so it is still checked (and fails), never silently skipped.

func RewriteTokenFields added in v0.3.4 

func RewriteTokenFields(root, filePath string, lineNumber int, identity, set map[string]string) error

RewriteTokenFields rewrites the CANARY token on one line of one file, preserving the line's comment prefix and terminator. identity keys (e.g. {"REQ": "CBIN-101"} or {"BUG": "BUG-API-001"}) are verified against the parsed token before any edit; a mismatch (moved/stale line) returns ErrTokenMoved. set replaces existing fields' values or appends new fields before the terminator. The result is round-trip validated with ParseTokenLine and written via safewrite (Force, Backup).

func RewriteTokensAtomic added in v0.3.5 

func RewriteTokensAtomic(root string, ops []RewriteOp) (func() error, error)

RewriteTokensAtomic stages every op (read the file, compute its rewritten bytes via computeRewrite) before writing anything. If staging any op fails -- a missing file, a moved/stale token line -- nothing is written to disk at all: a `bug update` or `prioritize` touching several files must never leave some rewritten and others untouched because a later op in the batch turned out to be invalid.

If a write partway through the batch fails, writes 0..k-1 are rolled back (restored from their pre-images) before the error is returned, so a mid-batch I/O failure never leaves a partially-rewritten tree either.

On success it returns a restore func the caller can invoke if a LATER operation -- typically the corresponding database write -- fails, undoing every file this call wrote and putting the tree back exactly as it found it.

func Run 

func Run(cfg Config, stdout, stderr io.Writer) (exitCode int)

Run runs the full scan pipeline: load config, scan, optional update-stale, write outputs, --strict. Verification is not a scan concern: `scan --verify` delegates wholly to the dedicated verifier (pkg/cmds/verify), never to this function. It writes to stdout/stderr via the given writers (use os.Stdout, os.Stderr from CLI). Returns exit code: 0 success, 2 --strict failure, 3 parse/IO error.

func RunFromArgs 

func RunFromArgs(root, out, csv, skipExpr string, strict, updateStale, projectOnly bool, staleDays int) int

RunFromArgs is a convenience that builds Config from flag-like args and runs with os.Stdout/os.Stderr. Used by tools/canary main. skipExpr is the regex string; if empty, default is used. staleDays overrides the staleness window in days; 0 means "use config/default". Verification is not a scan concern: tools/canary's --verify spelling delegates directly to pkg/cmds/verify.Run rather than passing through here.

func ScanDiagramRefs 

func ScanDiagramRefs(root string, skip *regexp.Regexp, reg *sources.Registry, ignorePatterns *ignore.GitIgnore) ([]DiagramRef, []ScanIssue, error)

ScanDiagramRefs walks root for .md/.markdown/.mmd files and extracts all requirement-ID mentions inside mermaid diagrams. Paths in the result are root-relative with forward slashes. ignorePatterns (from LoadCanaryIgnore) is honored the same way Scan honors it: relative path matched first, dirs skipped via SkipDir; nil means no .canaryignore patterns apply.

A file that cannot be read is reported as a ScanIssue rather than silently skipped, so callers can tell an empty result from an incomplete one.

func ScanDigest added in v0.3.3 

func ScanDigest(seed string, files []FileDigest) string

ScanDigest folds a seed and a file-digest set into one digest: the seed line first, then the sorted "path:hash" lines. Two scans of the same tree under the same seed produce the same value; any change to any indexed file, or to the seed itself (parser schema, project.yaml, .canaryignore -- see SnapshotSeed), changes it.

V4-04/P0.3: before the seed existed, this digest bound only token-file bytes -- editing .canary/project.yaml (id_pattern/source registry semantics), .canaryignore, or the parser's own schema left the digest unchanged, so a "fresh" index could silently disagree with how its tokens would now be parsed. Every caller must pass the same seed the index was built with (SnapshotSeed(root)); see ComputeScanDigest in pkg/cmds/index/index.go and its use in `canary next`'s freshness check.

func ScanMigrateNotes added in v0.3.0 

func ScanMigrateNotes(root string, skip *regexp.Regexp, ignorePatterns *ignore.GitIgnore, reg *sources.Registry) ([]MigrateNote, []ScanIssue, error)

ScanMigrateNotes walks root and extracts CANARY:MIGRATE guidance notes from every non-binary file the scanner would otherwise read. Paths in the result are root-relative with forward slashes. ignorePatterns (from LoadCanaryIgnore) is honored the same way Scan honors it: relative path matched first, dirs skipped via SkipDir; nil means no .canaryignore patterns apply. Files larger than 1MB are skipped.

A file that cannot be read is reported as a ScanIssue rather than silently skipped, so callers can tell an empty result from an incomplete one.

func ScanSummaryLine 

func ScanSummaryLine(rep Report) string

ScanSummaryLine returns a single parseable line for rep so agents get metrics without reading status.json.

func ScanTokenRecords added in v0.3.3 

func ScanTokenRecords(root string, skip *regexp.Regexp, projectFilter *regexp.Regexp, ignorePatterns *ignore.GitIgnore, reg *sources.Registry) ([]TokenRecord, []FileDigest, []ScanIssue, error)

ScanTokenRecords walks root and returns every CANARY token it finds, one record per token, alongside the digest of every file that contributed at least one token and the issues the scan hit. skip, projectFilter, ignorePatterns and reg may be nil.

Issues are returned rather than swallowed: a caller that is building an index (as opposed to printing a report) can refuse to commit anything when the tree contains a token the parser rejects.

func SerializeToken added in v0.3.3 

func SerializeToken(fields []Field) (string, error)

SerializeToken renders canonical form: `// CANARY: K=V; K="quoted"`. Values that could otherwise be misread are quoted with `\`→`\\` and `"`→`\"` escapes; `;` is legal only inside quotes. Control characters (C0 and DEL), oversized values, unknown ASPECT/STATUS enum members and malformed REQ ids are refused.

func SkipMatch added in v0.3.5 

func SkipMatch(skip *regexp.Regexp, path string) bool

SkipMatch reports whether path matches skip, a slash-anchored pattern. path is normalized with filepath.ToSlash first so an OS-native separator (Windows' backslash) can never let a directory the pattern was written to exclude slip through unmatched. nil-safe: a nil skip regexp never matches.

This is the single canonical skip check for every scanner in this repository (records.go, mermaid.go, migrate_notes.go, pkg/cmds/onboard, pkg/upgrade) — no caller should call skip.MatchString directly on an OS-native path.

func SnapshotSeed added in v0.3.5 

func SnapshotSeed(root string) string

SnapshotSeed binds a scan digest to everything besides token-file bytes that determines how those bytes are interpreted: the parser schema version, the registry/config semantics in <root>/.canary/project.yaml (id_pattern, source definitions), and the exclusion rules in <root>/.canaryignore. Two roots with byte-identical token files but different project.yaml or .canaryignore content -- or scanned by two binaries with different TokenSchemaVersion -- must never produce the same seed, or a "fresh" index could describe a tree it no longer accurately parses (V4-04/P0.3).

A missing file contributes the literal "absent"; a read error other than not-exist contributes the literal "unreadable" rather than failing here -- the index refuses to build on an unreadable .canaryignore elsewhere (LoadCanaryIgnore), so this function stays total and deterministic and lets that caller report the real error.

func Stale 

func Stale(rep Report, maxAge time.Duration, refTime time.Time) []string

Stale returns diagnostics for TESTED/BENCHED tokens older than maxAge. If refTime is zero, time.Now().UTC() is used.

func StateSkipRegex added in v0.3.3 

func StateSkipRegex() *regexp.Regexp

StateSkipRegex returns the default skip path regex extended with canary's own `.canary` state directory. See stateSkipPattern.

func ValidStatus added in v0.3.4 

func ValidStatus(s string) bool

ValidStatus reports whether s is one of the token grammar's known STATUS enum members. Implementation-lifecycle statuses (STUB, IMPL, TESTED, BENCHED, ...) and bug-lifecycle statuses (OPEN, IN_PROGRESS, FIXED, ...) share one set, matching what SerializeToken itself enforces on a STATUS field -- callers validate against this instead of guessing a subset so a value that this package would refuse to serialize is never accepted.

func VerifiedRequirements added in v0.3.3 

func VerifiedRequirements(rep Report, recs []evidence.Record, projectID, commit string) []string

VerifiedRequirements returns, sorted, the IDs of every requirement in rep whose declared features all have passing evidence for projectID at commit. It asks pkg/evidence.Complete -- the single completion function -- once per requirement, so a peer's export and this project's own `canary verify` answer the same question the same way.

The result is always non-nil (an empty slice when nothing is verified) so a caller can hand it straight to Report.Verified, where nil carries the distinct meaning "not checked".

func WorkingTreeClean added in v0.3.4 

func WorkingTreeClean(root string) (clean bool, status string, err error)

WorkingTreeClean reports whether root's git working tree matches HEAD -- including untracked files, which `git diff --exit-code` would miss. status carries the porcelain lines when dirty, for diagnostics.

func WriteCSV 

func WriteCSV(path string, rep Report) error

WriteCSV writes rep to path as CSV, atomically, one row per (requirement, feature, file). A feature's tests and benches are joined with "|" inside single cells rather than zipped column-wise against the file list: the old zip paired unrelated files, tests, and benches by shared row index, asserting adjacencies that never existed. Every cell is passed through csvGuard so no value can be interpreted as a spreadsheet formula, and encoding/csv quotes any value containing a comma, quote, or newline so the output is decodable by an independent reader.

func WriteJSON 

func WriteJSON(path string, rep Report) error

WriteJSON writes rep to path as JSON, atomically. The encode targets an in-memory buffer (which cannot short-write), and the file is committed by writeOut, whose error -- a failed create, write, or close -- is returned: a partial or unwritable report never passes silently.

Types

type Config 

type Config struct {
	Root        string
	Out         string
	CSV         string
	Strict      bool
	SkipRegex   *regexp.Regexp
	UpdateStale bool
	ProjectOnly bool
	// StaleDays overrides the staleness window (in days) used by --strict and
	// --update-stale. When <= 0, the effective threshold falls back to
	// .canary/project.yaml's verification.staleness_days, then to 30 days.
	StaleDays int
}

Config holds scanner run options. Used by both the standalone tools/canary binary and the canary scan CLI subcommand so scan works from any CWD.

type DiagramRef 

type DiagramRef struct {
	ReqID string
	File  string
	Line  int // 1-based
}

DiagramRef records one requirement-ID mention inside a mermaid diagram.

func ExtractDiagramRefs 

func ExtractDiagramRefs(relPath, content string, reg *sources.Registry) []DiagramRef

ExtractDiagramRefs finds requirement IDs inside ```mermaid fenced blocks. relPath ending in .mmd is treated as a whole-file mermaid diagram (no fences required). IDs are normalized through reg (flatfile padding); reg nil means sources.Default() (the legacy CBIN series).

type Feature 

type Feature struct {
	Feature string   `json:"feature"`
	Aspect  string   `json:"aspect"`
	Status  string   `json:"status"`
	Files   []string `json:"files"`
	Tests   []string `json:"tests"`
	Benches []string `json:"benches"`
	Owner   string   `json:"owner,omitempty"`
	Updated string   `json:"updated"`
	// Priority carries the token's declared PRIORITY (1 = highest). It is a
	// pointer so "declared nothing" stays distinguishable from "declared 0":
	// a consumer that needs a number applies its own default (5, the one
	// `canary index` uses) only when this is nil, rather than inventing one
	// for a token that did declare a value. An unparsable PRIORITY reads as
	// absent, matching `canary index`'s leniency.
	//
	// Additive: absent from a token means absent here.
	Priority *int `json:"priority,omitempty"`
	// DependsOn carries the token's DEPENDS_ON requirement IDs verbatim, so
	// verification can resolve external/peer dependency state without
	// re-parsing the tree. Additive: absent from a token means absent here.
	DependsOn []string `json:"depends_on,omitempty"`
}

Feature is one feature/aspect/status aggregation.

type Field added in v0.3.3 

type Field struct{ Key, Value string }

Field is one key/value pair of a CANARY token, in source order. Values are held decoded: quoting and escaping belong to the wire form only.

func ParseTokenLine added in v0.3.3 

func ParseTokenLine(line string) (fields []Field, ok bool, err error)

ParseTokenLine parses one line; ok=false when the line holds no CANARY token. CANARY:MIGRATE guidance lines are free text, not KV tokens, and so report ok=false as well. A malformed token returns ok=true with err set.

type FileDigest added in v0.3.3 

type FileDigest struct {
	Path string
	Hash string
}

FileDigest pairs a scanned file with the digest of its contents.

type MigrateNote added in v0.3.0 

type MigrateNote struct {
	File   string   `json:"file"`
	Line   int      `json:"line"` // 1-based
	Text   string   `json:"text"`
	ReqIDs []string `json:"req_ids,omitempty"` // registry-pattern matches inside Text, normalized
}

MigrateNote is one CANARY:MIGRATE guidance comment.

func ExtractMigrateNotes added in v0.3.0 

func ExtractMigrateNotes(relPath, content string, reg *sources.Registry) []MigrateNote

ExtractMigrateNotes finds CANARY:MIGRATE guidance comments in content. relPath is recorded verbatim on each note. reg nil means sources.Default() (the legacy CBIN series). A MIGRATE line is free text, not a KV token, and is never fed to parseKV — this is what keeps a MIGRATE line from ever aborting a scan.

type Report 

type Report struct {
	GeneratedAt    string        `json:"generated_at"`
	Requirements   []Requirement `json:"requirements"`
	Summary        Summary       `json:"summary"`
	MigrationNotes []MigrateNote `json:"migration_notes,omitempty"`
	Issues         []ScanIssue   `json:"issues,omitempty"`
	// Verified lists, sorted, the requirement IDs every one of whose declared
	// features had passing evidence at the scanned commit. It is the export a
	// peer project reads to answer "is this requirement done?" -- declarations
	// alone (STATUS=TESTED) never answer that.
	//
	// It is a pointer so absence is distinguishable from emptiness: nil means
	// this scan could not check evidence at all (no store, or no commit to
	// bind it to) and the key is omitted, which a reader must treat as
	// unknown. A non-nil empty slice is emitted as "verified": [] and means
	// the check ran and nothing passed.
	Verified *[]string `json:"verified,omitempty"`
}

Report is the scan output (JSON/CSV shape). Fields are only ever added: status.json is a cross-package wire contract.

func Scan 

func Scan(root string, skip *regexp.Regexp, projectFilter *regexp.Regexp, ignorePatterns *ignore.GitIgnore, reg *sources.Registry) (Report, error)

Scan walks root and returns a Report. skip, projectFilter, ignorePatterns and reg may be nil. Files the scanner cannot fully process are recorded in Report.Issues and skipped; only walk errors abort.

type Requirement 

type Requirement struct {
	ID        string    `json:"id"`
	Source    string    `json:"source,omitempty"`
	TicketURL string    `json:"ticket_url,omitempty"`
	Diagrams  []string  `json:"diagrams,omitempty"` // filled by Task 4
	Features  []Feature `json:"features"`
}

Requirement groups features by requirement ID.

type RewriteOp added in v0.3.5 

type RewriteOp struct {
	FilePath   string
	LineNumber int
	Identity   map[string]string
	Set        map[string]string
}

RewriteOp describes one token rewrite for RewriteTokensAtomic. FilePath is index-relative and is joined under the call's root, matching RewriteTokenFields.

type ScanIssue added in v0.3.3 

type ScanIssue struct {
	Path   string `json:"path"`
	Reason string `json:"reason"` // "binary" | "file_too_large" | "line_too_large" | "read_error" | "parse_error"
	Detail string `json:"detail,omitempty"`
}

ScanIssue records a file the scanner could not fully process. Issues never abort a scan: the file is skipped and the rest of the tree still scans.

type Summary 

type Summary struct {
	ByStatus           map[string]int `json:"by_status"`
	ByAspect           map[string]int `json:"by_aspect"`
	TotalTokens        int            `json:"total_tokens"`
	UniqueRequirements int            `json:"unique_requirements"`
}

Summary holds aggregate counts.

type TokenRecord added in v0.3.3 

type TokenRecord struct {
	// File is the path the token was read from, relative to the scan root.
	File string
	// Line is the 1-based line the token sits on.
	Line int
	// Raw is the token body verbatim, without the "CANARY:" marker.
	Raw string
	// ReqID is the registry-normalized requirement ID.
	ReqID string
	// Fields holds every declared key, upper-cased, values decoded.
	Fields map[string]string
	// ContentHash is the hex SHA-256 of the whole file this token came from.
	ContentHash string
}

TokenRecord is one CANARY token exactly where it was found. Scan folds tokens into per-requirement aggregates, which is the right shape for a report and the wrong shape for an index: the index needs the line each token sits on, its verbatim text, and every field it declared. Both views come out of the same parser, so the index can never disagree with the scan about what a token says.

func (TokenRecord) Field added in v0.3.3 

func (r TokenRecord) Field(key string) string

Field returns the named field, or "" when the token did not declare it. It never invents a value: an absent UPDATED reads as "" so a caller cannot mistake today's date for something the author wrote.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.