Documentation
¶
Overview ¶
Package manifest provides file manifest and hash tracking for cache invalidation.
This package implements a ManifestManager that scans directories, computes file hashes (SHA256), and detects changes between scans. It is used by the graph cache (CB-12) and hash verification (CB-14) subsystems.
Design Principles ¶
Security is paramount - all paths are validated to prevent directory traversal. Performance is optimized with mtime-first checking before computing hashes. Reliability is ensured through atomic hash computation with TOCTOU protection.
Thread Safety ¶
ManifestManager is safe for concurrent use. Individual Manifest structs are NOT safe for concurrent modification after creation.
Index ¶
Constants ¶
const ( // DefaultMaxFileSize is the default maximum file size for hashing (100MB). DefaultMaxFileSize = 100 * 1024 * 1024 // DefaultMaxRetries is the default number of retries for atomic hashing. DefaultMaxRetries = 3 )
Default configuration values.
Variables ¶
var ( // ErrPathTraversal is returned when a path escapes the project root. // This is a security error that prevents access to files outside the // validated project boundary. ErrPathTraversal = errors.New("path escapes project root") // ErrFileTooLarge is returned when a file exceeds MaxFileSize. // Large files are skipped to prevent memory exhaustion during hashing. ErrFileTooLarge = errors.New("file too large to hash") // ErrFileUnstable is returned when a file changes during hashing after // exhausting all retry attempts. This indicates the file is being actively // written to and cannot be reliably hashed. ErrFileUnstable = errors.New("file changed during hashing") // ErrInvalidHash is returned when a stored hash is malformed. // Valid hashes are exactly 64 lowercase hexadecimal characters. ErrInvalidHash = errors.New("invalid hash format") // ErrSymlinkCycle is returned when symlink following detects a cycle. // This prevents infinite loops when traversing symlinked directories. ErrSymlinkCycle = errors.New("symlink cycle detected") // ErrInvalidRoot is returned when the project root path is invalid. ErrInvalidRoot = errors.New("invalid project root") )
Sentinel errors for manifest operations.
var ( // DefaultIncludes specifies patterns for common source file types. DefaultIncludes = []string{ "**/*.go", "**/*.py", "**/*.ts", "**/*.tsx", "**/*.js", "**/*.jsx", "**/*.html", "**/*.css", } // DefaultExcludes specifies patterns for commonly excluded directories and files. DefaultExcludes = []string{ "vendor/**", "node_modules/**", ".git/**", "**/testdata/**", "**/*_test.go", "**/*.test.ts", "**/*.spec.ts", } )
Default glob patterns for common source files.
Functions ¶
This section is empty.
Types ¶
type Changes ¶
type Changes struct {
// Added contains relative paths of files that exist in the new
// manifest but not in the old manifest.
Added []string `json:"added,omitempty"`
// Modified contains relative paths of files that exist in both
// manifests but have different hashes.
Modified []string `json:"modified,omitempty"`
// Deleted contains relative paths of files that exist in the old
// manifest but not in the new manifest.
Deleted []string `json:"deleted,omitempty"`
}
Changes represents the differences between two manifests.
func (*Changes) HasChanges ¶
HasChanges returns true if there are any added, modified, or deleted files.
type FileEntry ¶
type FileEntry struct {
// Path is the relative path from project root.
Path string `json:"path"`
// Hash is the SHA256 hash of the file contents.
// Format: 64 lowercase hexadecimal characters.
Hash string `json:"hash"`
// Mtime is the file modification time in Unix nanoseconds.
Mtime int64 `json:"mtime"`
// Size is the file size in bytes.
Size int64 `json:"size"`
}
FileEntry represents a single file in the manifest.
type GlobMatcher ¶
type GlobMatcher struct {
// contains filtered or unexported fields
}
GlobMatcher provides file path matching against include/exclude patterns.
Patterns use glob syntax with ** for recursive matching:
- * matches any sequence of non-separator characters
- ** matches any sequence of characters including separators
- ? matches any single non-separator character
- [abc] matches one of the characters in brackets
Thread Safety: GlobMatcher is safe for concurrent use after creation.
func NewGlobMatcher ¶
func NewGlobMatcher(includes, excludes []string) *GlobMatcher
NewGlobMatcher creates a matcher with the given include and exclude patterns.
If includes is empty, all files are included by default. If excludes is empty, no files are excluded.
func (*GlobMatcher) Match ¶
func (m *GlobMatcher) Match(path string) bool
Match returns true if the path should be included.
A path is included if:
- It matches at least one include pattern (or includes is empty), AND
- It does not match any exclude pattern
The path should use forward slashes as separators for consistency.
type Hasher ¶
type Hasher interface {
// HashFile computes SHA256 of file contents.
//
// Returns lowercase hex string (64 chars) on success.
// Returns error if file cannot be read or exceeds size limit.
HashFile(path string) (string, error)
// HashFileAtomic computes hash with TOCTOU protection.
//
// Verifies that mtime is unchanged after hashing. If the file changes
// during hashing, the operation is retried up to maxRetries times.
//
// Returns a complete FileEntry on success, including hash, mtime, and size.
// Returns ErrFileUnstable if the file keeps changing after all retries.
HashFileAtomic(path string, maxRetries int) (FileEntry, error)
}
Hasher defines the interface for file hashing operations.
type ManagerOption ¶
type ManagerOption func(*ManifestManager)
ManagerOption is a functional option for configuring ManifestManager.
func WithExcludes ¶
func WithExcludes(patterns ...string) ManagerOption
WithExcludes sets the exclude glob patterns.
func WithFollowSymlinks ¶
func WithFollowSymlinks(follow bool) ManagerOption
WithFollowSymlinks enables or disables following symlinks.
func WithHasher ¶
func WithHasher(h Hasher) ManagerOption
WithHasher sets a custom hasher implementation.
func WithIncludes ¶
func WithIncludes(patterns ...string) ManagerOption
WithIncludes sets the include glob patterns.
func WithMaxFileSize ¶
func WithMaxFileSize(bytes int64) ManagerOption
WithMaxFileSize sets the maximum file size for hashing.
func WithMaxRetries ¶
func WithMaxRetries(n int) ManagerOption
WithMaxRetries sets the maximum retry count for atomic hashing.
type Manifest ¶
type Manifest struct {
// ProjectRoot is the absolute path to the project root directory.
ProjectRoot string `json:"project_root"`
// Files maps relative file paths to their entries.
// Keys are relative paths from ProjectRoot.
Files map[string]FileEntry `json:"files"`
// Errors contains files that failed during scanning.
// These files are not included in the Files map.
Errors []ScanError `json:"errors,omitempty"`
// CreatedAtMilli is the Unix timestamp in milliseconds when the
// manifest was first created.
CreatedAtMilli int64 `json:"created_at_milli"`
// UpdatedAtMilli is the Unix timestamp in milliseconds when the
// manifest was last updated.
UpdatedAtMilli int64 `json:"updated_at_milli"`
// Incomplete is true if the scan was cancelled before completion.
// When true, the Files map contains only a partial result.
Incomplete bool `json:"incomplete,omitempty"`
}
Manifest represents the state of all tracked files in a project.
A Manifest is created by scanning a project directory and recording the hash and metadata for each file. Manifests are compared using Diff() to detect changes.
func NewManifest ¶
NewManifest creates an empty manifest for the given project root.
func (*Manifest) ErrorCount ¶
ErrorCount returns the number of files that failed scanning.
type ManifestManager ¶
type ManifestManager struct {
// contains filtered or unexported fields
}
ManifestManager provides file manifest creation and comparison.
Thread Safety: ManifestManager is safe for concurrent use.
func NewManifestManager ¶
func NewManifestManager(opts ...ManagerOption) *ManifestManager
NewManifestManager creates a new ManifestManager with the given options.
Default configuration:
- maxFileSize: 100MB
- followSymlinks: false
- maxRetries: 3
- includes: DefaultIncludes
- excludes: DefaultExcludes
func (*ManifestManager) Diff ¶
func (m *ManifestManager) Diff(old, new *Manifest) *Changes
Diff compares two manifests and returns the changes.
Description:
Compares the Files maps of old and new manifests to identify added, modified, and deleted files.
Inputs:
old - The previous manifest (may be nil). new - The current manifest (must not be nil).
Outputs:
*Changes - The differences between old and new. Never nil.
Behavior:
- If old is nil, all files in new are considered added
- Errors fields are not compared
- Comparison is based on hash, not mtime
func (*ManifestManager) QuickCheck ¶
func (m *ManifestManager) QuickCheck(ctx context.Context, root string, entry FileEntry) (changed bool, err error)
QuickCheck determines if a file has changed since it was last hashed.
Description:
Uses mtime-first optimization: checks mtime before computing hash. If mtime is unchanged, assumes file is unchanged (fast path). If mtime changed, recomputes hash and compares.
Inputs:
ctx - Context for cancellation. root - Absolute path to the project root directory. entry - The FileEntry to check.
Outputs:
changed - True if the file has changed (or was deleted). err - Non-nil if an unexpected error occurred.
Behavior:
- If file is deleted, returns (true, nil)
- If mtime unchanged, returns (false, nil) without hashing
- If mtime changed, hashes and compares
func (*ManifestManager) Scan ¶
Scan walks a project directory and creates a manifest of all matching files.
Description:
Recursively walks the directory, applying include/exclude patterns, and computes hashes for matching files. Non-fatal errors (permission denied, large files) are recorded in the manifest's Errors field.
Inputs:
ctx - Context for cancellation. If cancelled, returns partial manifest. root - Absolute path to the project root directory.
Outputs:
*Manifest - The scan result. Never nil. error - Non-nil if root is invalid or cannot be accessed.
Behavior:
- Symlinks are NOT followed unless WithFollowSymlinks(true)
- Files larger than maxFileSize are skipped (added to Errors)
- Permission errors are recorded but don't stop scanning
- Context cancellation sets Incomplete=true and returns partial result
type SHA256Hasher ¶
type SHA256Hasher struct {
// contains filtered or unexported fields
}
SHA256Hasher implements Hasher using SHA256.
func NewSHA256Hasher ¶
func NewSHA256Hasher(maxFileSize int64) *SHA256Hasher
NewSHA256Hasher creates a new SHA256Hasher with the given size limit.
If maxFileSize is 0, no size limit is enforced. If maxFileSize is negative, it defaults to DefaultMaxFileSize.
func (*SHA256Hasher) HashFile ¶
func (h *SHA256Hasher) HashFile(path string) (string, error)
HashFile computes SHA256 of file contents.
Description:
Opens the file, reads its contents, and computes the SHA256 hash. Uses streaming to avoid loading the entire file into memory.
Inputs:
path - Absolute or relative path to the file.
Outputs:
string - Lowercase hexadecimal hash (64 characters). error - Non-nil if file cannot be read or exceeds size limit.
Errors:
ErrFileTooLarge - File size exceeds maxFileSize limit. os errors - File doesn't exist, permission denied, etc.
func (*SHA256Hasher) HashFileAtomic ¶
func (h *SHA256Hasher) HashFileAtomic(path string, maxRetries int) (FileEntry, error)
HashFileAtomic computes hash with TOCTOU race detection.
Description:
Computes the file hash while verifying that the file hasn't changed during the operation. Uses stat-before and stat-after to detect modifications. Retries if the file changes during hashing.
Inputs:
path - Absolute or relative path to the file. maxRetries - Maximum number of retry attempts (0 means no retries).
Outputs:
FileEntry - Complete entry with path, hash, mtime, and size. error - Non-nil if hashing fails after all retries.
Errors:
ErrFileUnstable - File changed during hashing after all retries. ErrFileTooLarge - File size exceeds maxFileSize limit. os errors - File doesn't exist, permission denied, etc.
Algorithm:
- Lstat file to get initial mtime and size
- Compute hash (streaming)
- Lstat file again to get final mtime and size
- If mtime and size unchanged, return FileEntry
- If changed, retry (up to maxRetries)
- If still changing after retries, return ErrFileUnstable
type ScanError ¶
type ScanError struct {
// Path is the relative path to the file that failed.
Path string `json:"path"`
// Err is the underlying error.
Err error `json:"error"`
}
ScanError represents a non-fatal error during scanning.
When a file cannot be processed (e.g., permission denied), it is recorded as a ScanError and scanning continues. The Manifest's Errors field contains all such errors.
func (ScanError) MarshalJSON ¶
MarshalJSON implements json.Marshaler. Serializes the error as its string representation.
Source Files