Documentation
¶
Overview ¶
Package storage provides a generic layered YAML store engine.
Both internal/config and internal/project compose a Store[T] with their own schema types. The store handles file discovery (static paths or walk-up), per-file loading with migrations, N-way merge with provenance tracking, and scoped writes with atomic I/O.
storage is low-level infrastructure: consumers compose a Store[T] behind a domain interface (see .claude/rules/store-backed-package.md) rather than exposing the store or constructing one at call sites.
Index ¶
- Variables
- func GenerateDefaultsYAML[T Schema]() string
- func ValidateDirectories() error
- type Field
- type FieldKind
- type FieldSet
- type KindFunc
- type LayerInfo
- type Migration
- type NormalizeOption
- type Option
- func WithCacheDir() Option
- func WithConfigDir() Option
- func WithDataDir() Option
- func WithDefaultFilename(name string) Option
- func WithDefaults(yaml string) Option
- func WithDefaultsFromStruct[T Schema]() Option
- func WithDirs(dirs ...string) Option
- func WithDotDefault() Option
- func WithFilenames(names ...string) Option
- func WithHeader(header string) Option
- func WithLock() Option
- func WithMigrations[T Schema](fns ...Migration[T]) Option
- func WithPaths(dirs ...string) Option
- func WithStateDir() Option
- func WithWalkUp(anchorDir string) Option
- type Options
- type Schema
- type Store
- func (s *Store[T]) Get(path string, out any) (bool, error)
- func (s *Store[T]) Has(path string) bool
- func (s *Store[T]) Layers() []LayerInfo
- func (s *Store[T]) MarkForWrite(path string) error
- func (s *Store[T]) MarkSeedForWrite()
- func (s *Store[T]) Options() Options
- func (s *Store[T]) Provenance(path string) (LayerInfo, bool)
- func (s *Store[T]) ProvenanceMap() map[string]string
- func (s *Store[T]) Read() *T
- func (s *Store[T]) Refresh() error
- func (s *Store[T]) Remove(path string) (bool, error)
- func (s *Store[T]) Set(path string, value any) error
- func (s *Store[T]) Txn(fn func(*Tx[T]) error) error
- func (s *Store[T]) Write() error
- func (s *Store[T]) WriteTargets() ([]WriteTarget, error)
- func (s *Store[T]) WriteTo(path string) error
- type TargetSource
- type Tx
- func (tx *Tx[T]) Get(path string, out any) (bool, error)
- func (tx *Tx[T]) Has(path string) bool
- func (tx *Tx[T]) MarkForWrite(path string) error
- func (tx *Tx[T]) Remove(path string) (bool, error)
- func (tx *Tx[T]) Set(path string, value any) error
- func (tx *Tx[T]) Write() error
- func (tx *Tx[T]) WriteTo(path string) error
- type WriteTarget
Constants ¶
This section is empty.
Variables ¶
var ErrAnchorNotAncestor = errors.New("walk-up anchor is not the current working directory or an ancestor of it")
ErrAnchorNotAncestor reports a walk-up anchor that is neither the current working directory nor one of its ancestors. An upward walk from CWD can never reach such an anchor and would escape to the filesystem root, so discovery refuses it as a caller programming error. The supported "no walk-up" case is an empty anchor, which disables walk-up entirely.
var ErrMigrationType = errors.New("migration has wrong type for Store[T]")
ErrMigrationType reports a migration whose store type does not match the Store[T] it was wired into (WithMigrations[T] not tied to New[T]'s T). Construction aborts rather than silently skipping the legacy-key cleanup.
var ErrMultiDocument = errors.New("expected a single YAML document")
ErrMultiDocument reports a YAML file containing more than one document. Config files are single-document; silently using only the first document would drop the rest, so the file is rejected loudly instead.
var ErrNonMappingRoot = errors.New("expected a mapping at the document root")
ErrNonMappingRoot reports a YAML document whose root node is not a mapping. Every layer must be a mapping so paths resolve against keyed fields.
var ErrSchemaDecode = errors.New("value no longer decodes into schema")
ErrSchemaDecode reports that grafting a value at a path produced a merged tree that no longer decodes into the schema type T. Set rejects the mutation and leaves the tree and snapshot untouched rather than persisting a tree that would poison the file on the next Write.
Functions ¶
func GenerateDefaultsYAML ¶ added in v0.6.0
GenerateDefaultsYAML walks the struct tags of T and produces a YAML string containing all fields that have a non-empty `default` tag. The output is suitable for WithDefaults — it provides the same base-layer behavior as a handwritten YAML constant, but derived from the struct definition.
Type coercion ensures YAML types match the Go field type:
- KindBool → YAML bool (true/false)
- KindInt → YAML int
- KindStringSlice → YAML sequence (comma-separated tag → []string)
- KindMap → YAML mapping (comma-separated key=value tag → map[string]string; split on the first "=" per entry, so values may contain "=" but not ",")
- KindDuration → YAML string (e.g. "30s")
- KindTime → YAML string (RFC3339Nano, e.g. "2026-01-01T00:00:00Z")
- KindText → YAML string
func ValidateDirectories ¶ added in v0.3.2
func ValidateDirectories() error
ValidateDirectories resolves all four XDG-style directories and returns an error if any two resolve to the same path. This catches misconfiguration (e.g. CLAWKER_DATA_DIR accidentally pointing at the config directory). TODO belongs in config package
Types ¶
type Field ¶ added in v0.6.0
type Field interface {
Path() string // Dotted YAML path (e.g. "build.image").
Kind() FieldKind // Data type classification.
Label() string // Short human-readable name (from `label` tag or derived from YAML key).
Description() string // Help text (from `desc` tag).
Default() string // Default value hint (from `default` tag), may be empty.
Required() bool // Whether this field must have a value (from `required:"true"` tag).
MergeTag() string // Merge strategy hint (from `merge` tag): "union", "overwrite", or "".
}
Field describes a single schema field's metadata. Concrete implementations are created by NewField or NormalizeFields.
type FieldKind ¶ added in v0.6.0
type FieldKind int
FieldKind classifies a schema field's data type for schema consumers (TUI editors, doc generators, CLI help).
const ( KindText FieldKind = iota // string KindBool // bool or *bool KindSelect // constrained string (options set by consumer) KindInt // int, int64 KindStringSlice // []string KindDuration // time.Duration KindTime // time.Time (serialized as an RFC3339Nano scalar) KindMap // map[string]string (only — other map types must register via KindFunc) KindStructSlice // []struct (non-string slice of structs) // KindLast is the boundary for storage-defined kinds. Consumer packages // define domain-specific kinds starting here: // // const KindMyType storage.FieldKind = storage.KindLast + 1 KindLast )
type FieldSet ¶ added in v0.6.0
type FieldSet interface {
All() []Field // All fields in discovery order.
Get(path string) Field // Lookup by dotted path; returns nil if not found.
Group(prefix string) []Field // Fields whose path starts with prefix + ".".
Len() int // Number of fields.
}
FieldSet is an ordered, indexed collection of Field values.
func NewFieldSet ¶ added in v0.6.0
NewFieldSet creates a FieldSet from a slice of Field values. The slice order is preserved by FieldSet.All.
func NormalizeFields ¶ added in v0.6.0
func NormalizeFields[T any](v T, opts ...NormalizeOption) FieldSet
NormalizeFields reflects over v's exported struct fields and produces a FieldSet containing schema metadata derived from struct tags:
- `yaml:"name"` — dotted path key (falls back to lowercased field name)
- `label:"Display Name"` — human-readable label (falls back to YAML key)
- `desc:"Help text"` — field description
- `default:"value"` — default value hint
- `required:"true"` — marks load-bearing fields that must have a value
Go type → FieldKind mapping:
- string → KindText
- bool, *bool → KindBool
- int, int64 → KindInt
- []string → KindStringSlice
- time.Duration → KindDuration
- struct, *struct → recursed (not a leaf field)
- map[string]string → KindMap
- []struct → KindStructSlice
- unrecognized types → KindFunc (if registered) → panic
NormalizeFields does NOT extract runtime values — only schema metadata. Panics if v is not a struct or pointer-to-struct, or if any exported field has an unsupported type and no KindFunc claims it.
type KindFunc ¶ added in v0.6.0
KindFunc classifies a reflect.Type that the normalizer does not recognize. Return (kind, true) to claim the type. Return (0, false) to fall through to the default panic.
type LayerInfo ¶
type LayerInfo struct {
Filename string // which filename matched (e.g., "clawker.yaml")
Path string // resolved absolute path
Data map[string]any // decoded view of this file's data (read-only copy)
}
LayerInfo describes a discovered file layer. Data is a decoded map view of the layer's node tree (comments dropped) for inspection — the engine itself holds the node.
type Migration ¶
Migration is a caller-provided function that inspects and optionally mutates a store's fields via its Get/Set/Remove member functions. It returns true if it changed anything (the store then re-saves the touched fields) and an error if the transform could not be applied — a non-nil error aborts construction rather than silently skipping the migration. Because it edits fields on the store's own node tree, comments on untouched fields are carried along by the tree — they survive the re-save without any extra work.
func dropLegacyKey(s *storage.Store[Settings]) (bool, error) {
return s.Remove("monitoring.legacy_port")
}
type NormalizeOption ¶ added in v0.6.0
type NormalizeOption func(*normalizeOpts)
NormalizeOption configures NormalizeFields behavior.
func WithKindFunc ¶ added in v0.6.0
func WithKindFunc(fn KindFunc) NormalizeOption
WithKindFunc registers a classifier for domain-specific types. When NormalizeFields encounters a Go type it does not recognize, it calls fn before panicking. This lets consumer packages define custom FieldKind values (starting at KindLast) without modifying the storage package.
type Option ¶
type Option func(*Options)
Option configures store construction via New.
func WithCacheDir ¶
func WithCacheDir() Option
WithCacheDir adds the resolved cache directory to the explicit path list. Resolution: CLAWKER_CACHE_DIR > XDG_CACHE_HOME > ~/.cache/clawker
func WithConfigDir ¶
func WithConfigDir() Option
WithConfigDir adds the resolved config directory to the explicit path list. Resolution: CLAWKER_CONFIG_DIR > XDG_CONFIG_HOME > ~/.config/clawker
func WithDataDir ¶
func WithDataDir() Option
WithDataDir adds the resolved data directory to the explicit path list. Resolution: CLAWKER_DATA_DIR > XDG_DATA_HOME > ~/.local/share/clawker
func WithDefaultFilename ¶ added in v0.6.0
WithDefaultFilename sets the filename used when writing to a directory with no existing file layers. Without this, filenames[0] is used, which may be a local override variant rather than the main file.
func WithDefaults ¶
WithDefaults provides a YAML string as the lowest-priority base layer. The string is parsed and merged before any discovered files. The same constant can be used for scaffolding (clawker init) and defaults.
func WithDefaultsFromStruct ¶ added in v0.6.0
WithDefaultsFromStruct generates a defaults YAML string from the `default` struct tags of T and registers it as the lowest-priority base layer. This is equivalent to WithDefaults(GenerateDefaultsYAML[T]()).
func WithDirs ¶ added in v0.2.4
WithDirs adds directories to be probed with dual placement discovery. Each directory uses the same dual-placement logic as walk-up: if a .clawker/ subdirectory exists, it probes .clawker/{filename} (dir form); otherwise it probes .{filename} (flat dotfile form). Both .yaml and .yml extensions are accepted. Directories are probed in the order given (first = highest priority). Priority: walk-up > dirs > explicit paths (WithPaths/WithConfigDir/etc.).
func WithDotDefault ¶ added in v0.6.0
func WithDotDefault() Option
WithDotDefault enables dual-placement dot-prefix logic in defaultWritePath. When the store has no file layers and falls back to CWD, the filename is written as .{filename} (or .clawker/{filename} if .clawker/ exists) instead of the raw filename. Use this for stores discovered via walk-up where files are dot-prefixed by convention.
func WithFilenames ¶
WithFilenames sets the ordered list of filenames to discover. All filenames must share the same schema type T. At each walk-up level the first filename in the list takes merge precedence when discovered at the same depth.
func WithHeader ¶ added in v0.12.11
WithHeader stamps the given text as a comment block at the top of the file on every Write, one comment line per input line ("# " prefixes are added by the YAML encoder — pass raw text). The header is re-applied on each write — it survives field-merge mutations and is idempotent: an existing comment line matching a header line's `key:` directive prefix (or the whole line, for lines without a colon) is replaced rather than duplicated, so a header whose value changes between writers (e.g. a version-pinned $schema URL) never stacks up. Unrelated comment lines are preserved. An empty header disables stamping, leaving the file comment-free.
func WithMigrations ¶
WithMigrations registers precondition-based migration functions. Each migration runs independently against every discovered file layer's own node tree. Migrations that return true trigger an atomic re-save of that file.
func WithPaths ¶
WithPaths adds explicit directories to the discovery path list. Files are probed as {dir}/{filename} for each requested filename.
func WithStateDir ¶
func WithStateDir() Option
WithStateDir adds the resolved state directory to the explicit path list. Resolution: CLAWKER_STATE_DIR > XDG_STATE_HOME > ~/.local/state/clawker
func WithWalkUp ¶
WithWalkUp enables bounded walk-up discovery from CWD up to anchorDir (inclusive). The store resolves CWD via os.Getwd(); anchorDir is a plain directory supplied by the caller (the storage engine holds no project-domain knowledge of how it was chosen). At each level the store checks for .clawker/{filename} (dir form) first, then .{filename} (flat dotfile form). Walk-up never proceeds past anchorDir.
anchorDir must be CWD or an ancestor of it. A non-ancestor anchor (a path beside or below CWD, a relative path, or a garbage path) is a caller programming error: store construction fails with an error wrapping ErrAnchorNotAncestor rather than letting the walk escape to the filesystem root. An empty anchorDir disables walk-up, so discovery falls back to explicit paths only.
type Options ¶ added in v0.12.11
type Options struct {
// Filenames is the ordered list of filenames to discover (WithFilenames).
Filenames []string
// Defaults is the raw YAML string for the base layer (WithDefaults).
Defaults string
// WalkUpAnchor bounds walk-up discovery from CWD up to this dir
// (inclusive); empty disables walk-up (WithWalkUp).
WalkUpAnchor string
// Dirs are directories probed with dual placement, highest priority
// first (WithDirs).
Dirs []string
// Paths are explicit directories probed without dual placement
// (WithPaths, WithConfigDir, WithDataDir, WithStateDir, WithCacheDir).
Paths []string
// Lock enables flock-based advisory locking for writes (WithLock).
Lock bool
// DotDefault applies the dual-placement dot prefix in the CWD write
// fallback (WithDotDefault).
DotDefault bool
// DefaultFilename is the filename used when writing to a location with
// no existing file; defaults to Filenames[0] (WithDefaultFilename).
DefaultFilename string
// Header is stamped as a comment block at the top of the file on every
// write; empty disables it (WithHeader).
Header string
// contains filtered or unexported fields
}
Options holds the accumulated construction configuration. Callers build it through the With* functional options; a store's resolved configuration is inspectable afterwards via Store.Options().
type Schema ¶ added in v0.6.0
type Schema interface {
Fields() FieldSet
}
Schema is the contract that schema types implement to expose their field metadata. Store is constrained to Schema, making field descriptions a compile-time requirement for all stored types.
type Store ¶
type Store[T Schema] struct { // contains filtered or unexported fields }
Store is a generic layered YAML store engine. Both internal/config and internal/project compose a Store[T] with their own schema types. The store handles file discovery, per-file loading with migrations, N-way merge with provenance, and scoped writes.
Internally, the store is node-native: every layer and the merged tree are yaml.Node trees, so comments are carried from load through merge to write. The typed struct T is decoded from the merged node and published as an immutable snapshot via atomic.Pointer. Readers get the current snapshot lock-free; a writer encodes the new value, grafts it into the node tree at the target path, and atomically swaps in the re-decoded snapshot. A write grafts the changed value into the target layer's own node tree, so the destination file keeps its comments and no other layer's comments leak in.
Load: file → node tree → merge → decode → immutable snapshot Set: encode value → graft into merged node → re-decode → atomic swap Write: merged node value → graft into target layer node → encode → file
func New ¶ added in v0.6.0
New constructs a store. seed is an explicit YAML string forming the virtual layer, merged on top of defaults ("" for none). File discovery, migrations, and all other options work normally.
The virtual layer (defaults + seed string) is the lowest-priority data source. Discovered file layers override it. Virtual-layer fields are NOT dirty after construction — a Write persists only explicit mutations, so schema defaults are never materialized into a user's file (where they would pin the current binary's defaults forever). A caller that wants the seed persisted (e.g. writing a preset to a new file) opts in with MarkSeedForWrite before Write/WriteTo.
With no options, the store has no file discovery — useful for seeding a new value that will be written via WriteTo.
func (*Store[T]) Get ¶
Get decodes the in-memory value at a dotted field path (e.g. "build.image") into out, a pointer to a typed destination — like yaml.Unmarshal, so the caller gets a real typed value:
var rules []EgressRule
store.Get("rules", &rules)
The first return is false when the path is absent. It reads the merged node tree, so it can see keys outside the typed schema (e.g. legacy keys a migration removes). A nil out checks presence without decoding (prefer Has for a pure presence test).
func (*Store[T]) Has ¶ added in v0.12.11
Has reports whether a value exists at the dotted path in the in-memory tree, without decoding it. It reads the merged node tree, so it sees keys outside the typed schema — the natural presence check for migrations.
func (*Store[T]) Layers ¶
Layers returns information about the discovered file layers. Layers are ordered from highest priority (index 0) to lowest.
func (*Store[T]) MarkForWrite ¶ added in v0.6.0
MarkForWrite adds a dotted field path to the write set so the next Write includes it regardless of whether Set detected a change.
Use this when persisting a value to a specific layer file where the merged result is already identical (e.g. writing the current winning value to a lower-priority layer). Normal Set-based dirty tracking won't catch this because the merged tree didn't change.
func (*Store[T]) MarkSeedForWrite ¶ added in v0.12.11
func (s *Store[T]) MarkSeedForWrite()
MarkSeedForWrite flags every field whose winning layer is the virtual seed/defaults layer as dirty, so the next Write/WriteTo persists it. This is the explicit opt-in for flows that materialize a seeded store into a file (e.g. writing a preset-populated project config during init). Ordinary file-backed stores never call it — their Writes persist only explicit Set/Remove mutations, keeping schema defaults out of user files.
func (*Store[T]) Options ¶ added in v0.12.11
Options returns a copy of the store's resolved construction options so callers can introspect how the store discovers and writes files (e.g. whether walk-up is enabled, which directories are probed). Slices are cloned; mutating the returned value does not affect the store.
func (*Store[T]) Provenance ¶ added in v0.6.0
Provenance returns the layer that provided the winning value for the given dotted field path (e.g. "build.image", "security.docker_socket"). Returns the LayerInfo and true if provenance is known, or zero value and false for fields that came from defaults or have no provenance record.
func (*Store[T]) ProvenanceMap ¶ added in v0.6.0
ProvenanceMap returns a mapping of dotted field paths to their source layer paths. Virtual layer fields (defaults) have an empty path.
func (*Store[T]) Read ¶ added in v0.3.2
func (s *Store[T]) Read() *T
Read returns the current immutable snapshot. The returned pointer is safe to hold, inspect, and pass around — it will never be mutated by the store. Set publishes new snapshots via atomic swap; existing readers are unaffected.
Lock-free: uses atomic.Pointer.Load.
func (*Store[T]) Refresh ¶ added in v0.6.0
Refresh re-discovers layer files, re-reads them from disk, and re-merges into a fresh snapshot. This picks up external modifications to existing files and newly created files found via discovery that weren't written by this store.
Pending mutations are discarded: Refresh resets dirty tracking to the fresh on-disk state, so call it before mutating, not between Set and Write.
Note: Write() already remerges and injects new layers for files it writes, so Refresh is only needed for external changes (e.g. another process modified a file).
func (*Store[T]) Remove ¶ added in v0.12.11
Remove deletes a dotted field path from the in-memory node tree (e.g. "agent.editor"), marks it for the next Write, and refreshes the snapshot. This "unsets" a field so a lower-priority layer can show through on next load, and is how a migration drops a legacy key. Empty parent maps are NOT pruned. Returns true if the key was found and removed.
func (*Store[T]) Set ¶
Set writes value at a dotted field path (e.g. "build.image") in the in-memory merged node, marks it dirty for the next Write, and refreshes the snapshot. value is a Go value (string, bool, int, slice, map) encoded faithfully to a YAML node. Paths outside the typed schema are allowed (used by migrations); for schema paths a value whose kind doesn't match the field is rejected.
Changes are not persisted until Write is called.
func (*Store[T]) Txn ¶ added in v0.12.11
Txn runs fn with a transaction handle while holding the store's transaction lock, serializing the whole closure against other Txn callers. Use it to make a compound read-modify-write atomic with respect to other such sequences:
store.Txn(func(tx *Tx[Schema]) error {
var rules []Rule
if _, err := tx.Get("rules", &rules); err != nil { return err }
rules = append(rules, r)
if err := tx.Set("rules", rules); err != nil { return err }
return tx.Write()
})
The per-op lock (s.mu) keeps each Get/Set/Write internally consistent, but it is released between calls — so two interleaved Get→Set→Write sequences can lose an update. Txn closes that gap. The handle's methods take s.mu per-op as usual; the transaction lock is separate, so there is no re-entrancy. Do NOT nest Txn.
func (*Store[T]) Write ¶
Write persists dirty fields to disk, then refreshes layer data from the written files so that subsequent Layers() calls return current values.
Only fields mutated since the last Write (via Set or Remove) are written. Set fields are merged into the target file; deleted fields are removed from it. This ensures per-field precision in multi-layer setups.
Each dirty field is routed to the layer it originated from (via provenance). Fields without provenance route to the highest-priority layer. To direct every dirty field at one explicit file instead, use WriteTo.
Write sequence per target: read the file's current on-disk content → merge set fields → remove deleted fields → atomic write (temp+rename). If locking is enabled (WithLock), the whole read-modify-write cycle per file runs inside a cross-process flock.
After a successful write, dirty tracking is cleared and layer data is refreshed from disk.
func (*Store[T]) WriteTargets ¶ added in v0.12.11
func (s *Store[T]) WriteTargets() ([]WriteTarget, error)
WriteTargets enumerates the store's valid write locations: the walk-up target when walk-up is enabled (the closest discovered walk-up layer for the write filename, or the CWD dual-placement candidate when none is in play), one candidate per configured directory, and every discovered file layer. Locations the store cannot rediscover (e.g. CWD for a store without walk-up) are never offered. Duplicates collapse into the first occurrence; order is walk-up, dirs, explicit paths, then remaining layers.
func (*Store[T]) WriteTo ¶ added in v0.6.0
WriteTo persists all dirty fields to the given absolute path instead of routing them by provenance. Use it to write a new file or a known path outside the discovered layer set (e.g. materializing a preset). The write merges into the file's current on-disk content — see Write.
type TargetSource ¶ added in v0.12.11
type TargetSource int
TargetSource classifies where a candidate write location came from.
const ( // TargetWalkUp is the CWD dual-placement candidate of a walk-up store. TargetWalkUp TargetSource = iota // TargetDir is a dual-placement candidate from a WithDirs directory. TargetDir // TargetPath is a candidate inside an explicit directory // (WithPaths, WithConfigDir, WithDataDir, WithStateDir, WithCacheDir). TargetPath // TargetLayer is an already-discovered file layer. TargetLayer )
type Tx ¶ added in v0.12.11
type Tx[T Schema] struct { // contains filtered or unexported fields }
Tx is the mutation handle a Txn closure receives. Its Get/Set/Remove/Write/ MarkForWrite forward to the store, but because the handle is reachable ONLY from inside Txn, a read-modify-write expressed as `tx.Get → tx.Set → tx.Write` is visibly transactional at the call site — the transaction lock is held for the whole closure. The handle holds no extra lock; each method takes s.mu per-op as usual.
func (*Tx[T]) MarkForWrite ¶ added in v0.12.11
MarkForWrite forces path into the write set. See Store.MarkForWrite.
type WriteTarget ¶ added in v0.12.11
type WriteTarget struct {
Source TargetSource
Path string // absolute file path
Filename string // which configured filename the target serves (e.g. "clawker.yaml")
}
WriteTarget is a candidate write location derived from the store's own configuration. Every target is rediscoverable: a file written to Path is picked up by an identically-configured store on reload. UIs offering "save to..." destinations must use these rather than inventing locations.