Documentation
¶
Overview ¶
Package storage is the on-disk toolbox for agents that stage large files to local disk: crash-safe atomic writes, an in-band-hashed streaming copy primitive, a resumable content-addressed chunk stager, plus per-account manifest / version-rotation / graveyard / path-sanitize helpers.
Everything here is transport-neutral: no HTTP, oauth2, Drive, or Range/resume semantics. Agents that need those drive the request themselves and hand the response body to CopyToFileAtomic (or the ChunkStage) as an io.Reader.
Index ¶
- Constants
- Variables
- func AtomicWriteJSON(dst string, payload []byte) error
- func CopyToFileAtomic(ctx context.Context, r io.Reader, dst string, h hash.Hash) (written int64, hexsum string, err error)
- func MoveAtomic(src, dst string) error
- func ParseStamp(iso string) (time.Time, error)
- func SanitizeName(in string) string
- func SanitizePath(in string) string
- func SplitVersionName(name string) (iso, display string, ok bool)
- type ChunkStage
- type Graveyard
- func (g Graveyard) Bury(fileID, livePath, displayName string, at time.Time) (string, error)
- func (g Graveyard) FileGraveyardDir(fileID string) string
- func (g Graveyard) GraveyardPath(fileID, entryName string) string
- func (g Graveyard) ListGraveyard(fileID string) ([]string, error)
- func (g Graveyard) ListGraveyardIDs() ([]string, error)
- func (g Graveyard) Sweep(keepDays int, now time.Time) (int, error)
- type Manifest
- type ManifestFile
- type ManifestStore
- func (s *ManifestStore) AccountID() string
- func (s *ManifestStore) Delete(fileID string)
- func (s *ManifestStore) Get(fileID string) (ManifestFile, bool)
- func (s *ManifestStore) MarkDeleted(fileID string, at time.Time)
- func (s *ManifestStore) Save() error
- func (s *ManifestStore) Set(fileID string, mf ManifestFile)
- func (s *ManifestStore) SetAccountID(id string)
- func (s *ManifestStore) Snapshot() map[string]ManifestFile
- type VersionLayout
- func (v VersionLayout) FileVersionsDir(fileID string) string
- func (v VersionLayout) ListVersions(fileID string) ([]string, error)
- func (v VersionLayout) Prune(fileID string, keep int) ([]string, error)
- func (v VersionLayout) Rotate(fileID, livePath, displayName string, at time.Time) (string, error)
- func (v VersionLayout) VersionPath(fileID, versionName string) string
Constants ¶
const MaxNameLen = 200
MaxNameLen caps each path component at 200 bytes. Most filesystems allow 255, but trimming earlier leaves headroom for the "<iso>__" prefix the version directory layout prepends.
Variables ¶
var ErrHashMismatch = errors.New("storage: hash mismatch")
ErrHashMismatch is returned (wrapped) when a computed digest disagrees with the expected one. Callers use errors.Is(err, ErrHashMismatch). Kept generic (not md5/sha-specific) so every consumer shares one sentinel.
Functions ¶
func AtomicWriteJSON ¶
AtomicWriteJSON writes payload to dst via tmp+rename, with fsync. Used by the manifest writer and the chunk-stage bitmap sidecar.
func CopyToFileAtomic ¶
func CopyToFileAtomic(ctx context.Context, r io.Reader, dst string, h hash.Hash) (written int64, hexsum string, err error)
CopyToFileAtomic streams r into dst atomically and durably, computing a hash in-band when h != nil. It writes to dst+".part", fsyncs it, renames .part -> dst, then fsyncs the parent directory so the rename survives a crash. It returns the number of bytes written and, if h != nil, the lowercase-hex digest of exactly those bytes (no second pass over the data).
This is the transport-neutral core of a streamed download: it has no HTTP, Range, or resume semantics. Callers needing those (e.g. an agent's StreamDownload) build the request themselves and pass the response body as r. On any error the .part file is removed best-effort so a retry starts clean. It does NOT compare the digest to an expected value — the caller does that against whatever authority it has (an upstream-provided md5, a catalog sha, …) and may use ErrHashMismatch for a consistent error.
func MoveAtomic ¶
MoveAtomic renames src → dst, creating dst's parent dir if needed. Used by versions.go to rotate live files into _versions/ and by graveyard.go to move soft-deleted files out of the live tree.
func ParseStamp ¶
ParseStamp converts an ISO entry-timestamp prefix ("2026-05-14T03-00-00Z", as produced by Rotate/Bury and returned by SplitVersionName) back to a time.Time. Exported so out-of-module callers (e.g. an agent listing versions over HTTP) can render the deletion/rotation time without duplicating the filesystem-safe stamp format.
func SanitizeName ¶
SanitizeName replaces every byte that would be unsafe on a POSIX filesystem (or that breaks shell pasting) with "_". This is path-component safety only — callers split on "/" before calling and reassemble after.
Rejected bytes:
- control chars 0x00..0x1f
- path separators / \
- shell-breaking whitespace \r \n \t
Note: spaces are intentionally kept. Drive lets users name files with spaces and we want to preserve that for human-friendly _versions/ entries.
func SanitizePath ¶
SanitizePath splits on "/", sanitizes each component, and re-joins. Empty components are dropped (collapses "a//b" → "a/b").
func SplitVersionName ¶
SplitVersionName extracts the ISO timestamp + display name from a rotated filename. Returns ok=false if the layout doesn't match "<iso>__<name>".
Types ¶
type ChunkStage ¶
type ChunkStage struct {
// contains filtered or unexported fields
}
ChunkStage stages a single content-addressed file as a set of fixed-size chunks written out of order and resumably. It is the disk half of a mesh/peer transfer: a peer pulls the chunks it is missing and pwrites each at its byte offset; a bitmap sidecar records which chunks have landed so a restart mid-transfer resumes instead of re-pulling everything.
Layout under dir (the sha256hex is the content address):
<sha256hex>.part the assembled file, pre-sized to `size`
<sha256hex>.part.bitmap JSON sidecar: {sha256,size,chunk_size,received}
Durability contract: WriteChunk fsyncs the data BEFORE marking the bitmap, so a crash can only ever leave a landed-but-unmarked chunk (re-pulled on resume, harmless) — never a marked-but-absent chunk. Complete re-hashes the whole assembled file and only renames it into place if the full sha256 matches, so a torn or reordered write can never be published.
A ChunkStage is safe for concurrent WriteChunk/Bitmap/Missing calls.
func OpenStage ¶
func OpenStage(dir, sha256hex string, size, chunkSize int64) (*ChunkStage, error)
OpenStage creates or reopens the stage for content sha256hex of the given size, chunked at chunkSize bytes. The .part file is created (or reopened) and pre-sized to size; the bitmap sidecar is loaded when its geometry matches, else initialized empty. Reopening a partially-filled stage returns it with the prior Bitmap()/Missing() intact so a transfer resumes.
func (*ChunkStage) Bitmap ¶
func (s *ChunkStage) Bitmap() []bool
Bitmap returns a copy of the received-chunk flags (index → landed).
func (*ChunkStage) Close ¶
func (s *ChunkStage) Close() error
Close releases the underlying file descriptor without completing the transfer, leaving the .part and bitmap on disk for a later OpenStage to resume. Safe to call more than once. Not required after a successful Complete (which already closed the file).
func (*ChunkStage) Complete ¶
func (s *ChunkStage) Complete(finalDst string) error
Complete verifies the assembled .part's full sha256 equals the stage's content address and, on match, atomically renames it to finalDst (fsyncing the parent dir) and drops the bitmap sidecar. On a hash mismatch it returns an error wrapping ErrHashMismatch and LEAVES the .part in place so the caller can re-pull the corrupt chunks and retry. Complete may be retried.
func (*ChunkStage) Missing ¶
func (s *ChunkStage) Missing() []int
Missing returns the indices of chunks not yet received, ascending.
func (*ChunkStage) WriteChunk ¶
WriteChunk pwrites the bytes read from r at index*chunkSize and marks the chunk received. r is bounded to the chunk's true length (chunkSize, or the short final chunk), so an over-long reader cannot bleed into the next chunk. Writing an already-received chunk is allowed and idempotent (it re-lands the bytes and re-marks). The data is fsynced before the bitmap is persisted.
ctx cancels the (potentially slow, network-backed) read+write promptly. The stage mutex is held only to validate/snapshot geometry and to persist the mark — NOT across the blocking copy+fsync — so a stalled peer on one chunk never wedges Bitmap()/Missing()/other WriteChunk calls. pwrite at distinct offsets is safe without the lock; the durability contract (fsync-before-mark) is preserved because the fsync completes before the re-lock that marks.
type Graveyard ¶
type Graveyard struct {
AccountRoot string
}
Graveyard handles soft-delete + retention for files that vanished from Drive. Layout:
<accountRoot>/graveyard/<fileID>/<iso>__<sanitizedName>
Backup walker calls Bury() once per missing file. A periodic prune job (or end-of-backup sweep) calls Sweep() with the keep-days window from the repo config.
func NewGraveyard ¶
func (Graveyard) Bury ¶
Bury moves livePath into graveyard/<fileID>/<iso>__<displayName>. If livePath doesn't exist, Bury is a no-op (caller may have already buried it on an earlier run before crash-recovery).
func (Graveyard) FileGraveyardDir ¶
FileGraveyardDir returns the directory under graveyard/<fileID> holding one file's soft-deleted copies. Symmetric with VersionLayout.FileVersionsDir.
func (Graveyard) GraveyardPath ¶
GraveyardPath returns the absolute path to one specific soft-deleted entry. Caller is responsible for stat'ing it (a concurrent Sweep may have pruned it). Mirrors VersionLayout.VersionPath.
func (Graveyard) ListGraveyard ¶
ListGraveyard returns the soft-delete entry filenames for one file, newest last (the ISO-timestamp prefix sorts lexicographically). Empty slice (no error) if the dir doesn't exist. Mirrors VersionLayout.ListVersions.
func (Graveyard) ListGraveyardIDs ¶
ListGraveyardIDs returns the fileIDs that currently have at least one recoverable soft-deleted entry — i.e. the immediate subdirs of graveyard/. Because Sweep removes a subdir once its last entry expires, this is exactly the still-recoverable set, obtained in a single ReadDir. Empty slice (no error) if graveyard/ doesn't exist.
type Manifest ¶
type Manifest struct {
AccountID string `json:"account_id"`
UpdatedAt time.Time `json:"updated_at"`
Files map[string]ManifestFile `json:"files"`
}
Manifest is the per-account file index that powers the 3-tier idempotency check in the backup walker. One file per account_id, stored at <BACKUP_ROOT>/<account_id>/manifest.json. A previous-good copy lives at manifest.json.bak — written by promoting the current file before each rename.
type ManifestFile ¶
type ManifestFile struct {
Path string `json:"path"`
ModifiedTime time.Time `json:"modified_time"`
MD5 string `json:"md5,omitempty"`
Size int64 `json:"size"`
HeadRevisionID string `json:"head_revision_id,omitempty"`
Version int64 `json:"version,omitempty"`
MimeType string `json:"mime_type"`
VersionCount int `json:"version_count,omitempty"`
Versions []string `json:"versions,omitempty"`
// DeletedAt is set when the file disappeared from Drive. The file's
// content is moved to graveyard/ and its entry is retained in the
// manifest until the graveyard sweeper drops it (graveyard_keep_days).
DeletedAt *time.Time `json:"deleted_at,omitempty"`
}
ManifestFile is one file's entry in the manifest. Binary files populate MD5; native files (Sheets/Docs) leave it empty and rely on HeadRevisionID + Version + Size for change detection.
type ManifestStore ¶
type ManifestStore struct {
// contains filtered or unexported fields
}
ManifestStore is the threadsafe owner of one account's manifest. Backup runner instantiates it at job start, calls Get/Set during the walk, then Save() at the end. No partial writes — Save() builds the full new JSON, promotes the current file to .bak, then renames the temp into place.
func NewManifestStore ¶
func NewManifestStore(backupRoot, accountID string) (*ManifestStore, error)
NewManifestStore opens (or fresh-initializes) the manifest for the given account. Missing file → fresh empty Manifest; corrupt JSON → fall back to .bak; both missing → fresh empty. Never errors on missing input.
func (*ManifestStore) AccountID ¶
func (s *ManifestStore) AccountID() string
AccountID returns the manifest's account_id (after first Save it's guaranteed correct; before first Save it may be empty).
func (*ManifestStore) Delete ¶
func (s *ManifestStore) Delete(fileID string)
Delete removes the entry entirely (used after graveyard prune).
func (*ManifestStore) Get ¶
func (s *ManifestStore) Get(fileID string) (ManifestFile, bool)
Get returns a copy of the file's entry (ok=false if absent).
func (*ManifestStore) MarkDeleted ¶
func (s *ManifestStore) MarkDeleted(fileID string, at time.Time)
MarkDeleted stamps DeletedAt on the entry without removing it. Used when a file vanishes from Drive — we keep the manifest row until the graveyard retention window expires.
func (*ManifestStore) Save ¶
func (s *ManifestStore) Save() error
Save serializes the manifest and writes it atomically:
- Marshal current state.
- If manifest.json exists, rename it to manifest.json.bak (atomic).
- Write the new state to a sibling tmp file, fsync, rename → manifest.json.
On a crash mid-Save:
- between (2) and (3): manifest.json missing, .bak holds last good copy.
- during (3) tmp write: tmp file orphaned (no effect on current).
- after rename: new file in place. Recovery sees it.
func (*ManifestStore) Set ¶
func (s *ManifestStore) Set(fileID string, mf ManifestFile)
Set inserts/updates the entry. Concurrent-safe.
func (*ManifestStore) SetAccountID ¶
func (s *ManifestStore) SetAccountID(id string)
SetAccountID stamps the account_id on the manifest. Called once at store init.
func (*ManifestStore) Snapshot ¶
func (s *ManifestStore) Snapshot() map[string]ManifestFile
Snapshot returns a deep-ish copy of all entries (map value semantics; the caller MAY mutate the returned map without affecting the store). Use this in the walker's "files in manifest but missing from Drive" pass.
type VersionLayout ¶
type VersionLayout struct {
AccountRoot string // BACKUP_ROOT/<account_id>
}
VersionLayout exposes the on-disk geometry for one account's version dir tree. Callers don't see paths directly; they hand a (fileID, srcPath, name) tuple and we put bits where they belong.
func NewVersionLayout ¶
func NewVersionLayout(accountRoot string) VersionLayout
func (VersionLayout) FileVersionsDir ¶
func (v VersionLayout) FileVersionsDir(fileID string) string
FileVersionsDir returns the directory under _versions/<fileID> holding rotated copies of one file. Caller does NOT need to mkdir — Rotate does.
func (VersionLayout) ListVersions ¶
func (v VersionLayout) ListVersions(fileID string) ([]string, error)
ListVersions returns the version filenames for one file, newest last. Empty slice (no error) if the dir doesn't exist.
func (VersionLayout) Prune ¶
func (v VersionLayout) Prune(fileID string, keep int) ([]string, error)
Prune trims _versions/<fileID>/ down to `keep` newest entries (by timestamp prefix). Returns the slice of entries pruned (just for the job log).
keep <= 0 → no-op (treat as "keep all"). Operators set keep_versions=0 in the UI when they want unlimited retention.
func (VersionLayout) Rotate ¶
Rotate moves a live file into _versions/<fileID>/<iso>__<name>. Atomic rename, no copy. Returns the version entry name (e.g. "2026-05-14T03-00-00Z__foo.md") for the caller to append to manifest.Versions.
Pass "" for at to use time.Now().UTC().
func (VersionLayout) VersionPath ¶
func (v VersionLayout) VersionPath(fileID, versionName string) string
VersionPath returns the absolute path to one specific version. Caller is responsible for stat'ing it (could be deleted by a concurrent prune, though job-runner takes a global mutex to avoid that).
Source Files