storage

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 2, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package storage owns the coordination layer between writer, exporter, retention, and the read API for on-disk JSONL files.

The package introduces three load-bearing abstractions:

  • FileID — canonical identity of a (date, key_hash) bucket. The same trace file can exist on disk as either `<keyhash>.jsonl` (active or post-rotation but pre-compression) or `<keyhash>.jsonl.gz` (post-compression). The SQLite jsonl_path column ALWAYS stores the canonical `.jsonl` form; FileID is the in-process value that maps between the two without baking the extension into callers.

  • Lease (lease.go) — refcount-based hold preventing eviction from removing a file while a reader or writer still has it open.

  • Coordinator (coordinator.go) — the always-on monitor that owns inventory, status, leases, and (when knobs are set) eviction.

This file declares the FileID type and the parsing helpers. The types declared here have no goroutines, no I/O lifecycle, and no shared state — they are pure value types.

Index

Constants

This section is empty.

Variables

View Source
var ErrFileBeingDeleted = errors.New("storage: file is being deleted by retention")

ErrFileBeingDeleted is returned by AcquireLease when the eviction path has begun deleting the requested FileID. Callers (writer during fileFor, exporter during Phase 2, gzip background worker) MUST treat this as "try again later" rather than as a hard error — it is a normal racing-with-retention outcome at scale, not a data corruption signal.

Functions

This section is empty.

Types

type Config

type Config struct {
	// DataDir is the absolute root passed through to walkAndStat
	// and FileID.{CanonicalPath,MediaSubtree,FSExists}. Required.
	DataDir string

	// Initial retention values. After New, callers mutate these via
	// UpdateConfig — the in-memory atomic.Pointer is the live source
	// of truth, not the Config struct.
	MaxBytes      int64
	MaxAgeDays    int
	WarnAtPercent int

	// TickInterval is the monitor cadence. Default 1 hour.
	TickInterval time.Duration

	// EvictionCapPerTick bounds work per tick for both eviction and
	// orphan reconciliation. Default 1000 files.
	EvictionCapPerTick int

	// IdleEvictionLag is the grace period: files modified within
	// this window are NOT evicted. Default 5 minutes. Gives
	// just-rotated or just-released files breathing room.
	IdleEvictionLag time.Duration
}

Config is the static configuration for a Coordinator. Set once at construction by New(); immutable after Start.

Operator-mutable retention thresholds (max_bytes, max_age_days, warn_at_percent) are stored in a SEPARATE atomic.Pointer on the Coordinator and updated via UpdateConfig(); they are deliberately NOT part of Config so the static / dynamic split is enforced by the API shape.

type Coordinator

type Coordinator struct {
	// contains filtered or unexported fields
}

Coordinator is the storage coordination primitive: lease arbitration plus (added by coordinator.go in B1.7) monitor, status, inventory, and eviction. This file declares only the lease-state portion; the remaining state and the public constructor live in coordinator.go.

A zero-value Coordinator is usable for lease operations — maps are lazily initialized under the mutex. atomic.Pointer fields are also nil-safe to load (returns nil pointer; consumers handle).

func New

func New(cfg Config, store Store, counters Counters) (*Coordinator, error)

New constructs a Coordinator with the given static config + store backend + (optional) counters. Validates config; applies defaults for zero-value fields. Does NOT start the monitor goroutine — the caller invokes Start(ctx) when ready.

store may be nil for unit tests that exercise lease semantics without a database. counters may be nil; the Coordinator nil-checks.

Returns error on:

  • empty DataDir
  • WarnAtPercent outside [0, 100] (0 means "use default 80")
  • negative duration fields
  • negative integer fields

func (*Coordinator) AcquireLease

func (c *Coordinator) AcquireLease(fid FileID) (*Lease, error)

AcquireLease borrows a refcounted hold on fid. The caller must call Release on the returned Lease exactly once.

Returns ErrFileBeingDeleted if eviction is mid-delete on this fid; the caller's correct response is to skip the operation that required the lease (writer: return the error and let the trace route to a new bucket on the next finalize; exporter: skip the group with a warn log; gzip worker: log + return).

func (*Coordinator) Inventory

func (c *Coordinator) Inventory() []FileInfo

Inventory returns a snapshot of the most recent walkAndStat result. Used by the Landing OPS strip + Settings storage card for the "current files" view (potential v0.2 expansion). For v0.1.1 the returned slice is what monitor.runTick last produced; callers should not mutate.

Returns nil if the monitor hasn't ticked yet.

func (*Coordinator) Start

func (c *Coordinator) Start(ctx context.Context) error

Start launches the monitor goroutine. Runs an immediate tick so Status() returns real data within milliseconds of Start (not at the end of the first ticker interval). Then ticks every tickInterval until ctx is canceled.

Returns ctx.Err() when canceled — main.go's graceful-shutdown sequence waits on this return.

Call exactly once per Coordinator. v0.1.1 doesn't enforce this with a mutex/atomic; B1.7's New constructor + the wire-up in cmd/api-log/main.go owns the single-call contract.

func (*Coordinator) Status

func (c *Coordinator) Status() Status

Status returns the most recently published status, or a "pending" snapshot when the monitor hasn't ticked yet (or hasn't been started). Always safe to call; never blocks; never returns nil because Status is a value type.

This is THE single source of truth for /healthz, Landing OPS, Settings storage card, and the startup banner. Adding a sibling "what's our current size?" computation anywhere else in the code would be the patchwork v5 was meant to prevent — route through this method.

func (*Coordinator) UpdateConfig

func (c *Coordinator) UpdateConfig(retention RetentionConfig) error

UpdateConfig atomically swaps the dynamic retention thresholds — the subset operators mutate via PUT /api/config/retention without restart.

Validates inputs the same way New does, then publishes via atomic.Pointer. Recomputes Status SYNCHRONOUSLY against the cached inventory so /healthz reflects the new thresholds immediately; eviction itself runs at next tick.

Mirrors the save_attachments pattern (api/config.go) — same shape: PUT updates immediately, side effects pick it up on the next polling cycle.

Passing the zero-value RetentionConfig (all fields 0) disables retention without restarting the monitor.

type Counters

type Counters interface {
	AddEvictedTraces(n int64)
	AddEvictedBytes(n int64)
}

Counters is the narrow interface the Coordinator needs from the metrics package. Declared here (not imported from internal/counters) to keep storage free of cross-package dependencies on the metrics surface. The real *counters.Counters value satisfies it via duck typing; tests pass nil for a no-op.

All methods are safe to call concurrently and safe to call on a nil Counters (the Coordinator nil-checks before each call).

type FileID

type FileID struct {
	// DataDir is the absolute root configured via storage.data_dir.
	// Kept on FileID so MediaSubtree, CanonicalPath, and FSExists
	// don't need it threaded as a separate argument.
	DataDir string

	// Date is the UTC date the writer used at bucket-creation time,
	// formatted as YYYY-MM-DD. Source-of-truth is the JSONL file's
	// parent directory name, not the trace's TsStart (those can
	// differ when finalize crosses a day boundary).
	Date string

	// KeyHash8 is the first 8 hex characters of the SHA-256 of the
	// client's Authorization / x-api-key, computed by
	// internal/ids.KeyHashShort. 8 hex chars ≈ 32 bits, sufficient
	// to keep per-tenant files separate at every realistic scale
	// (collision probability is ~1.2e-5 at 1000 tenants).
	KeyHash8 string
}

FileID identifies one trace-storage bucket: the per-(date, key_hash) file the writer rotates into. A FileID has a deterministic mapping to one canonical SQLite path and to either or both filesystem forms (`.jsonl` plain, `.jsonl.gz` compressed). It is the single source of identity across writer / exporter / retention / read API.

func FileIDFromPath

func FileIDFromPath(dataDir, path string) (FileID, error)

FileIDFromPath parses a canonical jsonl_path (the SQLite form) back into a FileID. The path may be the plain `.jsonl` form OR the compressed `.jsonl.gz` form — both produce the same FileID because the suffix is metadata, not identity.

Returns an error when:

  • path doesn't sit under dataDir (a defense against accidentally calling this with a path from a different deployment)
  • the relative path doesn't match `<date>/<keyhash8>.jsonl{,.gz}`
  • date doesn't parse as YYYY-MM-DD
  • keyhash isn't 8 lowercase hex characters

Used by:

  • retention/reconcile when reading SQLite jsonl_path rows
  • the background gzip worker (compressInPlace) to acquire a lease
  • tests + tooling that need to round-trip a path

func (FileID) CanonicalPath

func (f FileID) CanonicalPath() string

CanonicalPath returns the canonical filesystem form of this bucket — the `.jsonl` (uncompressed) path. SQLite's jsonl_path column always stores this exact string. The actual on-disk file may be the `.jsonl` form (active) or the `.jsonl.gz` form (compressed), or both transiently during rotation; use FSExists to determine the current on-disk reality.

func (FileID) FSExists

func (f FileID) FSExists() (string, error)

FSExists returns the actual on-disk form of this bucket, if any.

Returns:

  • (path, nil) when exactly one form exists (either `.jsonl` or `.jsonl.gz`). Plain wins over `.gz` when both exist transiently during rotation: compressInPlace writes `.gz` first, then removes the plain file, so seeing both means rotation is mid-step and the plain form is still authoritative.
  • ("", nil) when neither form exists. This is the "already gone" signal that callers like deleteIfIdle should treat as success.
  • ("", err) when stat returns an error OTHER than fs.ErrNotExist. Callers MUST distinguish — treating permission / I/O errors as "gone" would silently delete SQLite rows for files we just couldn't read.

func (FileID) MediaSubtree

func (f FileID) MediaSubtree() string

MediaSubtree returns the absolute path to this bucket's media directory: `<DataDir>/<Date>/<KeyHash8>/media/`. Extraction in internal/media writes files under this tree as `media/<trace_id>/<idx>.<ext>`; retention deletes the whole subtree alongside the JSONL file.

type FileInfo

type FileInfo struct {
	FileID    FileID
	FSPath    string    // most recently scanned on-disk form
	SizeBytes int64     // total across .jsonl + .jsonl.gz (sums during rotation)
	ModTime   time.Time // latest mtime across forms; zero if no JSONL on disk
	MediaSize int64     // recursive bytes under MediaSubtree()
}

FileInfo describes one trace-storage bucket — its on-disk size, modification time, and media subtree size. Produced by walkAndStat and consumed by status computation, retention eviction, and the orphan reconciliation tick.

FSPath is informational: when both `.jsonl` and `.jsonl.gz` exist transiently during rotation, walkAndStat records the latest form seen. Callers that need the live filesystem form at action time (e.g. eviction's os.Remove) should call FileID.FSExists() rather than trusting a possibly-stale FSPath.

type Inventory

type Inventory struct {
	Files      []FileInfo
	TotalBytes int64 // sum of SizeBytes + MediaSize over Files (the retention-managed footprint)
}

Inventory is the snapshot walkAndStat returns. Files is unsorted — callers (eviction sorts by mtime ASC, status sums) decide their own ordering.

type Lease

type Lease struct {
	// contains filtered or unexported fields
}

Lease is a refcounted hold on a single FileID. Multiple callers may hold concurrent leases on the same FileID; eviction blocks (returns false from markDeleting) until every lease is released.

Lease values are returned by Coordinator.AcquireLease and must have Release called exactly once. Release is idempotent — calling it more than once is a no-op rather than a panic, so it's safe to chain `defer lease.Release()` even in code paths that might release explicitly first.

func (*Lease) Release

func (l *Lease) Release()

Release releases the hold. Idempotent. Safe to call from defer even after an explicit Release().

type RetentionConfig

type RetentionConfig struct {
	MaxBytes      int64
	MaxAgeDays    int
	WarnAtPercent int
}

RetentionConfig is the dynamic subset of retention settings — operators mutate this via PUT /api/config/retention without a restart. The Coordinator holds it in an atomic.Pointer so the monitor's tick loop sees the latest value with no lock.

Static settings (data_dir, tick_interval, eviction_cap_per_tick, idle_eviction_lag) live on the immutable Config in coordinator.go (B1.7) — they require restart to change.

type Status

type Status struct {
	// DataDirBytes is the current retention-managed footprint
	// (sum of JSONL + media across all (date, keyhash) buckets).
	DataDirBytes int64

	// MaxBytes mirrors the active RetentionConfig.MaxBytes for ops
	// visibility; 0 when byte cap is disabled.
	MaxBytes int64

	// MaxAgeDays mirrors the active RetentionConfig.MaxAgeDays; 0
	// when age cap is disabled.
	MaxAgeDays int

	// UsagePct = 100 * DataDirBytes / MaxBytes, clamped to [0, 100+].
	// 0 when MaxBytes == 0 (no byte cap, so the percentage is
	// meaningless). The value can exceed 100 briefly between writer
	// activity and the next eviction tick.
	UsagePct int

	// State summarizes the storage situation in one word so the
	// Landing OPS strip can color-code without doing arithmetic:
	//
	//   "disabled" — both knobs are 0; monitor still runs but no
	//                eviction can fire.
	//   "ok"       — under thresholds.
	//   "warning"  — usage at or above warn_at_percent of max_bytes.
	//   "critical" — usage at or above 100% of max_bytes.
	//   "pending"  — monitor hasn't ticked yet; Status() returns
	//                this when atomic.Pointer is still nil at first
	//                load.
	State string

	// LastEvictionTs is the wall-clock time of the most recent
	// eviction tick that actually deleted something. Zero value
	// (time.Time{}) means "no eviction has run yet OR no files
	// were eligible at last tick".
	LastEvictionTs time.Time

	// LastEvictedBytes is the byte total deleted by the most recent
	// eviction tick. Zero when LastEvictionTs is zero.
	LastEvictedBytes int64

	// EvictionCapHit is true when the most recent tick exhausted the
	// per-tick eviction cap (default 1000 files) with more eligible
	// files remaining. Operator-facing signal that another tick is
	// needed to fully catch up — useful during the first-enable
	// migration.
	EvictionCapHit bool

	// NextEvictionEst is a best-effort projection of when the byte
	// cap would be hit at the current growth rate. Always zero in
	// v0.1.1; v0.2 may populate it from inventory deltas.
	NextEvictionEst time.Time

	// EngineRunning is true when monitor.Start has been called and
	// the goroutine is alive. Distinguishes "monitor hasn't started
	// yet" (e.g. unit test instantiation without Start) from
	// "monitor is up but hasn't ticked yet".
	EngineRunning bool
}

Status is the publishable snapshot of storage state. The monitor goroutine atomically stores a Status after each tick; readers (healthz, Landing OPS strip, Settings storage card) atomically load it. The struct is value-only (no methods, no internal references) so it can be JSON-marshaled directly.

All sizes are "retention-managed bytes" — the JSONL + media footprint that walkAndStat returns in Inventory.TotalBytes. SQLite, tmp/, admin_token, runtime_overrides.json are explicitly EXCLUDED. Adopters who want raw disk usage run `du -sb data/` separately; documented in retention.md.

type Store

type Store interface {
	// ListDistinctJSONLPaths returns every distinct jsonl_path
	// stored in the traces table, in unspecified order. Backed by
	// the idx_jsonl_path index added by B2's schema migration.
	ListDistinctJSONLPaths(ctx context.Context) ([]string, error)

	// DeleteByJSONLPath removes all trace rows referencing the
	// given canonical path. Returns affected row count.
	DeleteByJSONLPath(ctx context.Context, jsonlPath string) (int64, error)
}

Store is the narrow interface monitor + eviction need from the trace database. Declared here (not imported from internal/store/sqlite) to keep this package free of database dependencies — the sqlite Store satisfies it via duck typing.

B2 wires the real implementation; tests pass a fake.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL