storage

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 4, 2026 License: LGPL-2.1 Imports: 29 Imported by: 0

README

mod/storage

mod/storage is the durable data layer. It stores canonical blobs and trees in Pebble, metadata and indexes in SQLite, and regenerated serving artifacts in a bounded hot-file cache. It is the source of truth for published versions.

Place in the Runtime

flowchart TB
  rescan["mod/rescan"] --> publish["PublishStaged"]
  publish --> sqlite["SQLite index"]
  publish --> pebble["Pebble objects"]
  server["mod/server"] --> read["read APIs"]
  read --> sqlite
  read --> pebble
  read --> hot["hot artifact cache"]

Responsibilities

  • Validate staged entries and blobs before commit.
  • Write missing blobs and tree objects into Pebble.
  • Commit version metadata, artifacts, detections, events, and key sources into SQLite.
  • Serve blobs, trees, versions, artifacts, key lists, feeds, and checksums.
  • Maintain hot artifacts and enforce storage quotas.
  • Provide maintenance operations for inspect, prune, vacuum, compaction, and cache rebuild.

Publish Boundary

sequenceDiagram
  participant Rescan as rescan
  participant Store as storage
  participant Pebble as Pebble
  participant SQLite as SQLite
  Rescan->>Store: PublishStaged
  Store->>Store: validate tree and staged blobs
  Store->>Pebble: write missing objects
  Store->>SQLite: commit version metadata
  SQLite-->>Rescan: version visible

Contracts

  • PublishStaged is the visibility boundary. Partial staged data must not become public.
  • Blobs are addressed by BLAKE3-24 and verified before write.
  • Hot-cache files are disposable; Pebble and SQLite are durable truth.
  • Blob read/write paths materialize whole blobs, so configured per-file limits and in-flight read budgets are required.
  • Generated cache rebuilds must be deterministic for the same stored tree and listener context.

Important Files

  • obj.go, open.go: storage object and startup.
  • staged.go, version.go: publish path and version metadata.
  • pebble.go, pebblestore/: object store integration.
  • sqliteindex/: SQL schema and index queries.
  • hot.go, quota.go, maintenance.go: hot cache, quota, and maintenance.
  • validate.go: archive and storage limit enforcement.

Operational Notes

For tuning compression, memtables, block cache, CGO builds, and hot-cache verification, use the root STORAGE-TUNING.md. This README describes package boundaries; tuning profiles describe operator choices.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidRef = errors.New("invalid reference")

ErrInvalidRef marks malformed key/version input that callers map to 400.

Functions

This section is empty.

Types

type ArtifactBuilderInterface 

type ArtifactBuilderInterface interface {
	Build(ctx context.Context, writer io.Writer) error
}

ArtifactBuilderInterface writes artifact content to a writer.

type BlobSpoolObj 

type BlobSpoolObj struct {
	// contains filtered or unexported fields
}

BlobSpoolObj is a temporary directory under temp/ for staged blobs of one publication. Close removes it best-effort even when publish fails.

func (*BlobSpoolObj) Close 

func (obj *BlobSpoolObj) Close(_ context.Context) error

Close idempotently removes the spool directory and all contents. Cleanup is best-effort and independent of ctx cancellation: an aborted ingest cycle or shutdown must still delete the temp blob-spool directory, otherwise it leaks on disk until the next start.

func (*BlobSpoolObj) RootPath 

func (obj *BlobSpoolObj) RootPath() string

RootPath returns the spool directory path for callers writing staged blobs.

type HotFileObj 

type HotFileObj struct {
	Path      string
	File      *os.File
	SizeBytes uint64
	BodyHash  core.HashObj
	// contains filtered or unexported fields
}

HotFileObj holds an open descriptor of a materialized hot file. cleanup releases the shared-file reference after File is closed.

func (*HotFileObj) Close 

func (obj *HotFileObj) Close() error

Close closes the descriptor and runs cleanup for the shared-file refcount. Errors are joined; nil receiver and repeated calls are safe.

type InspectObj 

type InspectObj struct {
	RootPath            string
	SQLitePath          string
	PebblePath          string
	HotPath             string
	VersionCount        uint64
	BlobCount           uint64
	ArtifactCount       uint64
	HistoryEventCount   uint64
	PebbleDiskBytes     uint64
	PebbleRealDiskBytes uint64
	SQLiteDiskBytes     uint64
	HotBytes            uint64
}

InspectObj holds a snapshot of paths, entity counts and disk usage.

type Obj 

type Obj struct {
	// contains filtered or unexported fields
}

Obj combines the durable SQLite index, the Pebble blob/tree store and the hot artifact cache. writeMu serializes all durable-state mutations.

func New 

func New(ctx context.Context, configObj *stcfg.ConfigObj, logArr ...zerolog.Logger) (*Obj, error)

New opens storage, prepares the layout, cleans temp, opens SQLite/Pebble and runs Recover. The config is copied so ownership is not shared with the caller.

func (*Obj) ArtifactDigest 

func (obj *Obj) ArtifactDigest(ctx context.Context, builderObj ArtifactBuilderInterface) (core.ArtifactDigestObj, error)

ArtifactDigest runs a builder once and computes artifact identity: blake3-24, sha256, sha1, size, and ETag. It uses the same hashing scheme as hot-build serving, so body_hash matches on rebuild. Storage owns artifact hashing; overlay provides the builder and rescan calls this before RegisterArtifact.

func (*Obj) CanonicalTree 

func (obj *Obj) CanonicalTree(stagedEntries []core.StagedEntryObj, key string, version string) ([]core.TreeEntryObj, core.HashObj, error)

CanonicalTree normalizes staged entries and computes the tree_hash exactly like PublishStaged. Rescan uses it for cheap version-skip before materialization and reuses canonical entries for artifact builds. It is a pure calculation over config and input entries and does not touch disk.

func (*Obj) Close 

func (obj *Obj) Close(ctx context.Context) error

Close stops storage idempotently. It cancels the root context, waits for active operations/background GC and closes Pebble/index. Repeated calls wait for the same completion; ctx bounds the wait.

func (*Obj) CollectGarbage 

func (obj *Obj) CollectGarbage(ctx context.Context) error

CollectGarbage runs incremental GC for unreachable blobs and trees.

func (*Obj) ContentChecksum 

func (obj *Obj) ContentChecksum(ctx context.Context) (core.HashObj, error)

ContentChecksum returns the deterministic index content checksum for replica and state comparison.

func (*Obj) CountVersions 

func (obj *Obj) CountVersions(ctx context.Context, key string) (uint64, error)

CountVersions returns active version count without loading versions.

func (*Obj) DeleteKey 

func (obj *Obj) DeleteKey(ctx context.Context, key string) (uint64, uint64, error)

DeleteKey deletes all versions of a key under one writeMu for --prune --force. Each version removes its hot files and unreferenced objects; later RepairBlobRefs sweeps remaining orphan blobs.

func (*Obj) DeleteVersion 

func (obj *Obj) DeleteVersion(ctx context.Context, key string, version string) error

DeleteVersion unconditionally removes a version under writeMu. It deletes metadata, hot files, and unreferenced objects; missing versions are no-op.

func (*Obj) DistinctKeys 

func (obj *Obj) DistinctKeys(ctx context.Context, afterKey string, limit int) ([]string, error)

DistinctKeys returns one keyset page of unique version keys with key > afterKey. It lets --prune diff storage against release_mirrors without loading all versions into RAM.

func (*Obj) DurableBytes 

func (obj *Obj) DurableBytes() uint64

DurableBytes returns an estimate of durable disk usage: Pebble live data plus SQLite.

func (*Obj) EnsureArtifactFile 

func (obj *Obj) EnsureArtifactFile(ctx context.Context, keyObj core.ArtifactKeyObj, builderObj ArtifactBuilderInterface) (*HotFileObj, error)

EnsureArtifactFile returns a valid materialized hot file or builds it. Singleflight shares builds by identity so concurrent readers do not rebuild the same artifact. Final hash and size are checked against metadata; degraded or stale artifacts are rejected.

func (*Obj) FilterMissingBlobs 

func (obj *Obj) FilterMissingBlobs(ctx context.Context, hashArr []core.HashObj) ([]core.HashObj, error)

FilterMissingBlobs returns the subset of hashArr absent from durable storage. Brother sync fetches only missing blobs; the tree is the manifest and shared blobs are skipped locally.

func (*Obj) GetArtifact 

func (obj *Obj) GetArtifact(ctx context.Context, keyObj core.ArtifactKeyObj) (core.ArtifactObj, bool, error)

GetArtifact returns artifact metadata by key; the second result reports whether it exists.

func (*Obj) GetDetection 

func (obj *Obj) GetDetection(ctx context.Context, key string, version string) (core.DetectionObj, bool, error)

GetDetection returns format detection for a version; the second result reports whether it exists.

func (*Obj) GetKeySource 

func (obj *Obj) GetKeySource(ctx context.Context, key string) (core.KeySourceObj, bool, error)

GetKeySource returns the durable key-to-source binding; the second result reports whether it exists.

func (*Obj) GetVersion 

func (obj *Obj) GetVersion(ctx context.Context, key string, version string) (core.VersionObj, bool, error)

GetVersion returns version metadata by key and version; the second result reports whether it exists.

func (*Obj) Inspect 

func (obj *Obj) Inspect(ctx context.Context) (InspectObj, error)

Inspect returns a read-only storage snapshot with paths, table counts, and durable plus hot bytes.

func (*Obj) KeyDeletionEstimate 

func (obj *Obj) KeyDeletionEstimate(ctx context.Context, key string) (uint64, uint64, error)

KeyDeletionEstimate is a --prune dry-run for one key. It returns version count and an upper reclaim estimate; shared blobs may be counted more than once.

func (*Obj) LatestVersion 

func (obj *Obj) LatestVersion(ctx context.Context, key string) (core.VersionObj, bool, error)

LatestVersion returns the latest active key version; the second result reports whether any exists.

func (*Obj) ListArtifacts 

func (obj *Obj) ListArtifacts(ctx context.Context, key string, version string) ([]core.ArtifactObj, error)

ListArtifacts returns all artifacts for a version for release_detail.

func (*Obj) ListKeySources 

func (obj *Obj) ListKeySources(ctx context.Context) ([]core.KeySourceObj, error)

ListKeySources returns all bindings used by boot name-to-URL checks.

func (*Obj) ListPublishFeed 

func (obj *Obj) ListPublishFeed(ctx context.Context, key string, limit int) ([]core.FeedEventObj, error)

ListPublishFeed returns newest-first publish events for Atom feeds. An empty key means all keys; the method is read-only and does not take writeMu.

func (*Obj) ListVersions 

func (obj *Obj) ListVersions(ctx context.Context, key string, includeDeleted bool) ([]core.VersionObj, error)

ListVersions returns all key versions newest-first; includeDeleted includes upstream-deleted versions.

func (*Obj) ListVersionsKeyset 

func (obj *Obj) ListVersionsKeyset(ctx context.Context, key string, includeDeleted bool, afterSeq int64, afterVersion string, limit int) ([]core.VersionObj, error)

ListVersionsKeyset returns newest-first versions by (upstream_seq, version) cursor without OFFSET. Serve pagination and full walks use this O(log n + limit) path.

func (*Obj) ListVersionsKeysetBefore 

func (obj *Obj) ListVersionsKeysetBefore(ctx context.Context, key string, includeDeleted bool, beforeSeq int64, beforeVersion string, limit int) ([]core.VersionObj, error)

ListVersionsKeysetBefore returns versions newer than the (beforeSeq, beforeVersion) cursor, ascending (closest first). It backs the "newer" pager direction; the serve layer reverses and trims for display.

func (*Obj) ListVersionsPage 

func (obj *Obj) ListVersionsPage(ctx context.Context, key string, includeDeleted bool, limit int, offset int) ([]core.VersionObj, error)

ListVersionsPage returns one newest-first page without loading the full version set into RAM.

func (*Obj) MarkUpstreamDeleted 

func (obj *Obj) MarkUpstreamDeleted(ctx context.Context, key string, version string) error

MarkUpstreamDeleted handles versions that disappeared upstream according to history_policy.deletion. Delete mode physically removes the version; keep mode sets upstream_deleted, writes history, and refreshes latest.

func (*Obj) MaxUpstreamSeq 

func (obj *Obj) MaxUpstreamSeq(ctx context.Context, key string) (int64, error)

MaxUpstreamSeq returns the maximum stored upstream position for a key; 0 means the key has no versions. Rescan pre-assigns positions from this base before publishing a listing batch.

func (*Obj) NewBlobSpool 

func (obj *Obj) NewBlobSpool(ctx context.Context) (*BlobSpoolObj, error)

NewBlobSpool creates a temp/ spool for staged blobs before PublishStaged. Callers must close the spool; PublishStaged does it through cleanupSpool.

func (*Obj) Publish 

func (obj *Obj) Publish(ctx context.Context, publishObj core.PublishObj) (core.PublishResultObj, error)

Publish stores a version from inline entries, with content already in memory. It normalizes the tree, verifies links under writeMu, writes missing objects within durable budget, and commits metadata. The tree object is written last, so the version is unreachable until commit; identical republishes return Skipped.

func (*Obj) PublishStaged 

func (obj *Obj) PublishStaged(ctx context.Context, spoolObj *BlobSpoolObj, stagedObj core.StagedPublishObj) (core.PublishResultObj, error)

PublishStaged stores a version from staged spool blobs. It normalizes the tree, verifies blobs and symlinks, writes missing Pebble objects under durable budget, and commits. The spool is removed in all cases; the tree is written last, so the version is unreachable before commit.

func (*Obj) PutDetection 

func (obj *Obj) PutDetection(ctx context.Context, key string, version string, detectionObj core.DetectionObj) error

PutDetection upserts format detection for an already published version. Used by rescan heal to fix legacy rows whose go-zip viability was computed after publication; the versions FK rejects unknown versions.

func (*Obj) PutKeySource 

func (obj *Obj) PutKeySource(ctx context.Context, keySourceObj core.KeySourceObj) error

PutKeySource writes a binding under writeMu only when content changes. This avoids WAL churn on every rescan cycle; BoundTS is filled automatically when absent.

func (*Obj) ReadBlob 

func (obj *Obj) ReadBlob(ctx context.Context, hashObj core.HashObj) ([]byte, error)

ReadBlob reads a durable blob under an in-flight slot. Build paths can carry a snapshot in ctx so all reads use the same cut; verify_on_read follows policy.

func (*Obj) ReadTree 

func (obj *Obj) ReadTree(ctx context.Context, treeHashObj core.HashObj) ([]core.TreeEntryObj, error)

ReadTree reads and decodes a tree under an in-flight slot, using ctx snapshot and verify_on_read like ReadBlob.

func (*Obj) Recover 

func (obj *Obj) Recover(ctx context.Context) error

Recover runs startup recovery under writeMu and all build slots. It cleans interrupted temp files before serving begins.

func (*Obj) RegisterArtifact 

func (obj *Obj) RegisterArtifact(ctx context.Context, artifactObj core.ArtifactObj) error

RegisterArtifact registers one version artifact under writeMu in one transaction. Digests are computed beforehand by ArtifactDigest, then metadata and a history event are written.

func (*Obj) RegisterMetrics 

func (obj *Obj) RegisterMetrics(meterObj metric.Meter) error

RegisterMetrics registers observable Pebble metrics in the cache meter. A nil meter is a no-op; one callback captures db.Metrics and event counters per pass. db.Metrics takes Pebble's global DB mutex and walks LSM levels, so scrape intervals should stay at several seconds or more.

func (*Obj) RepairBlobRefs 

func (obj *Obj) RepairBlobRefs(ctx context.Context) error

RepairBlobRefs fully rebuilds blob_refs from trees under writeMu.

func (*Obj) ResurrectVersion 

func (obj *Obj) ResurrectVersion(ctx context.Context, key string, version string) error

ResurrectVersion clears upstream_deleted for a keep-mode version that returned with the same tree. Blobs and artifacts are already present, so it records a publish event, refreshes latest, and is idempotent.

func (*Obj) RewriteSet 

func (obj *Obj) RewriteSet(ctx context.Context, key string, version string) ([]core.HashObj, error)

RewriteSet returns blob hashes rewritten by overlay and needed for artifact rebuilds.

func (*Obj) SetHealPending 

func (obj *Obj) SetHealPending(ctx context.Context, key string, version string, pending bool) error

SetHealPending toggles the incomplete-materialization flag for a version. It is one-shot: rescan clears it after a heal attempt to avoid endless retries on healthy versions.

func (*Obj) SetKeyListingMode 

func (obj *Obj) SetKeyListingMode(ctx context.Context, key string, mode string) error

SetKeyListingMode pins the sticky git listing mode of a key (” resets it). PutKeySource never touches the mode, so rediscovery cannot silently switch a tags-mode key.

func (*Obj) SmallestArtifactByDescriptor 

func (obj *Obj) SmallestArtifactByDescriptor(ctx context.Context, materializerID string, artifactKind string, formatVersion uint32) (core.ArtifactObj, bool, error)

SmallestArtifactByDescriptor returns the smallest artifact for a descriptor triple for startup self-test.

func (*Obj) TouchVersionVerified 

func (obj *Obj) TouchVersionVerified(ctx context.Context, key string, version string, verifiedTS time.Time, upstreamRef string) error

TouchVersionVerified updates deep-verification time and/or adopts upstream_ref. A zero verifiedTS and an empty upstreamRef leave their existing fields unchanged.

func (*Obj) UpdateArtifactDigest 

func (obj *Obj) UpdateArtifactDigest(ctx context.Context, artifactObj core.ArtifactObj) error

UpdateArtifactDigest rewrites artifact digest metadata under writeMu for --rebuild-cache drift repair. It does not write history because this is metadata repair, not a new publication.

func (*Obj) Vacuum 

func (obj *Obj) Vacuum(ctx context.Context) (VacuumResultObj, error)

Vacuum reclaims disk with full Pebble compaction, WAL checkpoint, and SQLite VACUUM. It runs under writeMu during stopped maintenance; sizes are the real physical footprint.

func (*Obj) VerifyIntegrity 

func (obj *Obj) VerifyIntegrity(ctx context.Context) (core.IntegrityReportObj, error)

VerifyIntegrity performs a full integrity check under writeMu. It runs SQLite checks, validates blob_refs against trees, re-hashes reachable Pebble blobs, and counts orphans.

type VacuumResultObj 

type VacuumResultObj struct {
	BeforeBytes uint64
	AfterBytes  uint64
	FreedBytes  uint64
}

VacuumResultObj reports durable physical size before and after --vacuum.

Source Files

View all Source files

Directories

Path Synopsis
internal
hotverify

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.