storage

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 26, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package storage abstracts where dump objects and their sidecar metadata physically live. The dump catalog (internal/dumps) holds a Store and addresses objects by opaque keys (e.g. "<id>.dump", "<id>.meta.json") rather than filesystem paths, so the same catalog logic works over a local directory or an object store (S3 and S3-compatible services such as MinIO, R2).

This package is a stdlib-only leaf: it defines the Store interface plus a shared not-found sentinel. Concrete backends live in sibling files (local.go, s3.go) and may import third-party SDKs; the interface itself does not.

Index

Constants

This section is empty.

Variables

View Source
var ErrNotFound = errors.New("storage: object not found")

ErrNotFound is returned by Get when the requested key is absent. (Delete and Stat treat a missing key as a non-error — see their contracts below.) Backends MUST wrap their native "no such object" error with this sentinel (via fmt.Errorf("...: %w", ErrNotFound) or by returning it directly) so the catalog and app layers can distinguish "you asked for something that isn't here" (a user error) from a transient transport failure (a system error).

Functions

func RunStoreSuite

func RunStoreSuite(t *testing.T, newStore func(t *testing.T) Store)

RunStoreSuite exercises the Store contract against a backend. Every Store implementation runs this same table, so correctness is a property of implementing the interface — not a per-backend afterthought. newStore returns a fresh, empty Store for each subtest (e.g. a t.TempDir-rooted local store, or a uniquely-prefixed bucket view).

It lives in a non-test file so both the local unit test and the S3 integration test (separate build tags) can call it.

Types

type S3Options

type S3Options struct {
	Bucket   string
	Prefix   string // optional key prefix within the bucket
	Region   string
	Endpoint string // optional custom endpoint for S3-compatible services (MinIO, R2)
}

S3Options configures an S3 (or S3-compatible) Store. Credentials are NOT here: the SDK resolves them from the standard chain (env vars, shared config, instance/role), keeping the siphon config file free of secrets.

type Store

type Store interface {
	// Put writes the full contents of r under key, durably and atomically: the
	// key either resolves to the complete object or does not resolve at all — a
	// reader that fails mid-stream, or a cancelled context, must not leave a
	// partial object visible under key. Overwriting an existing key is allowed
	// and replaces it. Put reads r to EOF.
	Put(ctx context.Context, key string, r io.Reader) error

	// Get opens key for reading. The returned ReadCloser is a one-shot forward
	// stream — callers must not assume it is seekable — and must be closed. A
	// missing key returns an error wrapping ErrNotFound.
	Get(ctx context.Context, key string) (io.ReadCloser, error)

	// Delete removes key. Deleting a key that does not exist is NOT an error
	// (delete is idempotent), so callers can prune without racing existence.
	Delete(ctx context.Context, key string) error

	// List returns every key currently present in the store, in no guaranteed
	// order.
	List(ctx context.Context) ([]string, error)

	// Stat reports the size in bytes and existence of key. A missing key returns
	// (0, false, nil) — absence is not an error for Stat. A transport failure
	// returns a non-nil error.
	Stat(ctx context.Context, key string) (size int64, exists bool, err error)
}

Store is the durable key→bytes substrate behind the dump catalog.

Keys are opaque, caller-chosen strings. A backend may map a key onto a file name or an object key, but callers never depend on that mapping.

All methods take a context: object I/O is network I/O for remote backends and must be cancellable. A cancelled context aborts the operation.

func NewLocal

func NewLocal(root string) (Store, error)

NewLocal returns a Store backed by the directory root, creating it (0700) if absent. This preserves siphon's pre-Phase-G on-disk layout: keys are written verbatim as files under root, so an existing local catalog keeps working with no migration.

func NewS3

func NewS3(ctx context.Context, opt S3Options) (Store, error)

NewS3 builds an S3-backed Store. It loads AWS config (region, credentials) from the default chain and applies the optional custom endpoint for S3-compatible services. The bucket is assumed to exist.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL