storage

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 10, 2026 License: AGPL-3.0 Imports: 19 Imported by: 0

Documentation

Overview

Package storage provides shithub's storage abstractions:

  • ObjectStore: a small interface for S3-compatible object storage (works against MinIO in dev/test and DigitalOcean Spaces in prod).
  • reposfs: filesystem-backed bare git repository helpers (sharded layout, atomic helpers, strict path validation).
  • Quota: a placeholder type for disk-quota plumbing (no enforcement yet — wired up in later sprints).

Path validation is the security boundary: every entry point that takes owner/repo names from outside the package goes through RepoPath, which rejects unsafe inputs against a strict whitelist.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound is returned by Get/Stat for absent keys and by reposfs
	// helpers when a path doesn't exist.
	ErrNotFound = errors.New("storage: not found")

	// ErrPreconditionFailed is returned by Put when an If-None-Match check fails.
	ErrPreconditionFailed = errors.New("storage: precondition failed")

	// ErrInvalidPath is returned by RepoPath (and any helper that takes
	// owner/repo names) for inputs that violate the whitelist.
	ErrInvalidPath = errors.New("storage: invalid path")

	// ErrAlreadyExists is returned by Move and InitBare when the destination
	// is already populated. Lets callers distinguish a race from corruption.
	ErrAlreadyExists = errors.New("storage: already exists")

	// ErrEscapesRoot is returned by Delete and Move when a path resolves
	// outside the configured storage root. Hard fail — never silently ignore.
	ErrEscapesRoot = errors.New("storage: path escapes root")
)

Functions

func WriteAtomic 

func WriteAtomic(path string, src io.Reader) error

WriteAtomic writes src to path via a tempfile in the same directory, fsyncs, and renames. A crash between write and rename leaves the temp file behind (callers may sweep these on startup) but never a partial file at path.

The temp file MUST live on the same mount as path so the rename is atomic — callers should not pass paths that cross mount points.

Types

type ListOpts 

type ListOpts struct {
	// ContinuationToken resumes pagination from a prior page.
	ContinuationToken string
	// MaxKeys caps the page size. Zero means backend default.
	MaxKeys int
	// Recursive, when false, treats "/" as a delimiter and surfaces common
	// prefixes (folders) in ListResult.CommonPrefixes.
	Recursive bool
}

ListOpts controls a List.

type ListResult 

type ListResult struct {
	Objects               []ObjectMeta
	CommonPrefixes        []string
	NextContinuationToken string
	IsTruncated           bool
}

ListResult is one page of a List.

type MemoryStore 

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is an in-process ObjectStore implementation. Used by tests. Honors the same If-None-Match semantics as the s3 backend.

func NewMemoryStore 

func NewMemoryStore() *MemoryStore

NewMemoryStore constructs an empty in-memory store.

func (*MemoryStore) Delete 

func (m *MemoryStore) Delete(_ context.Context, key string) error

Delete implements ObjectStore. Idempotent.

func (*MemoryStore) Get 

func (m *MemoryStore) Get(_ context.Context, key string) (io.ReadCloser, ObjectMeta, error)

Get implements ObjectStore.

func (*MemoryStore) List 

func (m *MemoryStore) List(_ context.Context, prefix string, opts ListOpts) (ListResult, error)

List implements ObjectStore. ContinuationToken is the last key returned in the previous page; results are sorted lexicographically.

func (*MemoryStore) Put 

func (m *MemoryStore) Put(_ context.Context, key string, body io.Reader, opts PutOpts) (PutResult, error)

Put implements ObjectStore.

func (*MemoryStore) SignedURL 

func (m *MemoryStore) SignedURL(_ context.Context, key string, ttl time.Duration, method string) (string, error)

SignedURL implements ObjectStore. Tests can rely on the prefix to distinguish memory-backed URLs from real ones.

func (*MemoryStore) Stat 

func (m *MemoryStore) Stat(_ context.Context, key string) (ObjectMeta, error)

Stat implements ObjectStore.

type ObjectMeta 

type ObjectMeta struct {
	Key          string
	Size         int64
	ETag         string
	ContentType  string
	LastModified time.Time
}

ObjectMeta is the metadata returned by Get/Stat.

type ObjectStore 

type ObjectStore interface {
	// Put writes body to key. Returns the resulting object's metadata
	// (etag, size). Honors opts.IfNoneMatch (when "*", fails with
	// ErrPreconditionFailed if the key already exists).
	Put(ctx context.Context, key string, body io.Reader, opts PutOpts) (PutResult, error)

	// Get returns a reader for the object at key. Caller must Close.
	// Returns ErrNotFound when key is absent.
	Get(ctx context.Context, key string) (io.ReadCloser, ObjectMeta, error)

	// Stat returns metadata for key without fetching the body.
	// Returns ErrNotFound when key is absent.
	Stat(ctx context.Context, key string) (ObjectMeta, error)

	// Delete removes key. Returns nil for a missing key (idempotent).
	Delete(ctx context.Context, key string) error

	// List enumerates objects under prefix. Pagination via opts.ContinuationToken.
	List(ctx context.Context, prefix string, opts ListOpts) (ListResult, error)

	// SignedURL returns a pre-signed URL for the given method ("GET" or
	// "PUT") on key, valid for ttl. The URL grants direct browser/client
	// access without exposing credentials — used for avatar/attachment
	// uploads and large downloads in later sprints.
	SignedURL(ctx context.Context, key string, ttl time.Duration, method string) (string, error)
}

ObjectStore is the abstract interface every storage backend implements. Implementations: s3 (any S3-compatible endpoint via minio-go) and memory (in-process map for tests).

type PutOpts 

type PutOpts struct {
	ContentType string
	// IfNoneMatch, when "*", causes Put to fail with ErrPreconditionFailed
	// if the destination already exists. Other values are not supported.
	IfNoneMatch string
	// ContentLength, when > 0, is passed to the backend as a hint. When 0,
	// the backend buffers / streams as needed.
	ContentLength int64
}

PutOpts controls a Put.

type PutResult 

type PutResult struct {
	ETag string
	Size int64
}

PutResult is what Put returns.

type Quota 

type Quota struct {
	Used  int64 // bytes currently used
	Limit int64 // bytes allowed (0 = unlimited)
}

Quota is the disk-usage budget for a user or org. Recorded in the DB under users.disk_quota_* and orgs.disk_quota_* (added when those tables exist). S04 wires the type only — enforcement lives in a future policy package called from the push pipeline (S14) and attachment uploads.

func (Quota) Available 

func (q Quota) Available() int64

Available returns Limit - Used, clamped at zero. Returns -1 when the quota is unlimited.

func (Quota) WouldExceed 

func (q Quota) WouldExceed(n int64) bool

WouldExceed reports whether writing additional bytes n would push past the limit. Always false for an unlimited quota.

type RepoFS 

type RepoFS struct {
	// contains filtered or unexported fields
}

RepoFS owns the on-disk layout for bare git repositories. All callers that touch repo paths route through this type so the path-validation rules live in exactly one place.

func NewRepoFS 

func NewRepoFS(root string) (*RepoFS, error)

NewRepoFS validates root (must be absolute, must exist, must be a directory) and returns the layer.

func (*RepoFS) CloneBareShared 

func (r *RepoFS) CloneBareShared(ctx context.Context, src, dst string) error

CloneBareShared clones src → dst as a bare repo with object alternates pointing back at src. Disk usage of the result is essentially refs + a small overhead; objects live in src's `objects/` until the fork is detached (S16 hard-delete cascade repacks each fork before removing the source).

Both paths must be contained in r.root and on the same volume — the same-volume requirement is what makes alternates safe (S04).

On success the dst directory exists with `git init --bare` shape plus an `objects/info/alternates` file pointing at src/objects. On failure the dst directory is removed so a retry sees a clean slate.

func (*RepoFS) Delete 

func (r *RepoFS) Delete(path string) error

Delete removes the bare repo at path. Refuses paths outside root.

func (*RepoFS) Exists 

func (r *RepoFS) Exists(path string) (bool, error)

Exists reports whether path exists. Validates that path is under root.

func (*RepoFS) InitBare 

func (r *RepoFS) InitBare(ctx context.Context, path string) error

InitBare creates a bare git repository at path. Default branch is "trunk" — there is no path through this package that creates a bare repo with a different initial branch.

The parent directory tree is created on demand. ErrAlreadyExists is returned if path is non-empty.

func (*RepoFS) Move 

func (r *RepoFS) Move(oldPath, newPath string) error

Move atomically renames oldPath to newPath. Both must be under root. If newPath already exists, returns ErrAlreadyExists rather than overwriting (avoids silent corruption on concurrent moves).

func (*RepoFS) RepoPath 

func (r *RepoFS) RepoPath(owner, name string) (string, error)

RepoPath returns the absolute disk path for the bare repository at (owner, name). Validates inputs and guarantees the result is rooted at r.root. Both inputs are lowercased before path construction.

func (*RepoFS) Root 

func (r *RepoFS) Root() string

Root returns the absolute root path. Useful for logging and `storage check`.

func (*RepoFS) SetPreciousObjects 

func (r *RepoFS) SetPreciousObjects(ctx context.Context, path string) error

SetPreciousObjects marks a bare repo's objects as not-prunable. The canonical foot-gun for forks is source-repo `git gc` removing objects that forks reach via alternates; setting this on the source after a fork is created prevents that. Idempotent.

type S3Config 

type S3Config struct {
	Endpoint        string // host[:port], no scheme
	Region          string
	AccessKeyID     string
	SecretAccessKey string
	Bucket          string
	UseSSL          bool
	ForcePathStyle  bool
}

S3Config configures the S3-compatible client. Mirrors config.S3StorageConfig.

type S3Store 

type S3Store struct {
	// contains filtered or unexported fields
}

S3Store is an ObjectStore backed by any S3-compatible endpoint.

func NewS3Store 

func NewS3Store(cfg S3Config) (*S3Store, error)

NewS3Store constructs the client. The bucket must already exist; this constructor does not create it (deploy/dev scripts seed buckets).

func (*S3Store) Delete 

func (s *S3Store) Delete(ctx context.Context, key string) error

Delete implements ObjectStore. Returns nil for missing keys (idempotent).

func (*S3Store) Get 

func (s *S3Store) Get(ctx context.Context, key string) (io.ReadCloser, ObjectMeta, error)

Get implements ObjectStore.

func (*S3Store) List 

func (s *S3Store) List(ctx context.Context, prefix string, opts ListOpts) (ListResult, error)

List implements ObjectStore. Recursive=false uses "/" as a delimiter.

func (*S3Store) Put 

func (s *S3Store) Put(ctx context.Context, key string, body io.Reader, opts PutOpts) (PutResult, error)

Put implements ObjectStore.

func (*S3Store) SignedURL 

func (s *S3Store) SignedURL(ctx context.Context, key string, ttl time.Duration, method string) (string, error)

SignedURL implements ObjectStore.

func (*S3Store) Stat 

func (s *S3Store) Stat(ctx context.Context, key string) (ObjectMeta, error)

Stat implements ObjectStore.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.