blob

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package blob provides immutable pack-object storage: the BlobStore contract plus the S3/MinIO implementation (gocloud.dev s3blob).

Local-filesystem BlobStore backend (gocloud.dev fileblob) for local mode (`serve --data-dir`): packs are plain files under a directory, so a test binary needs zero external dependencies. This file also holds the shared bucket helpers (upload/download/list over *gcblob.Bucket) so the atomic tmp+rename download contract has exactly one implementation.

Index

Constants

View Source
const CoverHandoffTimeout = 10 * time.Second

CoverHandoffTimeout bounds the floor-generation bump a caller runs after a restore (Codex round 69): the bump must COMMIT while the last cover row still lives, and after a broken tail refresh that row's remaining lifetime can be as short as minTTL − coverLapseBudget ≈ 16s (the loss is reported at covered+budget; the row lives to covered+TTL). Ten seconds fits inside that worst case with slack, and trivially inside the fresh full TTL a successful tail refresh leaves.

View Source
const EnvExpectedBucketOwner = "OPENSTORAGE_S3_EXPECTED_BUCKET_OWNER"

EnvExpectedBucketOwner optionally pins every S3 request to a bucket owner (issue #58): when set to a 12-digit AWS account ID, the client middleware sends the bucket-owner condition on EVERY operation, so a bucket whose global name was re-registered by another account yields 403 Access Denied instead of silently serving/accepting private packs. Read here (not in cmd config) so every process that opens the store — serve, git hooks (inheriting the serve environment), and the purge CLI — enforces it uniformly; a probe-only or serve-only check would leave the rest of the data plane owner-blind. Empty (the default) attaches no middleware and changes no behavior.

View Source
const LFSRoot = "lfs/"

LFSRoot is the listing prefix covering every repo's LFS objects (the orphan sweeper scans it like PacksRoot, issue #51). LFS keys always use the BASE RepoID (ADR-35): namespaces share one object space, so there is no "+suffix" variant to cover.

View Source
const PacksRoot = "packs/"

PacksRoot is the listing prefix covering every repo's packs (the orphan-pack sweeper scans it in one listing instead of enumerating repos). Namespaced repos (`<repo>+import`/`<repo>+ephemeral`) keep fully separate pack spaces keyed by the full name, so each key's repo segment identifies its authoritative pack registry row set.

Variables

View Source
var ErrCapExceeded = errors.New("blob: stream exceeds the reserved cap")

ErrCapExceeded reports a capped download whose stream exceeded the caller's reserved size (Codex round 117 on #104): on a mutable legacy key, the object can be replaced by a larger version between the caller's Stat and the GET, and streaming past the reservation would consume unbudgeted replica disk.

View Source
var ErrCoverContinuityLost = errors.New("cover continuity lost")

ErrCoverContinuityLost marks work canceled by the cover-continuity machinery (Codex security round 107 on #104): callers must be able to distinguish a copy that erred BECAUSE its cover lapsed — whose committed destination is an unsafe, unaccounted commit exactly like a renewErr verdict — from ordinary op failures, because WithRenewedCover reports a canceled op through opErr with renewErr nil.

View Source
var ErrNotFound = errors.New("blob: object not found")

ErrNotFound is returned (wrapped, match with errors.Is) by Stat and DownloadFile for missing objects, so callers can distinguish absence from transport failures without depending on gocloud error codes.

View Source
var ErrSignedURLUnsupported = errors.New("blob: signed URLs unsupported by this backend")

ErrSignedURLUnsupported is returned by SignedURL on backends without presign support (fileblob). Callers (the LFS batch handler) fall back to proxying the transfer through the server.

Functions

func EnsurePublished

func EnsurePublished(ctx context.Context, b BlobStore, key, path string, onRestore func(context.Context, int64) error, onRestored func(context.Context) error) (restored bool, err error)

EnsurePublished re-uploads the local file at path to key when key is absent. It is the writer half of the orphan-sweep coordination (issue #5), called strictly AFTER a successful RegisterPacks: registration follows upload, so an orphan sweeper that deleted the freshly uploaded key just before the registration committed would otherwise leave a registered pack with no bytes. Ordering proof: the sweeper deletes (D) then re-reads the registry (C); the writer registers (R) then verifies here (V). If C observes R the sweeper restores the pack itself; if C ran before R then D happened before V, so V observes the missing key and re-uploads. Both writers produce identical bytes (pack keys are content-addressed), so the overlap is harmless either way.

The verification runs on a bounded context detached from the caller's cancellation: the registration it covers has already committed, so a client hanging up right after the commit must not skip the check — a skipped verification could leave the persistent pack row pointing at a blob a concurrent sweep just deleted (Codex round 14).

A restore re-materializes bytes the usage counter already includes (Codex round 58): a quota floor scanned during the absence subtracted the full counter from a listing missing them, hiding equally sized rowless legacy bytes. onRestore (nil-ok) runs BEFORE the repair upload so the caller can account the returning bytes first (a cover reservation — see the LFS repair in githttp); an onRestore error aborts the restore, leaving the registered-row-without-bytes state the next retry heals. The returned restored flag tells the caller to invalidate cached floors (bump the shared floor generation) after the bytes are back. onRestored (nil-ok) is the invalidation HANDOFF (Codex round 70): it runs under a live, freshly proven cover and is retried — alternating with cover renewals — until it commits or continuity is lost, so a transient ledger failure cannot strand durable restored bytes with neither cover nor invalidation.

func LFSKey

func LFSKey(repoID, oid string) string

LFSKey is the immutable object key for a repo's LFS object with the given sha256 oid. Keys are repo-scoped (docs/explanation/git-protocol-and-namespaces.md): forks inherit LFS pointers only, never the objects, which this keying yields automatically.

LFSKey is the CANONICAL object key: bytes here are written only by the server's digest-verified paths (verified proxy uploads, adoption copies, registered-object restores) — never by presigned PUTs. The `o2/` segment makes the spelling unreachable by any v0.1.x presigned upload URL, which signed exactly the bare LFSLegacyKey spelling (Codex security round 78 on issue #51): a still-valid legacy URL can overwrite ONLY the legacy key, so registered content — always canonical — is immune to replay for the URL's whole lifetime, with no quiesce-window reasoning. The oid is caller-validated 64-hex (lfsOIDPattern / isLFSOID); the assert rules out path tricks reaching the canonical namespace.

func LFSLegacyKey

func LFSLegacyKey(repoID, oid string) string

LFSLegacyKey is the v0.1.x object spelling — the only one old presigned PUT URLs can address. It is never registered and never served for registered objects: it exists as the adoption SOURCE (rowless pre-#51 blobs live here until adopted, then the source is deleted) and as the read-compat location for not-yet-adopted rowless objects.

func LFSPrefix

func LFSPrefix(repoID string) string

LFSPrefix is the listing prefix for a repo's LFS objects.

func PackKey

func PackKey(repo, id string) string

PackKey is the immutable object key for a repo's pack with the given id.

func PackPrefix

func PackPrefix(repo string) string

PackPrefix is the listing prefix for a repo's packs.

func QuotaRepoGroup

func QuotaRepoGroup(repo string) []string

QuotaRepoGroup returns repo's quota accounting group: the BASE repository first (suffix stripped), then every docs/explanation/git-protocol-and-namespaces.md namespace that rolls up under it. The ref-store quota ledger (issue #29) keys reservations under group[0] and sums usage over the whole group, mirroring StorageUsageBytes' rollup so both usage sources measure the same unit.

func StorageUsageBytes

func StorageUsageBytes(ctx context.Context, b BlobStore, repo string) (int64, error)

StorageUsageBytes sums the sizes of every object under the pack prefixes of repo's BASE repository and all of its namespaces PLUS the repo's LFS prefix (issue #51): registered packs and LFS objects, not-yet-swept orphans, AND retained superseded blobs — exactly the durable growth the per-repo storage quota (issue #21) bounds. The suffix, if any, is stripped first, so every namespace write is measured against the same rolled-up usage. Shared by the pre-receive hook, the REST pack writers, the LFS batch/transfer gates, and the git surface's pressure check.

Since issue #29 this listing is the admission check's usage FLOOR rather than its sole source: the ref-store ledger (ref.ReservePackBytes) charges max(counter, this listing) plus in-flight reservations, closing the concurrent-push overshoot this listing alone admitted. Retained superseded bytes (issue #31) are counted by BOTH sources — here as lingering prefix objects, there as pack_retained rows — until a sweep reclaims them. LFS bytes are likewise counted by both once uploads settle (ref.RegisterLFSObject); before #51's ledger existed, only this listing could see them, which is one more reason the floor stays.

func WithRenewedCover

func WithRenewedCover(ctx context.Context, renew func(context.Context) error, op func(context.Context) error, handoff func(context.Context) error) (opErr, renewErr error)

WithRenewedCover runs op while keeping a restore's cover reservation alive (Codex round 59): it takes the initial cover itself (round 64), renews it every coverRenewInterval — each attempt deadline-bounded and continuity-checked — for op's duration, and after op succeeds it alternates fresh cover refreshes with bounded attempts of the caller's handoff (the floor invalidation) until the handoff COMMITS under a live cover or continuity is lost (Codex round 70 — a single best-effort handoff could fail and strand durable bytes with neither cover nor invalidation). Renewal failures are tolerated while a recent success still covers the bytes; once coverLapseBudget passes without one, op's context is canceled — the restore aborts rather than publishing uncovered bytes (Codex round 62). opErr reports op's own failure; renewErr reports a handoff that never landed.

Types

type BlobStore

type BlobStore interface {
	// UploadFile streams a local file to key.
	UploadFile(ctx context.Context, key, path string) error
	// DownloadFile fetches key into destPath atomically (tmp + rename), so a
	// partially downloaded object is never visible at destPath. Missing keys
	// yield an error matching ErrNotFound via errors.Is (the hydration
	// missing-pack retry keys on it).
	DownloadFile(ctx context.Context, key, destPath string) error
	// DownloadFileCapped is DownloadFile bounded to capBytes of local
	// disk (Codex round 117 on #104): one GET, atomic rename — but a
	// stream exceeding capBytes aborts with ErrCapExceeded instead of
	// consuming unreserved space (the caller reserved capBytes BEFORE a
	// window in which a mutable-key object can be replaced by a larger
	// version).
	DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error
	// List returns all objects under prefix.
	List(ctx context.Context, prefix string) ([]ListedObject, error)
	// ListPages streams objects under prefix in lexicographic key order,
	// invoking fn once per page of at most pageSize objects, so callers
	// scanning large prefixes (the orphan-pack sweeper) hold one page at a
	// time instead of the whole listing. fn must not retain the slice; a
	// non-nil fn error aborts the scan and is returned unwrapped.
	ListPages(ctx context.Context, prefix string, pageSize int, fn func([]ListedObject) error) error
	// Delete removes the object at key; missing objects yield an error
	// matching ErrNotFound via errors.Is. Objects are immutable, so Delete
	// exists solely for the orphan-pack sweeper (issue #5): reaping blobs
	// that were uploaded but never registered because of a crash.
	Delete(ctx context.Context, key string) error
	// Copy duplicates srcKey to dstKey inside the store (server-side S3
	// CopyObject / local file copy — never staged through local disk, so
	// pack-sized copies work regardless of temp-dir capacity). A missing
	// srcKey yields an error matching ErrNotFound via errors.Is. Used by
	// the orphan-pack sweeper's quarantine handshake.
	Copy(ctx context.Context, dstKey, srcKey string) error
	// SignedURL returns a presigned URL for the given HTTP method
	// (http.MethodGet or http.MethodPut) on key, valid for expiry.
	// Backends without presign support return ErrSignedURLUnsupported.
	SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)
	// Stat returns the size of the object at key; missing objects yield an
	// error matching ErrNotFound via errors.Is.
	Stat(ctx context.Context, key string) (int64, error)
	// StatFingerprint returns the size plus an opaque content-version
	// fingerprint for the object at key: any overwrite changes it (S3
	// ETag — content-derived, so an attacker cannot preserve it across a
	// same-length rewrite; file backends fall back to mtime+size).
	// Verification flows that hash an object across multiple reads use
	// fingerprint equality to prove the bytes were not REPLACED between
	// reads (issue #51, Codex round 77): ranged reads spanning a swap
	// can splice old-prefix+new-suffix into a digest that matches even
	// though the stored bytes differ. Missing objects yield an error
	// matching ErrNotFound via errors.Is.
	StatFingerprint(ctx context.Context, key string) (int64, string, error)
	// ReadRange returns exactly the length bytes at offset from the object
	// at key — a ranged point read, never the whole object (repack fold
	// selection reads exact object counts from 12-byte pack headers with
	// it). Missing keys yield an error matching ErrNotFound via errors.Is;
	// an object shorter than offset+length is a non-ErrNotFound error.
	ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
	// EnsureBucket idempotently creates the backing bucket.
	EnsureBucket(ctx context.Context) error
	Close() error
}

BlobStore is the pack-object storage contract. Keys are opaque to callers except through PackKey/PackPrefix. Objects are immutable once written.

func Open

func Open(ctx context.Context, blobURL string) (BlobStore, error)

Open dispatches on the URL scheme to the matching BlobStore backend:

s3://bucket?region=&endpoint=  → OpenS3
file:///abs/dir                → local filesystem bucket

BlobStore already carries the lifecycle (EnsureBucket/Close), so there is no separate lifecycle interface. Open never calls EnsureBucket: hooks reconstruct stores from env URLs; the serve seam owns bucket creation.

type File

type File struct {
	// contains filtered or unexported fields
}

File implements BlobStore on a local directory via gocloud fileblob.

func OpenFile

func OpenFile(blobURL string) (*File, error)

OpenFile opens a file:// blob URL, e.g. file:///abs/dir (empty host, so the directory is always absolute). The root directory is created up front — hooks reconstruct stores from env URLs without calling EnsureBucket, and a missing directory must not fail them. MetadataDontWrite keeps listings clean: no .attrs sidecars next to pack files.

func (*File) Close

func (b *File) Close() error

func (*File) Copy

func (b *File) Copy(ctx context.Context, dstKey, srcKey string) error

Copy duplicates srcKey to dstKey (local file copy inside the bucket dir).

func (*File) Delete

func (b *File) Delete(ctx context.Context, key string) error

Delete removes the object at key (ErrNotFound when absent).

func (*File) DownloadFile

func (b *File) DownloadFile(ctx context.Context, key, destPath string) error

DownloadFile fetches key into destPath atomically (tmp + rename).

func (*File) DownloadFileCapped

func (b *File) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error

DownloadFileCapped fetches key into destPath atomically, aborting with ErrCapExceeded once the stream would exceed capBytes on disk.

func (*File) EnsureBucket

func (b *File) EnsureBucket(_ context.Context) error

EnsureBucket idempotently creates the backing directory (the file:// analogue of S3 CreateBucket at serve startup).

func (*File) List

func (b *File) List(ctx context.Context, prefix string) ([]ListedObject, error)

List returns all objects under prefix.

func (*File) ListPages

func (b *File) ListPages(ctx context.Context, prefix string, pageSize int, fn func([]ListedObject) error) error

ListPages streams objects under prefix in key order, one page at a time.

func (*File) ReadRange

func (b *File) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)

ReadRange returns exactly the length bytes at offset from the object at key (a seeked local read; ErrNotFound when absent).

func (*File) SignedURL

func (b *File) SignedURL(_ context.Context, key, method string, expiry time.Duration) (string, error)

SignedURL is unsupported on the filesystem backend: it returns ErrSignedURLUnsupported so callers fall back to proxying transfers through the server (docs/explanation/git-protocol-and-namespaces.md local mode).

func (*File) Stat

func (b *File) Stat(ctx context.Context, key string) (int64, error)

Stat returns the size of the object at key (ErrNotFound when absent).

func (*File) StatFingerprint

func (b *File) StatFingerprint(ctx context.Context, key string) (int64, string, error)

StatFingerprint — see BlobStore.

func (*File) UploadFile

func (b *File) UploadFile(ctx context.Context, key, path string) error

UploadFile streams a local file to key.

type ListedObject

type ListedObject struct {
	Key  string
	Size int64
	// ModTime is the backend's last-modified time for the object (S3
	// LastModified / file mtime). The orphan-pack sweeper uses it as the
	// upload age: objects are immutable, so it never moves after publish.
	ModTime time.Time
}

ListedObject is one blob under a prefix.

type S3

type S3 struct {
	// contains filtered or unexported fields
}

S3 implements BlobStore on a gocloud S3 bucket plus the raw client (needed for EnsureBucket at serve startup).

func OpenS3

func OpenS3(ctx context.Context, blobURL string) (*S3, error)

OpenS3 opens a blob URL, e.g. s3://spike?endpoint=http://127.0.0.1:9000&s3ForcePathStyle=true&region=us-east-1 Credentials come from the standard AWS env vars, and the optional expected-bucket-owner enforcement comes from EnvExpectedBucketOwner (a malformed value is boot-fatal). The URL is parsed by hand (bucket = host, endpoint/region = query) so we control path-style behavior deterministically against MinIO regardless of gocloud URL-param renames.

func (*S3) Close

func (b *S3) Close() error

func (*S3) Copy

func (b *S3) Copy(ctx context.Context, dstKey, srcKey string) error

Copy duplicates srcKey to dstKey: server-side CopyObject for objects up to the single-request limit, multipart UploadPartCopy beyond it (still fully server-side — bytes never flow through this process).

func (*S3) Delete

func (b *S3) Delete(ctx context.Context, key string) error

Delete removes the object at key (ErrNotFound when absent).

func (*S3) DownloadFile

func (b *S3) DownloadFile(ctx context.Context, key, destPath string) error

DownloadFile fetches key into destPath atomically (tmp + rename).

func (*S3) DownloadFileCapped

func (b *S3) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error

DownloadFileCapped fetches key into destPath atomically, aborting with ErrCapExceeded once the stream would exceed capBytes on disk.

func (*S3) EnsureBucket

func (b *S3) EnsureBucket(ctx context.Context) error

EnsureBucket creates the bucket, tolerating "already exists/owned by you" (so `serve` works against a fresh MinIO). A least-privilege deployment role for a pre-provisioned bucket has no s3:CreateBucket, so AccessDenied on create is not proof the bucket is missing: probe it with HeadBucket and accept when reachable. Any other create error, or a denied create with an unreachable bucket, stays boot-fatal. With an expected bucket owner configured (issue #58), EVERY accepted outcome additionally passes an owner-conditioned HeadBucket probe, so squatted names and credential/account drift fail boot instead of 403ing data operations.

func (*S3) List

func (b *S3) List(ctx context.Context, prefix string) ([]ListedObject, error)

List returns all objects under prefix.

func (*S3) ListPages

func (b *S3) ListPages(ctx context.Context, prefix string, pageSize int, fn func([]ListedObject) error) error

ListPages streams objects under prefix in key order, one page at a time.

func (*S3) ReadRange

func (b *S3) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)

ReadRange returns exactly the length bytes at offset from the object at key (a ranged GET; ErrNotFound when absent).

func (*S3) SignedURL

func (b *S3) SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)

SignedURL returns a presigned S3 URL for method (GET or PUT) on key. ContentType is deliberately left unset in the options: gocloud's aws-sdk-go-v2 presigner returns Unimplemented when asked to sign a Content-Type for PUT, and an unsigned Content-Type means git-lfs's `Content-Type: application/octet-stream` PUT verifies against MinIO/S3 SigV4 without extra configuration.

With expected-bucket-owner enforcement on, presigning runs through the same client middleware as every other operation and the presigner hoists the x-amz-* header into a SIGNED lowercase query parameter (x-amz-expected-bucket-owner=...), the spelling S3 documents and enforces on presigned requests (aws-sdk-go-v2 #2508). The URL is self-contained: redeemers send no extra header, and stripping or altering the parameter breaks the SigV4 signature.

func (*S3) Stat

func (b *S3) Stat(ctx context.Context, key string) (int64, error)

Stat returns the size of the object at key (ErrNotFound when absent).

func (*S3) StatFingerprint

func (b *S3) StatFingerprint(ctx context.Context, key string) (int64, string, error)

StatFingerprint — see BlobStore.

func (*S3) UploadFile

func (b *S3) UploadFile(ctx context.Context, key, path string) error

UploadFile streams a local file to key.

func (*S3) VersioningEnabled

func (b *S3) VersioningEnabled(ctx context.Context) (bool, error)

VersioningEnabled reports whether the backing bucket has ever had S3 versioning turned on (status Enabled OR Suspended — suspension keeps all previously stored versions, and unversioned deletes still only hide the current version behind a delete marker). The orphan-pack sweeper's delete mode refuses such buckets: its deletes would reclaim nothing while reporting that they did.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL