Documentation
¶
Overview ¶
Package blob provides immutable pack-object storage: the BlobStore contract plus the S3/MinIO implementation (gocloud.dev s3blob).
Local-filesystem BlobStore backend (gocloud.dev fileblob) for local mode (`serve --data-dir`): packs are plain files under a directory, so a test binary needs zero external dependencies. This file also holds the shared bucket helpers (upload/download/list over *gcblob.Bucket) so the atomic tmp+rename download contract has exactly one implementation.
Index ¶
- Constants
- Variables
- func EnsurePublished(ctx context.Context, b BlobStore, key, path string, ...) (restored bool, err error)
- func LFSKey(repoID, oid string) string
- func LFSLegacyKey(repoID, oid string) string
- func LFSPrefix(repoID string) string
- func PackKey(repo, id string) string
- func PackPrefix(repo string) string
- func QuotaRepoGroup(repo string) []string
- func StorageUsageBytes(ctx context.Context, b BlobStore, repo string) (int64, error)
- func WithRenewedCover(ctx context.Context, renew func(context.Context) error, ...) (opErr, renewErr error)
- type BlobStore
- type File
- func (b *File) Close() error
- func (b *File) Copy(ctx context.Context, dstKey, srcKey string) error
- func (b *File) Delete(ctx context.Context, key string) error
- func (b *File) DownloadFile(ctx context.Context, key, destPath string) error
- func (b *File) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error
- func (b *File) EnsureBucket(_ context.Context) error
- func (b *File) List(ctx context.Context, prefix string) ([]ListedObject, error)
- func (b *File) ListPages(ctx context.Context, prefix string, pageSize int, ...) error
- func (b *File) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
- func (b *File) SignedURL(_ context.Context, key, method string, expiry time.Duration) (string, error)
- func (b *File) Stat(ctx context.Context, key string) (int64, error)
- func (b *File) StatFingerprint(ctx context.Context, key string) (int64, string, error)
- func (b *File) UploadFile(ctx context.Context, key, path string) error
- type ListedObject
- type S3
- func (b *S3) Close() error
- func (b *S3) Copy(ctx context.Context, dstKey, srcKey string) error
- func (b *S3) Delete(ctx context.Context, key string) error
- func (b *S3) DownloadFile(ctx context.Context, key, destPath string) error
- func (b *S3) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error
- func (b *S3) EnsureBucket(ctx context.Context) error
- func (b *S3) List(ctx context.Context, prefix string) ([]ListedObject, error)
- func (b *S3) ListPages(ctx context.Context, prefix string, pageSize int, ...) error
- func (b *S3) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
- func (b *S3) SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)
- func (b *S3) Stat(ctx context.Context, key string) (int64, error)
- func (b *S3) StatFingerprint(ctx context.Context, key string) (int64, string, error)
- func (b *S3) UploadFile(ctx context.Context, key, path string) error
- func (b *S3) VersioningEnabled(ctx context.Context) (bool, error)
Constants ¶
const CoverHandoffTimeout = 10 * time.Second
CoverHandoffTimeout bounds the floor-generation bump a caller runs after a restore (Codex round 69): the bump must COMMIT while the last cover row still lives, and after a broken tail refresh that row's remaining lifetime can be as short as minTTL − coverLapseBudget ≈ 16s (the loss is reported at covered+budget; the row lives to covered+TTL). Ten seconds fits inside that worst case with slack, and trivially inside the fresh full TTL a successful tail refresh leaves.
const EnvExpectedBucketOwner = "OPENSTORAGE_S3_EXPECTED_BUCKET_OWNER"
EnvExpectedBucketOwner optionally pins every S3 request to a bucket owner (issue #58): when set to a 12-digit AWS account ID, the client middleware sends the bucket-owner condition on EVERY operation, so a bucket whose global name was re-registered by another account yields 403 Access Denied instead of silently serving/accepting private packs. Read here (not in cmd config) so every process that opens the store — serve, git hooks (inheriting the serve environment), and the purge CLI — enforces it uniformly; a probe-only or serve-only check would leave the rest of the data plane owner-blind. Empty (the default) attaches no middleware and changes no behavior.
const LFSRoot = "lfs/"
LFSRoot is the listing prefix covering every repo's LFS objects (the orphan sweeper scans it like PacksRoot, issue #51). LFS keys always use the BASE RepoID (ADR-35): namespaces share one object space, so there is no "+suffix" variant to cover.
const PacksRoot = "packs/"
PacksRoot is the listing prefix covering every repo's packs (the orphan-pack sweeper scans it in one listing instead of enumerating repos). Namespaced repos (`<repo>+import`/`<repo>+ephemeral`) keep fully separate pack spaces keyed by the full name, so each key's repo segment identifies its authoritative pack registry row set.
Variables ¶
var ErrCapExceeded = errors.New("blob: stream exceeds the reserved cap")
ErrCapExceeded reports a capped download whose stream exceeded the caller's reserved size (Codex round 117 on #104): on a mutable legacy key, the object can be replaced by a larger version between the caller's Stat and the GET, and streaming past the reservation would consume unbudgeted replica disk.
var ErrCoverContinuityLost = errors.New("cover continuity lost")
ErrCoverContinuityLost marks work canceled by the cover-continuity machinery (Codex security round 107 on #104): callers must be able to distinguish a copy that erred BECAUSE its cover lapsed — whose committed destination is an unsafe, unaccounted commit exactly like a renewErr verdict — from ordinary op failures, because WithRenewedCover reports a canceled op through opErr with renewErr nil.
var ErrNotFound = errors.New("blob: object not found")
ErrNotFound is returned (wrapped, match with errors.Is) by Stat and DownloadFile for missing objects, so callers can distinguish absence from transport failures without depending on gocloud error codes.
var ErrSignedURLUnsupported = errors.New("blob: signed URLs unsupported by this backend")
ErrSignedURLUnsupported is returned by SignedURL on backends without presign support (fileblob). Callers (the LFS batch handler) fall back to proxying the transfer through the server.
Functions ¶
func EnsurePublished ¶
func EnsurePublished(ctx context.Context, b BlobStore, key, path string, onRestore func(context.Context, int64) error, onRestored func(context.Context) error) (restored bool, err error)
EnsurePublished re-uploads the local file at path to key when key is absent. It is the writer half of the orphan-sweep coordination (issue #5), called strictly AFTER a successful RegisterPacks: registration follows upload, so an orphan sweeper that deleted the freshly uploaded key just before the registration committed would otherwise leave a registered pack with no bytes. Ordering proof: the sweeper deletes (D) then re-reads the registry (C); the writer registers (R) then verifies here (V). If C observes R the sweeper restores the pack itself; if C ran before R then D happened before V, so V observes the missing key and re-uploads. Both writers produce identical bytes (pack keys are content-addressed), so the overlap is harmless either way.
The verification runs on a bounded context detached from the caller's cancellation: the registration it covers has already committed, so a client hanging up right after the commit must not skip the check — a skipped verification could leave the persistent pack row pointing at a blob a concurrent sweep just deleted (Codex round 14).
A restore re-materializes bytes the usage counter already includes (Codex round 58): a quota floor scanned during the absence subtracted the full counter from a listing missing them, hiding equally sized rowless legacy bytes. onRestore (nil-ok) runs BEFORE the repair upload so the caller can account the returning bytes first (a cover reservation — see the LFS repair in githttp); an onRestore error aborts the restore, leaving the registered-row-without-bytes state the next retry heals. The returned restored flag tells the caller to invalidate cached floors (bump the shared floor generation) after the bytes are back. onRestored (nil-ok) is the invalidation HANDOFF (Codex round 70): it runs under a live, freshly proven cover and is retried — alternating with cover renewals — until it commits or continuity is lost, so a transient ledger failure cannot strand durable restored bytes with neither cover nor invalidation.
func LFSKey ¶
LFSKey is the immutable object key for a repo's LFS object with the given sha256 oid. Keys are repo-scoped (docs/explanation/git-protocol-and-namespaces.md): forks inherit LFS pointers only, never the objects, which this keying yields automatically.
LFSKey is the CANONICAL object key: bytes here are written only by the server's digest-verified paths (verified proxy uploads, adoption copies, registered-object restores) — never by presigned PUTs. The `o2/` segment makes the spelling unreachable by any v0.1.x presigned upload URL, which signed exactly the bare LFSLegacyKey spelling (Codex security round 78 on issue #51): a still-valid legacy URL can overwrite ONLY the legacy key, so registered content — always canonical — is immune to replay for the URL's whole lifetime, with no quiesce-window reasoning. The oid is caller-validated 64-hex (lfsOIDPattern / isLFSOID); the assert rules out path tricks reaching the canonical namespace.
func LFSLegacyKey ¶
LFSLegacyKey is the v0.1.x object spelling — the only one old presigned PUT URLs can address. It is never registered and never served for registered objects: it exists as the adoption SOURCE (rowless pre-#51 blobs live here until adopted, then the source is deleted) and as the read-compat location for not-yet-adopted rowless objects.
func PackPrefix ¶
PackPrefix is the listing prefix for a repo's packs.
func QuotaRepoGroup ¶
QuotaRepoGroup returns repo's quota accounting group: the BASE repository first (suffix stripped), then every docs/explanation/git-protocol-and-namespaces.md namespace that rolls up under it. The ref-store quota ledger (issue #29) keys reservations under group[0] and sums usage over the whole group, mirroring StorageUsageBytes' rollup so both usage sources measure the same unit.
func StorageUsageBytes ¶
StorageUsageBytes sums the sizes of every object under the pack prefixes of repo's BASE repository and all of its namespaces PLUS the repo's LFS prefix (issue #51): registered packs and LFS objects, not-yet-swept orphans, AND retained superseded blobs — exactly the durable growth the per-repo storage quota (issue #21) bounds. The suffix, if any, is stripped first, so every namespace write is measured against the same rolled-up usage. Shared by the pre-receive hook, the REST pack writers, the LFS batch/transfer gates, and the git surface's pressure check.
Since issue #29 this listing is the admission check's usage FLOOR rather than its sole source: the ref-store ledger (ref.ReservePackBytes) charges max(counter, this listing) plus in-flight reservations, closing the concurrent-push overshoot this listing alone admitted. Retained superseded bytes (issue #31) are counted by BOTH sources — here as lingering prefix objects, there as pack_retained rows — until a sweep reclaims them. LFS bytes are likewise counted by both once uploads settle (ref.RegisterLFSObject); before #51's ledger existed, only this listing could see them, which is one more reason the floor stays.
func WithRenewedCover ¶
func WithRenewedCover(ctx context.Context, renew func(context.Context) error, op func(context.Context) error, handoff func(context.Context) error) (opErr, renewErr error)
WithRenewedCover runs op while keeping a restore's cover reservation alive (Codex round 59): it takes the initial cover itself (round 64), renews it every coverRenewInterval — each attempt deadline-bounded and continuity-checked — for op's duration, and after op succeeds it alternates fresh cover refreshes with bounded attempts of the caller's handoff (the floor invalidation) until the handoff COMMITS under a live cover or continuity is lost (Codex round 70 — a single best-effort handoff could fail and strand durable bytes with neither cover nor invalidation). Renewal failures are tolerated while a recent success still covers the bytes; once coverLapseBudget passes without one, op's context is canceled — the restore aborts rather than publishing uncovered bytes (Codex round 62). opErr reports op's own failure; renewErr reports a handoff that never landed.
Types ¶
type BlobStore ¶
type BlobStore interface {
// UploadFile streams a local file to key.
UploadFile(ctx context.Context, key, path string) error
// DownloadFile fetches key into destPath atomically (tmp + rename), so a
// partially downloaded object is never visible at destPath. Missing keys
// yield an error matching ErrNotFound via errors.Is (the hydration
// missing-pack retry keys on it).
DownloadFile(ctx context.Context, key, destPath string) error
// DownloadFileCapped is DownloadFile bounded to capBytes of local
// disk (Codex round 117 on #104): one GET, atomic rename — but a
// stream exceeding capBytes aborts with ErrCapExceeded instead of
// consuming unreserved space (the caller reserved capBytes BEFORE a
// window in which a mutable-key object can be replaced by a larger
// version).
DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error
// List returns all objects under prefix.
List(ctx context.Context, prefix string) ([]ListedObject, error)
// ListPages streams objects under prefix in lexicographic key order,
// invoking fn once per page of at most pageSize objects, so callers
// scanning large prefixes (the orphan-pack sweeper) hold one page at a
// time instead of the whole listing. fn must not retain the slice; a
// non-nil fn error aborts the scan and is returned unwrapped.
ListPages(ctx context.Context, prefix string, pageSize int, fn func([]ListedObject) error) error
// Delete removes the object at key; missing objects yield an error
// matching ErrNotFound via errors.Is. Objects are immutable, so Delete
// exists solely for the orphan-pack sweeper (issue #5): reaping blobs
// that were uploaded but never registered because of a crash.
Delete(ctx context.Context, key string) error
// Copy duplicates srcKey to dstKey inside the store (server-side S3
// CopyObject / local file copy — never staged through local disk, so
// pack-sized copies work regardless of temp-dir capacity). A missing
// srcKey yields an error matching ErrNotFound via errors.Is. Used by
// the orphan-pack sweeper's quarantine handshake.
Copy(ctx context.Context, dstKey, srcKey string) error
// SignedURL returns a presigned URL for the given HTTP method
// (http.MethodGet or http.MethodPut) on key, valid for expiry.
// Backends without presign support return ErrSignedURLUnsupported.
SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)
// Stat returns the size of the object at key; missing objects yield an
// error matching ErrNotFound via errors.Is.
Stat(ctx context.Context, key string) (int64, error)
// StatFingerprint returns the size plus an opaque content-version
// fingerprint for the object at key: any overwrite changes it (S3
// ETag — content-derived, so an attacker cannot preserve it across a
// same-length rewrite; file backends fall back to mtime+size).
// Verification flows that hash an object across multiple reads use
// fingerprint equality to prove the bytes were not REPLACED between
// reads (issue #51, Codex round 77): ranged reads spanning a swap
// can splice old-prefix+new-suffix into a digest that matches even
// though the stored bytes differ. Missing objects yield an error
// matching ErrNotFound via errors.Is.
StatFingerprint(ctx context.Context, key string) (int64, string, error)
// ReadRange returns exactly the length bytes at offset from the object
// at key — a ranged point read, never the whole object (repack fold
// selection reads exact object counts from 12-byte pack headers with
// it). Missing keys yield an error matching ErrNotFound via errors.Is;
// an object shorter than offset+length is a non-ErrNotFound error.
ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
// EnsureBucket idempotently creates the backing bucket.
EnsureBucket(ctx context.Context) error
Close() error
}
BlobStore is the pack-object storage contract. Keys are opaque to callers except through PackKey/PackPrefix. Objects are immutable once written.
func Open ¶
Open dispatches on the URL scheme to the matching BlobStore backend:
s3://bucket?region=&endpoint= → OpenS3 file:///abs/dir → local filesystem bucket
BlobStore already carries the lifecycle (EnsureBucket/Close), so there is no separate lifecycle interface. Open never calls EnsureBucket: hooks reconstruct stores from env URLs; the serve seam owns bucket creation.
type File ¶
type File struct {
// contains filtered or unexported fields
}
File implements BlobStore on a local directory via gocloud fileblob.
func OpenFile ¶
OpenFile opens a file:// blob URL, e.g. file:///abs/dir (empty host, so the directory is always absolute). The root directory is created up front — hooks reconstruct stores from env URLs without calling EnsureBucket, and a missing directory must not fail them. MetadataDontWrite keeps listings clean: no .attrs sidecars next to pack files.
func (*File) DownloadFile ¶
DownloadFile fetches key into destPath atomically (tmp + rename).
func (*File) DownloadFileCapped ¶
DownloadFileCapped fetches key into destPath atomically, aborting with ErrCapExceeded once the stream would exceed capBytes on disk.
func (*File) EnsureBucket ¶
EnsureBucket idempotently creates the backing directory (the file:// analogue of S3 CreateBucket at serve startup).
func (*File) ListPages ¶
func (b *File) ListPages(ctx context.Context, prefix string, pageSize int, fn func([]ListedObject) error) error
ListPages streams objects under prefix in key order, one page at a time.
func (*File) ReadRange ¶
ReadRange returns exactly the length bytes at offset from the object at key (a seeked local read; ErrNotFound when absent).
func (*File) SignedURL ¶
func (b *File) SignedURL(_ context.Context, key, method string, expiry time.Duration) (string, error)
SignedURL is unsupported on the filesystem backend: it returns ErrSignedURLUnsupported so callers fall back to proxying transfers through the server (docs/explanation/git-protocol-and-namespaces.md local mode).
func (*File) StatFingerprint ¶
StatFingerprint — see BlobStore.
type ListedObject ¶
type ListedObject struct {
Key string
Size int64
// ModTime is the backend's last-modified time for the object (S3
// LastModified / file mtime). The orphan-pack sweeper uses it as the
// upload age: objects are immutable, so it never moves after publish.
ModTime time.Time
}
ListedObject is one blob under a prefix.
type S3 ¶
type S3 struct {
// contains filtered or unexported fields
}
S3 implements BlobStore on a gocloud S3 bucket plus the raw client (needed for EnsureBucket at serve startup).
func OpenS3 ¶
OpenS3 opens a blob URL, e.g. s3://spike?endpoint=http://127.0.0.1:9000&s3ForcePathStyle=true®ion=us-east-1 Credentials come from the standard AWS env vars, and the optional expected-bucket-owner enforcement comes from EnvExpectedBucketOwner (a malformed value is boot-fatal). The URL is parsed by hand (bucket = host, endpoint/region = query) so we control path-style behavior deterministically against MinIO regardless of gocloud URL-param renames.
func (*S3) Copy ¶
Copy duplicates srcKey to dstKey: server-side CopyObject for objects up to the single-request limit, multipart UploadPartCopy beyond it (still fully server-side — bytes never flow through this process).
func (*S3) DownloadFile ¶
DownloadFile fetches key into destPath atomically (tmp + rename).
func (*S3) DownloadFileCapped ¶
DownloadFileCapped fetches key into destPath atomically, aborting with ErrCapExceeded once the stream would exceed capBytes on disk.
func (*S3) EnsureBucket ¶
EnsureBucket creates the bucket, tolerating "already exists/owned by you" (so `serve` works against a fresh MinIO). A least-privilege deployment role for a pre-provisioned bucket has no s3:CreateBucket, so AccessDenied on create is not proof the bucket is missing: probe it with HeadBucket and accept when reachable. Any other create error, or a denied create with an unreachable bucket, stays boot-fatal. With an expected bucket owner configured (issue #58), EVERY accepted outcome additionally passes an owner-conditioned HeadBucket probe, so squatted names and credential/account drift fail boot instead of 403ing data operations.
func (*S3) ListPages ¶
func (b *S3) ListPages(ctx context.Context, prefix string, pageSize int, fn func([]ListedObject) error) error
ListPages streams objects under prefix in key order, one page at a time.
func (*S3) ReadRange ¶
ReadRange returns exactly the length bytes at offset from the object at key (a ranged GET; ErrNotFound when absent).
func (*S3) SignedURL ¶
func (b *S3) SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)
SignedURL returns a presigned S3 URL for method (GET or PUT) on key. ContentType is deliberately left unset in the options: gocloud's aws-sdk-go-v2 presigner returns Unimplemented when asked to sign a Content-Type for PUT, and an unsigned Content-Type means git-lfs's `Content-Type: application/octet-stream` PUT verifies against MinIO/S3 SigV4 without extra configuration.
With expected-bucket-owner enforcement on, presigning runs through the same client middleware as every other operation and the presigner hoists the x-amz-* header into a SIGNED lowercase query parameter (x-amz-expected-bucket-owner=...), the spelling S3 documents and enforces on presigned requests (aws-sdk-go-v2 #2508). The URL is self-contained: redeemers send no extra header, and stripping or altering the parameter breaks the SigV4 signature.
func (*S3) StatFingerprint ¶
StatFingerprint — see BlobStore.
func (*S3) UploadFile ¶
UploadFile streams a local file to key.
func (*S3) VersioningEnabled ¶
VersioningEnabled reports whether the backing bucket has ever had S3 versioning turned on (status Enabled OR Suspended — suspension keeps all previously stored versions, and unversioned deletes still only hide the current version behind a delete marker). The orphan-pack sweeper's delete mode refuses such buckets: its deletes would reclaim nothing while reporting that they did.