blob

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package blob is the content-addressed binary store behind every repository (decision record 0004). Blobs are keyed by their lowercase hex SHA-256; the SHA-512, SHA-1 and MD5 that package clients also use are computed in the same pass during the upload and returned by Commit so the caller can record them in the item index, which is where checksum deploy, checksum search and checksum files read them.

Uploads are sessions rather than single calls because two clients need to append across HTTP requests: Docker chunked uploads (PATCH then PUT) and the JFrog CLI's multipart upload. A session is identified by an opaque ID that ResumeUpload accepts in a later request. One writer per session is assumed; the HTTP layer serialises requests for the same upload.

Backends: blob/fs (local disk), blob/s3 (S3-compatible object storage), blob/gcs (Google Cloud Storage) and blob/azure (Azure Blob Storage), plus blob/mirror, one store over several of them. All are held to the same behaviour by blob/blobtest.

Role

platform layer: the content-addressed blob port. No module owns blob storage, so the port and the hashing and digest checks that every backend shares sit in the platform, with the backends below it. In the artifacts module the item, registry, multipart and build services, the format handlers and the HTTP servers write and serve content through Store, and garbage collection walks and deletes through it; the composition root opens a store for each configured tier. Blob, the row of the metadata store's blob index, is declared here as well, so that the mirror's repair worker, which reads it through its own port, imports no store. It imports the kernel and the standard library.

Inputs

  • The bytes of an upload, appended to a session over one or more requests, and the digests a client declares: before the body through WithExpectedSHA256 (Docker's monolithic PUT ?digest=, X-Checksum-Sha256), and at Commit as want.
  • Upload IDs for ResumeUpload, and the SHA-256 that names a stored blob for Open, Stat, Exists, Delete and SignURL.
  • For a Signer, the route (Via) and SignOptions: how long the URL stays valid, and the file name and media type it carries as response overrides.

Outputs

  • Store and Upload, the contract every backend implements: begin, resume, append, commit and abort an upload; open a blob for ranged reads, stat, check, delete and walk. Info and Digests describe a blob.
  • Signer, Via (ViaStorage, ViaCDN), SignOptions and SignedDisposition: signed download URLs for repositories that set downloadRedirect or cdnRedirect, naming the file in the form 7.x's signed URLs ask for.
  • PartSizer, which backends that write fixed-size parts implement, so the mirror can size its chunks to every member's parts.
  • UploadOption, UploadOptions, WithExpectedSHA256 and ResolveOptions, for BeginUpload.
  • Hasher (NewHasher, Write, Sum, MarshalBinary, UnmarshalBinary), CopyHashed and Verify: the one-pass hashing and the digest check that every backend shares.
  • The validators ValidateSHA256, ValidateDigests and ValidUploadID, and NewUploadID.
  • The sentinels ErrNotFound, ErrDigestMismatch, ErrUploadUnknown, ErrInvalidDigest, ErrCannotSign and ErrHasherState. Backends wrap them, and callers test them with errors.Is.
  • Blob, the blob index row: a blob's checksums, size, creation time, reference count, and its State in garbage collection with the time it was marked or removed; BlobState with BlobLive, BlobDeleting and BlobDeleted. The metadata store keeps the rows, and the mirror's BlobIndex returns them.

Invariants

  • A blob is keyed by its lowercase hex SHA-256, and ValidateSHA256 accepts nothing else. Backends build file and object names from that digest and from upload IDs, so the validators are also the guard against path traversal.
  • SHA-512, SHA-1 and MD5 come from the same pass as the SHA-256, and Commit returns all four. CopyHashed feeds the Hasher only the bytes a backend stored, so after a short or failed write the session still matches its content and the client continues from Size.
  • One writer per upload session: the HTTP layer serialises a session's requests. A session accepts appends from whoever presents its ID, so NewUploadID draws IDs from crypto/rand.
  • Committing content that is already stored succeeds and discards the session, so the same bytes committed twice, or from two sessions at once, end as one blob.
  • The hasher state is persisted: fs keeps it in each session's state file and the object-store backends in their saved sessions, and UnmarshalBinary reads it back in a later request or process. Its encoding (the SHA-256, SHA-512, SHA-1 and MD5 states, in that order, each behind a two-byte big-endian length) must not change, and TestUnitHasherStateKnownAnswer pins it. A change still compiles, and a round trip within one build cannot see it. At run time the sessions saved before it would no longer restore: fs rehashes their data and carries on, while the object-store backends report ErrUploadUnknown and the client starts the upload again.
  • Error texts reach clients, for example in the JFrog CLI's multipart upload status, so they stay byte-identical.

References

  • Decision 0004: docs/research/decisions/0004-storage-sql-metadata-and-blobs.md ("Blobs": content addressing, the checksums of one pass and the upload sessions).
  • Decision 0010: docs/research/decisions/0010-retention-trash-and-gc.md (the blob row's states while garbage collection deletes a blob).
  • Decision 0027: docs/research/decisions/0027-cloud-blob-backends.md (the Google Cloud Storage and Azure backends behind this contract).
  • Decision 0028: docs/research/decisions/0028-blob-mirroring.md (one store over several; its chunks follow every member's part size).
  • Decision 0029: docs/research/decisions/0029-cdn-delivery-and-download-redirect.md ("Signing contract": Signer, Via and SignOptions).
  • Decision 0024: docs/research/decisions/0024-cold-storage-tier.md (the cold tier, on the same backends).
  • Research 18 and 19: docs/research/18-cloud-blob-storage.md and docs/research/19-cdn-and-download-redirect.md.
  • Decision 0044: docs/research/decisions/0044-modular-monolith-migration.md, "Dependency rules": the platform imports no module.
  • Target layout: docs/research/architecture-options/5-modular-monolith.md, lines 146 to 151 (platform/blob and its subpackages) and line 1049 (the mapping row).
  • Plan: docs/implementation/plan.md#p2-stores, task 3, #p2b-cloud-blob-storage-and-mirroring, #p4b-reference-deployments-and-cdn-delivery, #p7-npm-go-maven-locals (prework 1) and #p9d-modular-monolith-migration, epic PLT (https://github.com/weaveplatform/artifactweave/issues/938).
  • https://github.com/weaveplatform/artifactweave/issues/980
  • https://github.com/weaveplatform/artifactweave/issues/986

Status

Implemented P2: the store contract, upload sessions with the saved hasher state, the validators and the sentinels, and Blob, the blob index row. P2b added PartSizer, by which the mirror sizes its chunks (issue 39). P3 gave the blob row its garbage-collection states (issue 64). P4b replaced the presigner with the signing capability of download and CDN redirects: Signer, Via, SignOptions, Disposition and ErrCannotSign (issue 119). P7 made SHA-512 a digest of every upload (issue 296). Moved here in P9d. Disposition gave way to SignedDisposition, the form 7.x's signed URLs ask for (issue 1577).

Testing

  • TestUnitHasherSaveRestoreContinues: a state saved part way and restored gives the digests of one pass over the whole stream.
  • TestUnitHasherStateKnownAnswer: the state saved after a fixed input has the framing above and a fixed SHA-256, and once restored it continues to the digests shasum and md5 give for the whole input.
  • TestUnitHasherRejectsBadState: an empty, truncated, overlong or corrupt state fails with ErrHasherState and leaves the hasher as it was.
  • TestUnitVerify, TestUnitValidate and TestUnitResolveOptions: the digest check, the validators and the options, with their sentinels; an upload ID with path characters is invalid.
  • TestUnitCopyHashedKeepsHashInStepWithWrites: after a short write, a write error or a read error, the hash covers exactly the stored bytes.
  • TestUnitNewUploadID: an ID has the shape ValidUploadID accepts, and a failing random source is an error.

Every backend runs the blobtest contract: fs, s3, gcs, azure and mirror. The metadata store's blob index contract runs on its in-memory, SQLite and PostgreSQL stores.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotFound reports a blob that is not stored.
	ErrNotFound = errors.New("blob: not found")
	// ErrDigestMismatch reports content whose digest differs from the one the
	// client declared. The upload is aborted; nothing is committed.
	ErrDigestMismatch = errors.New("blob: digest mismatch")
	// ErrUploadUnknown reports an upload ID that does not name a live session:
	// never issued, already committed or aborted, or malformed.
	ErrUploadUnknown = errors.New("blob: unknown upload")
	// ErrInvalidDigest reports a digest that is not lowercase hex of the right
	// length for its algorithm.
	ErrInvalidDigest = errors.New("blob: invalid digest")
	// ErrCannotSign reports a SignURL for a route the store does not sign:
	// Signs(via) is false.
	ErrCannotSign = errors.New("blob: store cannot sign this route")
)

Sentinel errors. Backends wrap them, so test with errors.Is.

View Source
var ErrHasherState = errors.New("blob: invalid hasher state")

ErrHasherState reports a saved hasher state that cannot be restored.

Functions

func CopyHashed

func CopyHashed(w io.Writer, r io.Reader, h *Hasher) (int64, error)

CopyHashed copies r to w in chunks, feeding h only with bytes w accepted, so a short or failed write leaves the hash state matching what is stored. It returns the number of bytes stored.

func NewUploadID

func NewUploadID() (string, error)

NewUploadID returns a random 32-character hex upload ID. IDs are unguessable because a session accepts appends from whoever presents it.

func SignedDisposition

func SignedDisposition(filename string) string

SignedDisposition is the Content-Disposition a signed download URL asks the object store or CDN to answer with, for every backend and route: attachment;filename="<name>", with no space after the semicolon and no filename* part, the name as it is. Every 7.x signed URL in a capture asks for that form, on CloudFront (RTFACT-23307 on 7.4.3, sbt/sbt#6614) and on S3 (RTFACT-24153, adoptium/adoptium-support#923 on 7.71.1), and a store answered a '+' in the name raw (adoptium/installer#766); a streamed download names the file in its own form. No capture shows a name outside printable ASCII, and the signer is not in 7.x's OSS sources; such a name is written as filename* alone, as 7.161.26's RequestResponseHelper.setHeaderForNonAscii names it on a streamed download, so that the override every store echoes stays ASCII [I].

func ValidUploadID

func ValidUploadID(id string) bool

ValidUploadID reports whether id has the shape NewUploadID produces. Backends use upload IDs in file and object names.

func ValidateDigests

func ValidateDigests(d Digests) error

ValidateDigests checks every non-empty field of d.

func ValidateSHA256

func ValidateSHA256(s string) error

ValidateSHA256 rejects anything but 64 lowercase hex characters. Backends build file and object names from the digest, so this is also the guard against path traversal.

func Verify

func Verify(want, got Digests) error

Verify compares every non-empty field of want with got.

Types

type Blob

type Blob struct {
	Checksums kernel.Checksums
	Size      int64
	Created   time.Time
	Refs      int64
	// State is where the row stands in garbage collection (decision 0010);
	// only live rows serve content.
	State BlobState
	// Deleted is when garbage collection marked the content for deletion
	// (deleting) or removed it (deleted). A row revived by a deploy keeps the
	// time until it is pruned, so the deploy knows the content it found may
	// have been removed after it was written and checks it is still there.
	Deleted time.Time
}

Blob is the index record of one stored content blob. Items reference blobs by SHA256; Refs counts those references (and those of trash entries and bundle artifacts) so garbage collection removes a blob only when nothing can serve it again. SHA512, SHA1 and MD5 are recorded so checksum deploy, the checksum searches and checksum files resolve without reading content.

type BlobState

type BlobState string

BlobState is a blob index row's stage in garbage collection. A row is marked deleting before its content is removed and becomes deleted (a tombstone) once it has been, so a deploy of the same content can never reference content that is being removed (decision 0010).

const (
	BlobLive     BlobState = "live"
	BlobDeleting BlobState = "deleting"
	BlobDeleted  BlobState = "deleted"
)

Blob index row states.

type Digests

type Digests struct {
	SHA256 string
	SHA512 string
	SHA1   string
	MD5    string
}

Digests are lowercase hex checksums. In Commit's want argument an empty field means "not declared"; in a returned Info an empty SHA512, SHA1 or MD5 means the backend does not keep it (the item index does).

type Hasher

type Hasher struct {
	// contains filtered or unexported fields
}

Hasher computes all four digests in one pass and can save and restore its state, so a resumed upload continues hashing instead of re-reading data.

func NewHasher

func NewHasher() *Hasher

NewHasher returns a Hasher at the start of the stream.

func (*Hasher) MarshalBinary

func (h *Hasher) MarshalBinary() ([]byte, error)

MarshalBinary saves the state of all four digests.

func (*Hasher) Sum

func (h *Hasher) Sum() Digests

Sum returns the digests of everything written so far.

func (*Hasher) UnmarshalBinary

func (h *Hasher) UnmarshalBinary(b []byte) error

UnmarshalBinary restores a state saved by MarshalBinary.

func (*Hasher) Write

func (h *Hasher) Write(p []byte) (int, error)

Write feeds p to every digest. It never fails.

type Info

type Info struct {
	Size    int64
	Digests Digests
}

Info describes a stored blob.

type PartSizer

type PartSizer interface {
	PartSize() int64
}

PartSizer is implemented by backends that write an upload in parts of a fixed size. Bytes short of a whole part stay pending in the upload's session, which is saved after every Append, so a caller that appends in multiples of PartSize keeps that saved state small. The mirror sizes its chunks by it.

type SignOptions

type SignOptions struct {
	// Via is the route.
	Via Via
	// TTL is how long the URL stays valid once handed out.
	TTL time.Duration
	// Filename and ContentType, when set, travel as response overrides, so
	// the object store answers with the item's name and media type rather
	// than the blob's: the 302 that points at the URL describes no body.
	Filename    string
	ContentType string
}

SignOptions describe one signed download URL.

type Signer

type Signer interface {
	Signs(via Via) bool
	SignURL(ctx context.Context, sha256 string, o SignOptions) (string, error)
}

Signer is implemented by backends that can hand a client a time-limited URL for a blob, used when a repository sets downloadRedirect or cdnRedirect. Signs is fixed when the store is built, from its configuration and credentials, so a caller knows the route is available before it tries; SignURL returns ErrCannotSign for a route Signs refuses.

type Store

type Store interface {
	// BeginUpload starts an upload session.
	BeginUpload(ctx context.Context, opts ...UploadOption) (Upload, error)
	// ResumeUpload reopens a session started earlier, possibly by another
	// request. It returns ErrUploadUnknown for an ID that is not live.
	ResumeUpload(ctx context.Context, id string) (Upload, error)
	// Open returns the content for random access, so HTTP handlers can serve
	// ranges with http.ServeContent. The caller closes it. Info carries Size
	// and Digests.SHA256; SHA512, SHA1 and MD5 may be empty.
	Open(ctx context.Context, sha256 string) (io.ReadSeekCloser, Info, error)
	// Stat returns a blob's Info without opening it.
	Stat(ctx context.Context, sha256 string) (Info, error)
	// Exists reports whether a blob is stored.
	Exists(ctx context.Context, sha256 string) (bool, error)
	// Delete removes a blob. Deleting a missing blob succeeds, because garbage
	// collection and a concurrent delete can race and both mean "gone".
	Delete(ctx context.Context, sha256 string) error
	// Walk calls fn for every stored blob in an unspecified order and stops at
	// the first error fn returns, which Walk returns. Upload sessions are not
	// visited. It exists for garbage collection and storage accounting.
	Walk(ctx context.Context, fn func(Info) error) error
}

Store is the content-addressed blob store.

type Upload

type Upload interface {
	// ID names the session for ResumeUpload.
	ID() string
	// Size is the number of bytes appended so far.
	Size() int64
	// Append writes r to the end of the upload and returns how many bytes it
	// stored. On error the bytes counted are stored and hashed, so the session
	// stays consistent and the client may continue from Size.
	Append(ctx context.Context, r io.Reader) (int64, error)
	// Commit finishes the upload. Every non-empty field of want is compared
	// with the computed digest; a mismatch aborts the session and returns
	// ErrDigestMismatch. Content that is already stored is not written again
	// and the session is discarded, so committing the same bytes twice, or
	// from two sessions at once, succeeds both times.
	Commit(ctx context.Context, want Digests) (Info, error)
	// Abort discards the session. Aborting a session that is already gone
	// succeeds.
	Abort(ctx context.Context) error
}

Upload is one upload session.

type UploadOption

type UploadOption func(*UploadOptions)

UploadOption configures BeginUpload.

func WithExpectedSHA256

func WithExpectedSHA256(sha256 string) UploadOption

WithExpectedSHA256 declares the content digest up front. Commit still verifies it.

type UploadOptions

type UploadOptions struct {
	// ExpectedSHA256 is the digest the client declared before sending the
	// body (Docker's monolithic PUT ?digest=, X-Checksum-Sha256). Backends
	// that cannot rename use it to write straight to the final key.
	ExpectedSHA256 string
}

UploadOptions is the resolved form of the options, exported for backends.

func ResolveOptions

func ResolveOptions(opts ...UploadOption) (UploadOptions, error)

ResolveOptions applies opts and validates them.

type Via

type Via int

Via is the route a signed download URL takes (decision 0029).

const (
	// ViaStorage is a URL on the object store itself.
	ViaStorage Via = iota
	// ViaCDN is a URL on a CDN in front of the object store.
	ViaCDN
)

func (Via) String

func (v Via) String() string

String names the route for logs.

Directories

Path Synopsis
Package azure is the Azure Blob Storage blob store (decision 0027).
Package azure is the Azure Blob Storage blob store (decision 0027).
Package blobtest is the contract every blob.Store backend must satisfy.
Package blobtest is the contract every blob.Store backend must satisfy.
Package cold will wrap the cold blob tier for archiving and stage the content that restore jobs bring back.
Package cold will wrap the cold blob tier for archiving and stage the content that restore jobs bring back.
Package fs is the local-disk blob store.
Package fs is the local-disk blob store.
Package gcs is the Google Cloud Storage blob store (decision 0027).
Package gcs is the Google Cloud Storage blob store (decision 0027).
Package mirror is a blob store that keeps each blob on several member stores at once (decision 0028): different clouds, regions or accounts, any mix of the fs, s3, gcs and azure backends.
Package mirror is a blob store that keeps each blob on several member stores at once (decision 0028): different clouds, regions or accounts, any mix of the fs, s3, gcs and azure backends.
Package s3 is the S3-compatible blob store (AWS S3, MinIO, Ceph RGW and similar).
Package s3 is the S3-compatible blob store (AWS S3, MinIO, Ceph RGW and similar).
Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027).
Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL