Help us shape the future of Go! Take the 2026 Go Developer Survey

blob

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package blob is the content-addressed binary store behind every repository (decision record 0004). Blobs are keyed by their lowercase hex SHA-256; the SHA-512, SHA-1 and MD5 that package clients also use are computed in the same pass during the upload and returned by Commit so the caller can record them in the item index, which is where checksum deploy, checksum search and checksum files read them.

Uploads are sessions rather than single calls because two clients need to append across HTTP requests: Docker chunked uploads (PATCH then PUT) and the JFrog CLI's multipart upload. A session is identified by an opaque ID that ResumeUpload accepts in a later request. One writer per session is assumed; the HTTP layer serialises requests for the same upload.

Backends: blob/fs (local disk), blob/s3 (S3-compatible object storage), blob/gcs (Google Cloud Storage) and blob/azure (Azure Blob Storage), plus blob/mirror, one store over several of them. All are held to the same behaviour by blob/blobtest.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotFound reports a blob that is not stored.
	ErrNotFound = errors.New("blob: not found")
	// ErrDigestMismatch reports content whose digest differs from the one the
	// client declared. The upload is aborted; nothing is committed.
	ErrDigestMismatch = errors.New("blob: digest mismatch")
	// ErrUploadUnknown reports an upload ID that does not name a live session:
	// never issued, already committed or aborted, or malformed.
	ErrUploadUnknown = errors.New("blob: unknown upload")
	// ErrInvalidDigest reports a digest that is not lowercase hex of the right
	// length for its algorithm.
	ErrInvalidDigest = errors.New("blob: invalid digest")
	// ErrCannotSign reports a SignURL for a route the store does not sign:
	// Signs(via) is false.
	ErrCannotSign = errors.New("blob: store cannot sign this route")
)

Sentinel errors. Backends wrap them, so test with errors.Is.

View Source
var ErrHasherState = errors.New("blob: invalid hasher state")

ErrHasherState reports a saved hasher state that cannot be restored.

Functions

func CopyHashed

func CopyHashed(w io.Writer, r io.Reader, h *Hasher) (int64, error)

CopyHashed copies r to w in chunks, feeding h only with bytes w accepted, so a short or failed write leaves the hash state matching what is stored. It returns the number of bytes stored.

func Disposition

func Disposition(filename string) string

Disposition is the Content-Disposition naming a downloaded file, in the form Artifactory sends: a quoted name for old clients and an RFC 6266 UTF-8 name for the rest. A streamed download and a signed URL's override use the same value.

func NewUploadID

func NewUploadID() (string, error)

NewUploadID returns a random 32-character hex upload ID. IDs are unguessable because a session accepts appends from whoever presents it.

func ValidUploadID

func ValidUploadID(id string) bool

ValidUploadID reports whether id has the shape NewUploadID produces. Backends use upload IDs in file and object names.

func ValidateDigests

func ValidateDigests(d Digests) error

ValidateDigests checks every non-empty field of d.

func ValidateSHA256

func ValidateSHA256(s string) error

ValidateSHA256 rejects anything but 64 lowercase hex characters. Backends build file and object names from the digest, so this is also the guard against path traversal.

func Verify

func Verify(want, got Digests) error

Verify compares every non-empty field of want with got.

Types

type Digests

type Digests struct {
	SHA256 string
	SHA512 string
	SHA1   string
	MD5    string
}

Digests are lowercase hex checksums. In Commit's want argument an empty field means "not declared"; in a returned Info an empty SHA512, SHA1 or MD5 means the backend does not keep it (the item index does).

type Hasher

type Hasher struct {
	// contains filtered or unexported fields
}

Hasher computes all four digests in one pass and can save and restore its state, so a resumed upload continues hashing instead of re-reading data.

func NewHasher

func NewHasher() *Hasher

NewHasher returns a Hasher at the start of the stream.

func (*Hasher) MarshalBinary

func (h *Hasher) MarshalBinary() ([]byte, error)

MarshalBinary saves the state of all four digests.

func (*Hasher) Sum

func (h *Hasher) Sum() Digests

Sum returns the digests of everything written so far.

func (*Hasher) UnmarshalBinary

func (h *Hasher) UnmarshalBinary(b []byte) error

UnmarshalBinary restores a state saved by MarshalBinary.

func (*Hasher) Write

func (h *Hasher) Write(p []byte) (int, error)

Write feeds p to every digest. It never fails.

type Info

type Info struct {
	Size    int64
	Digests Digests
}

Info describes a stored blob.

type PartSizer

type PartSizer interface {
	PartSize() int64
}

PartSizer is implemented by backends that write an upload in parts of a fixed size. Bytes short of a whole part stay pending in the upload's session, which is saved after every Append, so a caller that appends in multiples of PartSize keeps that saved state small. The mirror sizes its chunks by it.

type SignOptions

type SignOptions struct {
	// Via is the route.
	Via Via
	// TTL is how long the URL stays valid once handed out.
	TTL time.Duration
	// Filename and ContentType, when set, travel as response overrides, so
	// the object store answers with the item's name and media type rather
	// than the blob's: the 302 that points at the URL describes no body.
	Filename    string
	ContentType string
}

SignOptions describe one signed download URL.

type Signer

type Signer interface {
	Signs(via Via) bool
	SignURL(ctx context.Context, sha256 string, o SignOptions) (string, error)
}

Signer is implemented by backends that can hand a client a time-limited URL for a blob, used when a repository sets downloadRedirect or cdnRedirect. Signs is fixed when the store is built, from its configuration and credentials, so a caller knows the route is available before it tries; SignURL returns ErrCannotSign for a route Signs refuses.

type Store

type Store interface {
	// BeginUpload starts an upload session.
	BeginUpload(ctx context.Context, opts ...UploadOption) (Upload, error)
	// ResumeUpload reopens a session started earlier, possibly by another
	// request. It returns ErrUploadUnknown for an ID that is not live.
	ResumeUpload(ctx context.Context, id string) (Upload, error)
	// Open returns the content for random access, so HTTP handlers can serve
	// ranges with http.ServeContent. The caller closes it. Info carries Size
	// and Digests.SHA256; SHA512, SHA1 and MD5 may be empty.
	Open(ctx context.Context, sha256 string) (io.ReadSeekCloser, Info, error)
	// Stat returns a blob's Info without opening it.
	Stat(ctx context.Context, sha256 string) (Info, error)
	// Exists reports whether a blob is stored.
	Exists(ctx context.Context, sha256 string) (bool, error)
	// Delete removes a blob. Deleting a missing blob succeeds, because garbage
	// collection and a concurrent delete can race and both mean "gone".
	Delete(ctx context.Context, sha256 string) error
	// Walk calls fn for every stored blob in an unspecified order and stops at
	// the first error fn returns, which Walk returns. Upload sessions are not
	// visited. It exists for garbage collection and storage accounting.
	Walk(ctx context.Context, fn func(Info) error) error
}

Store is the content-addressed blob store.

type Upload

type Upload interface {
	// ID names the session for ResumeUpload.
	ID() string
	// Size is the number of bytes appended so far.
	Size() int64
	// Append writes r to the end of the upload and returns how many bytes it
	// stored. On error the bytes counted are stored and hashed, so the session
	// stays consistent and the client may continue from Size.
	Append(ctx context.Context, r io.Reader) (int64, error)
	// Commit finishes the upload. Every non-empty field of want is compared
	// with the computed digest; a mismatch aborts the session and returns
	// ErrDigestMismatch. Content that is already stored is not written again
	// and the session is discarded, so committing the same bytes twice, or
	// from two sessions at once, succeeds both times.
	Commit(ctx context.Context, want Digests) (Info, error)
	// Abort discards the session. Aborting a session that is already gone
	// succeeds.
	Abort(ctx context.Context) error
}

Upload is one upload session.

type UploadOption

type UploadOption func(*UploadOptions)

UploadOption configures BeginUpload.

func WithExpectedSHA256

func WithExpectedSHA256(sha256 string) UploadOption

WithExpectedSHA256 declares the content digest up front. Commit still verifies it.

type UploadOptions

type UploadOptions struct {
	// ExpectedSHA256 is the digest the client declared before sending the
	// body (Docker's monolithic PUT ?digest=, X-Checksum-Sha256). Backends
	// that cannot rename use it to write straight to the final key.
	ExpectedSHA256 string
}

UploadOptions is the resolved form of the options, exported for backends.

func ResolveOptions

func ResolveOptions(opts ...UploadOption) (UploadOptions, error)

ResolveOptions applies opts and validates them.

type Via

type Via int

Via is the route a signed download URL takes (decision 0029).

const (
	// ViaStorage is a URL on the object store itself.
	ViaStorage Via = iota
	// ViaCDN is a URL on a CDN in front of the object store.
	ViaCDN
)

func (Via) String

func (v Via) String() string

String names the route for logs.

Directories

Path Synopsis
Package azure is the Azure Blob Storage blob store (decision 0027).
Package azure is the Azure Blob Storage blob store (decision 0027).
Package blobtest is the contract every blob.Store backend must satisfy.
Package blobtest is the contract every blob.Store backend must satisfy.
Package fs is the local-disk blob store.
Package fs is the local-disk blob store.
Package gcs is the Google Cloud Storage blob store (decision 0027).
Package gcs is the Google Cloud Storage blob store (decision 0027).
Package mirror is a blob store that keeps each blob on several member stores at once (decision 0028): different clouds, regions or accounts, any mix of the fs, s3, gcs and azure backends.
Package mirror is a blob store that keeps each blob on several member stores at once (decision 0028): different clouds, regions or accounts, any mix of the fs, s3, gcs and azure backends.
Package s3 is the S3-compatible blob store (AWS S3, MinIO, Ceph RGW and similar).
Package s3 is the S3-compatible blob store (AWS S3, MinIO, Ceph RGW and similar).
Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027).
Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL