Documentation
¶
Overview ¶
Package blob is the content-addressed binary store behind every repository (decision record 0004). Blobs are keyed by their lowercase hex SHA-256; the SHA-512, SHA-1 and MD5 that package clients also use are computed in the same pass during the upload and returned by Commit so the caller can record them in the item index, which is where checksum deploy, checksum search and checksum files read them.
Uploads are sessions rather than single calls because two clients need to append across HTTP requests: Docker chunked uploads (PATCH then PUT) and the JFrog CLI's multipart upload. A session is identified by an opaque ID that ResumeUpload accepts in a later request. One writer per session is assumed; the HTTP layer serialises requests for the same upload.
Backends: blob/fs (local disk), blob/s3 (S3-compatible object storage), blob/gcs (Google Cloud Storage) and blob/azure (Azure Blob Storage), plus blob/mirror, one store over several of them. All are held to the same behaviour by blob/blobtest.
Index ¶
- Variables
- func CopyHashed(w io.Writer, r io.Reader, h *Hasher) (int64, error)
- func Disposition(filename string) string
- func NewUploadID() (string, error)
- func ValidUploadID(id string) bool
- func ValidateDigests(d Digests) error
- func ValidateSHA256(s string) error
- func Verify(want, got Digests) error
- type Digests
- type Hasher
- type Info
- type PartSizer
- type SignOptions
- type Signer
- type Store
- type Upload
- type UploadOption
- type UploadOptions
- type Via
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotFound reports a blob that is not stored. ErrNotFound = errors.New("blob: not found") // ErrDigestMismatch reports content whose digest differs from the one the // client declared. The upload is aborted; nothing is committed. ErrDigestMismatch = errors.New("blob: digest mismatch") // ErrUploadUnknown reports an upload ID that does not name a live session: // never issued, already committed or aborted, or malformed. ErrUploadUnknown = errors.New("blob: unknown upload") // ErrInvalidDigest reports a digest that is not lowercase hex of the right // length for its algorithm. ErrInvalidDigest = errors.New("blob: invalid digest") // ErrCannotSign reports a SignURL for a route the store does not sign: // Signs(via) is false. ErrCannotSign = errors.New("blob: store cannot sign this route") )
Sentinel errors. Backends wrap them, so test with errors.Is.
var ErrHasherState = errors.New("blob: invalid hasher state")
ErrHasherState reports a saved hasher state that cannot be restored.
Functions ¶
func CopyHashed ¶
CopyHashed copies r to w in chunks, feeding h only with bytes w accepted, so a short or failed write leaves the hash state matching what is stored. It returns the number of bytes stored.
func Disposition ¶
Disposition is the Content-Disposition naming a downloaded file, in the form Artifactory sends: a quoted name for old clients and an RFC 6266 UTF-8 name for the rest. A streamed download and a signed URL's override use the same value.
func NewUploadID ¶
NewUploadID returns a random 32-character hex upload ID. IDs are unguessable because a session accepts appends from whoever presents it.
func ValidUploadID ¶
ValidUploadID reports whether id has the shape NewUploadID produces. Backends use upload IDs in file and object names.
func ValidateDigests ¶
ValidateDigests checks every non-empty field of d.
func ValidateSHA256 ¶
ValidateSHA256 rejects anything but 64 lowercase hex characters. Backends build file and object names from the digest, so this is also the guard against path traversal.
Types ¶
type Digests ¶
Digests are lowercase hex checksums. In Commit's want argument an empty field means "not declared"; in a returned Info an empty SHA512, SHA1 or MD5 means the backend does not keep it (the item index does).
type Hasher ¶
type Hasher struct {
// contains filtered or unexported fields
}
Hasher computes all four digests in one pass and can save and restore its state, so a resumed upload continues hashing instead of re-reading data.
func (*Hasher) MarshalBinary ¶
MarshalBinary saves the state of all four digests.
func (*Hasher) UnmarshalBinary ¶
UnmarshalBinary restores a state saved by MarshalBinary.
type PartSizer ¶
type PartSizer interface {
PartSize() int64
}
PartSizer is implemented by backends that write an upload in parts of a fixed size. Bytes short of a whole part stay pending in the upload's session, which is saved after every Append, so a caller that appends in multiples of PartSize keeps that saved state small. The mirror sizes its chunks by it.
type SignOptions ¶
type SignOptions struct {
// Via is the route.
Via Via
// TTL is how long the URL stays valid once handed out.
TTL time.Duration
// Filename and ContentType, when set, travel as response overrides, so
// the object store answers with the item's name and media type rather
// than the blob's: the 302 that points at the URL describes no body.
Filename string
ContentType string
}
SignOptions describe one signed download URL.
type Signer ¶
type Signer interface {
Signs(via Via) bool
SignURL(ctx context.Context, sha256 string, o SignOptions) (string, error)
}
Signer is implemented by backends that can hand a client a time-limited URL for a blob, used when a repository sets downloadRedirect or cdnRedirect. Signs is fixed when the store is built, from its configuration and credentials, so a caller knows the route is available before it tries; SignURL returns ErrCannotSign for a route Signs refuses.
type Store ¶
type Store interface {
// BeginUpload starts an upload session.
BeginUpload(ctx context.Context, opts ...UploadOption) (Upload, error)
// ResumeUpload reopens a session started earlier, possibly by another
// request. It returns ErrUploadUnknown for an ID that is not live.
ResumeUpload(ctx context.Context, id string) (Upload, error)
// Open returns the content for random access, so HTTP handlers can serve
// ranges with http.ServeContent. The caller closes it. Info carries Size
// and Digests.SHA256; SHA512, SHA1 and MD5 may be empty.
Open(ctx context.Context, sha256 string) (io.ReadSeekCloser, Info, error)
// Stat returns a blob's Info without opening it.
Stat(ctx context.Context, sha256 string) (Info, error)
// Exists reports whether a blob is stored.
Exists(ctx context.Context, sha256 string) (bool, error)
// Delete removes a blob. Deleting a missing blob succeeds, because garbage
// collection and a concurrent delete can race and both mean "gone".
Delete(ctx context.Context, sha256 string) error
// Walk calls fn for every stored blob in an unspecified order and stops at
// the first error fn returns, which Walk returns. Upload sessions are not
// visited. It exists for garbage collection and storage accounting.
Walk(ctx context.Context, fn func(Info) error) error
}
Store is the content-addressed blob store.
type Upload ¶
type Upload interface {
// ID names the session for ResumeUpload.
ID() string
// Size is the number of bytes appended so far.
Size() int64
// Append writes r to the end of the upload and returns how many bytes it
// stored. On error the bytes counted are stored and hashed, so the session
// stays consistent and the client may continue from Size.
Append(ctx context.Context, r io.Reader) (int64, error)
// Commit finishes the upload. Every non-empty field of want is compared
// with the computed digest; a mismatch aborts the session and returns
// ErrDigestMismatch. Content that is already stored is not written again
// and the session is discarded, so committing the same bytes twice, or
// from two sessions at once, succeeds both times.
Commit(ctx context.Context, want Digests) (Info, error)
// Abort discards the session. Aborting a session that is already gone
// succeeds.
Abort(ctx context.Context) error
}
Upload is one upload session.
type UploadOption ¶
type UploadOption func(*UploadOptions)
UploadOption configures BeginUpload.
func WithExpectedSHA256 ¶
func WithExpectedSHA256(sha256 string) UploadOption
WithExpectedSHA256 declares the content digest up front. Commit still verifies it.
type UploadOptions ¶
type UploadOptions struct {
// ExpectedSHA256 is the digest the client declared before sending the
// body (Docker's monolithic PUT ?digest=, X-Checksum-Sha256). Backends
// that cannot rename use it to write straight to the final key.
ExpectedSHA256 string
}
UploadOptions is the resolved form of the options, exported for backends.
func ResolveOptions ¶
func ResolveOptions(opts ...UploadOption) (UploadOptions, error)
ResolveOptions applies opts and validates them.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package azure is the Azure Blob Storage blob store (decision 0027).
|
Package azure is the Azure Blob Storage blob store (decision 0027). |
|
Package blobtest is the contract every blob.Store backend must satisfy.
|
Package blobtest is the contract every blob.Store backend must satisfy. |
|
Package fs is the local-disk blob store.
|
Package fs is the local-disk blob store. |
|
Package gcs is the Google Cloud Storage blob store (decision 0027).
|
Package gcs is the Google Cloud Storage blob store (decision 0027). |
|
Package mirror is a blob store that keeps each blob on several member stores at once (decision 0028): different clouds, regions or accounts, any mix of the fs, s3, gcs and azure backends.
|
Package mirror is a blob store that keeps each blob on several member stores at once (decision 0028): different clouds, regions or accounts, any mix of the fs, s3, gcs and azure backends. |
|
Package s3 is the S3-compatible blob store (AWS S3, MinIO, Ceph RGW and similar).
|
Package s3 is the S3-compatible blob store (AWS S3, MinIO, Ceph RGW and similar). |
|
Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027).
|
Package session is the upload session shared by the object-store blob backends (s3, gcs, azure; decision 0027). |