Documentation
¶
Overview ¶
Package blob defines the object storage abstraction that holds site files, metadata and user uploads. Implementations live in sub-packages (fs, s3, gcs, azure); backends.Open builds one from configuration.
Keys are "/"-separated, never start with "/", and are relative to the configured bucket prefix (implementations add/strip the prefix).
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotFound is returned when an object does not exist. ErrNotFound = errors.New("blob: not found") // ErrNotSupported is returned by optional operations (e.g. presigning on fs). ErrNotSupported = errors.New("blob: operation not supported") // ErrInvalidKey is returned for keys that are empty, absolute or contain "..". ErrInvalidKey = errors.New("blob: invalid key") )
Functions ¶
func ContentTypeByExt ¶
ContentTypeByExt returns the content type for a file name's extension, or "".
func DetectContentType ¶
DetectContentType guesses a content type from a file name (extension) and falls back to sniffing the first 512 bytes when head is non-nil.
Types ¶
type Description ¶
type Description struct {
Type string `json:"type"`
Bucket string `json:"bucket,omitempty"`
Container string `json:"container,omitempty"`
Account string `json:"account,omitempty"`
Prefix string `json:"prefix,omitempty"`
Region string `json:"region,omitempty"`
Endpoint string `json:"endpoint,omitempty"`
}
Description is the public storage description included in /info.
type DigestReporter ¶
type DigestReporter interface {
// StatSHA256 returns the object's metadata (as Stat) together with the
// lowercase hex SHA-256 of its full content as verified by the storage
// service, or "" when the service holds no such digest for this object
// version. A digest always describes the object the returned info
// describes. It returns ErrNotFound for a missing object and
// ErrNotSupported when the store cannot report digests at all (callers
// then read the object).
StatSHA256(ctx context.Context, key string) (*ObjectInfo, string, error)
}
DigestReporter is optionally implemented by stores whose service keeps a SHA-256 it verified itself for some objects, so callers can learn an object's digest without reading it: S3 objects uploaded with x-amz-checksum-sha256 (as presigned uploads are where flexible checksums are enabled, see PresignPutOptions).
type ListResult ¶
type ListResult struct {
Objects []ObjectInfo
// Prefixes are the "directories" (ending in "/") found when a delimiter is used.
Prefixes []string
}
ListResult is the result of a List call.
type ObjectInfo ¶
type ObjectInfo struct {
Key string `json:"key"`
Size int64 `json:"size"`
ETag string `json:"etag"` // opaque, changes whenever content changes; never empty
ContentType string `json:"content_type,omitempty"`
ModTime time.Time `json:"mod_time"`
}
ObjectInfo describes a stored object.
type PresignPutOptions ¶
type PresignPutOptions struct {
// ContentType is signed into the URL and stored with the object; ""
// derives it from the key's extension.
ContentType string
// Size is the exact body size in bytes, or -1 when unknown.
Size int64
// SHA256 is the lowercase or uppercase hex SHA-256 of the body; "" when
// unknown. Ignored by backends that cannot enforce it.
SHA256 string
}
PresignPutOptions describes the object a presigned upload URL accepts.
What a backend can bind into the URL differs (see each implementation):
- s3: Content-Type and the size are signed (Content-Length, or Content-MD5 of the empty body for Size 0), so S3 rejects any other size; SHA256 is signed as x-amz-checksum-sha256, which S3 verifies against the body, wherever flexible checksums are enabled (AWS S3 by default; custom endpoints only with AWS_REQUEST_CHECKSUM_CALCULATION).
- gcs: Content-Type and x-goog-content-length-range: Size,Size are signed (V4), so GCS rejects any other size; GCS cannot verify SHA-256.
- azure: a SAS cannot restrict the size or content of an upload; only the blob is fixed. Callers must verify uploads afterwards (the deploy gateway compares sizes at commit and deletes mismatching objects).
The zero value is not "anything goes": Size 0 means an empty body. Use Size -1 when the size is unknown.
type PresignedRequest ¶
type PresignedRequest struct {
Method string `json:"method"`
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"` // must be sent verbatim
ExpiresAt time.Time `json:"expires_at"`
}
PresignedRequest is an upload/download URL usable without credentials.
type Store ¶
type Store interface {
// Get opens an object for reading. The caller must close the reader.
Get(ctx context.Context, key string) (io.ReadCloser, *ObjectInfo, error)
// GetRange reads length bytes starting at offset (length < 0 = to end).
GetRange(ctx context.Context, key string, offset, length int64) (io.ReadCloser, *ObjectInfo, error)
// Stat returns object metadata or ErrNotFound.
Stat(ctx context.Context, key string) (*ObjectInfo, error)
// Put writes an object, replacing any existing one. size may be -1 if unknown.
Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) (*ObjectInfo, error)
// Delete removes an object. Deleting a missing object is not an error.
Delete(ctx context.Context, key string) error
// DeletePrefix removes every object whose key starts with prefix.
DeletePrefix(ctx context.Context, prefix string) error
// List returns all objects under prefix (all pages). With delimiter "/",
// objects in "sub-directories" are folded into Prefixes; with "" the
// listing is recursive. Results are sorted by key.
List(ctx context.Context, prefix, delimiter string) (*ListResult, error)
// PresignPut returns a URL that accepts a PUT of the object's bytes as
// described by opts (content type, and as far as the backend can enforce
// them, size and SHA-256). The returned Headers must be sent verbatim.
// Returns ErrNotSupported when the backend cannot presign.
PresignPut(ctx context.Context, key string, opts PresignPutOptions, ttl time.Duration) (*PresignedRequest, error)
// PresignGet returns a time-limited download URL or ErrNotSupported.
PresignGet(ctx context.Context, key string, ttl time.Duration) (*PresignedRequest, error)
// Ping checks connectivity/permissions cheaply (used by /readyz).
Ping(ctx context.Context) error
// Describe returns non-secret details advertised to CLIs for direct sync
// (type, bucket/container, account, prefix, region).
Describe() Description
// Close releases resources.
Close() error
}
Store is an object store. Implementations must be safe for concurrent use.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package azure implements blob.Store on Azure Blob Storage (and Azurite).
|
Package azure implements blob.Store on Azure Blob Storage (and Azurite). |
|
Package backends constructs a blob.Store from configuration.
|
Package backends constructs a blob.Store from configuration. |
|
Package blobtest is a conformance test-suite for blob.Store implementations.
|
Package blobtest is a conformance test-suite for blob.Store implementations. |
|
Package fs implements blob.Store on a local directory tree.
|
Package fs implements blob.Store on a local directory tree. |
|
Package gcs implements blob.Store on Google Cloud Storage.
|
Package gcs implements blob.Store on Google Cloud Storage. |
|
internal
|
|
|
blobutil
Package blobutil holds helpers shared by the blob.Store implementations: key/prefix validation, content-type detection on streams, exact-size enforcement, chunked reading for multi-part uploads, synthetic ETags, presign TTL clamping and small concurrency utilities.
|
Package blobutil holds helpers shared by the blob.Store implementations: key/prefix validation, content-type detection on streams, exact-size enforcement, chunked reading for multi-part uploads, synthetic ETags, presign TTL clamping and small concurrency utilities. |
|
Package s3 implements blob.Store on Amazon S3 and S3-compatible object stores (MinIO, Cloudflare R2, Ceph RGW, Backblaze B2, ...).
|
Package s3 implements blob.Store on Amazon S3 and S3-compatible object stores (MinIO, Cloudflare R2, Ceph RGW, Backblaze B2, ...). |