storage

package
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package storage is Gombit's object storage contract: a driver-neutral Storage interface for files (uploads, avatars, reports, exports), so application code never depends on a filesystem path or an S3 SDK call.

info, err := store.Put(ctx, "avatars/42.png", r, storage.PutOptions{
	ContentType: "image/png",
})
body, info, err := store.Open(ctx, "avatars/42.png")
defer body.Close()
err = store.Delete(ctx, "avatars/42.png")

The contract, which every driver keeps and the storagetest suite checks:

  • Streaming. Put reads an io.Reader to EOF and Open returns an io.ReadCloser; neither requires the whole object in memory, a seekable source, or a known length.
  • Atomic writes. A Put that fails (its reader errors, its context ends, its declared size is wrong) leaves the key as it was; no one ever reads a partial object. The exception is ErrUnknownOutcome: a remote backend was sent the object and never confirmed or denied storing it, so the key holds the old object or the new one, whole.
  • Portable keys. ValidateKey is the same rule on every driver, and it rejects any key that could climb out of a prefix ("..", a leading '/', a backslash).
  • Classified errors. ErrNotFound, ErrInvalidKey, ErrInvalidOptions, ErrSizeMismatch, ErrUnsupported, ErrUnavailable, ErrUnknownOutcome, and ErrExists match with errors.Is on every driver, and MapError turns them into D10 errors for a handler.

Authorization stays with the application: decide who may read or write an object before calling Storage, and before handing out a URL.

Index

Constants

View Source
const DefaultContentType = "application/octet-stream"

DefaultContentType is the content type of an object stored without one.

View Source
const FilenameMetadata = "filename"

FilenameMetadata is the metadata name of an object's filename, for display and Content-Disposition (storage/upload stores the client's filename under it; presign.Handler serves it).

View Source
const MaxContentTypeBytes = 256

MaxContentTypeBytes bounds PutOptions.ContentType, parameters included.

View Source
const MaxKeyBytes = storagekey.MaxKeyBytes

MaxKeyBytes is the longest key, in bytes (S3's limit).

View Source
const MaxMetadataBytes = 2048

MaxMetadataBytes bounds a PutOptions.Metadata, measured as the metadata headers of an S3 request: each header name ("x-amz-meta-" and the name) plus each value as it is sent, a value that is not printable ASCII being RFC 2047 encoded (mime.BEncoding, words of at most 75 characters), S3's documented encoding for non-ASCII metadata. That is the strictest measure among S3-compatible services (MinIO counts exactly this; AWS the decoded UTF-8).

View Source
const MaxPublicationToken = 60000

MaxPublicationToken is the longest token a Publisher returns: it fits a TEXT column on every supported database.

View Source
const MaxSegmentBytes = storagekey.MaxSegmentBytes

MaxSegmentBytes is the longest segment of a key (the part between '/'s), in bytes. AWS S3 itself allows longer, but MinIO, an S3-compatible service that stores objects as files, refuses a longer segment (XMinioInvalidObjectName), so a portable key stays within it.

View Source
const MaxURLExpiry = 7 * 24 * time.Hour

MaxURLExpiry is the longest a signed URL may live: S3's limit for a presigned URL (SigV4), applied on every driver so a lifetime that works in development works in production.

View Source
const SignedUploadTimeout = time.Hour

SignedUploadTimeout is how long after a direct upload's grant expires its upload may still be writing the object. A grant bounds only when the PUT may start, so the app's own storage route (presign.Handler) aborts a PUT still running this long after its URL expired: nothing it serves is published later. S3 cannot be told to abort a presigned PUT it has started, which is why direct uploads under storage/claims are staged: the client only ever writes a staging key (upload.StagingKey).

Variables

View Source
var (
	// ErrNotFound: no object is stored under the key.
	ErrNotFound = errors.New("storage: object not found")
	// ErrInvalidKey: the key breaks the key rules (see ValidateKey).
	ErrInvalidKey = errors.New("storage: invalid object key")
	// ErrInvalidOptions: a PutOptions or URLOptions value is invalid (a
	// malformed content type or metadata, a negative size or expiry).
	ErrInvalidOptions = errors.New("storage: invalid options")
	// ErrSizeMismatch: the bytes read differ in length from the declared
	// PutOptions.Size.
	ErrSizeMismatch = errors.New("storage: object length differs from its declared size")
	// ErrUnsupported: the driver cannot do this (a URL from a driver with no
	// URL scheme).
	ErrUnsupported = errors.New("storage: operation not supported by this driver")
	// ErrNotPublic: a public URL was asked for an object that is not
	// public (its key is not under the store's public prefix).
	ErrNotPublic = errors.New("storage: object is not public")
	// ErrUnavailable: the backend could not be reached or failed
	// transiently; retrying later may succeed.
	ErrUnavailable = errors.New("storage: backend unavailable")
	// ErrUnknownOutcome: a Put failed after sending the request that would
	// publish the object, and the backend never said whether it did (the
	// response was lost, the connection dropped, the service failed, or ctx
	// ended while waiting). The key holds either the previous object or the
	// new one, whole; which one is unknown. Only a driver whose backend is
	// across a network returns it. The error also matches the cause
	// (ErrUnavailable, or ctx's error). Stat the key, or Put again: a Put is
	// safe to repeat.
	ErrUnknownOutcome = errors.New("storage: outcome unknown: the object may or may not have been stored")
	// ErrExists: a Put with PutOptions.IfAbsent found an object already
	// stored under the key, and left it as it was.
	ErrExists = errors.New("storage: an object is already stored under the key")
)

Sentinels for errors.Is. Drivers wrap them (usually in an *Error), so application code classifies a failure the same way on every driver.

Functions

func CheckOwnable

func CheckOwnable(s Storage) error

CheckOwnable reports whether storage/claims can own s's objects: s must be a Publisher (its copies can be fenced) or a BoundedWriter (its writes end with their calls). Any other store fails with ErrUnsupported: nothing proves that a write it sent will not complete after its claim is gone.

func ContextReadCloser

func ContextReadCloser(ctx context.Context, rc io.ReadCloser) io.ReadCloser

ContextReadCloser returns rc bound to ctx, as the reader Storage.Open returns must be: once ctx has ended, every Read fails with ctx's error, and ContextReadCloser closes rc to end a Read in progress. That interruption needs rc's Close to be safe while a Read runs and to make that Read return, which an *os.File, an HTTP response body, and an io.Pipe provide (and a reader that never blocks, such as a bytes.Reader, does not need); a driver passes such a reader. With an rc whose Close cannot interrupt a Read, a Read blocked when ctx ends returns when rc lets it, and then fails with ctx's error. Close closes rc once. When rc can seek, so can the result.

func ContextReader

func ContextReader(ctx context.Context, r io.Reader) io.Reader

ContextReader returns r stopping at ctx: each Read first checks ctx and fails with its error once it has ended, without calling r. It does not interrupt a Read of r already in progress (r has no way to be interrupted), but when that Read returns after ctx has ended it fails with ctx's error, whatever r returned (EOF included). A driver copies a Put's reader through it so a canceled Put stops streaming instead of reading to EOF, as Storage.Put specifies.

func EscapeKey

func EscapeKey(key string) string

EscapeKey returns key as a URL path: each segment percent-encoded except for unreserved characters (letters, digits, '-', '.', '_', '~'), the '/' between segments kept. Every byte that could mean something else in a URL, or that a server might decode differently ('+', ';', '%'), is encoded.

func Exists

func Exists(ctx context.Context, s Storage, key string) (bool, error)

Exists reports whether key holds an object: Stat, with ErrNotFound as false.

func ExpectSize

func ExpectSize(r io.Reader, size int64) io.Reader

ExpectSize returns r limited to exactly size bytes, for a driver enforcing PutOptions.Size: a source that ends early, or has a byte past size, fails with ErrSizeMismatch, and a size of zero means an empty source. The read that reaches size looks one byte further, so the mismatch comes with it: read the result to EOF (io.Copy, io.ReadAll). io.ReadFull and io.CopyN stop at a byte count and can drop an error returned with the last bytes. A negative size fails every read with ErrInvalidOptions.

func Fence

func Fence(ctx context.Context, s Storage, token string) error

Fence makes sure the copy token names can never be published after it returns nil (Publisher.Fence). For a BoundedWriter it does nothing: an in-process copy publishes nothing after its Publish call's context ended, so a caller that bounded that context (a lease) need only wait for the bound. Any other store fails with ErrUnsupported: nothing can be proven about it.

func IsPublic

func IsPublic(publicPrefix, key string) bool

IsPublic reports whether key is public under publicPrefix: publicPrefix is not empty and key starts with it.

func KnownSize

func KnownSize(n int64) *int64

KnownSize is a PutOptions.Size of exactly n bytes.

func List

func List(ctx context.Context, s Storage, prefix string, fn func(ObjectInfo) error) error

List enumerates s's objects under prefix (Lister), or fails with ErrUnsupported when s cannot.

func MapError

func MapError(ctx context.Context, err error, notFound, internal string) error

MapError maps a storage error to a D10 category error for a handler:

  • ErrNotFound becomes not_found (with notFound as the message);
  • ErrExists becomes conflict (an IfAbsent Put found an object);
  • ErrInvalidOptions and ErrSizeMismatch become validation (the request sent a content type, metadata, or length that cannot be stored);
  • ErrUnavailable, ErrUnknownOutcome, and a context that ended (the request timed out or the client went away), become dependency_unavailable;
  • anything else, ErrInvalidKey and ErrNotPublic included, becomes internal (with internal as the message): keys and the URLs asked for are chosen by the server, so either is a server bug. Validate a key taken from a request with ValidateKey first, and answer not_found.

The driver's own error text never reaches the client.

func MetadataValueWireLen

func MetadataValueWireLen(value string) int

MetadataValueWireLen is how many bytes value takes in an S3 metadata header: its length when it is printable ASCII, otherwise the length of its RFC 2047 encoding (mime.BEncoding.Encode("UTF-8", value)), which an S3 driver must send.

func PreparePublish

func PreparePublish(ctx context.Context, s Storage, src, dst string) (string, error)

PreparePublish readies a copy of src to dst in s (Publisher), and returns its token. For a BoundedWriter nothing happens yet: the token names the keys, and Publish copies in the process. Any other store fails with ErrUnsupported.

func PutReader

func PutReader(ctx context.Context, r io.Reader, opts PutOptions) io.Reader

PutReader returns r as a driver's Put should read it: limited to opts.Size when one is declared (ExpectSize), and stopping when ctx ends (ContextReader). Read it to EOF.

func SizeFromContentLength

func SizeFromContentLength(n int64) *int64

SizeFromContentLength is the PutOptions.Size for an HTTP request body of Content-Length n: nil when unknown (-1), the exact length otherwise (zero included, so a request claiming an empty body cannot store one).

func ValidateKey

func ValidateKey(key string) error

ValidateKey reports whether key is a valid object key, as ErrInvalidKey (wrapped with the reason) when it is not.

A key is a '/'-separated path of one or more segments, the same on every driver, so a key that works in development works in production:

  • 1 to MaxKeyBytes bytes of valid UTF-8;
  • no leading or trailing '/', and no empty segment ("a//b");
  • no segment that is "." or "..", or that ends with '.' or a space: Windows strips a trailing dot or space from a path component, so "..." or ".. " would otherwise become ".." on a filesystem there, and "a." would collide with "a";
  • no segment longer than MaxSegmentBytes;
  • no backslash (a path separator on Windows), no control character (NUL, newline, DEL, the C1 controls), and no Unicode line separator or bidirectional control (which can split a log line or disguise a name in a listing: "invoice\u202efdp.exe").

A key is an address, not a filename a client chose: build keys on the server and keep a client's filename as metadata.

func ValidateMetadata

func ValidateMetadata(md map[string]string) error

ValidateMetadata reports whether md is valid user metadata, as ErrInvalidOptions when it is not: names of lower-case ASCII letters, digits, and '-' (portable as HTTP header suffixes, which are case-insensitive); values of UTF-8 without control characters; names and values together at most MaxMetadataBytes.

func ValidatePrefix

func ValidatePrefix(prefix string) error

ValidatePrefix reports whether prefix can start every key of a store (a driver's configured prefix, "myapp/"), as ErrInvalidKey when it cannot. A prefix is empty, or a valid key (see ValidateKey) followed by '/', shorter than MaxKeyBytes so that a key fits after it: prefix + key is held to the key rules, MaxKeyBytes included. config.Validate applies the same rule, so a configuration it accepts is one the driver accepts.

func ValidatePublicPrefix

func ValidatePublicPrefix(prefix string) error

ValidatePublicPrefix reports whether prefix can be a store's public prefix: empty (no object is public), or a key path ending with '/'.

func ValidatePutOptions

func ValidatePutOptions(opts PutOptions) error

ValidatePutOptions reports whether opts is valid, as ErrInvalidOptions (wrapped with the reason) when it is not: the content type must parse as a type/subtype media type of at most MaxContentTypeBytes, Size must not be negative, and Metadata must pass ValidateMetadata.

func ValidateURLOptions

func ValidateURLOptions(opts URLOptions) error

ValidateURLOptions reports whether opts is valid, as ErrInvalidOptions when it is not: a signed URL needs a positive Expires of at most MaxURLExpiry, and a public one none.

func ValidateUploadURLOptions

func ValidateUploadURLOptions(opts UploadURLOptions) error

ValidateUploadURLOptions reports whether opts is valid, as ErrInvalidOptions when it is not: a lifetime as for a signed URL, a non-negative size, and a content type and metadata as for Put.

func Wrap

func Wrap(op, key string, err error) error

Wrap returns err as an *Error for op on key, or nil when err is nil. An err that is already the *Error of op on key is returned unchanged; one for another operation or key is wrapped, so the outer envelope always names the call that failed.

Types

type BoundedWriter

type BoundedWriter interface {
	Storage
	// BoundedWrites declares the guarantee; it does nothing.
	BoundedWrites()
}

BoundedWriter is a Storage whose writes are bounded by their calls: a Put publishes, if at all, before it returns, and never once its context has ended (the driver checks it just before publishing). The local and memory drivers are. A store that writes through a remote service generally is not (a request it sent may complete later), and cannot declare it; to take part in storage/claims it must be a Publisher.

type DirectUploader

type DirectUploader interface {
	// UploadURL returns the request that stores exactly one object: under
	// key, of exactly opts.Size bytes, with opts.ContentType and
	// opts.Metadata, until opts.Expires after the start of the second it is
	// signed in (the model of storage.SignedURL). The backend refuses a
	// request that differs in any of these (a different length, type,
	// metadata, or key), so the grant is scoped to that one object. It
	// stores only where nothing is stored yet (S3 answers 412, the app's
	// storage route a D10 409 conflict, otherwise): once used,
	// the object cannot be replaced through the grant, so bytes checked
	// after the upload stay the bytes stored. It authorizes nobody: decide
	// who may upload first, and check what arrived (upload.Confirm) before
	// using it.
	UploadURL(ctx context.Context, key string, opts UploadURLOptions) (UploadRequest, error)
}

DirectUploader is a Storage that grants signed uploads: a request a client makes on its own, authorized by the grant alone (no session, no application credentials), that stores exactly one object. The application asks for one (after deciding the client may upload), hands it to the client, and the client sends the bytes.

Where the bytes go depends on the driver, and only S3 keeps them out of the application process. S3 grants a presigned PUT to the bucket: the bytes go straight to S3. The local and memory drivers (with a presign.Signer) grant a PUT to the application's own storage route (presign.Handler, mounted by framework.New): the bytes pass through the application process, streamed into the store, but not through the application's handlers or its request parsing. UploadURL answers ErrUnsupported for a store that grants neither.

type Error

type Error struct {
	Op  string // "put", "open", "stat", "delete", "url"
	Key string
	Err error
}

Error is a failed storage operation: what was attempted, on which key, and why. Err is the classified cause (one of the sentinels, possibly wrapping the driver's own error), reachable with errors.Is/As.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Unwrap

func (e *Error) Unwrap() error

type Lister

type Lister interface {
	// List calls fn with every object whose key starts with prefix ("" for
	// all), in an order the driver chooses, until fn returns an error,
	// which List then returns. Each ObjectInfo has the Key, Size, ETag, and
	// ModTime; ContentType and Metadata may be empty (an S3 listing does
	// not carry them: Stat the object for them). fn may delete the object
	// it is given. An object stored or deleted while List runs may or may
	// not be seen.
	List(ctx context.Context, prefix string, fn func(ObjectInfo) error) error
}

Lister is a Storage that can enumerate its objects, for inventories, audits, and migrations between stores. (Cleaning up abandoned uploads does not list the store, only the staging namespace that storage/upload reserves: see claims.SweepStaging.) The local, memory, and S3 drivers implement it.

type ObjectInfo

type ObjectInfo struct {
	// Key is the object's key.
	Key string
	// Size is the object's length in bytes.
	Size int64
	// ContentType is the object's media type (DefaultContentType when it
	// was stored without one).
	ContentType string
	// ETag identifies this version of the object's bytes, when the driver
	// has one; it changes when the object is replaced with different
	// bytes. A driver reports it from Put, Open, and Stat alike, or from
	// none of them (empty: unsupported). It is opaque: compare ETags, never
	// compute or interpret one. (S3's depends on how the object was stored
	// and on the bucket's encryption, and is not a checksum of the bytes in
	// general; the local and memory drivers happen to use a SHA-256, which
	// is not part of the contract.)
	ETag string
	// ModTime is when the object was last stored (for an object written
	// once, as uploads under generated keys are, when it was created).
	// Stat, Open, and List always report it; the ObjectInfo Put returns
	// may leave it zero when the backend does not say (S3's PutObject),
	// rather than cost a second request.
	ModTime time.Time
	// Metadata is the user metadata the object was stored with (nil when
	// none). Every ObjectInfo a driver returns has a map of its own: the
	// caller may change it without changing the stored object or any other
	// result.
	Metadata map[string]string
}

ObjectInfo describes a stored object.

func Publish

func Publish(ctx context.Context, s Storage, token string) (ObjectInfo, error)

Publish publishes the copy token names (PreparePublish). For a BoundedWriter, it copies within the process (Open, then Put with src's ContentType, Metadata and size, IfAbsent), under ctx: the copy publishes nothing once Publish has returned, or after ctx's deadline. Any other store fails with ErrUnsupported.

func (ObjectInfo) StoredFilename

func (o ObjectInfo) StoredFilename() string

StoredFilename is the object's filename (the FilenameMetadata metadata), for display and Content-Disposition; empty when it was stored without one. (upload.File, which embeds ObjectInfo, has it as its Filename field.)

type Publisher

type Publisher interface {
	// PreparePublish readies a copy of the object at src (its bytes,
	// ContentType and Metadata) to dst, and returns a token naming it (at
	// most MaxPublicationToken bytes, by construction). Nothing is published yet: record the
	// token durably before Publish, so that the copy can be fenced whatever
	// happens to the caller. A missing src is ErrNotFound. When it fails
	// but returns a token, something of the copy may remain (a request
	// whose answer was lost): record the token and Fence it.
	PreparePublish(ctx context.Context, src, dst string) (string, error)
	// Publish publishes the prepared copy at dst. A failure whose outcome
	// is unknown (the request sent, no definite answer) is
	// ErrUnknownOutcome: the copy may be published, now or later; Fence
	// settles it.
	Publish(ctx context.Context, token string) (ObjectInfo, error)
	// Fence makes sure the copy named by token can never be published
	// after it returns nil: it was published already, or never will be.
	// It also removes what it can of an unpublished copy, but proves no
	// more than that nothing will be published (on S3 a part still being
	// processed may be stored afterwards; it is never an object). It is
	// idempotent. An error means nothing is proven yet: try again.
	Fence(ctx context.Context, token string) error
}

Publisher is a Storage that copies an object in two steps, so that a copy whose outcome is unknown can be settled for good: S3, where the copy is a multipart upload whose parts are copied server-side, published by CompleteMultipartUpload and fenced by AbortMultipartUpload. PreparePublish, Publish and Fence (the functions) use it, and fall back to an in-process copy for a BoundedWriter; for any other store they fail with ErrUnsupported.

type PutOptions

type PutOptions struct {
	// ContentType is the object's media type ("image/png"), parameters
	// allowed ("text/plain; charset=utf-8"). Empty stores
	// DefaultContentType. It must be a type/subtype media type.
	ContentType string

	// Size, when set, is the exact length r will produce (zero for an empty
	// object). Put then fails with ErrSizeMismatch, storing nothing, if r
	// ends early or runs long. Nil means unknown: Put reads r to EOF. Build
	// it with KnownSize, or from an HTTP request's Content-Length with
	// SizeFromContentLength (-1, unknown, is nil).
	Size *int64

	// Metadata is small user metadata stored with the object and returned
	// in its ObjectInfo exactly as given. Names are lower-case ASCII
	// letters, digits, and '-'; values are UTF-8 without control
	// characters or leading or trailing spaces (HTTP trims header values);
	// see ValidateMetadata for the size limit. A driver whose backend
	// carries only ASCII (S3 headers) encodes values reversibly. Put copies
	// the map: changing it afterwards changes nothing stored.
	Metadata map[string]string

	// IfAbsent stores the object only where no object is stored under key:
	// when one is, Put fails with ErrExists and leaves it as it is. The
	// check and the store are one atomic step on every driver, against
	// concurrent Puts from this process and from any other sharing the
	// backend (the local driver publishes with a rename that refuses to
	// replace; S3 with a conditional write, "If-None-Match: *"), so of two
	// IfAbsent Puts to one empty key exactly one succeeds.
	IfAbsent bool
}

PutOptions describes an object being stored.

type Storage

type Storage interface {
	// Put stores the bytes read from r under key, replacing any object
	// already there. It streams: r is read to EOF in pieces, never required
	// to be seekable or of known length.
	//
	// Put is atomic. When it fails (r returns an error, ctx ends, the
	// length differs from a declared opts.Size), key keeps the object it
	// had before, or stays absent; no reader ever sees a partial object.
	// Concurrent Puts to one key leave one of them whole, and a reader
	// during a Put reads the previous object whole.
	//
	// The one exception is a failure that matches ErrUnknownOutcome: a
	// driver whose backend is across a network sent the request that
	// publishes the object and never learned whether it took effect. The
	// key then holds the previous object or the new one, whole; which, the
	// driver cannot say. Every other failure, ctx ending before the object
	// was sent included, leaves the key as it was.
	//
	// Put observes ctx between reads of r: once ctx has ended, it makes no
	// further Read, and it fails with ctx's error and stores nothing, even
	// if r then reaches EOF. (If ctx ends while a remote backend is
	// publishing the object, the error may match ErrUnknownOutcome as well
	// as ctx's error.) It cannot interrupt a Read of r already in
	// progress (an io.Reader has no way to be interrupted, and r is the
	// caller's): a source that can block must be made to return by its
	// owner, as an HTTP server closes a request body when the client goes
	// away, or a caller closes the pipe it writes.
	Put(ctx context.Context, key string, r io.Reader, opts PutOptions) (ObjectInfo, error)

	// Open returns the object's bytes as a stream, and its ObjectInfo. The
	// caller must Close the reader. A missing object is ErrNotFound. The
	// reader stays bound to ctx until it is closed: once ctx ends, Read
	// fails with ctx's error on every driver, and a Read blocked in the
	// backend returns (an HTTP response body does this by itself;
	// ContextReadCloser does it for a reader whose Close interrupts a Read),
	// so keep ctx alive for as long as the stream is read.
	Open(ctx context.Context, key string) (io.ReadCloser, ObjectInfo, error)

	// Stat returns the object's ObjectInfo without its bytes. A missing
	// object is ErrNotFound.
	Stat(ctx context.Context, key string) (ObjectInfo, error)

	// Delete removes the object. Deleting a missing object is not an error,
	// so a retried Delete is safe.
	Delete(ctx context.Context, key string) error

	// URL returns a URL a client can fetch the object from: a permanent
	// public URL for a public object (ErrNotPublic for a private one; see
	// IsPublic), or a signed one that works for any object until it
	// expires (see URLOptions; at most MaxURLExpiry). A driver that cannot
	// produce one returns ErrUnsupported. URL does not check that the
	// object exists, and it does not authorize anyone: decide who may have
	// the URL before asking for it.
	URL(ctx context.Context, key string, opts URLOptions) (string, error)
}

Storage is the driver-neutral object storage contract. Application code depends on it, never on a driver's API (a filesystem path, an S3 SDK call), so switching drivers changes configuration, not code.

Every method validates its key with ValidateKey and fails with ErrInvalidKey before touching the backend. Every method honors ctx: one that has already ended fails the call with ctx's error.

Every valid key is its own object on every driver: keys that differ only in case, a key that is a prefix of another ("a" and "a/b"), segments up to 255 bytes, and names a filesystem reserves ("CON", "c:") all store and read back as themselves. A driver on a filesystem must map keys to paths in a way that keeps this (it cannot just join the key to a directory).

Implementations must pass the storagetest conformance suite.

type URLOptions

type URLOptions struct {
	// Signed asks for a URL that grants access only until Expires has
	// passed. Unsigned asks for a permanent public URL.
	Signed bool
	// Expires is how long a signed URL is valid: it must be positive for a
	// signed URL, and zero for a public one (ErrInvalidOptions otherwise),
	// so a zero or negative lifetime can never widen into a permanent URL.
	// It is a whole number of seconds (ErrInvalidOptions otherwise),
	// counted from the start of the second the URL is signed in, as S3
	// counts it, on every driver.
	Expires time.Duration
}

URLOptions chooses the kind of URL URL returns. Build it with PublicURL or SignedURL.

func PublicURL

func PublicURL() URLOptions

PublicURL asks for a permanent URL to an object readable without credentials.

func SignedURL

func SignedURL(ttl time.Duration) URLOptions

SignedURL asks for a URL that grants access to the object for ttl, counted from the start of the second the URL is signed in, as S3 counts a presigned URL's lifetime: it expires at that second plus ttl, so it works for more than ttl minus one second and at most ttl after URL returns (less any time URL itself takes). ttl must be a positive whole number of seconds, at most MaxURLExpiry: URL fails with ErrInvalidOptions otherwise, rather than returning anything longer-lived or counted differently by another driver.

type UploadRequest

type UploadRequest struct {
	Method  string            `json:"method"`
	URL     string            `json:"url"`
	Header  map[string]string `json:"headers"`
	Expires time.Time         `json:"expires"`
}

UploadRequest is a direct upload for a client to make: send Method to URL with exactly the headers in Header and the object's bytes as the body, before Expires. The client's HTTP library sets Content-Length from the body (a browser's fetch does; it cannot be set by hand). Expires is when the grant stops working: its lifetime's end, or earlier on S3 when the credentials it was signed with expire first (temporary credentials, such as an IAM role's), since the grant ends with them.

func UploadURL

func UploadURL(ctx context.Context, s Storage, key string, opts UploadURLOptions) (UploadRequest, error)

UploadURL asks s for a direct upload URL (DirectUploader), or fails with ErrUnsupported when s cannot make one.

type UploadURLOptions

type UploadURLOptions struct {
	// Expires is how long the upload URL works, counted from the start of
	// the second it is signed in, as for SignedURL (so for more than
	// Expires minus a second, and at most Expires): a positive whole number
	// of seconds, at most MaxURLExpiry. Keep it short (minutes): the URL is
	// a bearer grant.
	Expires time.Duration
	// Size is the exact length the client will upload, in bytes (zero for
	// an empty object). A request of any other length is refused.
	Size int64
	// ContentType is the object's media type, which the client must send
	// as its Content-Type (DefaultContentType when empty). It is what the
	// client declares, not what the bytes are: check them afterwards.
	ContentType string
	// Metadata is stored with the object (the rules of
	// PutOptions.Metadata).
	Metadata map[string]string
}

UploadURLOptions describes the object a direct upload will store.

type UploadVerifier

type UploadVerifier interface {
	VerifyUpload(ctx context.Context, key string) error
}

UploadVerifier is a DirectUploader whose backend can keep more about an uploaded object than ObjectInfo describes, set by the client outside what its grant signed: S3 stores a PUT's Cache-Control, Content-Disposition, Content-Encoding, Content-Language and Expires headers, which SigV4 does not authenticate unless signed, and serves them back. VerifyUpload fails with ErrInvalidOptions when the object at key carries any of that, so the object can be refused before it is used. upload.Confirm calls it. A store without it (local, memory) keeps only what ObjectInfo describes, whatever headers the upload sent.

Directories

Path Synopsis
Package claims is the ownership protocol for stored files: which object a record owns, so that cleaning up never deletes a file a record refers to.
Package claims is the ownership protocol for stored files: which object a record owns, so that cleaning up never deletes a file a record refers to.
Package filefield is the runtime behind storage-backed model fields (types.File and types.Image): the code `gombit generate` writes for a resource with such a field calls it to grant uploads for the field, accept an uploaded file into a record, and describe it on reads.
Package filefield is the runtime behind storage-backed model fields (types.File and types.Image): the code `gombit generate` writes for a resource with such a field calls it to grant uploads for the field, accept an uploaded file into a record, and describe it on reads.
Package local is a storage.Storage on the local filesystem, for development and single-host deployments: objects are files under one root directory, streamed to and from disk.
Package local is a storage.Storage on the local filesystem, for development and single-host deployments: objects are files under one root directory, streamed to and from disk.
Package memory is an in-process storage.Storage for tests: objects live in a map, nothing touches the filesystem, and every behavior is deterministic.
Package memory is an in-process storage.Storage for tests: objects live in a map, nothing touches the filesystem, and every behavior is deterministic.
Package presign gives a store with no URL scheme of its own (the local and memory drivers) the URLs of the storage contract, served by the application: a Signer makes the URLs, and Handler serves them.
Package presign gives a store with no URL scheme of its own (the local and memory drivers) the URLs of the storage contract, served by the application: a Signer makes the URLs, and Handler serves them.
Package s3 is a storage.Storage on S3-compatible object storage: AWS S3, Cloudflare R2, MinIO, and other services that speak the S3 API.
Package s3 is a storage.Storage on S3-compatible object storage: AWS S3, Cloudflare R2, MinIO, and other services that speak the S3 API.
Package storagetest is the conformance suite every storage.Storage driver must pass.
Package storagetest is the conformance suite every storage.Storage driver must pass.
Package upload receives files from HTTP requests into a storage.Storage, safe by default:
Package upload receives files from HTTP requests into a storage.Storage, safe by default:

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL