Documentation
¶
Overview ¶
Package filesystem is a storage library for Go applications: one portable API over local disk, memory, S3-compatible object stores and any backend a Driver is written for.
Layers ¶
The package is the bottom of five layers, each usable without the ones above it:
- Driver: the small interface one backend implements (Stat, Read, Write, Delete, List, Capabilities). Drivers live in their own packages; the ones with third-party dependencies live in their own modules.
- Disk: the concrete type applications hold. Every portable behaviour is written once here rather than per driver: key normalisation and traversal rejection, content-type detection, SHA-256 and byte counting on writes, size limits, error classification, URL generation, and the interceptor chain.
- Manager: named disks, scoped and mirrored disks, cross-disk copy.
- upload (sub-package): how bytes arrive from browsers and phones.
- records (separate module): files as database rows attached to models.
Capabilities ¶
Anything only some backends can do (presigned URLs, server-side copy, per-object visibility, multipart uploads, stored metadata) is a capability. A driver declares its capabilities in one struct, and NewDisk refuses a driver whose declaration does not match the interfaces it implements, so no driver can claim a feature it does not have. Disk methods for a missing capability either fall back to a portable implementation (Copy, Move, DeletePrefix) or return an error whose Code is CodeUnsupported.
Interceptors ¶
Retry, timeouts, logging, metrics, tracing and read-only enforcement run as Interceptors around every Disk operation. They never wrap the driver, so they cannot hide its capabilities (the usual failure of Go decorators). See the layer sub-package for the built-in ones.
Keys ¶
Keys are slash-separated, relative and normalised by CleanKey before any driver sees them. A key containing a ".." segment is rejected, never resolved. Directories do not exist as objects: a "directory" is the set of keys sharing a prefix, which is how every object store behaves.
Index ¶
- Constants
- Variables
- func As[T any](d *Disk) (T, bool)
- func Batch(ctx context.Context, keys []string, parallel int, ...) error
- func CleanDir(raw string) (string, error)
- func CleanKey(raw string) (string, error)
- func EnvName(prefix, name string) string
- func EscapeKey(key string) string
- func JoinKey(parts ...string) string
- func KeyInDir(key, dir string) bool
- func NotModified(e Entry, opts ReadOptions) bool
- func ObjectSHA256(ctx context.Context, d *Disk, key string) (string, error)
- func ParseAllowedIP(s string) (netip.Prefix, error)
- func ParseSize(s string) (int64, error)
- func RejectParams(u *url.URL, allowed ...string) error
- func RelativeKey(key, dir string) string
- func RetryAfterOf(err error) (time.Duration, bool)
- func SameContent(src, dst Entry) bool
- func TrimCache(ctx context.Context, cache *Disk, maxBytes int64) (int, error)
- func ValidateChecksums(list []Checksum) error
- func ValidatePresign(opts PresignOptions) error
- type Abandoned
- type AbandonedFinder
- type AbandonedKind
- type AbandonedSweeper
- type AppendOptions
- type Appender
- type BatchError
- type ByteRange
- type CacheOptions
- type Call
- type Capabilities
- type Checksum
- type Code
- type ConditionalReader
- type Copier
- type DirMaker
- type Disk
- func CachedRead(name string, remote, cache *Disk, opts CacheOptions) (*Disk, error)
- func Failover(name string, primary *Disk, secondaries []*Disk, opts FailoverOptions) (*Disk, error)
- func Mirror(name string, primary *Disk, mirrors ...*Disk) (*Disk, error)
- func NewDisk(name string, driver Driver, opts ...Option) (*Disk, error)
- func Overlay(name string, upper, lower *Disk, opts ...Option) (*Disk, error)
- func Route(name string, mounts []Mount, opts ...Option) (*Disk, error)
- func (d *Disk) AbortMultipart(ctx context.Context, key, uploadID string) error
- func (d *Disk) Append(ctx context.Context, key string, body io.Reader, opts AppendOptions) (Entry, error)
- func (d *Disk) Capabilities() Capabilities
- func (d *Disk) Check(ctx context.Context) error
- func (d *Disk) CleanKey(key string) (string, error)
- func (d *Disk) Close() error
- func (d *Disk) CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) (Entry, error)
- func (d *Disk) Copy(ctx context.Context, src, dst string) (Entry, error)
- func (d *Disk) CreateMultipart(ctx context.Context, key string, opts WriteOptions) (string, error)
- func (d *Disk) DataURI(ctx context.Context, key string, maxBytes int64) (string, error)
- func (d *Disk) DefaultVisibility() Visibility
- func (d *Disk) Delete(ctx context.Context, key string) error
- func (d *Disk) DeleteKeys(ctx context.Context, keys []string) error
- func (d *Disk) DeletePrefix(ctx context.Context, dir string) error
- func (d *Disk) DirExists(ctx context.Context, dir string) (bool, error)
- func (d *Disk) DriverKey(key string) (string, error)
- func (d *Disk) Exists(ctx context.Context, key string) (bool, error)
- func (d *Disk) FS(ctx context.Context) fs.FS
- func (d *Disk) FindAbandoned(ctx context.Context, before time.Time) ([]Abandoned, error)
- func (d *Disk) Hold(ctx context.Context, key string) (Hold, error)
- func (d *Disk) List(ctx context.Context, dir string, opts ListOptions) iter.Seq2[Entry, error]
- func (d *Disk) ListPage(ctx context.Context, dir string, opts ListOptions) (Page, error)
- func (d *Disk) ListParts(ctx context.Context, key, uploadID string) ([]Part, error)
- func (d *Disk) MakeDir(ctx context.Context, dir string) error
- func (d *Disk) Move(ctx context.Context, src, dst string) (Entry, error)
- func (d *Disk) Name() string
- func (d *Disk) PresignPart(ctx context.Context, key, uploadID string, number int, expires time.Duration) (SignedRequest, error)
- func (d *Disk) PresignPost(ctx context.Context, p PostPolicy) (PresignedPost, error)
- func (d *Disk) PublicURL(ctx context.Context, key string) (string, error)
- func (d *Disk) Read(ctx context.Context, key string) (io.ReadCloser, error)
- func (d *Disk) ReadAll(ctx context.Context, key string) ([]byte, error)
- func (d *Disk) ReadIf(ctx context.Context, key string, opts ReadOptions) (io.ReadCloser, Entry, error)
- func (d *Disk) ReadRange(ctx context.Context, key string, rng ByteRange) (io.ReadCloser, error)
- func (d *Disk) ReadVersion(ctx context.Context, key, id string) (io.ReadCloser, error)
- func (d *Disk) Restore(ctx context.Context, key string, opts RestoreOptions) error
- func (d *Disk) RestoreVersion(ctx context.Context, key, id string) error
- func (d *Disk) Scope(name, prefix string) (*Disk, error)
- func (d *Disk) SetHold(ctx context.Context, key string, h Hold) error
- func (d *Disk) SetModTime(ctx context.Context, key string, t time.Time) error
- func (d *Disk) SetStorageClass(ctx context.Context, key, class string) error
- func (d *Disk) SetTags(ctx context.Context, key string, tags map[string]string) error
- func (d *Disk) SetVisibility(ctx context.Context, key string, v Visibility) error
- func (d *Disk) SignedURL(ctx context.Context, key string, opts PresignOptions) (SignedRequest, error)
- func (d *Disk) Stat(ctx context.Context, key string) (Entry, error)
- func (d *Disk) SweepAbandoned(ctx context.Context, before time.Time) (int, error)
- func (d *Disk) Tags(ctx context.Context, key string) (map[string]string, error)
- func (d *Disk) TempFile(ctx context.Context, key string, opts TempFileOptions) (string, error)
- func (d *Disk) UploadPart(ctx context.Context, key, uploadID string, number int, body io.Reader, ...) (Part, error)
- func (d *Disk) Versions(ctx context.Context, key string) ([]Version, error)
- func (d *Disk) Write(ctx context.Context, key string, body io.Reader, opts WriteOptions) (Entry, error)
- func (d *Disk) WriteBytes(ctx context.Context, key string, b []byte, opts WriteOptions) (Entry, error)
- type Driver
- type Entry
- type Error
- type FailoverEvent
- type FailoverOptions
- type HTTPHeaders
- type HTTPMethod
- type HeaderPinner
- type Hold
- type Holder
- type IPRestricter
- type Interceptor
- type KeyEncoder
- type KeyError
- type KeyEscaper
- type KeyMapper
- type KeysDeleter
- type ListOptions
- type LookupFunc
- type Manager
- type ModTimeSetter
- type Mount
- type Mover
- type MultipartUploader
- type Mux
- type Op
- type OpKind
- type Opener
- type Option
- func WithContentDetector(detect sniff.Detector) Option
- func WithDefaultVisibility(v Visibility) Option
- func WithInterceptors(interceptors ...Interceptor) Option
- func WithKeyNormalizer(fn func(string) (string, error)) Option
- func WithPublicURL(base string) Option
- func WithURLSigner(s URLSigner, methods ...HTTPMethod) Option
- func WithoutContentTypeDetection() Option
- type Page
- type Part
- type PartPresigner
- type PostPolicy
- type PostPresigner
- type PrefixDeleter
- type PresignOptions
- type PresignedPost
- type Presigner
- type PublicURLer
- type ReadOptions
- type RestoreOptions
- type Restorer
- type SignedRequest
- type StorageClasser
- type SyncAction
- type SyncItem
- type SyncOptions
- type SyncReport
- type Tagger
- type TempFileOptions
- type URLSigner
- type Unwrapper
- type Version
- type VersionLister
- type VersionReader
- type Visibility
- type VisibilitySetter
- type WriteOptions
Constants ¶
const ( MinPartNumber = 1 MaxPartNumber = 10000 )
MinPartNumber and MaxPartNumber bound part numbers. They are S3's limits, adopted by every S3-compatible backend and by browser uploaders.
const ( PostFieldKey = "key" PostFieldContentType = "Content-Type" PostFieldCacheControl = HeaderCacheControl PostFieldContentDisposition = HeaderContentDisposition PostFieldContentEncoding = HeaderContentEncoding // PostFileField is the form field holding the file; backends require // it to be the last field. PostFileField = "file" // PostFilenamePlaceholder in a key prefix upload's key field is // replaced by the backend with the uploaded file's name. PostFilenamePlaceholder = "${filename}" )
Form field names of a presigned POST, the HTML-form upload S3 and GCS accept (UP-04). A driver's PresignPost returns its fields under these names where the backend defines them, so a client builds the form the same way for every backend: every field of PresignedPost.Fields, in any order, then the file in PostFileField, last.
const ( HeaderCacheControl = "Cache-Control" HeaderContentDisposition = "Content-Disposition" HeaderContentEncoding = "Content-Encoding" )
The HTTP names of the HTTPHeaders fields.
const ( // ParamPrefix scopes the disk to a key prefix (see Disk.Scope). ParamPrefix = "prefix" // ParamPublicURL sets WithPublicURL. ParamPublicURL = "public_url" // ParamVisibility sets WithDefaultVisibility ("public" or "private"). ParamVisibility = "visibility" // ParamReadOnly ("true" or "false") adds the ReadOnly interceptor. ParamReadOnly = "readonly" )
Query parameters every DSN accepts, handled by the Mux itself.
const ( // EnvDisks lists the disk names, comma-separated ("media,backups"). EnvDisks = "DISKS" // EnvDefaultDisk names the default disk; empty means the first listed. EnvDefaultDisk = "DEFAULT_DISK" // EnvDSNSuffix ends each disk's DSN variable: <PREFIX><NAME>_DSN. EnvDSNSuffix = "_DSN" )
Variable names OpenManager reads, each after the caller's prefix.
const ( Byte int64 = 1 KB int64 = 1000 MB = 1000 * KB GB = 1000 * MB TB = 1000 * GB KiB int64 = 1 << 10 MiB int64 = 1 << 20 GiB int64 = 1 << 30 TiB int64 = 1 << 40 )
Size units for ParseSize. Decimal units are powers of 1000 and binary units powers of 1024, as the SI and IEC define them: "10MB" is 10,000,000 bytes and "10MiB" is 10,485,760. Units are case-sensitive, so "mb" (millibits, to the strict reader) is refused rather than guessed.
const CheckDir = ".filesystem-check"
CheckDir is the directory Check writes its probe object into. It is a fixed, recognisable name so an operator who finds a leftover probe (after a crash mid-check) knows what it is and that it is safe to delete.
const DefaultPageSize = 1000
DefaultPageSize is the page size used when ListOptions.Limit is 0. It is S3's maximum, so one page is one request on the most common backend.
const MaxKeyLength = 1024
MaxKeyLength is the longest key, in bytes, that CleanKey accepts. It is S3's limit, the strictest of the common object stores, so a key that is valid on one disk is valid on every disk.
const MaxPresignExpiry = 7 * 24 * time.Hour
MaxPresignExpiry is the longest lifetime a signed URL may have. It is the S3 SigV4 limit; a longer-lived URL is effectively a public link and should be one, through VisibilityPublic.
const OverlayWhiteouts = ".overlay-whiteouts"
OverlayWhiteouts is the directory of the upper disk where an overlay records deleted lower keys: deleting "a/b.txt" while the lower disk holds it writes an empty "<OverlayWhiteouts>/a/b.txt" on the upper disk. Keys under it, in any letter case, are reserved on an overlay disk and refused with CodeInvalidKey: on an upper disk that folds case, a key such as ".OVERLAY-WHITEOUTS/a.txt" would otherwise forge a whiteout.
Variables ¶
var ( ErrNotFound error = &codeSentinel{CodeNotFound} ErrExists error = &codeSentinel{CodeExists} ErrInvalidKey error = &codeSentinel{CodeInvalidKey} ErrInvalidArgument error = &codeSentinel{CodeInvalidArgument} ErrUnsupported error = &codeSentinel{CodeUnsupported} ErrReadOnly error = &codeSentinel{CodeReadOnly} ErrTooLarge error = &codeSentinel{CodeTooLarge} ErrRejected error = &codeSentinel{CodeRejected} ErrQuotaExceeded error = &codeSentinel{CodeQuotaExceeded} ErrPermission error = &codeSentinel{CodePermission} ErrConflict error = &codeSentinel{CodeConflict} ErrNotModified error = &codeSentinel{CodeNotModified} ErrUnknown error = &codeSentinel{CodeUnknown} )
Sentinels for errors.Is. Each matches every *Error with the same Code.
var AbandonedKindValues = []AbandonedKind{AbandonedMultipart, AbandonedStaging}
AbandonedKindValues lists every AbandonedKind.
var CodeValues = []Code{ CodeNotFound, CodeExists, CodeInvalidKey, CodeInvalidArgument, CodeUnsupported, CodeReadOnly, CodeTooLarge, CodeRejected, CodeQuotaExceeded, CodePermission, CodeUnavailable, CodeConflict, CodeNotModified, CodeUnknown, }
CodeValues lists every Code, for exhaustive tests and documentation.
var ErrSuperseded = Errorf(CodeConflict, "a concurrent attempt delivered the result")
ErrSuperseded is what a concurrent call of a hedgeable operation (see Op.Concurrent) returns when another call already delivered the result: the operation succeeded, through the other call. A hedging interceptor waits for the other call's answer instead of taking this one. Should it reach a caller anyway, its code is CodeConflict.
var HTTPMethodValues = []HTTPMethod{MethodGet, MethodPut}
HTTPMethodValues lists every HTTPMethod.
var OpKindValues = []OpKind{ OpStat, OpRead, OpWrite, OpDelete, OpDeletePrefix, OpDeleteKeys, OpList, OpCopy, OpMove, OpSetVisibility, OpPresign, OpPublicURL, OpMultipartCreate, OpMultipartUpload, OpMultipartPresign, OpMultipartListParts, OpMultipartComplete, OpMultipartAbort, OpSweepAbandoned, OpPresignPost, OpAppend, OpSetModTime, OpTags, OpSetTags, OpHold, OpSetHold, OpVersions, OpRestore, OpSetStorageClass, OpReadVersion, OpRestoreVersion, OpFindAbandoned, OpMakeDir, OpDirExists, }
OpKindValues lists every OpKind.
var SyncActions = []SyncAction{SyncCopy, SyncDelete}
SyncActions lists every SyncAction.
var VisibilityValues = []Visibility{VisibilityPublic, VisibilityPrivate}
VisibilityValues lists every Visibility.
Functions ¶
func As ¶
As is the typed escape hatch to a backend's native client: it returns the value of the first driver in the wrapper chain (following Unwrapper) that has a method Native() T. For example, As[*s3.Client](disk) on an S3 disk. It returns false when no driver offers a T.
func Batch ¶
func Batch(ctx context.Context, keys []string, parallel int, fn func(ctx context.Context, key string) error) error
Batch runs fn for every key with at most parallel calls at a time and collects the failures: nil when every call succeeded, otherwise a *BatchError listing each failed key with its error, in the order of keys. It is the helper behind the Disk's per-key fallbacks, and the shape to use for any per-key work an application fans out (copying a tree, re-encoding every image under a prefix).
A parallel below 1 runs one call at a time. Keys are passed as given, duplicates included. Once ctx ends no further call starts, and every key not yet started fails with the context's error, so the result still names every key that was not processed.
func CleanDir ¶
CleanDir normalises a directory argument (for List and DeletePrefix) the same way as CleanKey, except that the empty string, "/" and "." are valid and mean the root of the disk. The result never has a trailing slash.
func CleanKey ¶
CleanKey normalises a caller-supplied key and rejects unsafe ones. Every Disk method runs it before a driver sees the key, which is what makes path traversal impossible regardless of the driver's own care.
Normalisation: backslashes become slashes, leading slashes are dropped, empty and "." segments are removed, and runs of slashes collapse.
Rejection (CodeInvalidKey): an empty result, any ".." segment (rejected rather than resolved, so "a/../b" is an error and never silently "b"), invalid UTF-8, a control character (including NUL), or more than MaxKeyLength bytes.
func EnvName ¶
EnvName is the variable that holds the DSN of disk name under prefix: the name upper-cased, with "-" and "." as "_" ("STORAGE_" and "media-cdn" give "STORAGE_MEDIA_CDN_DSN").
func EscapeKey ¶
EscapeKey percent-escapes each segment of a clean key for use in a URL path, keeping the slashes between segments.
func JoinKey ¶
JoinKey joins already-clean key parts with the separator, skipping empty parts. It does not clean its input; wrapper drivers use it to prefix keys that the Disk has already cleaned.
func KeyInDir ¶
KeyInDir reports whether key lies under dir (a clean directory as returned by CleanDir, where "" is the root). A key equal to dir is not inside it.
func NotModified ¶
func NotModified(e Entry, opts ReadOptions) bool
NotModified reports whether opts say the caller's copy of the object e describes is current, by the HTTP rules: If-None-Match first (an exact ETag, or "*" for any object), else If-Modified-Since to the second. It is exported for drivers that implement ConditionalReader over a Stat.
func ObjectSHA256 ¶
ObjectSHA256 reads the object at key on d in full and returns the hex SHA-256 of its content: the digest Disk.Write records in Entry.SHA256, computed from the stored bytes rather than trusted from metadata. It is for integrity checks (Sync's Checksum, scrubbing records against the digest they stored), and costs a full read.
func ParseAllowedIP ¶
ParseAllowedIP parses PresignOptions.AllowedIP: a CIDR prefix in canonical form. Drivers and signers that enforce the restriction parse it with this one function, so every backend accepts and refuses the same values. An IPv4-mapped IPv6 prefix (::ffff:203.0.113.0/120) is returned as the IPv4 prefix it covers (203.0.113.0/24), because client addresses are compared unmapped and the mapped form would match none of them; one shorter than /96 also covers non-IPv4 addresses and is refused.
func ParseSize ¶
ParseSize reads a human size such as "10MB", "1.5GiB", "512KiB", "64B" or "1024" (bytes) into a byte count, for configuration values like an upload limit. It is strict: the number is a plain decimal (digits, at most one ".", no sign, exponent, separator or space), the unit is one of B, KB, MB, GB, TB (powers of 1000) or KiB, MiB, GiB, TiB (powers of 1024) written exactly so, nothing may follow it, and the result must be a whole number of bytes that fits an int64 ("1.5B" is refused). Anything else is CodeInvalidArgument, never a guess.
func RejectParams ¶
RejectParams returns CodeInvalidArgument naming the first query parameter of u that is not in allowed. Openers call it to refuse typos.
func RelativeKey ¶
RelativeKey returns key with the dir prefix removed. Callers must have checked KeyInDir first; for a key outside dir it returns key unchanged.
func RetryAfterOf ¶
RetryAfterOf returns the wait the backend asked for, from the outermost *Error in err's chain that carries one, and false when none does.
func SameContent ¶
SameContent reports whether dst holds src's content as far as the listings can tell: equal sizes, and equal SHA-256 digests when both have one, otherwise equal ETags when both have one. Anything less counts as different, so an object whose content cannot be compared is copied again rather than wrongly skipped. ETags are compared as strings: two backends that compute them differently (an S3 multipart ETag and a local one) never match, which costs a copy, never a skip.
func TrimCache ¶
TrimCache deletes the oldest copies on cache (by modification time, so by when they were copied or last revalidated) until the bytes it holds are at most maxBytes, and returns how many it deleted. Run it from a job (jobs.Func) to bound a CachedRead cache; deleting a copy only costs a fetch from remote on its next read. It lists the whole cache disk, so point it only at a disk used as a cache.
func ValidateChecksums ¶
ValidateChecksums checks list the way Disk.Write does before it reads a byte: every Name is 1 to 32 lowercase letters, digits or dashes and unique, every New is set, and every Want is hex of New's size. Layers that compute the same digests themselves (upload.Confirm reading an assembled object) call it so a bad list fails the same way on every path, with CodeInvalidArgument.
func ValidatePresign ¶
func ValidatePresign(opts PresignOptions) error
ValidatePresign checks PresignOptions: a known method, an expiry in (0, MaxPresignExpiry], Headers only on a PUT and free of CR and LF, AllowedIP empty or a canonical CIDR prefix (ParseAllowedIP), and Visibility empty or a known value on a PUT. Signers call it too, so the rule is enforced the same way on every path.
Types ¶
type Abandoned ¶
type Abandoned struct {
Kind AbandonedKind
// ID is the multipart upload ID, or the staging file's name inside
// the driver's staging directory.
ID string
// Key is the key a multipart upload was started for, as the backend
// driver received it (a scoped disk's prefix included). It is empty
// for a staging file, and for an upload whose record no longer names
// its key.
Key string
// LastActivity is the time the sweep compares with its cutoff: the
// last part received, or, where the backend records only that, when
// the upload started (S3, Azure); a staging file's last write.
LastActivity time.Time
}
Abandoned is one leftover a sweep would remove, as FindAbandoned reports it.
type AbandonedFinder ¶
type AbandonedFinder interface {
FindAbandoned(ctx context.Context, before time.Time) ([]Abandoned, error)
}
AbandonedFinder is the dry run of AbandonedSweeper: FindAbandoned returns what SweepAbandoned(ctx, before) would remove at this moment, applying exactly the same rules (the same cutoff, the same names it leaves alone), and changes nothing. Work that starts, completes or is aborted between the two calls makes them differ, so a sweep's count can be lower or higher than the preview's length. A driver implements both from one scan, so the two cannot drift apart.
type AbandonedKind ¶
type AbandonedKind string
AbandonedKind names what kind of leftover an Abandoned is.
const ( // AbandonedMultipart is a multipart upload never completed or aborted; // sweeping it aborts it and frees its parts. AbandonedMultipart AbandonedKind = "multipart" // AbandonedStaging is a staging file of a write interrupted before // it was renamed into place (local, sftp, ftp, webdav). AbandonedStaging AbandonedKind = "staging" )
The closed set of leftovers a sweep removes.
type AbandonedSweeper ¶
type AbandonedSweeper interface {
SweepAbandoned(ctx context.Context, before time.Time) (int, error)
}
AbandonedSweeper removes what unfinished work left behind in the backend: multipart uploads that were never completed or aborted, and any staging files of writes that were interrupted. Those cost storage and are invisible to List, so nothing else ever removes them.
Work counts as abandoned when its last activity (a part received, a staging file written) is before before; a backend that records only when an upload started uses that. Work at or after before is left alone, so a caller must choose before well beyond the longest upload it allows; a sweep with a recent before aborts uploads in progress. Stored objects are never touched. It returns how many uploads and staging files it removed; a failure part-way returns the count so far with the error, and a rerun continues.
type AppendOptions ¶
type AppendOptions struct {
// Visibility of an object Append creates; empty means the disk's
// default. An existing object keeps its own.
Visibility Visibility
// MaxSize, when positive, caps the object's size after the append: an
// append that would take it past MaxSize is refused with CodeTooLarge
// and appends nothing. A log or export then cannot grow without bound
// however many appends arrive.
MaxSize int64
}
AppendOptions control an append.
type Appender ¶
type Appender interface {
Append(ctx context.Context, key string, body io.Reader, opts AppendOptions) (Entry, error)
}
Appender adds bytes to the end of an object, creating it when missing. Contract, checked by the conformance suite: a body that fails to read appends nothing (the object is as it was), two appends to one key never interleave their bytes, and an append that would take the object past a positive opts.MaxSize is refused with CodeTooLarge without appending (the check and the append are one step, so concurrent appends cannot overshoot together).
type BatchError ¶
type BatchError struct {
Failures []KeyError
}
BatchError reports the keys a batch operation (Disk.DeleteKeys) could not process; every other key succeeded. errors.Is and CodeOf see the per-key errors, so errors.Is(err, ErrPermission) asks whether any key failed for lack of permission.
func (*BatchError) Error ¶
func (e *BatchError) Error() string
Error names the failure count and the first failure. A BatchError with no failures is a caller's bug (a driver built one by hand), but Error must not panic inside a log line, so it says so instead.
func (*BatchError) Unwrap ¶
func (e *BatchError) Unwrap() []error
Unwrap returns the per-key errors.
type ByteRange ¶
ByteRange selects part of an object for Read. Offset is the first byte; Length is the number of bytes, where 0 means "to the end of the object" (reading zero bytes is never useful, so the zero value reads everything from Offset). A range past the end of the object yields an empty read, not an error.
type CacheOptions ¶
type CacheOptions struct {
// TTL is how long a copy is served without asking remote. Required,
// positive.
TTL time.Duration
// OnCacheError, when set, is told of every cache failure the disk
// absorbs: a copy that could not be made, read, refreshed or dropped.
// No such failure fails the call, so a copy that could not be dropped
// after a write stays stale until TTL passes; this hook is how the
// application finds out (log it, count it, alert on it).
OnCacheError func(ctx context.Context, key string, err error)
// Now is the clock a copy's age is judged by against TTL, and the time
// a revalidated copy is marked fresh at; nil means time.Now. A fake
// clock makes TTL expiry testable without sleeping. A copy's age is
// Now minus its ModTime on cache, so cache's clock and this one must
// agree.
Now func() time.Time
}
CacheOptions configure CachedRead.
type Capabilities ¶
type Capabilities struct {
// Interface-backed capabilities.
Copy bool // implements Copier: server-side copy
Move bool // implements Mover: server-side or atomic rename
SetVisibility bool // implements VisibilitySetter; Stat reports Visibility
DeletePrefix bool // implements PrefixDeleter: bulk delete in few calls
DeleteKeys bool // implements KeysDeleter: many keys deleted in few calls
Presign bool // implements Presigner: backend-signed URLs
PublicURL bool // implements PublicURLer
Multipart bool // implements MultipartUploader: parts uploaded through the server
PresignPart bool // implements PartPresigner: parts uploaded directly by clients
PresignPost bool // implements PostPresigner: browser form uploads the backend checks
// SweepAbandoned: implements AbandonedSweeper, removing what unfinished
// work left in the backend (multipart uploads never completed or
// aborted, staging files of interrupted writes). Wrapper drivers that
// narrow a disk to part of a backend (scoped, mirror) never forward it,
// because the leftovers are backend-wide.
SweepAbandoned bool
// FindAbandoned: implements AbandonedFinder, listing what
// SweepAbandoned would remove without removing it (a dry run). Needs
// SweepAbandoned, and like it is never forwarded by a wrapper driver.
FindAbandoned bool
// Dirs: implements DirMaker. The backend has real directories, so
// MakeDir creates an empty one that DirExists then reports. Without
// it, MakeDir does nothing and a directory exists exactly while a key
// is under it (see Disk.MakeDir).
Dirs bool
Append bool // implements Appender: add bytes to the end of an object
SetModTime bool // implements ModTimeSetter: set an object's modification time
// ConditionalRead: implements ConditionalReader, answering
// If-None-Match and If-Modified-Since in the backend; without it
// Disk.ReadIf compares against a Stat.
ConditionalRead bool
Tags bool // implements Tagger: key-value tags stored with an object
Hold bool // implements Holder: legal holds and retention dates
Versions bool // implements VersionLister: an object's stored versions
// VersionRead: implements VersionReader, reading one stored version and
// making it the current content again. A separate bit from Versions, so
// a backend that can only list versions stays honest. Needs Versions,
// which is where version IDs come from.
VersionRead bool
StorageClass bool // implements StorageClasser: change an object's storage class
Restore bool // implements Restorer: bring an archived object back online
// Attribute capabilities, with no interface of their own.
Metadata bool // stores WriteOptions.Metadata and returns it from Stat
ContentType bool // stores WriteOptions.ContentType and returns it from Stat
Headers bool // stores WriteOptions.Headers and returns them from Stat
// PresignHeaders: the driver's presigned PUT (Presign) pins
// PresignOptions.Headers, so the backend refuses an upload without them
// and stores them. Needs Presign and Headers.
PresignHeaders bool
// PresignIP: the driver's presigned URLs (Presign) carry
// PresignOptions.AllowedIP, so the backend refuses a request from an
// address outside it. Needs Presign.
PresignIP bool
// PresignVisibility: the driver's presigned PUT (Presign) and POST
// form (PresignPost), whichever it declares, pin
// PresignOptions.Visibility and PostPolicy.Visibility, so the backend
// stores the upload with that visibility and refuses a request that
// changes it. Needs SetVisibility and one of Presign or PresignPost;
// a driver whose backend has no per-object ACL (S3 with bucket owner
// enforced, GCS with uniform access) does not declare it.
PresignVisibility bool
ConditionalWrite bool // honours WriteOptions.IfNotExists atomically
// ChecksumSHA256: the driver sends WriteOptions.SHA256 to the backend,
// which refuses a body that does not match it. The Disk verifies the
// digest itself on every driver; this adds the backend's own check.
ChecksumSHA256 bool
// CompareAndSwap: honours WriteOptions.IfMatch atomically, replacing an
// object only while its ETag is still the one the caller read.
CompareAndSwap bool
// CaseInsensitive: keys differing only in letter case address one
// object ("A.txt" and "a.txt"). Keys are case-sensitive on every other
// backend; a driver over a case-insensitive filesystem (macOS, Windows
// by default) reports it so callers can refuse such pairs themselves.
CaseInsensitive bool
}
Capabilities declares what a driver can do beyond the Driver interface. Interface-backed fields must match the optional interfaces the driver implements; NewDisk checks this. The Disk calls an optional interface only when its field is true, which is what lets a wrapper driver implement every interface yet report only what the driver underneath supports.
func (Capabilities) String ¶
func (c Capabilities) String() string
String renders the true capabilities, for logs and diagnostics.
type Checksum ¶
type Checksum struct {
// Name labels the digest in Entry.Checksums ("md5", "crc32c"):
// lowercase letters, digits and dashes, 1 to 32 characters, unique
// within one write.
Name string
// New returns a fresh hasher; it is called once per write attempt, so
// a retried write starts from zero. Required.
New func() hash.Hash
// Want, when set, is the hex digest the body must have (any case).
// A body that ends with another one fails the write with
// CodeInvalidArgument before the driver commits, exactly as
// WriteOptions.SHA256 does, so a corrupted body never replaces what
// was stored. Its length must match New's Size.
Want string
}
Checksum asks Disk.Write for one more digest of the body, computed while it streams, beside the SHA-256 it always computes (Entry.SHA256), and optionally verified: MD5 for an existing checksum column, CRC32C to compare with what GCS reports, BLAKE3 or xxHash for fast content addressing. The library names no algorithm and imports no hash package for it: the caller brings the hash.Hash.
Why a list of hashers rather than a fixed set: every application keeps the digest its other systems already use, and adding one should not need a release of this library. SHA-256 stays the one the Disk always computes, so Entry.SHA256, upload.Result.SHA256 and the records schema keep their meaning.
type Code ¶
type Code string
Code classifies an error independently of the backend that produced it. Callers branch on the Code (with errors.Is against the Err* sentinels, or with CodeOf) and never on a vendor error type, which is what lets an application swap S3 for local disk without touching its error handling.
const ( // CodeNotFound: the key does not exist (or the directory has no keys). CodeNotFound Code = "not_found" // CodeExists: a conditional write (WriteOptions.IfNotExists) found the // key already present, or a registry already holds the name. CodeExists Code = "exists" // CodeInvalidKey: the key is empty, too long, not valid UTF-8, contains a // control character, or contains a ".." segment. CodeInvalidKey Code = "invalid_key" // CodeInvalidArgument: an option or argument other than the key is // invalid (an unknown visibility, a non-positive expiry, …). CodeInvalidArgument Code = "invalid_argument" // CodeUnsupported: the driver does not have the capability the call // needs, and there is no portable fallback. CodeUnsupported Code = "unsupported" // CodeReadOnly: a mutating operation reached a read-only disk. CodeReadOnly Code = "read_only" // CodeTooLarge: a write exceeded WriteOptions.MaxSize. Nothing is stored. CodeTooLarge Code = "too_large" // CodeRejected: the content was refused by a policy, for example an // upload whose detected type is not allowed. Nothing is stored. CodeRejected Code = "rejected" // CodeQuotaExceeded: the owner of the scope has no storage left (see // records/quota). Distinct from CodeTooLarge, which is about one file. CodeQuotaExceeded Code = "quota_exceeded" // CodePermission: the backend refused the operation (credentials, ACL, // file mode). CodePermission Code = "permission" // reset). The only code the retry interceptor retries. A context that // ended (context.DeadlineExceeded or context.Canceled anywhere in the // chain) is classified here too, so a per-attempt deadline from // layer.Timeout is retried; layer.Retry still stops once the caller's // own context has ended. CodeUnavailable Code = "unavailable" // CodeConflict: a compare-and-swap write (WriteOptions.IfMatch) found // the object changed, or gone, since the caller read it. Nothing is // written; read it again and retry. CodeConflict Code = "conflict" // CodeNotModified: a conditional read (Disk.ReadIf) found the object // unchanged, so no body is returned. Not a failure: the caller's copy // is current. CodeNotModified Code = "not_modified" // CodeUnknown: an error the driver could not classify. CodeUnknown Code = "unknown" )
The closed set of error codes. Every driver maps its native errors into these; an error it cannot classify becomes CodeUnknown, never a new code.
func CodeOf ¶
CodeOf returns the Code of err, classified exactly as the Disk classifies a driver's error: the Code of the outermost *Error in its chain (an unknown one becomes CodeUnavailable when the chain holds a context error), the code of a bare sentinel, CodeNotFound, CodeExists or CodePermission for the io/fs sentinels, CodeUnavailable for a context that ended, or CodeUnknown for any other non-nil error. It returns "" for a nil error.
Interceptors see a driver's error before the Disk annotates it, so this is what lets layer.Retry recognise a bare context.DeadlineExceeded from an attempt that layer.Timeout cut short.
type ConditionalReader ¶
type ConditionalReader interface {
ReadIf(ctx context.Context, key string, opts ReadOptions) (io.ReadCloser, Entry, error)
}
ConditionalReader answers a conditional read in the backend (S3, GCS and Azure send the conditions with the GET), returning CodeNotModified and no body when the condition holds.
type Copier ¶
Copier copies an object inside the backend without streaming it through the application. dst is overwritten if present.
type DirMaker ¶
type DirMaker interface {
MakeDir(ctx context.Context, dir string) error
DirExists(ctx context.Context, dir string) (bool, error)
}
DirMaker is implemented by backends with real directories (local disk, SFTP, FTP, WebDAV). Directories arrive cleaned and never empty (the root always exists).
MakeDir creates dir and any missing parents, and succeeds when dir already exists; when an object already has dir's key it fails with CodeExists. DirExists reports whether dir exists as a directory, empty or not; an object at dir's key is not a directory.
type Disk ¶
type Disk struct {
// contains filtered or unexported fields
}
Disk is the handle applications use for one storage location. It is safe for concurrent use. Construct it with NewDisk; the zero value is not usable.
Every method cleans its key with CleanKey before the driver sees it, runs the driver call through the interceptor chain as one Op, and returns errors as *Error with the operation, disk name and key filled in.
func CachedRead ¶
func CachedRead(name string, remote, cache *Disk, opts CacheOptions) (*Disk, error)
CachedRead returns a Disk named name over remote whose reads are served from a copy on cache, typically a local disk in front of a slow or metered bucket. A read of a key not on cache copies the whole object there first, then serves the read (ranges included) from the copy; a copy younger than opts.TTL is served without asking remote. An older copy is revalidated with a Stat of remote: when cache stores metadata and remote reports the same ETag it was copied with, the copy is served (and, where cache can set modification times, made fresh again); otherwise it is copied anew. Writes, deletes, copies, moves, appends and completed multipart uploads go to remote and then drop cache's copy.
Semantics, which callers must accept:
- Freshness is bounded by TTL, not exact: a copy made while another process writes the key may be stale until it is revalidated. Use it for content that changes rarely or under new keys.
- Stat, List, URLs and every other capability are remote's and are never cached. Signed PUT URLs and form uploads are refused or not offered (an upload straight to remote would leave a stale copy).
- A conditional read (ReadIf) asks remote for the current ETag and serves the copy only when it was made from that ETag, else reads remote, so its entry and body always match.
- A cache that fails (full, unavailable) never fails a call: reads are served from remote, writes and deletes return remote's result, and OnCacheError is told.
- cache grows with every key read; bound it with TrimCache, run as a job.
cache must be a disk of its own: keys on it are remote's keys, and TrimCache deletes from it.
func Failover ¶
Failover returns a Disk named name that reads (Stat and Read) from primary and, when primary fails as ShouldFailover decides, from each secondary in turn, so reads survive an outage of the primary backend. Everything else (writes, deletes, listings, URLs, every capability) goes to primary alone.
Semantics, which callers must accept:
- Secondaries are only read. Keeping them in step is the application's job, typically with Mirror (write to all) or a backfill; a read that fails over returns whatever the secondary holds, which may be older than the primary's copy or missing (CodeNotFound).
- Listings are not failed over: a cursor from one backend means nothing to another.
- The disk has primary's capabilities and interceptors, except ConditionalRead, so conditional reads run as a Stat and a Read that both fail over, and SweepAbandoned and FindAbandoned (sweep the primary directly). A secondary runs its own interceptors.
- When every disk fails, the last secondary's error is returned.
func Mirror ¶
Mirror returns a Disk named name that reads from primary and writes to primary and then to every mirror. It is how an application moves to a new backend without downtime (mirror old to new, backfill, then switch the primary) or keeps a warm copy.
Semantics, which callers must accept:
- Not atomic across disks. A write succeeds on primary first; if a mirror then fails, the error is returned (CodeUnavailable wrapping the cause) and the object exists on primary only. Writes are overwrites, so retrying converges.
- Mirrors receive the bytes by reading the object back from primary, because the caller's body can be consumed only once.
- A mirror that folds letter case where the primary does not (local disk on macOS behind a memory or S3 primary) holds one object for keys that differ only in case; the primary holds two. Compare Capabilities().CaseInsensitive before mixing them.
- When the primary's driver signs its own URLs (S3, GCS, Azure), signed PUT URLs and presigned parts are refused: a client uploading straight to the backend would bypass the mirrors. A primary that cannot sign uses the disk's URLSigner (WithURLSigner), and those PUT URLs are issued: they name the mirror disk and are served by the application (fshttp.Signed), so the upload is written through the mirror as long as the application resolves that name to the mirror disk. Server-side multipart is allowed and replicated on completion.
- The mirror disk's capabilities are primary's, minus PresignPart, PresignPost, PresignHeaders, PresignVisibility (signed PUTs are refused, as above), Append, SetModTime, ConditionalRead, CompareAndSwap, Tags, Hold, Versions, VersionRead, StorageClass, Restore (in-place changes would skip the mirrors) and SweepAbandoned and FindAbandoned (each disk keeps its own leftovers; sweep them directly). MakeDir creates the directory on the primary, then on every mirror.
func NewDisk ¶
NewDisk builds a Disk over driver. name identifies the disk in errors, logs and signed URLs, and must match [A-Za-z0-9][A-Za-z0-9._-]{0,63}.
It fails with CodeInvalidArgument when the driver declares a capability it does not implement (see Capabilities), so a misdeclared driver is caught at start-up rather than at the first call.
func Overlay ¶
Overlay returns a Disk named name that layers upper over lower, copy on write: reads find a key on upper first and fall through to lower, writes go to upper only, and deleting a key lower holds leaves a whiteout on upper that hides it. lower is never changed through the overlay. It is how a preview environment, a test or a tenant sandbox starts from shared read-only data and keeps its changes to itself. opts configure the overlay disk itself; upper and lower keep their own interceptors.
Semantics, which callers must accept:
- Listings merge both disks in key order, upper winning a key both hold, and hide whited-out keys and directories whose lower keys are all whited out (checking such a directory costs one more listing).
- Not atomic across the two disks: a delete removes upper's copy, then writes the whiteout; if that fails, lower's copy shows again and the error is returned, and retrying converges.
- The overlay declares only what it honours on both layers: the write attributes upper stores (Metadata, ContentType, Headers, ChecksumSHA256) and CaseInsensitive when either folds case. Copy and Move stream through the Disk (a move of a lower key copies it up and whites it out), DeletePrefix and DeleteKeys go key by key, and the rest (URLs, visibility, conditional and in-place writes, multipart) is not offered: an object can sit on either layer.
func Route ¶
Route returns a Disk named name that dispatches each key to the mount with the longest Prefix containing it, so one handle can put "avatars/…" on a CDN-backed bucket, "exports/…" on cold storage and everything else on a default disk. opts configure the routed disk itself (interceptors around every routed call, a URL signer, …); each mounted disk keeps its own.
Semantics, which callers must accept:
- A key under no prefix, with no catch-all mount, does not exist: Stat and Read report CodeNotFound, Delete succeeds, and Write fails with CodeInvalidArgument.
- Keys of the catch-all under a mounted prefix are shadowed: unreachable and not listed, as with a filesystem mount.
- Copy and Move inside one mount use that disk (server-side where it can); across mounts they stream (CopyAcross, MoveAcross), and a move across mounts is not atomic.
- Listing a directory above mounts (the root, or "a" over a mount at "a/b") merges the catch-all's entries with the mounts, shown as directories, or with their keys when recursive.
- The routed disk declares a capability only when every mount has it: Copy, Move, SetVisibility, Presign (the mount can sign both GET and PUT, each through its driver or its URLSigner as WithURLSigner routes that method; signer URLs name the mount), PublicURL (the mount's driver or a WithPublicURL base), PresignHeaders (whichever of the two signs the mount's PUTs can pin headers), PresignIP (whichever signs each method enforces it), PresignVisibility (every mount's PUTs go to a driver that pins it), and the write attributes (Metadata, ContentType, Headers, ConditionalWrite, ChecksumSHA256, CompareAndSwap). DeleteKeys is always declared, each mount deleting its share its own way; CaseInsensitive when any mount folds case. Multipart, form uploads, appends, tags, holds, versions, version reads, storage classes, restores, conditional reads, modification times, bulk DeletePrefix, SweepAbandoned, FindAbandoned and real directories (Dirs) are not routed: use the mounted disk for those (the Disk falls back for DeletePrefix, conditional reads and DirExists, and MakeDir does nothing).
func (*Disk) AbortMultipart ¶
AbortMultipart discards an unfinished upload and its parts. Aborting an unknown upload succeeds, so it is safe in cleanup paths.
func (*Disk) Append ¶
func (d *Disk) Append(ctx context.Context, key string, body io.Reader, opts AppendOptions) (Entry, error)
Append adds body to the end of key, creating the object (with opts.Visibility, or the disk's default) when it does not exist, and returns the object's entry afterwards: logs, audit trails, chunked exports. With opts.MaxSize the object never grows past it: a body that alone exceeds it is cut off as it streams, and the driver refuses an append that would exceed it, both with CodeTooLarge and nothing appended. Needs the Append capability (local, memory, SFTP, Azure append blobs); object stores without appends answer CodeUnsupported. The body is a stream, so an append is never retried. Size and hash in the entry are the whole object's as the driver reports them, not the appended part's.
func (*Disk) Capabilities ¶
func (d *Disk) Capabilities() Capabilities
Capabilities returns what the disk's driver can do. Methods backed by a missing capability either fall back (Copy, Move, DeletePrefix) or return CodeUnsupported; this lets callers check up front.
func (*Disk) Check ¶
Check proves the disk works end to end: it writes a small random object, reads it back, compares the bytes and deletes it. Run it at start-up so a wrong credential or a missing bucket fails the deploy instead of the first user upload. A read-only disk cannot be checked this way and reports CodeReadOnly.
func (*Disk) CleanKey ¶
CleanKey returns key as this disk addresses it: passed through the key normalizer (WithKeyNormalizer), if any, then cleaned and checked like the package-level CleanKey. Code that compares keys it built with keys the disk returns (List, Stat) should clean them with this method.
func (*Disk) Close ¶
Close releases the driver's resources when it holds any (it implements io.Closer); otherwise it does nothing.
func (*Disk) CompleteMultipart ¶
func (d *Disk) CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) (Entry, error)
CompleteMultipart assembles the parts into the final object. parts must list every part with the ETag the backend returned for it.
func (*Disk) Copy ¶
Copy copies src to dst on this disk, replacing dst. With the Copy capability the backend copies server-side; otherwise the object streams through the application as a Read followed by a Write, keeping its content type, metadata and visibility where the driver stores them.
func (*Disk) CreateMultipart ¶
CreateMultipart starts a multipart upload of key and returns its upload ID. Metadata, content type and visibility are taken from opts and checked against the capabilities as for Write. Needs the Multipart capability.
func (*Disk) DataURI ¶
DataURI returns key as an RFC 2397 data URI, "data:<type>;base64,<bytes>", for inlining a tiny file (an icon, a placeholder image) into HTML, CSS or JSON without a second request.
It refuses, before reading a byte, anything Stat reports larger than maxBytes (CodeTooLarge) and stored active content (HTML, SVG, XML, JavaScript; CodeRejected), because a data URI of an active type runs script wherever it is embedded. The read itself stops at maxBytes + 1, since a backend's reported size can be stale, and the bytes are sniffed: content that is active whatever its stored type is refused too. The type is the stored one (only its charset parameter is kept), else the sniffed one. maxBytes must be positive (CodeInvalidArgument); the caller chooses it, since what counts as tiny depends on where the URI is embedded, and the base64 output is a third larger than the file.
func (*Disk) DefaultVisibility ¶
func (d *Disk) DefaultVisibility() Visibility
DefaultVisibility returns the visibility writes get when they do not choose one. For a driver without per-object visibility it is the visibility of every object on the disk.
func (*Disk) Delete ¶
Delete removes key. Deleting a missing key succeeds, so deletes are idempotent and safe to retry.
func (*Disk) DeleteKeys ¶
DeleteKeys deletes many keys: in few calls with the DeleteKeys capability (S3 deletes 1000 per request), otherwise one Delete per key. Missing keys succeed, as with Delete. Keys that could not be deleted, invalid ones included, come back in a *BatchError while every other key is deleted, so one bad key never stops a cleanup; nil means all are gone. Duplicate keys are deleted once.
func (*Disk) DeletePrefix ¶
DeletePrefix deletes every key under dir. With the DeletePrefix capability the backend does it in bulk; otherwise the Disk lists and deletes key by key. dir must not be the root: emptying a whole disk is never a prefix operation, so it is refused with CodeInvalidArgument.
func (*Disk) DirExists ¶
DirExists reports whether the directory dir exists. dir is cleaned like a key; the root always exists. An object at dir's key is not a directory.
With the Dirs capability the backend answers, so an empty directory made by MakeDir counts. Without it a directory exists exactly while a key is under it, and DirExists lists dir with a limit of one entry (one OpList).
func (*Disk) DriverKey ¶
DriverKey returns the key the backend driver receives for key: the key cleaned as the Disk cleans it, then rewritten by every key-changing wrapper on the way down (KeyMapper, following Unwrapper). On a disk scoped to "tenant", DriverKey("a.png") is "tenant/a.png". It exists so options a backend binds to one exact key can name that key without guessing at prefixes. It does not follow wrappers that send a key to one of several disks (Route, Overlay), which implement neither interface; there it returns the key at that level.
func (*Disk) Exists ¶
Exists reports whether key exists. A CodeNotFound error is false, nil; any other error is returned.
func (*Disk) FS ¶
FS returns a read-only io/fs view of the disk, so code written against the standard interfaces (html/template.ParseFS, http.FileServerFS, fs.WalkDir, testing/fstest) works on any backend. It implements fs.FS, fs.StatFS, fs.ReadDirFS and fs.ReadFileFS; fs.Sub gives a view of a directory.
io/fs has no context, so ctx governs every call made through the view: cancel it to stop them. Every call goes through the Disk, so keys are cleaned and interceptors run as for any other read.
Names are io/fs paths (fs.ValidPath: slash-separated, no "." or ".." elements, "." for the root). A name the Disk would clean into a different key (a backslash, a control character) does not exist in the view, so two names never reach one object. Directories are implied by keys, as in Disk.List: a directory exists while some key is below it, and the root always exists. Files report mode 0444 and directories fs.ModeDir|0555; FileInfo.Sys returns the Entry.
func (*Disk) FindAbandoned ¶
FindAbandoned is the dry run of SweepAbandoned: it returns what SweepAbandoned(ctx, before) would remove now, under the same rules, and removes nothing, so a maintenance job can report before it acts. The result is ordered by LastActivity, oldest first, then ID. Work that starts, completes or is aborted between the preview and the sweep makes them differ (see AbandonedFinder).
Needs the FindAbandoned capability; without it the call fails with CodeUnsupported. Like SweepAbandoned, a scoped or mirrored disk never has it: ask the disk it was derived from.
func (*Disk) List ¶
List iterates over every entry under dir, fetching pages lazily; each page is one intercepted (and so retryable) driver call. Breaking out of the loop stops fetching. A failure is yielded once as (Entry{}, err) and ends the iteration.
for e, err := range disk.List(ctx, "avatars", filesystem.ListOptions{Recursive: true}) {
if err != nil { return err }
…
}
func (*Disk) ListPage ¶
ListPage returns one page of the entries under dir (see ListOptions). dir is cleaned like a key, except that "" means the root.
func (*Disk) ListParts ¶
ListParts lists the parts uploaded so far, in part-number order, so a client can resume an interrupted upload.
func (*Disk) MakeDir ¶
MakeDir creates the directory dir (and any missing parents), succeeding when it already exists. dir is cleaned like a key; "" (the root) always exists, so it does nothing.
With the Dirs capability (local disk, and backends with real directories) the backend creates it: an empty directory DirExists reports until it is deleted. An object store has no directories, only keys that share a prefix, so without the capability MakeDir checks dir and does nothing else: no marker object is written, the directory appears when a key is written under it, and DirExists reports false until then. That is deliberate: a marker would be an object that List, DeletePrefix, quotas, mirrors and the records layer would all see, and that other tools reading the bucket would take for a file. Code that must see an empty directory checks Capabilities().Dirs.
Either way the call runs as one OpMakeDir, a mutation, so a read-only disk refuses it on every backend.
func (*Disk) Move ¶
Move renames src to dst on this disk, replacing dst. With the Move capability it is a backend rename; otherwise it is Copy then Delete of src, which is not atomic: a failure between the two leaves both keys.
func (*Disk) PresignPart ¶
func (d *Disk) PresignPart(ctx context.Context, key, uploadID string, number int, expires time.Duration) (SignedRequest, error)
PresignPart returns a signed PUT for one part, for clients that upload parts directly to the backend. Needs the PresignPart capability.
func (*Disk) PresignPost ¶
func (d *Disk) PresignPost(ctx context.Context, p PostPolicy) (PresignedPost, error)
PresignPost returns a form that lets a client upload straight to the backend under p, with every condition enforced by the backend. It needs the PresignPost capability (S3 and GCS); without it, CodeUnsupported. It counts as a mutation, so a read-only disk refuses it. The upload is still the client's: confirm it (upload.Confirm) before trusting it.
func (*Disk) PublicURL ¶
PublicURL returns a URL anyone can use to read key. With WithPublicURL it is the configured base plus the escaped key (no backend call); otherwise it needs the driver's PublicURL capability. It does not check that the object exists or is public: a URL for a private object simply fails when fetched.
func (*Disk) ReadIf ¶
func (d *Disk) ReadIf(ctx context.Context, key string, opts ReadOptions) (io.ReadCloser, Entry, error)
ReadIf reads key unless the caller's copy is current: it answers CodeNotModified, with no body, when opts.IfNoneMatch matches the object's ETag, or (with no IfNoneMatch) when the object was not modified after opts.IfModifiedSince. It returns the object's entry with the body, for the ETag and time the caller keeps for its next read. Drivers with the ConditionalRead capability decide in the backend; on others the Disk compares against a Stat before reading, which a concurrent overwrite between the two can outrun (the body is then newer than the entry). An object without an ETag never matches IfNoneMatch.
func (*Disk) ReadRange ¶
ReadRange opens part of key for reading (see ByteRange). The caller must close the reader. Offset and Length must not be negative.
func (*Disk) ReadVersion ¶
ReadVersion opens version id of key, an ID Versions reported. The caller must close the reader. An empty id is CodeInvalidArgument. Needs the VersionRead capability.
func (*Disk) Restore ¶
Restore starts bringing archived key back online for opts.Days. Needs the Restore capability; Days must be positive (CodeInvalidArgument).
func (*Disk) RestoreVersion ¶
RestoreVersion makes version id of key its current content again, as a new latest version; no version is lost. An empty id is CodeInvalidArgument. Needs the VersionRead capability.
func (*Disk) Scope ¶
Scope returns a Disk named name that sees only the keys under prefix on d: key "a.png" on the scoped disk is "<prefix>/a.png" on d. It shares d's driver, interceptors, URL signer and settings; a public URL base, if set, gains the prefix. Typical use is one scoped disk per tenant or feature.
The scoped disk needs its own name because signed URLs and logs identify a disk by name, and its keys differ from d's.
func (*Disk) SetHold ¶
SetHold sets key's hold: a legal hold, a retention date, both, or (the zero Hold) neither, where the backend allows lifting them. Needs the Hold capability.
func (*Disk) SetModTime ¶
SetModTime sets key's modification time to t, which Stat then reports (to the backend's precision, a second at worst). Needs the SetModTime capability (local, memory, SFTP); object stores, which stamp the upload time themselves, answer CodeUnsupported. A zero t is CodeInvalidArgument.
func (*Disk) SetStorageClass ¶
SetStorageClass moves key to storage class class, a name of the backend's own (S3 "STANDARD_IA", GCS "NEARLINE", Azure "Cool"); Stat then reports it in Entry.StorageClass. An empty class is CodeInvalidArgument; a class the backend does not know comes back from the driver. Needs the StorageClass capability.
func (*Disk) SetTags ¶
SetTags replaces key's tags with tags (nil or empty removes them all). A tag with an empty key, or a key or value holding a control character, is CodeInvalidArgument; backend limits (S3 allows ten tags) come back from the driver. Needs the Tags capability.
func (*Disk) SetVisibility ¶
SetVisibility changes the visibility of an existing key. It needs the SetVisibility capability.
func (*Disk) SignedURL ¶
func (d *Disk) SignedURL(ctx context.Context, key string, opts PresignOptions) (SignedRequest, error)
SignedURL returns a pre-authorised URL for downloading (MethodGet) or uploading (MethodPut) key. It uses the disk's URLSigner for the methods WithURLSigner routes to it (by default every download, and uploads when the driver cannot presign), else the driver's own signing when it has the Presign capability, else fails with CodeUnsupported. A method routed to the signer never falls back to the driver. A PUT URL counts as a mutation, so a read-only disk refuses to issue one.
func (*Disk) SweepAbandoned ¶
SweepAbandoned removes what unfinished work left in the backend before before: multipart uploads never completed or aborted, and staging files of interrupted writes (see AbandonedSweeper for exactly what counts). It returns how many it removed. Stored objects are never touched; to delete uploaded objects nobody confirmed, use upload.Sweep on their prefix.
Choose before well beyond the longest upload the application allows (a day ago is usual): a recent before aborts uploads still in progress. The library starts no background goroutine, so schedule it like upload.Sweep.
Needs the SweepAbandoned capability; without it the call fails with CodeUnsupported. A scoped or mirrored disk never has it: sweep the disk it was derived from.
func (*Disk) TempFile ¶
TempFile copies key to a new file on local disk and returns its path, for tools that need a path rather than a stream (ffmpeg, libvips, pdftotext). The file keeps the key's extension when it is short and alphanumeric, since such tools often go by it, and is readable by the current user only.
The caller owns the file and must remove it (defer os.Remove(path)). On any error nothing is left behind: a failed or refused copy removes its partial file. Errors carry the read operation, this disk and the key.
func (*Disk) UploadPart ¶
func (d *Disk) UploadPart(ctx context.Context, key, uploadID string, number int, body io.Reader, size int64) (Part, error)
UploadPart uploads one part through the server. A retry interceptor may retry it only when body implements io.Seeker and can actually seek; a body whose Seek fails (an *os.File on a pipe) is uploaded once as a stream, as in Write.
func (*Disk) Versions ¶
Versions returns key's stored versions, newest first. Needs the Versions capability (a versioned bucket).
func (*Disk) Write ¶
func (d *Disk) Write(ctx context.Context, key string, body io.Reader, opts WriteOptions) (Entry, error)
Write stores body under key, replacing any existing object, and returns the stored entry with Size and SHA256 computed by the Disk while streaming.
The Disk validates opts against the driver's capabilities first, so nothing is silently dropped: Metadata needs Capabilities.Metadata, Headers need Capabilities.Headers, IfNotExists needs Capabilities.ConditionalWrite, and a Visibility other than the disk default needs Capabilities.SetVisibility. When ContentType is empty it is detected from the first bytes and the key's extension.
A write is retried by a retry interceptor only when body implements io.Seeker and can actually seek (a pipe's *os.File cannot, so it is written once as a stream), because a consumed stream cannot be replayed.
func (*Disk) WriteBytes ¶
func (d *Disk) WriteBytes(ctx context.Context, key string, b []byte, opts WriteOptions) (Entry, error)
WriteBytes stores b under key. It is Write with a rewindable body, so it is always retryable.
type Driver ¶
type Driver interface {
Stat(ctx context.Context, key string) (Entry, error)
Read(ctx context.Context, key string, rng ByteRange) (io.ReadCloser, error)
Write(ctx context.Context, key string, body io.Reader, opts WriteOptions) (Entry, error)
Delete(ctx context.Context, key string) error
List(ctx context.Context, dir string, opts ListOptions) (Page, error)
Capabilities() Capabilities
}
Driver is the contract a storage backend implements. It is deliberately small: everything portable (key cleaning, content-type detection, hashing, size limits, fallbacks, URLs, interceptors) is the Disk's job, so a driver only moves bytes.
Contract every driver must honour, verified by the filesystemtest conformance suite:
- Keys and directories arrive already cleaned (see CleanKey).
- Stat and Read of a missing key return an error with CodeNotFound.
- Write streams body to completion and never closes it; the caller owns it. If reading body fails, nothing is stored and the read error is returned. A successful Write replaces any existing object atomically: readers see the old bytes or the new bytes, never a mix.
- Delete of a missing key succeeds.
- List returns entries in ascending key order and honours Cursor and Limit; a directory with no keys lists as empty, not CodeNotFound.
- Capabilities is constant for the driver's lifetime and true only for what the driver really does.
type Entry ¶
type Entry struct {
Key string
IsDir bool
Size int64
ModTime time.Time
ContentType string
ETag string
SHA256 string
// Checksums holds the extra digests WriteOptions.Checksums asked for,
// by Checksum.Name, in lowercase hex; nil when none were asked for.
Checksums map[string]string
Visibility Visibility
Metadata map[string]string
// Headers are empty when the driver does not store them.
Headers HTTPHeaders
// StorageClass is the backend's name for the object's storage class
// (S3 "STANDARD_IA", GCS "NEARLINE", Azure "Cool"), empty when the
// driver does not report one or the backend leaves its default unnamed.
// SetStorageClass changes it.
StorageClass string
}
Entry describes one stored object, or one directory in a non-recursive listing.
Field presence depends on the source:
- From Stat and Write: every field the backend can report. ContentType, ETag, Visibility and Metadata are empty when the driver does not store them (see Capabilities).
- From a non-recursive List: a directory appears once with IsDir set, Key set to the directory's key, and every other field zero.
- SHA256 is set by the Disk on a Write it streamed, and by drivers that store a SHA-256 (none of the built-in ones do on Stat).
- Checksums is set only by the Disk on a Write that asked for more digests (WriteOptions.Checksums), never by Stat or List.
func CopyAcross ¶
func CopyAcross(ctx context.Context, src *Disk, srcKey string, dst *Disk, dstKey string) (Entry, error)
CopyAcross copies srcKey on src to dstKey on dst. On the same disk it is Disk.Copy (server-side when supported); across disks it streams the bytes, keeping content type, and metadata and visibility where dst stores them.
type Error ¶
type Error struct {
Code Code
Op OpKind
Disk string
Key string
Err error
// RetryAfter is how long the backend asked the caller to wait before
// trying again (an HTTP Retry-After header on a 429 or 503, or the
// vendor's equivalent); 0 means it did not say. A driver sets it on a
// CodeUnavailable error with WithRetryAfter. layer.Retry waits at least
// this long before the next attempt, and fshttp and upload pass it on to
// their clients as a Retry-After header.
RetryAfter time.Duration
}
Error is the one error type the library returns. Drivers create it with NewError; the Disk fills in Op, Disk and Key before returning it, so a log line always says which operation on which disk and key failed.
Err is the underlying cause and may be a native SDK error: errors.As reaches it through Unwrap, which is the escape hatch for vendor-specific error handling.
func Errorf ¶
Errorf builds a classified error from a format string, for failures that have no underlying cause (validation in the Disk or a driver).
func NewError ¶
NewError is how a driver reports a classified failure, keeping the native cause (which may be nil) reachable through errors.As.
func (*Error) Error ¶
Error renders "filesystem: <op> <disk>:<key>: <code>: <cause>", omitting the parts that are empty.
func (*Error) Is ¶
Is makes errors.Is(err, ErrNotFound) true for every *Error whose Code is CodeNotFound, and likewise for each sentinel.
type FailoverEvent ¶
type FailoverEvent struct {
// Op is OpStat or OpRead.
Op OpKind
// Key is the key read.
Key string
// From names the disk that failed, To the disk tried next.
From, To string
// Err is From's error.
Err error
}
FailoverEvent describes one read that moved from a disk that failed to the next one.
type FailoverOptions ¶
type FailoverOptions struct {
// ShouldFailover decides which errors move a read to the next disk.
// nil means CodeUnavailable or CodeUnknown: a backend that is down or
// answered with something unclassifiable. An answer such as
// CodeNotFound or CodePermission is the disk's answer, not an outage,
// and is returned as it is.
ShouldFailover func(error) bool
// OnFailover, when set, is called each time a read moves on, to count
// or log outages of the primary.
OnFailover func(ctx context.Context, e FailoverEvent)
}
FailoverOptions configure Failover. The zero value fails over on CodeUnavailable and CodeUnknown and reports nothing.
type HTTPHeaders ¶
type HTTPHeaders struct {
// CacheControl, such as "public, max-age=31536000, immutable" for
// content-addressed keys.
CacheControl string
// ContentDisposition, such as `attachment; filename="report.pdf"`.
ContentDisposition string
// ContentEncoding, such as "gzip" for a pre-compressed object, which is
// stored and read back exactly as written.
ContentEncoding string
}
HTTPHeaders are stored with an object and sent with it when it is served (by the backend through its own URLs, or by fshttp), so a CDN caches it and a browser handles it as intended. Needs Capabilities.Headers. Values may not contain CR or LF, which would let a value inject another header when it is served.
func HTTPHeadersFrom ¶
func HTTPHeadersFrom(src http.Header) HTTPHeaders
HTTPHeadersFrom reads the HTTPHeaders fields from an HTTP header set, such as the request of a signed PUT.
func (HTTPHeaders) IsZero ¶
func (h HTTPHeaders) IsZero() bool
IsZero reports whether no header is set.
func (HTTPHeaders) SetOn ¶
func (h HTTPHeaders) SetOn(dst http.Header)
SetOn writes every non-empty field into dst under its HTTP name, leaving the others as they are. Signers use it to list the headers a client must send with a presigned PUT (SignedRequest.Header); servers use it to send stored headers with an object.
type HTTPMethod ¶
type HTTPMethod string
HTTPMethod is the method a signed request authorises.
const ( MethodGet HTTPMethod = http.MethodGet MethodPut HTTPMethod = http.MethodPut )
The closed set of methods a signed URL can authorise.
type HeaderPinner ¶
type HeaderPinner interface {
PinsHeaders() bool
}
HeaderPinner is implemented by a URLSigner whose PUT URLs pin PresignOptions.Headers: the URL carries them under its signature, and the server that accepts the upload refuses a request without them and stores them (signedurl.Signer with fshttp.Signed does). The Disk gives headers only to a signer whose PinsHeaders reports true, so a signer that does not know about them can never drop them silently.
type Hold ¶
type Hold struct {
// Legal protects the object until the hold is lifted, with no date.
Legal bool
// RetainUntil protects the object until this time; zero means none.
// Backends refuse to shorten a retention already set in compliance
// mode.
RetainUntil time.Time
}
Hold is an object's protection from deletion and overwrite, as object lock implements it (S3 Object Lock, GCS holds and retention, Azure immutability policies).
type Holder ¶
type Holder interface {
Hold(ctx context.Context, key string) (Hold, error)
SetHold(ctx context.Context, key string, h Hold) error
}
Holder is implemented by drivers that declare the Hold capability, so compliance code can protect an object without reaching for the backend's SDK. SetHold replaces the whole Hold (both fields); the Disk cleans the key and counts it as a mutation, so a read-only disk refuses it before the driver is called. A backend that refuses to lift or shorten a hold returns its error unchanged.
type IPRestricter ¶
type IPRestricter interface {
RestrictsIP() bool
}
IPRestricter is implemented by a URLSigner whose URLs carry PresignOptions.AllowedIP under their signature, and whose server refuses a request from an address outside it (signedurl.Signer with fshttp.Signed does). The Disk gives AllowedIP only to a signer whose RestrictsIP reports true, so a signer that does not know about it can never issue a URL that works from anywhere.
type Interceptor ¶
Interceptor runs around an operation. It must call call to proceed (zero, one or several times) and return its error or its own. It may inspect op before and after the call, but must not change Kind, Disk, Key or Target.
func ReadOnly ¶
func ReadOnly() Interceptor
ReadOnly is an interceptor that refuses every mutating operation (see Op.Mutating) with CodeReadOnly before it reaches the driver. Signed GET URLs and public URLs still work; signed PUT URLs do not, because they authorise a write.
It lives in the core rather than the layer package because configuration (Mux's readonly parameter) needs it and the core cannot import layer.
type KeyEncoder ¶
KeyEncoder maps keys to names a backend can store and back. Some backends refuse characters a key may hold (SFTP servers on Windows refuse : * ? < > | ", Azure refuses a blob name ending in a dot); a driver for such a backend holds a KeyEncoder, encodes every key it sends and decodes every name it lists. Keys reaching and leaving the Disk are never encoded, so applications and other drivers see the same keys.
DecodeKey(EncodeKey(k)) must return k for every clean key, and EncodeKey must not produce "/" or "..", so the encoded name keeps the key's segments.
type KeyEscaper ¶
KeyEscaper is the stock KeyEncoder: it percent-escapes (as "%3A") every character in Chars anywhere in a key, every character in SegmentEnd when it ends a segment (Windows and Azure refuse names ending in "." or " "), and '%' itself. The zero value escapes only '%'.
func (KeyEscaper) DecodeKey ¶
func (e KeyEscaper) DecodeKey(name string) (string, error)
DecodeKey implements KeyEncoder. It accepts only names EncodeKey produces: a malformed escape, an escape EncodeKey would not write ("%41" for "A", lower-case hex, "%2F" for "/"), or a raw character it would have escaped is CodeInvalidKey, so two names can never decode to one key and a name can never gain a segment.
func (KeyEscaper) EncodeKey ¶
func (e KeyEscaper) EncodeKey(key string) string
EncodeKey implements KeyEncoder.
type KeyMapper ¶
KeyMapper is implemented by wrapper drivers that change keys on the way to the driver they wrap (a scoped disk adds its prefix). Disk.DriverKey follows it, so a backend-specific option bound to one key (an S3 request option, say) can name the key the backend really receives.
type KeysDeleter ¶
KeysDeleter deletes many keys in as few calls as the backend allows (S3 deletes 1000 per request). Keys are already cleaned. Deleting a missing key succeeds. A partial failure is a *BatchError listing the keys that were not deleted, with every other key deleted; any other error means the call failed and no key can be assumed deleted.
type ListOptions ¶
type ListOptions struct {
// Recursive lists every key under the directory. When false, keys in
// sub-directories are folded into one IsDir entry per sub-directory.
Recursive bool
// Cursor resumes after a previous page: pass Page.NextCursor. Empty
// starts from the beginning.
Cursor string
// Limit caps the entries per page. 0 means DefaultPageSize.
Limit int
}
ListOptions control one page of a listing.
type LookupFunc ¶
LookupFunc reads one configuration value and reports whether it is set: os.LookupEnv in production, a map in tests. Taking it as an argument, rather than reading the process environment, keeps OpenManager free of hidden state.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager holds an application's named disks and its default disk. It is a value the application owns, not a global, so tests and multiple apps in one process never share disks. It is safe for concurrent use.
func NewManager ¶
NewManager returns a Manager holding disks. The first disk is the default until SetDefault changes it. Names must be unique.
func (*Manager) Add ¶
Add registers d under its name. A second disk with the same name is CodeExists. The first disk added becomes the default.
func (*Manager) Check ¶
Check runs Disk.Check on every disk and joins the failures, for a start-up or health probe.
func (*Manager) Close ¶
Close closes every disk and joins the failures. Disks that share a driver (scoped and mirror disks) may close it more than once; built-in drivers tolerate that.
func (*Manager) Default ¶
Default returns the default disk, or CodeNotFound when none is registered.
func (*Manager) SetDefault ¶
SetDefault makes the named disk the default.
type ModTimeSetter ¶
ModTimeSetter sets an object's modification time, for tools that sync or archive by date (rsync-style mirroring, restoring a backup).
type Mount ¶
type Mount struct {
// Prefix is the directory the disk appears under: key "a.png" on Disk
// is "<Prefix>/a.png" on the routed disk. "" mounts Disk at the root as
// the catch-all for keys under no other prefix.
Prefix string
// Disk serves the keys under Prefix, with its own interceptors,
// settings and capabilities.
Disk *Disk
}
Mount places a disk at a directory of a routed disk (see Route).
type Mover ¶
Mover renames an object inside the backend. dst is overwritten if present; src no longer exists afterwards.
type MultipartUploader ¶
type MultipartUploader interface {
CreateMultipart(ctx context.Context, key string, opts WriteOptions) (uploadID string, err error)
UploadPart(ctx context.Context, key, uploadID string, number int, body io.Reader, size int64) (Part, error)
ListParts(ctx context.Context, key, uploadID string) ([]Part, error)
CompleteMultipart(ctx context.Context, key, uploadID string, parts []Part) (Entry, error)
AbortMultipart(ctx context.Context, key, uploadID string) error
}
MultipartUploader uploads large objects in parts. Parts uploaded through the server use UploadPart; a driver that can also sign part uploads for clients implements PartPresigner, which together with this interface is the contract browser uploaders such as Uppy's S3 multipart plugin expect.
type Mux ¶
type Mux struct {
// contains filtered or unexported fields
}
Mux opens disks from DSN strings such as "s3://bucket?region=auto&prefix=tenant-a&visibility=public". Each driver package exports an Opener; the application registers the ones it imports.
Unlike a global registry filled by init functions, a Mux is a value: two parts of a program (or two tests) can hold different sets of drivers, and nothing is registered by importing a package.
func (*Mux) Open ¶
Open parses dsn, builds its driver with the registered opener and returns a Disk named name. opts are applied after the DSN's own settings, so code can add interceptors or a signer to a configured disk.
Every returned error names the disk ("disk media: …") and never contains the DSN itself, because DSNs often carry credentials. When the driver was built but the Disk could not be (an invalid Mux parameter value), the driver is closed so no connection leaks.
func (*Mux) OpenManager ¶
func (m *Mux) OpenManager(ctx context.Context, prefix string, lookup LookupFunc, opts ...Option) (*Manager, error)
OpenManager builds a Manager from configuration, twelve-factor style:
STORAGE_DISKS=media,backups STORAGE_MEDIA_DSN=s3://media?region=auto&visibility=public STORAGE_BACKUPS_DSN=file:///var/backups STORAGE_DEFAULT_DISK=media
with prefix "STORAGE_". opts apply to every disk. A missing or invalid setting is CodeInvalidArgument naming the variable, never its value (DSNs carry credentials), and the disks opened so far are closed. An error from opening a disk (an unknown scheme, a bad DSN parameter, a failed connection) keeps its code and names both the variable and the disk: "STORAGE_MEDIA_DSN: disk media: unknown DSN parameter …".
type Op ¶
type Op struct {
// Kind is the operation.
Kind OpKind
// Disk is the name of the Disk running it.
Disk string
// Key is the object key, the directory for List, DeletePrefix,
// MakeDir and DirExists, or empty for DeleteKeys and the abandoned
// sweeps.
Key string
// Keys are the keys of a DeleteKeys, else nil.
Keys []string
// Target is the destination key of a Copy or Move, else empty.
Target string
// Mutating is true when the operation changes data or authorises a
// change: Kind.Mutates(), plus a presign for MethodPut. Read-only
// enforcement keys off this.
Mutating bool
// Replayable is true when the call may safely run more than once. False
// for a write whose body cannot be rewound, so a retry interceptor must
// not retry it.
Replayable bool
// Streaming is true when the call returns a stream that outlives it (a
// Read): cancelling the context after the call returns would break the
// stream, so timeout interceptors leave such calls alone.
Streaming bool
// Attempt counts how many times the call has started, from 1.
Attempt int
// Bytes is set after a successful Write to the number of bytes stored.
Bytes int64
// Hedgeable is true when the call may run concurrently with itself
// (Stat, Read). An interceptor may call call again before an earlier
// call returned only on a hedgeable operation, after setting
// Concurrent, and must wait for every call it started before it
// returns.
Hedgeable bool
// Concurrent is set by an interceptor (layer.Hedge) before it starts
// calls of a hedgeable operation that may overlap. The Disk then keeps
// the result of the first call to succeed, and a later call that also
// succeeds discards its own result (closing its reader) and returns
// ErrSuperseded, so the call that returned nil is always the one whose
// result the Disk returns. Without it, calls are taken to run one
// after another, and each success replaces the previous result,
// closing its reader.
Concurrent bool
// contains filtered or unexported fields
}
Op describes one operation as it passes through the interceptor chain. Interceptors read it; the Disk owns it and updates the result fields.
func (*Op) OnClose ¶
func (op *Op) OnClose(fn func())
OnClose registers fn to run once when the caller closes the reader the operation returns (a Read), so an interceptor can release what the stream needed until then, such as an attempt's context. If the operation fails, or returns no reader, fn runs when it returns. Call it before returning from the interceptor, never from a concurrent attempt.
func (*Op) WrapBody ¶
WrapBody asks the Disk to pass the operation's byte stream through fn: the body of a Write, Append or UploadPart before the driver reads it, and the reader a Read returns before the caller reads it (its Close still closes the driver's reader). It is how an interceptor reaches bytes it otherwise never sees, to throttle or count them. Operations without a byte stream ignore it. fn must return a reader that yields exactly the bytes it was given; call WrapBody before calling call, because a body is opened when an attempt starts. An interceptor placed inside Retry runs, and registers fn, once per attempt, so fn should return a reader it already wrapped unchanged; one placed after a hedging interceptor must not call WrapBody at all, as hedged attempts run concurrently.
type OpKind ¶
type OpKind string
OpKind names a Disk operation. Interceptors switch on it, errors carry it, and metrics label by it.
const ( OpStat OpKind = "stat" OpRead OpKind = "read" OpWrite OpKind = "write" OpDelete OpKind = "delete" OpDeletePrefix OpKind = "delete_prefix" OpDeleteKeys OpKind = "delete_keys" OpList OpKind = "list" OpCopy OpKind = "copy" OpMove OpKind = "move" OpSetVisibility OpKind = "set_visibility" OpPresign OpKind = "presign" OpPublicURL OpKind = "public_url" OpMultipartCreate OpKind = "multipart_create" OpMultipartUpload OpKind = "multipart_upload_part" OpMultipartPresign OpKind = "multipart_presign_part" OpMultipartListParts OpKind = "multipart_list_parts" OpMultipartComplete OpKind = "multipart_complete" OpMultipartAbort OpKind = "multipart_abort" OpSweepAbandoned OpKind = "sweep_abandoned" OpFindAbandoned OpKind = "find_abandoned" OpMakeDir OpKind = "make_dir" OpDirExists OpKind = "dir_exists" OpPresignPost OpKind = "presign_post" OpAppend OpKind = "append" OpSetModTime OpKind = "set_mod_time" OpTags OpKind = "tags" OpSetTags OpKind = "set_tags" OpHold OpKind = "hold" OpSetHold OpKind = "set_hold" OpVersions OpKind = "versions" OpRestore OpKind = "restore" OpSetStorageClass OpKind = "set_storage_class" OpReadVersion OpKind = "read_version" OpRestoreVersion OpKind = "restore_version" )
The closed set of operations. Every Disk method that reaches a driver runs as exactly one of these (a portable fallback such as Copy without server side copy runs as the operations it is made of).
type Opener ¶
Opener builds a driver from a parsed DSN. It receives the URL with the Mux's own query parameters already removed, and must reject any parameter it does not understand, so a typo in configuration fails at start-up instead of being ignored.
type Option ¶
Option configures a Disk in NewDisk.
func WithContentDetector ¶
WithContentDetector adds detect to the content-type detection of writes that do not declare a type, for formats the built-in tables do not know (DICOM, STL, camera RAW, Parquet). It is consulted only when the built-in answer is application/octet-stream, so it can never relabel a type sniff.Detect already recognises (HTML stays HTML); see sniff.DetectWith. It has no effect under WithoutContentTypeDetection. Uploads detect with upload.Policy.Detector, so set it there too.
func WithDefaultVisibility ¶
func WithDefaultVisibility(v Visibility) Option
WithDefaultVisibility sets the visibility of writes that do not choose one. The default is VisibilityPrivate.
func WithInterceptors ¶
func WithInterceptors(interceptors ...Interceptor) Option
WithInterceptors appends interceptors; the first given runs outermost.
func WithKeyNormalizer ¶
WithKeyNormalizer makes the Disk pass every key and directory through fn before cleaning it (CleanKey), so spellings fn treats as equal address one object: contrib/keynorm.NFC makes the NFC and NFD forms of "café.txt" one key, as macOS and Linux clients produce them. fn runs before the traversal check, so a normaliser that turns other characters into "." or "/" cannot smuggle a ".." past it. An error from fn is CodeInvalidKey. Every key this disk is given is normalised, so objects already stored under another spelling cannot be reached through it, not even to move them: set it on an empty disk, or first rename the old keys through a Disk over the same driver without the normaliser.
func WithPublicURL ¶
WithPublicURL makes PublicURL return base + "/" + key, typically a CDN origin. It takes precedence over the driver's own public URL. base must be an absolute http or https URL; a trailing slash is ignored.
func WithURLSigner ¶
func WithURLSigner(s URLSigner, methods ...HTTPMethod) Option
WithURLSigner sets the signer SignedURL uses, typically a CDN's (contrib/cloudfront, contrib/cloudflare) or the HMAC signer of package signedurl.
Which methods it signs: with no methods given, the signer signs every download (MethodGet), even on a disk whose driver presigns (S3, GCS, Azure), so a CDN in front of a bucket hands out CDN links rather than raw bucket links; uploads (MethodPut) stay with the driver's own presigning, which sends the bytes straight to the backend, and go to the signer only when the driver cannot presign. Naming methods replaces that rule: the signer signs exactly those, and the driver (when it presigns) the rest, so WithURLSigner(s, MethodGet, MethodPut) routes uploads through the signer too, and WithURLSigner(s, MethodPut) keeps downloads on the driver.
Invariant: a method routed to the signer never falls back to the driver or to an unsigned link. A signer error, or a signer that cannot honour PresignOptions.Headers or AllowedIP, fails the call. An unknown method is CodeInvalidArgument.
func WithoutContentTypeDetection ¶
func WithoutContentTypeDetection() Option
WithoutContentTypeDetection stops the Disk from detecting the content type of writes that do not declare one; they are stored without a type.
type Page ¶
Page is one page of a listing, in ascending key order. NextCursor is empty on the last page.
func PageOf ¶
func PageOf(objects []Entry, dir string, opts ListOptions) Page
PageOf builds one listing page from every object under dir, for drivers that can enumerate their keys (memory, local disk) and so do not need to implement directory folding and cursors themselves. objects must hold only keys inside dir (see KeyInDir); their order does not matter.
It sorts by key, folds sub-directories into IsDir entries when the listing is not recursive, skips everything up to and including opts.Cursor, and cuts the page at opts.Limit (DefaultPageSize when 0). The cursor is the key of the last entry returned, so a listing stays correct when keys are added or deleted between pages.
type Part ¶
Part identifies one uploaded part of a multipart upload. Number starts at 1; ETag is what the backend returned for the part and must be passed back unchanged to CompleteMultipart.
type PartPresigner ¶
type PartPresigner interface {
PresignPart(ctx context.Context, key, uploadID string, number int, expires time.Duration) (SignedRequest, error)
}
PartPresigner signs a PUT for one part of a multipart upload so a client can send it straight to the backend.
type PostPolicy ¶
type PostPolicy struct {
// Key is the exact key the upload is stored under. Set Key or
// KeyPrefix, not both.
Key string
// KeyPrefix lets the client choose any key under this directory; the
// key field is then KeyPrefix + "/" + PostFilenamePlaceholder, which
// the client may keep (the backend substitutes the file's name) or
// replace with any key under the prefix. Never the root.
KeyPrefix string
// ContentType, when set, is the only Content-Type accepted; the
// client sends it as the PostFieldContentType field. Empty accepts any.
ContentType string
// MinSize and MaxSize bound the upload in bytes: MaxSize is required,
// and 0 ≤ MinSize ≤ MaxSize.
MinSize, MaxSize int64
// Expires is the form's lifetime, in (0, MaxPresignExpiry].
Expires time.Duration
// Headers are stored with the object (and pinned by the policy); they
// need Capabilities.Headers.
Headers HTTPHeaders
// Visibility is the visibility the uploaded object is stored with,
// pinned by the policy (S3's and GCS's acl field), under the rules of
// PresignOptions.Visibility: empty pins nothing (unchanged), and a
// value the disk must pin needs Capabilities.PresignVisibility.
Visibility Visibility
}
PostPolicy is what a presigned POST lets a client upload. Unlike a presigned PUT, the backend itself enforces every condition: the size range, the key, the content type, the stored headers and the expiry, so an oversized or mistyped upload is refused before it is stored.
type PostPresigner ¶
type PostPresigner interface {
PresignPost(ctx context.Context, p PostPolicy) (PresignedPost, error)
}
PostPresigner signs presigned POST forms. Options arrive validated and cleaned by the Disk. The returned Fields must include PostFieldKey, and PostFieldContentType when the policy fixes a type, plus whatever the backend needs (policy document, signature, credential).
type PrefixDeleter ¶
PrefixDeleter deletes every key under a directory in as few calls as the backend allows. dir is clean and never the root: the Disk refuses to delete a whole disk through DeletePrefix.
type PresignOptions ¶
type PresignOptions struct {
// Method is MethodGet (download) or MethodPut (direct upload).
Method HTTPMethod
// Expires is the URL's lifetime: required, positive, at most
// MaxPresignExpiry. There is deliberately no default, so every caller
// decides how long a link may live.
Expires time.Duration
// ContentType: for a PUT, the type the upload must declare; for a GET,
// an override for the response's Content-Type.
ContentType string
// ContentDisposition overrides the response's Content-Disposition on a
// GET, e.g. `attachment; filename="report.pdf"`.
ContentDisposition string
// Headers, for a PUT only, are stored with the uploaded object, exactly
// as WriteOptions.Headers are for a Write. The signed request pins
// them: SignedRequest.Header lists them, the client must send them
// unchanged, and the backend refuses an upload without them. The Disk
// issues such a URL only when that is guaranteed: through the driver
// when it declares Capabilities.PresignHeaders, through its URLSigner
// when the driver stores headers (Capabilities.Headers) and the signer
// implements HeaderPinner reporting true. Otherwise it is
// CodeUnsupported, never a URL that silently loses them. Non-empty on
// a GET is CodeInvalidArgument: a download overrides the response with
// ContentDisposition instead.
Headers HTTPHeaders
// AllowedIP, when set, restricts the URL to clients whose address is
// inside this CIDR prefix: "203.0.113.0/24", or "203.0.113.7/32" for
// one address. It must be in canonical form (no bits set past the
// prefix length), so what is signed is exactly what is meant. The Disk
// issues such a URL only when the restriction is enforced: through the
// driver when it declares Capabilities.PresignIP, through its URLSigner
// when the signer implements IPRestricter reporting true. Otherwise it
// is CodeUnsupported, never a URL that silently works from anywhere.
AllowedIP string
// Visibility, for a PUT only, is the visibility the uploaded object is
// stored with, signed into the request (S3's x-amz-acl, GCS's
// x-goog-acl), so the client cannot change it. Empty, the default,
// signs nothing and the object gets whatever the backend gives an
// upload that names none, exactly as before this field existed.
//
// It is checked like WriteOptions.Visibility: a value other than the
// disk's default needs Capabilities.SetVisibility, and on a disk
// without SetVisibility the default itself is accepted and signs
// nothing (that backend stores every object at the default). On a
// disk with SetVisibility any value needs a driver that pins it,
// Capabilities.PresignVisibility, and a URLSigner never does:
// otherwise it is CodeUnsupported, never a URL that silently stores
// the object some other way. Non-empty on a GET is
// CodeInvalidArgument.
//
// Who may ask for VisibilityPublic is the caller's decision: an
// upload ticket handed to a browser lets its holder create a public
// object, so upload.Presign issues one only under
// upload.Policy.AllowPublic.
Visibility Visibility
}
PresignOptions describe a signed URL.
type PresignedPost ¶
PresignedPost is a form a client posts to URL: every field in Fields, then the file in PostFileField. It expires at ExpiresAt.
type Presigner ¶
type Presigner interface {
Presign(ctx context.Context, key string, opts PresignOptions) (SignedRequest, error)
}
Presigner issues URLs signed by the backend itself (S3 SigV4, GCS V4). Options arrive validated by the Disk.
type PublicURLer ¶
PublicURLer returns the backend's own public URL for a key. A Disk configured with WithPublicURL ignores it in favour of the configured base (typically a CDN).
type ReadOptions ¶
type ReadOptions struct {
// IfNoneMatch is the ETag of the caller's copy; "*" matches any
// existing object. When it matches, the read answers CodeNotModified.
// It takes precedence over IfModifiedSince, as in HTTP.
IfNoneMatch string
// IfModifiedSince answers CodeNotModified when the object was not
// modified after it (compared to the second, as HTTP dates are).
IfModifiedSince time.Time
}
ReadOptions make a read conditional, as HTTP's conditional GET: the body is returned only when the object differs from the caller's copy.
type RestoreOptions ¶
type RestoreOptions struct {
// Days the restored copy stays readable before it returns to the
// archive. Required, positive.
Days int
}
RestoreOptions configure Restore.
type Restorer ¶
type Restorer interface {
Restore(ctx context.Context, key string, opts RestoreOptions) error
}
Restorer brings an archived object (S3 Glacier, Azure archive tier) back online for a while. The restore is asynchronous in the backend: Restore starts it, and reads succeed once the backend has finished.
type SignedRequest ¶
SignedRequest is a pre-authorised HTTP request a client can make without credentials. Header holds headers the client must send unchanged (some backends sign the content type of a PUT).
type StorageClasser ¶
StorageClasser moves an existing object to another storage class (S3 "STANDARD_IA" or "GLACIER", GCS "NEARLINE", Azure "Cool"), trading read cost and latency for storage cost. Class names are the backend's own and are passed through unchanged.
type SyncAction ¶
type SyncAction string
SyncAction is what Sync does with one object. It is a closed set (SyncActions lists it), so a report consumer can switch on it exhaustively. There is no "skip" action: an unchanged object has no SyncItem and is only counted in SyncReport.Unchanged.
const ( // SyncCopy copies a source object that is missing on the target or // differs from the target's object there. SyncCopy SyncAction = "copy" // SyncDelete deletes a target object with no source object at the // same relative key (SyncOptions.Delete only). SyncDelete SyncAction = "delete" )
The closed set of sync actions.
type SyncItem ¶
type SyncItem struct {
Action SyncAction
// Key is the object on the source for a copy, on the target for a
// delete.
Key string
// Target is the key on the target disk: where a copy lands, or the
// object a delete removes (then equal to Key).
Target string
// Size is the object's size: the source's for a copy, the target's
// for a delete.
Size int64
// Err is why the action failed; nil when it succeeded and always nil
// in a dry run.
Err error
}
SyncItem is one object Sync acted on, or would act on in a dry run.
type SyncOptions ¶
type SyncOptions struct {
// Delete also deletes target objects under the target directory that
// have no source object at the same relative key, making the target a
// mirror. Off by default: a sync never deletes what it was not asked
// to. It never deletes outside the target directory.
Delete bool
// DryRun reports every action through OnItem and in the report, and
// changes nothing on either disk.
DryRun bool
// Same decides whether a target object already holds the source
// object's content; nil means SameContent. It is the escape hatch
// for backends whose ETags are not comparable but that keep a digest
// of their own in Metadata.
Same func(src, dst Entry) bool
// Checksum reads both objects and compares their SHA-256 when Same
// says they differ but their sizes are equal: the listings of many
// backends (a local disk, or two backends whose ETags are computed
// differently) cannot prove equal content, and without it every such
// object is copied again on every run. It costs a full read of each
// such pair; a read failure counts as different, so the object is
// copied.
Checksum bool
// OnItem, when set, receives every action as it is done (or, in a dry
// run, planned), failures included, in source listing order, then
// the deletes in target listing order.
OnItem func(SyncItem)
}
SyncOptions configure Sync. The zero value copies new and changed objects and deletes nothing.
type SyncReport ¶
SyncReport counts what Sync did, or would do in a dry run.
func Sync ¶
func Sync(ctx context.Context, src *Disk, srcDir string, dst *Disk, dstDir string, opts SyncOptions) (SyncReport, error)
Sync makes the objects under dstDir on dst match those under srcDir on src: every source object missing on the target, or whose target object differs (SyncOptions.Same), is copied to the same relative key under dstDir; with Delete, target objects under dstDir with no source object are deleted. Both listings are read in full before anything changes, so a listing failure stops Sync with nothing done, and objects it copies are never listed back. Each object is handled on its own: a failed copy or delete is reported (OnItem, SyncReport.Failed) and the rest go on, and the returned error says how many failed with the first cause. A rerun after a failure copies only what is still missing or different.
A delete is limited to keys inside dstDir, checked again before every deletion, so no listing quirk can make Sync delete outside it. When both disks reach the same backend (a scoped disk and the disk it was derived from, a routed or mirrored disk and one of its targets), the two directories as that backend sees them must not overlap in either direction (CodeInvalidArgument): the target would be part of the source, and a sync with Delete would remove the source while copying it. Delete is refused (CodeInvalidArgument) whenever either side is a Route, Overlay, Mirror, CachedRead or Failover disk, or wraps one, since such a disk can reach targets no check can see; sync the disks underneath it instead.
On a target with the CaseInsensitive capability, source objects whose keys differ only in case would overwrite each other there on every run: none of them is copied, and each is reported as a failed copy with CodeConflict naming the other. Matching target objects is then case-insensitive too, so Delete never removes the object a copy wrote.
type Tagger ¶
type Tagger interface {
Tags(ctx context.Context, key string) (map[string]string, error)
SetTags(ctx context.Context, key string, tags map[string]string) error
}
Tagger stores key-value tags with an object, separately from its bytes and metadata, so they can change without rewriting it (S3 object tags, Azure blob index tags). Lifecycle rules and cost reports often key off them. SetTags replaces every tag.
type TempFileOptions ¶
type TempFileOptions struct {
// Dir is the directory the file is created in; "" means os.TempDir.
Dir string
// MaxSize, when positive, refuses an object larger than this many bytes
// with CodeTooLarge, checked from Stat before anything is written and
// again while copying, so an object that grew in between cannot fill
// the disk.
MaxSize int64
}
TempFileOptions configure Disk.TempFile. The zero value writes to the system temporary directory with no size limit.
type URLSigner ¶
type URLSigner interface {
SignURL(ctx context.Context, disk, key string, opts PresignOptions) (SignedRequest, error)
}
URLSigner signs URLs for disks whose driver cannot (local disk, memory), so that a signed download or upload link works the same in development as on S3. The signedurl sub-package provides an HMAC implementation and the fshttp sub-package serves its URLs.
type Unwrapper ¶
type Unwrapper interface {
Unwrap() Driver
}
Unwrapper is implemented by wrapper drivers (scoped, mirror) so that the typed escape hatch As can reach the backend driver underneath.
type Version ¶
type Version struct {
// ID names the version to the backend.
ID string
Size int64
ModTime time.Time
ETag string
// Latest is the version a plain Read returns.
Latest bool
// DeleteMarker records a deletion rather than content.
DeleteMarker bool
}
Version is one stored version of an object, newest first in the slice Versions returns.
type VersionLister ¶
VersionLister lists the versions a versioned bucket keeps of an object.
type VersionReader ¶
type VersionReader interface {
ReadVersion(ctx context.Context, key, id string) (io.ReadCloser, error)
RestoreVersion(ctx context.Context, key, id string) error
}
VersionReader reads one stored version of an object and makes a stored version the current content again, by the ID VersionLister reported. RestoreVersion keeps every version: the restored content becomes a new, latest version. A missing version, or a delete marker, is CodeNotFound.
type Visibility ¶
type Visibility string
Visibility is the portable access level of a stored object. Each driver maps it to its own mechanism: file modes on local disk, ACLs or separate buckets on object stores.
const ( // VisibilityPublic: anyone with the object's public URL may read it. VisibilityPublic Visibility = "public" // VisibilityPrivate: readable only through the backend's credentials or // a signed URL. The default for a Disk unless configured otherwise, // because exposing a file by accident is worse than having to opt in. VisibilityPrivate Visibility = "private" )
The closed set of visibilities.
func (Visibility) Valid ¶
func (v Visibility) Valid() bool
Valid reports whether v is one of VisibilityValues.
type VisibilitySetter ¶
type VisibilitySetter interface {
SetVisibility(ctx context.Context, key string, v Visibility) error
}
VisibilitySetter changes an existing object's visibility.
type WriteOptions ¶
type WriteOptions struct {
// ContentType is stored by drivers with Capabilities.ContentType. Empty
// means the Disk detects it from the first bytes and the key's extension.
ContentType string
// Visibility overrides the Disk's default visibility. Empty means the
// default.
Visibility Visibility
// Metadata is stored by drivers with Capabilities.Metadata; on any other
// driver a non-empty map is CodeUnsupported rather than silently lost.
Metadata map[string]string
// Headers are stored with the object by drivers with
// Capabilities.Headers; on any other driver non-empty headers are
// CodeUnsupported rather than silently lost.
Headers HTTPHeaders
// IfNotExists makes the write fail with CodeExists when the key already
// exists, atomically. Needs Capabilities.ConditionalWrite; without it
// the write is refused with CodeUnsupported before any byte is read.
//
// The condition is checked when the object is committed, not before the
// body is sent: the local, memory and S3 drivers (and GCS, Azure and
// SFTP) read or upload the whole body first, so a losing writer streams
// all of it before getting CodeExists. Stat first when bodies are large
// and collisions likely; that is a hint, and IfNotExists stays the
// guarantee.
IfNotExists bool
// IfMatch makes the write a compare-and-swap: it replaces the object
// only while its ETag is still IfMatch (the one the caller read), and
// fails with CodeConflict, writing nothing, when the object changed or
// was deleted in between. Two writers that read the same version can
// then never both win. Needs Capabilities.CompareAndSwap
// (CodeUnsupported otherwise); not with IfNotExists, and not for
// multipart uploads.
IfMatch string
// SizeHint is the body's length when known in advance. 0 means unknown
// (an empty body is still written correctly). A positive hint is also a
// declaration the Disk holds the body to: a body that ends early or
// runs past it fails the write with CodeInvalidArgument and nothing is
// stored, so a truncated upload (a dropped connection, a short
// Content-Length) is never kept as if it were whole. Drivers may use it
// to choose a single-request upload.
SizeHint int64
// SHA256, when set, is the hex SHA-256 the body must have (the client's
// declared digest, or one computed upstream). The Disk hashes every
// body anyway and refuses a mismatch with CodeInvalidArgument before
// the driver commits, on every driver, so a corrupted body never
// replaces what was stored. Drivers with Capabilities.ChecksumSHA256
// also send it to the backend, which verifies it on its side of the
// wire. Not for multipart uploads (CodeUnsupported).
SHA256 string
// Checksums are more digests the Disk computes while streaming, beside
// SHA-256, each optionally verified (Checksum.Want), and reports in
// Entry.Checksums: MD5, CRC32C, BLAKE3, whatever hash.Hash the caller
// brings. A malformed list is CodeInvalidArgument. Not for multipart
// uploads (CodeUnsupported). nil computes SHA-256 only.
Checksums []Checksum
// MaxSize, when positive, aborts the write with CodeTooLarge once the
// body exceeds this many bytes. Enforced by the Disk while streaming, so
// nothing larger is ever stored.
MaxSize int64
}
WriteOptions control a write. The zero value writes with the Disk's default visibility and a detected content type.
Source Files
¶
- batch.go
- cachedisk.go
- check.go
- checksum.go
- disk.go
- disk_append.go
- disk_batch.go
- disk_cond.go
- disk_datauri.go
- disk_dir.go
- disk_list.go
- disk_meta.go
- disk_multipart.go
- disk_post.go
- disk_sweep.go
- disk_temp.go
- disk_url.go
- disk_write.go
- doc.go
- driver.go
- entry.go
- errors.go
- failover.go
- iofs.go
- key.go
- keyenc.go
- listmerge.go
- manager.go
- mirror.go
- mux.go
- mux_env.go
- op.go
- overlay.go
- page.go
- readonly.go
- route.go
- scoped.go
- size.go
- sync.go
Directories
¶
| Path | Synopsis |
|---|---|
|
adapters
|
|
|
s3
module
|
|
|
contrib
|
|
|
imaging
module
|
|
|
svg
module
|
|
|
Package filesystemtest holds the conformance suite every driver must pass and a Fake disk with assertions for application tests.
|
Package filesystemtest holds the conformance suite every driver must pass and a Fake disk with assertions for application tests. |
|
Package fshttp serves disks over net/http: signed downloads and uploads (the URLs the signedurl package issues), and public files.
|
Package fshttp serves disks over net/http: signed downloads and uploads (the URLs the signedurl package issues), and public files. |
|
internal
|
|
|
covmerge
command
Command covmerge merges Go coverage profiles of the same packages run in different environments (the records suite on three database engines), counting a block covered when any run covered it.
|
Command covmerge merges Go coverage profiles of the same packages run in different environments (the records suite on three database engines), counting a block covered when any run covered it. |
|
mdwrap
command
Command mdwrap fails when a Markdown file contains hard-wrapped prose: a paragraph or list item broken across physical lines.
|
Command mdwrap fails when a Markdown file contains hard-wrapped prose: a paragraph or list item broken across physical lines. |
|
Package layer holds the built-in interceptors: cross-cutting behaviour that wraps every Disk operation without wrapping the driver, so no capability is ever hidden (see filesystem.Interceptor and ADR-0003).
|
Package layer holds the built-in interceptors: cross-cutting behaviour that wraps every Disk operation without wrapping the driver, so no capability is ever hidden (see filesystem.Interceptor and ADR-0003). |
|
Package local is a filesystem.Driver over a directory on local disk.
|
Package local is a filesystem.Driver over a directory on local disk. |
|
Package memory is an in-process filesystem.Driver that keeps objects in a map.
|
Package memory is an in-process filesystem.Driver that keeps objects in a map. |
|
records
module
|
|
|
Package scan checks uploaded files for malware.
|
Package scan checks uploaded files for malware. |
|
Package signedurl signs and verifies URLs for disks whose backend cannot sign its own (local disk, memory), so that signed downloads and direct uploads work the same in development as on S3.
|
Package signedurl signs and verifies URLs for disks whose backend cannot sign its own (local disk, memory), so that signed downloads and direct uploads work the same in development as on S3. |
|
Package sniff detects a file's MIME type from its first bytes, refined by its file name only where the bytes cannot tell.
|
Package sniff detects a file's MIME type from its first bytes, refined by its file name only where the bytes cannot tell. |
|
Package upload is how bytes arrive from browsers and phones: streaming server uploads, presigned direct uploads confirmed by the server, browser multipart uploads in the protocol Uppy's AWS S3 plugin speaks, and the sweeping of uploads nobody finished.
|
Package upload is how bytes arrive from browsers and phones: streaming server uploads, presigned direct uploads confirmed by the server, browser multipart uploads in the protocol Uppy's AWS S3 plugin speaks, and the sweeping of uploads nobody finished. |