upload

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package upload receives files from HTTP requests into a storage.Storage, safe by default:

  • the file streams into the store; the request is never buffered, and a file over Policy.MaxBytes fails as soon as the byte past the limit arrives (or before a byte is read, when the declared length is over);
  • the media type is detected from the file's first bytes and checked against Policy.Types; the client's Content-Type and the filename's extension are not trusted;
  • the key is generated by the server (Policy.Prefix plus a random id), so a client-controlled filename never becomes an object path: it is kept, cleaned, as metadata (FilenameMetadata);
  • a failed upload leaves nothing behind: a Put that fails stores nothing (the storage contract); one whose outcome is unknown (storage.ErrUnknownOutcome: a remote store may have stored it) has its generated key deleted; and a file stored before the request turned out to be invalid is deleted. When such a delete fails, the error says which key may still hold a file (*CleanupError).

Receive reads a multipart/form-data request (an HTML file input); ReceiveBody reads a request whose body is the file (PUT or POST with the file's bytes); Save applies the same policy to any reader. MapError turns their errors into D10 errors for a handler.

Authorize and Confirm are the direct-upload flow: Authorize grants the client one signed upload (a PUT of a declared size and type, under a generated key, to S3 directly or to the app's storage route for the local and memory drivers; see storage.DirectUploader); the client uploads; Confirm then checks what arrived, by its bytes, and deletes it if it breaks the policy.

Index

Constants

View Source
const (
	// DefaultField is the multipart form field Receive reads the file from
	// when Policy.Field is empty.
	DefaultField = "file"
	// FilenameMetadata is the metadata name the client's filename, cleaned,
	// is stored under (storage.FilenameMetadata).
	FilenameMetadata = storage.FilenameMetadata
	// MaxFilenameBytes bounds the stored filename; a longer one is cut at a
	// character boundary.
	MaxFilenameBytes = 255
	// SniffBytes is how many leading bytes a Detector sees (at most; fewer
	// when the file is shorter).
	SniffBytes = 512
	// MaxFormBytes bounds the rest of a multipart request, on its own: its
	// part headers, boundaries, and fields other than the file. A file
	// smaller than Policy.MaxBytes does not make room for more of them.
	MaxFormBytes = 64 << 10
)
View Source
const DefaultGrantExpiry = 15 * time.Minute

DefaultGrantExpiry is how long a direct upload grant works unless Policy.GrantExpiry says otherwise: long enough to start a large upload, short enough that a leaked grant is soon useless.

View Source
const DefaultUploadTimeout = time.Hour

DefaultUploadTimeout is how long Save and Receive may take to store a file under Policy.Claims unless Policy.UploadTimeout says otherwise.

View Source
const StagingPrefix = "_staging/"

StagingPrefix starts the keys uploads under Policy.Claims are staged at (StagingKey): direct uploads always, and Save's on a remote store. It is a reserved namespace at the root of the store: claims.SweepStaging deletes any object under it whose key has no live claim, whoever stored it. Store nothing else there; Policy prefixes may not start with it.

Variables

View Source
var (
	// ErrTooLarge: the file, or the request carrying it, is over the
	// policy's limit.
	ErrTooLarge = errors.New("upload: too large")
	// ErrType: the file's detected media type is not one the policy
	// accepts.
	ErrType = errors.New("upload: media type not allowed")
	// ErrNoFile: a multipart request has no file in the policy's field.
	ErrNoFile = errors.New("upload: no file")
	// ErrMalformed: the request is not a well-formed upload (not
	// multipart, a broken body, or more than one file).
	ErrMalformed = errors.New("upload: malformed request")
	// ErrExpired: a key with no live claim under Policy.Claims (Confirm):
	// its upload expired, was discarded (a failed write abandons the
	// uploads it named), or was never granted. Upload the file again.
	ErrExpired = errors.New("upload: the upload has expired or was discarded")
)

Functions

func CleanFilename

func CleanFilename(name string) string

CleanFilename returns a client-sent filename as safe metadata: its last path element (after '/' or '\'), without control, line-separator, or bidirectional-control characters, with "=?" (which metadata refuses) broken up, invalid UTF-8 replaced, surrounding space trimmed, and at most MaxFilenameBytes long. "." and ".." become empty. The result names a file for display; it is never an object key or a path.

func MapError

func MapError(ctx context.Context, err error) error

MapError maps an upload error to a D10 error for a handler: ErrTooLarge (or an http.MaxBytesError) is 413 payload_too_large; ErrType, ErrNoFile, ErrMalformed and ErrExpired are validation errors; anything else is a storage error (storage.MapError).

func StagingKey

func StagingKey(key string) string

StagingKey is where a direct upload of key is uploaded under Policy.Claims: StagingPrefix and key. Only Confirm writes key itself.

Types

type Claimer

type Claimer interface {
	// Pending claims key for scope, which nothing has been stored under
	// yet, leased until until: Save publishes nothing under key after it.
	Pending(ctx context.Context, key, scope string, until time.Time) error
	// Stage claims key for scope, for a direct upload to StagingKey(key)
	// (Authorize).
	Stage(ctx context.Context, key, scope string, until time.Time) error
	// Promote moves a key staged for scope to promoting, leased until
	// until, and reports whether it did: only then may Confirm copy the
	// staged object to key.
	Promote(ctx context.Context, key, scope string, until time.Time) (bool, error)
	// Lookup reports the scope key is claimed for, and whether that claim
	// is live (not being deleted, nor gone).
	Lookup(ctx context.Context, key string) (scope string, live bool, err error)
	// Publishing records the token of the promotion's copy on the key's
	// promoting claim, before it is published (storage.PreparePublish).
	Publishing(ctx context.Context, key, token string) error
	// Unpromote moves a promoting key back to pending (nothing was copied).
	Unpromote(ctx context.Context, key string) error
	// Abandon deletes the object of key and its claim if the claim is
	// still pending, and reports whether it did; it leaves any other key
	// alone.
	Abandon(ctx context.Context, key string) (bool, error)
}

Claimer is the ownership protocol uploads take part in; *claims.Claims implements it.

type CleanupError

type CleanupError struct {
	Key string
	Err error // why the delete failed
}

CleanupError reports a key that may still hold a file after a failed upload: one stored before the request turned out to be invalid, or one a Put of unknown outcome may have stored, whose delete failed too. The upload's error carries it (errors.As) alongside the failure it reports, which it does not change; delete Key when the store is reachable again.

func (*CleanupError) Error

func (e *CleanupError) Error() string

func (*CleanupError) Unwrap

func (e *CleanupError) Unwrap() error

type Detector

type Detector func(head []byte) string

Detector returns a file's media type (with parameters, if any) from its first bytes: at most SniffBytes of them, fewer when the file is shorter. It gets a copy of them, its own to keep or change: nothing it does to it changes the file stored.

type File

type File struct {
	storage.ObjectInfo
	// Filename is the client's filename, cleaned (empty when it sent none):
	// for display and Content-Disposition only, never a path. It is the
	// stored FilenameMetadata (ObjectInfo.StoredFilename).
	Filename string
}

File is a stored upload.

func Confirm

func Confirm(ctx context.Context, store storage.Storage, key string, p Policy) (File, error)

Confirm checks the direct upload at key (a Grant's Key) against p once the client says it is done: it must be stored (ErrNoFile otherwise), at most p.MaxBytes long (ErrTooLarge), and of a type p accepts as detected from its bytes, the same media type it was declared as (ErrType: a declared image/png whose bytes are HTML is refused, since the declared type is what the store serves it as). A file that fails is deleted.

Under Policy.Claims, the upload was staged (Authorize): Confirm promotes the claim (Claimer.Promote), checks the staged object, and publishes a copy of it at key (promote: prepared, recorded on the claim, then published, under the promotion's lease, Policy.UploadTimeout), then deletes the staged object. A refused staged object is deleted and the claim goes back to pending, so the grant can upload again. A copy whose outcome is unknown leaves the claim promoting with the copy recorded (it may exist): a retried Confirm then checks key as it is, and the claims protocol fences the copy before it ever forgets the key. A key that is not a staged upload awaiting confirmation (already promoted or held: a retried confirmation; or stored by Save) is checked as it is, and a refusal deletes nothing.

key must be one the application granted (Policy.Prefix is checked, as a guard): Confirm checks the file, not who may claim it.

func Receive

func Receive(store storage.Storage, r *http.Request, p Policy) (File, error)

Receive stores the file of a multipart/form-data request (the part in p.Field with a filename; an empty file input sends none) and reads the rest of the request, so that a second file is refused. Everything but the file (part headers, boundaries, other fields) may be at most MaxFormBytes, on its own: other fields fail with ErrTooLarge as soon as they pass it, and the request as a whole is checked at its end, which is the end of the body (bytes after the closing boundary count too), not the closing boundary. The request may be at most p.MaxBytes plus MaxFormBytes long: a longer declared length fails before a byte is read, and a longer body when it gets there. If the request turns out to be invalid after the file was stored, the file is deleted.

func ReceiveBody

func ReceiveBody(store storage.Storage, r *http.Request, p Policy) (File, error)

ReceiveBody stores a request's body as the file: its declared length, if any, must be at most p.MaxBytes (checked before a byte is read) and must match the body. The filename, if any, is the filename parameter of the request's Content-Disposition header; the Content-Type header is not trusted.

func Save

func Save(ctx context.Context, store storage.Storage, src io.Reader, filename string, p Policy) (File, error)

Save stores the file src under a generated key, applying p: its first bytes decide its media type, and it fails with ErrTooLarge at the byte past p.MaxBytes. filename is the client's name for it (metadata only; empty when none). A file Save refuses is not stored.

type Grant

type Grant struct {
	// Key is where the file will be: Policy.Prefix and a random id.
	Key string `json:"key"`
	// Request is the upload for the client to make.
	Request storage.UploadRequest `json:"upload"`
}

Grant is a direct upload granted by Authorize: the client sends Request (the file's bytes as its body), and the application keeps Key to Confirm afterwards.

func Authorize

func Authorize(ctx context.Context, store storage.Storage, p Policy, size int64, contentType, filename string) (Grant, error)

Authorize grants a direct upload of one file the client declares: size bytes (at most p.MaxBytes, else ErrTooLarge) of contentType (one p.Types accepts, else ErrType), with the client's filename kept as metadata. The key is generated, and the store refuses any other length or type; the declared type is only a claim, which Confirm checks against the bytes. Decide that the caller may upload before calling it, and remember the Key (with who asked) to Confirm the same one: a grant is not a record that the upload happened.

Under Policy.Claims the upload is staged: the key is claimed (Stage), and the request uploads to StagingKey(key), never to the key itself, which only Confirm writes, by copying the staged object once it passes. A PUT the application cannot end (S3 checks a presigned URL's expiry when the request starts) can then only ever publish a staging object.

It fails with storage.ErrUnsupported when the store has no direct uploads (storage.DirectUploader); Receive is the fallback.

type Policy

type Policy struct {
	// MaxBytes is the largest file accepted, in bytes. Required.
	MaxBytes int64
	// Types lists the media types accepted, as detected from the file's
	// content: "image/png", or "image/*" for a whole top-level type.
	// Required: "*/*" accepts any type, and must be asked for.
	Types []string
	// Prefix starts every generated key ("avatars/"): empty, or a valid
	// key path ending with '/'.
	Prefix string
	// Field is the multipart form field holding the file (DefaultField
	// when empty). Receive only.
	Field string
	// Detect detects the media type (http.DetectContentType when nil). The
	// default recognizes common images, audio, video, PDF, archives, and
	// fonts, and reports text as text/plain or text/html; it never reports
	// image/svg+xml or application/json. Supply a Detector for types it
	// does not recognize.
	Detect Detector
	// Metadata is stored with every file, besides FilenameMetadata (which
	// it must not set).
	Metadata map[string]string
	// GrantExpiry is how long a direct upload grant (Authorize) works,
	// counted from the start of the second it is signed in, as for any
	// signed URL (DefaultGrantExpiry when zero; whole seconds, at most
	// storage.MaxURLExpiry).
	GrantExpiry time.Duration
	// UploadTimeout bounds how long the application may take to write a
	// claimed key under Claims (DefaultUploadTimeout when zero): Save's and
	// Receive's Put, and Confirm's copy of a staged upload, are aborted
	// then, before they publish, so the key's claim can lease it until a
	// known time (storage/claims). Unused without Claims.
	UploadTimeout time.Duration
	// Claims, when set, puts uploads under an ownership protocol
	// (storage/claims: a *claims.Claims). Each generated key is claimed
	// (Pending) before anything is stored or granted under it, and a
	// failure to claim fails the upload. A file that fails (an invalid
	// request, a refused confirmation) is deleted through Abandon, which
	// deletes it only while its claim is pending: never a file a record
	// holds, nor one under a key that was never claimed.
	Claims Claimer
	// Scope names who the uploads are for under Claims (a field:
	// "document.attachment"), at most 255 bytes. Each key is claimed for
	// it, and Confirm accepts only a key claimed for the same scope: a
	// grant from one field cannot be attached to another, whatever their
	// prefixes. Give every policy sharing a store under Claims its own.
	Scope string
}

Policy is what an upload must satisfy and where it is stored.

func (Policy) Validate

func (p Policy) Validate() error

Validate reports whether p is a usable policy (the check every function here makes first): a positive MaxBytes, Types listed, a valid Prefix and Metadata, a GrantExpiry within storage.MaxURLExpiry.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL