Documentation
¶
Overview ¶
Package upload receives files without buffering them: streaming multipart parsing with per-route limits, content sniffing that must agree with the file extension, random stored names, an optional virus-scan step (ClamAV) before files become visible, resumable uploads (tus 1.0) and presigned direct-to-storage uploads.
form, err := upload.Receive(c, disk, upload.Images(5<<20))
if err != nil {
return err
}
avatar := form.Files[0] // Path, Name, Type, Size, SHA256
Rules say what a route accepts (a zero Rules rejects everything); Receive parses a request, Save stores a file from any reader, Tus serves resumable uploads and Presign / Direct hand out direct-to-storage uploads. Files land on a storage.Disk.
See https://github.com/AlexGalhardo/vukem/blob/main/docs/subsystems.md#uploads.
Index ¶
- Constants
- Variables
- func DetectableTypes() []string
- func Direct(cfg DirectConfig) (http.Handler, error)
- func Extensions(mimeType string) []string
- func SanitizeName(name string) string
- type ClamAVScanner
- type DirectConfig
- type File
- type Form
- type InfectedError
- type Presigned
- type Presigner
- type Rules
- type Scanner
- type TusConfig
- type TusHandler
- type TusInfo
Examples ¶
Constants ¶
const ( DefaultMaxFiles = 1 DefaultMaxFields = 32 DefaultMaxFieldSize = 64 << 10 DefaultQuarantineDir = "quarantine" )
Defaults applied by Rules when a limit is left at zero.
const DefaultMaxPixels = 25_000_000
DefaultMaxPixels is the width × height Images accepts (25 megapixels, about 100 MB once decoded to RGBA).
const (
DefaultPresignTTL = 15 * time.Minute
)
Defaults for presigned uploads.
const (
TusVersion = "1.0.0"
)
tus protocol constants (https://tus.io/protocols/resumable-upload).
Variables ¶
var ( // ErrFileTooLarge reports a file larger than Rules.MaxFileSize (413). ErrFileTooLarge = errs.New(http.StatusRequestEntityTooLarge, "The file is too large.") // ErrFieldTooLarge reports a form field larger than Rules.MaxFieldSize (413). ErrFieldTooLarge = errs.New(http.StatusRequestEntityTooLarge, "A form field is too large.") // ErrTooManyFields reports more than Rules.MaxFields fields (413). ErrTooManyFields = errs.New(http.StatusRequestEntityTooLarge, "Too many form fields.") // ErrNotMultipart reports a request that is not multipart/form-data (415). ErrNotMultipart = errs.New(http.StatusUnsupportedMediaType, "The request must be multipart/form-data.") )
Errors returned (as client-safe *errs.HTTPError values) by Receive and Save.
var ErrInfected = errors.New("upload: file is infected")
ErrInfected is wrapped by Scanner errors for infected files.
var ErrUndetectableType = errors.New("upload: this type cannot be detected from content")
ErrUndetectableType reports a Rules.Types entry this package cannot detect from a file's content. It is a configuration error, not a client's: the rule would reject every file of that type instead of accepting it, silently, because detection is by content and the detector cannot name the type at all.
The detectable set is the one Extensions knows (see Extensions), which is what http.DetectContentType can return. AVIF, HEIC, JPEG XL and TIFF are **not** in it, measured on Go 1.27.2. An application that needs one of them has to sniff with a real detector and call Save or [Rules.Check] with the type it found.
Functions ¶
func DetectableTypes ¶ added in v0.30.0
func DetectableTypes() []string
DetectableTypes are the media types this package can recognise from a file's content, sorted. Rules.Types accepts only these: detection is by content with http.DetectContentType, so a type it cannot name would make the rule reject every file of that type.
Measured on Go 1.27.2, these are **not** in it and so cannot be allowed: image/avif, image/heic, image/heif, image/jxl, image/tiff. The detector answers application/octet-stream for all of them.
That is also the escape hatch. "application/octet-stream" is detectable -- it is what the detector says when it cannot tell -- so a service handling AVIF or TIFF allows it here and checks the type itself with a real sniffer. Doing that deliberately is the point; having a rule name image/avif and quietly match nothing is not.
func Direct ¶
func Direct(cfg DirectConfig) (http.Handler, error)
Direct returns a handler for POST {"filename", "content_type", "size"} that answers with a Presigned upload as JSON. Missing Presigner or Authorize, or unusable Rules, are boot errors.
func Extensions ¶
Extensions returns the file extensions accepted for a detected MIME type.
func SanitizeName ¶
SanitizeName returns a display-safe version of a client file name: the last path element (either separator), without control and format (bidi, zero-width) characters, quotes or leading dots, at most 200 characters; "file" when nothing is left. It is meant for metadata (Content-Disposition, listings), never for storage paths.
Example ¶
package main
import (
"fmt"
"github.com/AlexGalhardo/vukem/upload"
)
func main() {
fmt.Println(upload.SanitizeName("../../etc/pass\x00wd.txt"))
}
Output: passwd.txt
Types ¶
type ClamAVScanner ¶
type ClamAVScanner struct {
// Addr is clamd's host:port.
Addr string
// Timeout bounds dialing and each read or write (default 30s).
Timeout time.Duration
// ChunkSize is the INSTREAM chunk size (default 64 KiB). Keep it
// below clamd's StreamMaxLength.
ChunkSize int
}
ClamAVScanner scans files with clamd's INSTREAM command over TCP.
func ClamAV ¶
func ClamAV(addr string) *ClamAVScanner
ClamAV returns a Scanner talking to clamd at addr (for example "127.0.0.1:3310").
type DirectConfig ¶
type DirectConfig struct {
Presigner Presigner
Rules Rules
// TTL of the presigned URLs (default 15 minutes, at most 1 hour).
TTL time.Duration
// Authorize runs first; an error denies the request (required).
Authorize func(r *http.Request) error
}
DirectConfig configures Direct.
type File ¶
type File struct {
// Field is the form field name.
Field string `json:"field"`
// Name is the client's file name, sanitized. It is metadata only and
// never part of Path.
Name string `json:"name"`
// Path is where the file was stored on the disk (random name).
Path string `json:"path"`
// Type is the MIME type detected from the content.
Type string `json:"type"`
// Size in bytes.
Size int64 `json:"size"`
// SHA256 is the hex digest of the content.
SHA256 string `json:"sha256"`
}
File is a stored upload.
func Save ¶
func Save(ctx context.Context, d storage.Disk, rules Rules, field, filename string, src io.Reader) (File, error)
Save validates and stores one file read from src under rules (the same checks as Receive): use it for files from other sources, such as a finished tus upload. field names the input in validation errors.
Example ¶
package main
import (
"bytes"
"context"
"fmt"
"strings"
"github.com/AlexGalhardo/vukem/storage"
"github.com/AlexGalhardo/vukem/upload"
)
func main() {
ctx := context.Background()
disk := storage.NewMemory("/storage")
rules := upload.Images(2 << 20)
rules.Dir = "avatars"
// A 1x1 PNG header: Images also checks the declared dimensions.
png := []byte("\x89PNG\r\n\x1a\n" + "\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x00\x00\x00\x00:~\x9bU")
f, err := upload.Save(ctx, disk, rules, "avatar", "../../Me.png", bytes.NewReader(png))
if err != nil {
fmt.Println(err)
return
}
// The stored name is random; the client's name is sanitized metadata.
fmt.Println(f.Type, f.Name, strings.HasPrefix(f.Path, "avatars/"))
// Content that does not match an allowed type is rejected.
_, err = upload.Save(ctx, disk, rules, "avatar", "evil.png", strings.NewReader("<script>"))
fmt.Println(err != nil)
}
Output: image/png Me.png true true
func Verify ¶
Verify checks an object already on d (a finished direct upload) against rules: detected type, extension of key, size and, for images, the declared dimensions (Rules.MaxPixels). Only keys Presign could have issued for rules are accepted (400 otherwise, without touching the disk). It deletes the object when it fails and returns its File description otherwise.
type Form ¶
Form is a received multipart form.
func Receive ¶
Receive reads the multipart body of c part by part and streams each file to d: nothing is buffered beyond a small read buffer, whatever the file size. It enforces rules (sizes → 413, types, extensions and file count → 422) and removes every file it wrote when any part fails, so a rejected request leaves nothing behind. Files are stored under rules.Dir with random names; client file names are kept only as File.Name.
Nothing may read the body before Receive: MethodOverride leaves multipart bodies unread, and Ctx.Input/FormFile must not be called.
type InfectedError ¶
type InfectedError struct {
Signature string
}
InfectedError names the signature a scanner found.
func (*InfectedError) Unwrap ¶
func (e *InfectedError) Unwrap() error
Unwrap makes errors.Is(err, ErrInfected) true.
type Presigned ¶
type Presigned struct {
// Key is the object's path on the disk (random name under Rules.Dir).
Key string `json:"key"`
// URL, Method and Headers describe the request to send.
URL string `json:"url"`
Method string `json:"method"`
Headers map[string]string `json:"headers,omitempty"`
// ExpiresAt is when the URL stops working.
ExpiresAt time.Time `json:"expires_at"`
}
Presigned is a direct upload the client performs itself.
func Presign ¶
func Presign( ctx context.Context, p Presigner, rules Rules, filename, contentType string, size int64, ttl time.Duration, ) (Presigned, error)
Presign checks a planned upload against rules (declared type allowed, extension matching it, 0 < size <= MaxFileSize) and returns a presigned PUT for a fresh random key. The client declares the type, so call Verify once the client reports the upload done: it sniffs the stored content and deletes objects that break the rules.
type Presigner ¶
type Presigner interface {
PresignPut(
ctx context.Context,
key, contentType string,
maxSize int64,
ttl time.Duration,
) (url string, headers http.Header, err error)
}
Presigner issues URLs that let a client upload one object straight to storage. The signature must bind the key, the content type and the maximum size, and expire after ttl. drivers/s3's PresignPut signs the key and expiry only, so an adapter over it must add the size and type conditions (or rely on Verify, which runs after the upload).
type Rules ¶
type Rules struct {
// MaxFileSize is the largest file in bytes (required).
MaxFileSize int64
// MaxSize caps the whole request body; 0 = MaxFiles*MaxFileSize + 1 MiB.
// It replaces the router-wide body limit for this request.
MaxSize int64
// MaxFiles is the number of files accepted (default 1).
MaxFiles int
// MaxPixels bounds width × height of JPEG, PNG, GIF and WebP files,
// read from their header after storing (decompression bombs); files of
// those types without a readable header are rejected. Images sets
// DefaultMaxPixels; 0 skips the check.
MaxPixels int64
// Types are the allowed MIME types, matched against the type detected
// from the content (http.DetectContentType); the client's
// Content-Type is ignored (required).
Types []string
// Extensions optionally narrow the accepted file name extensions
// (".jpg"). The extension must always match the detected type.
Extensions []string
// Dir is the directory files are stored under.
Dir string
// MaxFields and MaxFieldSize bound non-file fields (defaults 32, 64 KiB).
MaxFields int
MaxFieldSize int64
// Scanner, when set, scans every file in QuarantineDir before moving
// it to Dir; infected files are deleted and rejected.
Scanner Scanner
// QuarantineDir holds files until they are scanned (default
// "quarantine"). Quarantined names start with a dot, so
// storage.Handler never serves them.
QuarantineDir string
}
Rules restrict what Receive, Save, Verify and Presign accept. MaxFileSize and Types are required: a zero Rules rejects every upload (fail closed).
type Scanner ¶
Scanner inspects a file before it becomes visible. Scan returns an error wrapping ErrInfected for malware; any other error means the scan could not run, and the upload is rejected (fail closed).
type TusConfig ¶
type TusConfig struct {
// Dir holds partial uploads (opened with os.Root; created if needed).
// Keep it outside any publicly served directory.
Dir string
// BasePath is the URL path the handler is mounted at ("/uploads/tus/");
// upload URLs are BasePath + id.
BasePath string
// MaxSize is the largest Upload-Length accepted (required).
MaxSize int64
// Authorize runs before every request except OPTIONS; an error denies
// it (its status when it has one, 403 otherwise). Required: upload
// endpoints are never open by accident.
Authorize func(r *http.Request) error
// OnComplete receives a finished upload. open reads its bytes (close
// it before returning). It typically calls Save to validate the
// content and move it to a disk. The partial data is deleted after it
// returns; an error is sent to the client (422 for validation errors).
OnComplete func(ctx context.Context, info TusInfo, open func() (io.ReadCloser, error)) error
// Expiry is how long an unfinished upload is kept (default 24h).
// Call Cleanup periodically (scheduler) to delete expired uploads.
Expiry time.Duration
// Now is the clock (default time.Now).
Now func() time.Time
}
TusConfig configures Tus.
type TusHandler ¶
type TusHandler struct {
// contains filtered or unexported fields
}
TusHandler serves the tus 1.0 core protocol with the creation, expiration and termination extensions.
func Tus ¶
func Tus(cfg TusConfig) (*TusHandler, error)
Tus returns a tus 1.0 handler; mount it at cfg.BasePath (a subtree pattern, "/uploads/tus/"). It is a boot error to omit MaxSize, Authorize, OnComplete, Dir or BasePath. Close it on shutdown.
func (*TusHandler) Cleanup ¶
func (h *TusHandler) Cleanup(ctx context.Context) (int, error)
Cleanup deletes expired uploads and orphaned data older than Expiry, returning how many uploads it removed. Schedule it (hourly is plenty).
func (*TusHandler) ServeHTTP ¶
func (h *TusHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP implements http.Handler.