Help us shape the future of Go! Take the 2026 Go Developer Survey

upload

package
v0.30.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: MIT Imports: 36 Imported by: 0

Documentation

Overview

Package upload receives files without buffering them: streaming multipart parsing with per-route limits, content sniffing that must agree with the file extension, random stored names, an optional virus-scan step (ClamAV) before files become visible, resumable uploads (tus 1.0) and presigned direct-to-storage uploads.

form, err := upload.Receive(c, disk, upload.Images(5<<20))
if err != nil {
	return err
}
avatar := form.Files[0] // Path, Name, Type, Size, SHA256

Rules say what a route accepts (a zero Rules rejects everything); Receive parses a request, Save stores a file from any reader, Tus serves resumable uploads and Presign / Direct hand out direct-to-storage uploads. Files land on a storage.Disk.

See https://github.com/AlexGalhardo/vukem/blob/main/docs/subsystems.md#uploads.

Index

Examples

Constants

View Source
const (
	DefaultMaxFiles      = 1
	DefaultMaxFields     = 32
	DefaultMaxFieldSize  = 64 << 10
	DefaultQuarantineDir = "quarantine"
)

Defaults applied by Rules when a limit is left at zero.

View Source
const DefaultMaxPixels = 25_000_000

DefaultMaxPixels is the width × height Images accepts (25 megapixels, about 100 MB once decoded to RGBA).

View Source
const (
	DefaultPresignTTL = 15 * time.Minute
)

Defaults for presigned uploads.

View Source
const (
	TusVersion = "1.0.0"
)

tus protocol constants (https://tus.io/protocols/resumable-upload).

Variables

View Source
var (
	// ErrFileTooLarge reports a file larger than Rules.MaxFileSize (413).
	ErrFileTooLarge = errs.New(http.StatusRequestEntityTooLarge, "The file is too large.")
	// ErrFieldTooLarge reports a form field larger than Rules.MaxFieldSize (413).
	ErrFieldTooLarge = errs.New(http.StatusRequestEntityTooLarge, "A form field is too large.")
	// ErrTooManyFields reports more than Rules.MaxFields fields (413).
	ErrTooManyFields = errs.New(http.StatusRequestEntityTooLarge, "Too many form fields.")
	// ErrNotMultipart reports a request that is not multipart/form-data (415).
	ErrNotMultipart = errs.New(http.StatusUnsupportedMediaType, "The request must be multipart/form-data.")
)

Errors returned (as client-safe *errs.HTTPError values) by Receive and Save.

View Source
var ErrInfected = errors.New("upload: file is infected")

ErrInfected is wrapped by Scanner errors for infected files.

View Source
var ErrUndetectableType = errors.New("upload: this type cannot be detected from content")

ErrUndetectableType reports a Rules.Types entry this package cannot detect from a file's content. It is a configuration error, not a client's: the rule would reject every file of that type instead of accepting it, silently, because detection is by content and the detector cannot name the type at all.

The detectable set is the one Extensions knows (see Extensions), which is what http.DetectContentType can return. AVIF, HEIC, JPEG XL and TIFF are **not** in it, measured on Go 1.27.2. An application that needs one of them has to sniff with a real detector and call Save or [Rules.Check] with the type it found.

Functions

func DetectableTypes added in v0.30.0

func DetectableTypes() []string

DetectableTypes are the media types this package can recognise from a file's content, sorted. Rules.Types accepts only these: detection is by content with http.DetectContentType, so a type it cannot name would make the rule reject every file of that type.

Measured on Go 1.27.2, these are **not** in it and so cannot be allowed: image/avif, image/heic, image/heif, image/jxl, image/tiff. The detector answers application/octet-stream for all of them.

That is also the escape hatch. "application/octet-stream" is detectable -- it is what the detector says when it cannot tell -- so a service handling AVIF or TIFF allows it here and checks the type itself with a real sniffer. Doing that deliberately is the point; having a rule name image/avif and quietly match nothing is not.

func Direct

func Direct(cfg DirectConfig) (http.Handler, error)

Direct returns a handler for POST {"filename", "content_type", "size"} that answers with a Presigned upload as JSON. Missing Presigner or Authorize, or unusable Rules, are boot errors.

func Extensions

func Extensions(mimeType string) []string

Extensions returns the file extensions accepted for a detected MIME type.

func SanitizeName

func SanitizeName(name string) string

SanitizeName returns a display-safe version of a client file name: the last path element (either separator), without control and format (bidi, zero-width) characters, quotes or leading dots, at most 200 characters; "file" when nothing is left. It is meant for metadata (Content-Disposition, listings), never for storage paths.

Example
package main

import (
	"fmt"

	"github.com/AlexGalhardo/vukem/upload"
)

func main() {
	fmt.Println(upload.SanitizeName("../../etc/pass\x00wd.txt"))
}
Output:
passwd.txt

Types

type ClamAVScanner

type ClamAVScanner struct {
	// Addr is clamd's host:port.
	Addr string
	// Timeout bounds dialing and each read or write (default 30s).
	Timeout time.Duration
	// ChunkSize is the INSTREAM chunk size (default 64 KiB). Keep it
	// below clamd's StreamMaxLength.
	ChunkSize int
}

ClamAVScanner scans files with clamd's INSTREAM command over TCP.

func ClamAV

func ClamAV(addr string) *ClamAVScanner

ClamAV returns a Scanner talking to clamd at addr (for example "127.0.0.1:3310").

func (*ClamAVScanner) Scan

func (s *ClamAVScanner) Scan(ctx context.Context, r io.Reader) error

Scan implements Scanner.

type DirectConfig

type DirectConfig struct {
	Presigner Presigner
	Rules     Rules
	// TTL of the presigned URLs (default 15 minutes, at most 1 hour).
	TTL time.Duration
	// Authorize runs first; an error denies the request (required).
	Authorize func(r *http.Request) error
}

DirectConfig configures Direct.

type File

type File struct {
	// Field is the form field name.
	Field string `json:"field"`
	// Name is the client's file name, sanitized. It is metadata only and
	// never part of Path.
	Name string `json:"name"`
	// Path is where the file was stored on the disk (random name).
	Path string `json:"path"`
	// Type is the MIME type detected from the content.
	Type string `json:"type"`
	// Size in bytes.
	Size int64 `json:"size"`
	// SHA256 is the hex digest of the content.
	SHA256 string `json:"sha256"`
}

File is a stored upload.

func Save

func Save(ctx context.Context, d storage.Disk, rules Rules, field, filename string, src io.Reader) (File, error)

Save validates and stores one file read from src under rules (the same checks as Receive): use it for files from other sources, such as a finished tus upload. field names the input in validation errors.

Example
package main

import (
	"bytes"
	"context"
	"fmt"
	"strings"

	"github.com/AlexGalhardo/vukem/storage"
	"github.com/AlexGalhardo/vukem/upload"
)

func main() {
	ctx := context.Background()
	disk := storage.NewMemory("/storage")

	rules := upload.Images(2 << 20)
	rules.Dir = "avatars"

	// A 1x1 PNG header: Images also checks the declared dimensions.
	png := []byte("\x89PNG\r\n\x1a\n" + "\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x00\x00\x00\x00:~\x9bU")
	f, err := upload.Save(ctx, disk, rules, "avatar", "../../Me.png", bytes.NewReader(png))
	if err != nil {
		fmt.Println(err)
		return
	}

	// The stored name is random; the client's name is sanitized metadata.
	fmt.Println(f.Type, f.Name, strings.HasPrefix(f.Path, "avatars/"))

	// Content that does not match an allowed type is rejected.
	_, err = upload.Save(ctx, disk, rules, "avatar", "evil.png", strings.NewReader("<script>"))
	fmt.Println(err != nil)
}
Output:
image/png Me.png true
true

func Verify

func Verify(ctx context.Context, d storage.Disk, key string, rules Rules) (File, error)

Verify checks an object already on d (a finished direct upload) against rules: detected type, extension of key, size and, for images, the declared dimensions (Rules.MaxPixels). Only keys Presign could have issued for rules are accepted (400 otherwise, without touching the disk). It deletes the object when it fails and returns its File description otherwise.

type Form

type Form struct {
	Values url.Values
	Files  []File
}

Form is a received multipart form.

func Receive

func Receive(c *web.Ctx, d storage.Disk, rules Rules) (*Form, error)

Receive reads the multipart body of c part by part and streams each file to d: nothing is buffered beyond a small read buffer, whatever the file size. It enforces rules (sizes → 413, types, extensions and file count → 422) and removes every file it wrote when any part fails, so a rejected request leaves nothing behind. Files are stored under rules.Dir with random names; client file names are kept only as File.Name.

Nothing may read the body before Receive: MethodOverride leaves multipart bodies unread, and Ctx.Input/FormFile must not be called.

type InfectedError

type InfectedError struct {
	Signature string
}

InfectedError names the signature a scanner found.

func (*InfectedError) Error

func (e *InfectedError) Error() string

Error implements error.

func (*InfectedError) Unwrap

func (e *InfectedError) Unwrap() error

Unwrap makes errors.Is(err, ErrInfected) true.

type Presigned

type Presigned struct {
	// Key is the object's path on the disk (random name under Rules.Dir).
	Key string `json:"key"`
	// URL, Method and Headers describe the request to send.
	URL     string            `json:"url"`
	Method  string            `json:"method"`
	Headers map[string]string `json:"headers,omitempty"`
	// ExpiresAt is when the URL stops working.
	ExpiresAt time.Time `json:"expires_at"`
}

Presigned is a direct upload the client performs itself.

func Presign

func Presign(
	ctx context.Context,
	p Presigner,
	rules Rules,
	filename, contentType string,
	size int64,
	ttl time.Duration,
) (Presigned, error)

Presign checks a planned upload against rules (declared type allowed, extension matching it, 0 < size <= MaxFileSize) and returns a presigned PUT for a fresh random key. The client declares the type, so call Verify once the client reports the upload done: it sniffs the stored content and deletes objects that break the rules.

type Presigner

type Presigner interface {
	PresignPut(
		ctx context.Context,
		key, contentType string,
		maxSize int64,
		ttl time.Duration,
	) (url string, headers http.Header, err error)
}

Presigner issues URLs that let a client upload one object straight to storage. The signature must bind the key, the content type and the maximum size, and expire after ttl. drivers/s3's PresignPut signs the key and expiry only, so an adapter over it must add the size and type conditions (or rely on Verify, which runs after the upload).

type Rules

type Rules struct {
	// MaxFileSize is the largest file in bytes (required).
	MaxFileSize int64
	// MaxSize caps the whole request body; 0 = MaxFiles*MaxFileSize + 1 MiB.
	// It replaces the router-wide body limit for this request.
	MaxSize int64
	// MaxFiles is the number of files accepted (default 1).
	MaxFiles int
	// MaxPixels bounds width × height of JPEG, PNG, GIF and WebP files,
	// read from their header after storing (decompression bombs); files of
	// those types without a readable header are rejected. Images sets
	// DefaultMaxPixels; 0 skips the check.
	MaxPixels int64
	// Types are the allowed MIME types, matched against the type detected
	// from the content (http.DetectContentType); the client's
	// Content-Type is ignored (required).
	Types []string
	// Extensions optionally narrow the accepted file name extensions
	// (".jpg"). The extension must always match the detected type.
	Extensions []string
	// Dir is the directory files are stored under.
	Dir string
	// MaxFields and MaxFieldSize bound non-file fields (defaults 32, 64 KiB).
	MaxFields    int
	MaxFieldSize int64
	// Scanner, when set, scans every file in QuarantineDir before moving
	// it to Dir; infected files are deleted and rejected.
	Scanner Scanner
	// QuarantineDir holds files until they are scanned (default
	// "quarantine"). Quarantined names start with a dot, so
	// storage.Handler never serves them.
	QuarantineDir string
}

Rules restrict what Receive, Save, Verify and Presign accept. MaxFileSize and Types are required: a zero Rules rejects every upload (fail closed).

func Images

func Images(maxSize int64) Rules

Images allows JPEG, PNG, GIF and WebP files up to maxSize bytes and DefaultMaxPixels.

type Scanner

type Scanner interface {
	Scan(ctx context.Context, r io.Reader) error
}

Scanner inspects a file before it becomes visible. Scan returns an error wrapping ErrInfected for malware; any other error means the scan could not run, and the upload is rejected (fail closed).

type TusConfig

type TusConfig struct {
	// Dir holds partial uploads (opened with os.Root; created if needed).
	// Keep it outside any publicly served directory.
	Dir string
	// BasePath is the URL path the handler is mounted at ("/uploads/tus/");
	// upload URLs are BasePath + id.
	BasePath string
	// MaxSize is the largest Upload-Length accepted (required).
	MaxSize int64
	// Authorize runs before every request except OPTIONS; an error denies
	// it (its status when it has one, 403 otherwise). Required: upload
	// endpoints are never open by accident.
	Authorize func(r *http.Request) error
	// OnComplete receives a finished upload. open reads its bytes (close
	// it before returning). It typically calls Save to validate the
	// content and move it to a disk. The partial data is deleted after it
	// returns; an error is sent to the client (422 for validation errors).
	OnComplete func(ctx context.Context, info TusInfo, open func() (io.ReadCloser, error)) error
	// Expiry is how long an unfinished upload is kept (default 24h).
	// Call Cleanup periodically (scheduler) to delete expired uploads.
	Expiry time.Duration
	// Now is the clock (default time.Now).
	Now func() time.Time
}

TusConfig configures Tus.

type TusHandler

type TusHandler struct {
	// contains filtered or unexported fields
}

TusHandler serves the tus 1.0 core protocol with the creation, expiration and termination extensions.

func Tus

func Tus(cfg TusConfig) (*TusHandler, error)

Tus returns a tus 1.0 handler; mount it at cfg.BasePath (a subtree pattern, "/uploads/tus/"). It is a boot error to omit MaxSize, Authorize, OnComplete, Dir or BasePath. Close it on shutdown.

func (*TusHandler) Cleanup

func (h *TusHandler) Cleanup(ctx context.Context) (int, error)

Cleanup deletes expired uploads and orphaned data older than Expiry, returning how many uploads it removed. Schedule it (hourly is plenty).

func (*TusHandler) Close

func (h *TusHandler) Close() error

Close releases the upload directory.

func (*TusHandler) ServeHTTP

func (h *TusHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler.

type TusInfo

type TusInfo struct {
	ID       string            `json:"id"`
	Length   int64             `json:"length"`
	Offset   int64             `json:"-"`
	Metadata map[string]string `json:"metadata,omitempty"`
	Created  time.Time         `json:"created"`
	Expires  time.Time         `json:"expires"`
}

TusInfo describes a resumable upload.

func (TusInfo) Filename

func (i TusInfo) Filename() string

Filename returns the "filename" metadata value sent by the client (untrusted: pass it through Save, which sanitizes it).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL