upload

package
v1.7.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package upload provides the shared chunked archive upload flow.

Index

Constants

View Source
const (
	// UploadFootprintFactor is the number of declared-payload-sized files that
	// can exist on the store filesystem at once, read off the code paths in
	// chunk.go: every chunk is published under the session directory, merge
	// copies all of them into `.merged-*.zip` while they are still present
	// (two copies at the merge peak), and the backup finalizer stages a third
	// copy in ./data before the restore, while the chunks and archive.zip are
	// still there. Three is the measured peak rather than a round number: the
	// rename that publishes archive.zip replaces the merge scratch file, so
	// those two never coexist.
	UploadFootprintFactor int64 = 3

	// StagingExpansionFactor bounds extracted or restored content as a
	// multiple of the declared payload. Extraction is bounded per package
	// rather than per byte: internal/plugin and web/api/admin cap a theme or
	// plugin at 512 MiB extracted, and web/backup accepts a backup whose
	// entries expand to at most backup.MaxArchiveSize (4 GiB), the same value
	// as the largest payload this store admits.
	StagingExpansionFactor int64 = 2

	// StagingExpansionFloor keeps the budget honest for small payloads: a few
	// megabytes of ZIP can expand to the 512 MiB extraction cap, so a pure
	// factor would under-count them. The value matches maxPluginExtractedSize
	// and maxThemeExtractedSize. The one case it does not fully cover is a
	// small backup whose restore expands to the 4 GiB backup cap; that restore
	// runs on the next startup, after Complete has already released the
	// session's chunks and archive, so the reserve floor absorbs it instead.
	StagingExpansionFloor int64 = 512 << 20

	// FreeSpaceReserve is the floor left untouched on the store filesystem for
	// everything else living beside the uploads: the database and its
	// journal/WAL, logs, runtime data directories and the staged restore that
	// runs on the next startup. 1 GiB covers those writes on the supported
	// deployments while staying small next to the 8 GiB reservation budget.
	FreeSpaceReserve int64 = 1 << 30
)

The store reservation counts the declared payload, but the flow holds several copies of those bytes before the session is released, and the store root shares its filesystem with the primary database, its journal, the logs and the extracted theme/plugin trees. Admission therefore has to reserve more than the payload itself.

View Source
const ChunkSize int64 = 5 * 1024 * 1024

Variables

View Source
var DefaultStore = &Store{
	Root:    filepath.Join(".", "data", ".uploading"),
	MaxSize: backup.MaxArchiveSize,
}
View Source
var ErrBusy = errors.New("upload store busy; retry later")
View Source
var ErrNoSpace = errors.New("insufficient free disk space for upload")

ErrNoSpace reports that the store refused an upload because the filesystem holding the store root cannot absorb the flow's worst-case footprint, or because it could not answer the free-space query at all.

View Source
var ErrNotFound = errors.New("upload not found")
View Source
var ErrQuota = errors.New("upload quota exceeded")

Functions

func RequiredFreeSpace added in v0.1.20

func RequiredFreeSpace(size int64) int64

RequiredFreeSpace is the free space a payload of size bytes needs before it can be admitted. It is exported so operators and tests can reason about the budget without duplicating the formula.

Types

type CleanupStats added in v0.1.20

type CleanupStats struct {
	// Sessions and ReservedBytes describe the declared reservations, not the
	// bytes currently allocated; the free-space budget is what bounds the
	// allocation those reservations imply.
	Sessions         int           `json:"sessions"`
	ReservedBytes    int64         `json:"reserved_bytes"`
	OldestSessionAge time.Duration `json:"oldest_session_age"`
	LastScan         time.Time     `json:"last_scan"`
	// LastScanDurationNS measures the most recent successful reservation scan,
	// excluding time spent waiting for the store lock. Fast scans can measure
	// zero on clocks with coarse resolution.
	LastScanDurationNS int64 `json:"last_scan_duration_ns"`
	// LastSuccess is the last cleanup pass that completed without error.
	LastSuccess time.Time `json:"last_success"`
	// LastError is the most recent cleanup failure, empty after a success.
	LastError string `json:"last_error,omitempty"`
	// ReclaimedFiles and ReclaimedBytes are running totals for the abandoned
	// scratch files and empty reservation directories that reconciliation has
	// removed since the process started.
	ReclaimedFiles int   `json:"reclaimed_files"`
	ReclaimedBytes int64 `json:"reclaimed_bytes"`
	// Contention reports how often the store lock refused a caller and how long
	// it was held. It is the evidence for whether the single lock needs
	// splitting, which is why it is measured before anything is changed; see
	// contention.go.
	Contention ContentionStats `json:"contention"`
}

CleanupStats is a point-in-time view of the store for operators. Session counts and reservations are refreshed by every successful scan, so reading the stats performs no disk I/O and never waits for an in-flight upload.

type ContentionStats added in v0.1.20

type ContentionStats struct {
	// Busy counts refused lock acquisitions by operation: the direct evidence
	// of one caller blocking another.
	Busy map[string]int `json:"busy"`
	// Hold counts completed lock holds by operation, with total and worst-case
	// duration. Complete is the one to watch: it holds the lock across the whole
	// installation, which is the longest hold by construction.
	Hold map[string]OpTiming `json:"hold"`
}

ContentionStats answers one question with data instead of a guess: is the single store lock actually refusing unrelated work?

Every write path -- chunk saves, merge, plugin and theme installation, backup finalization -- shares one non-queuing lock, so a caller that finds it held is answered 429 and has to retry. Splitting that lock into per-session exclusion plus a small global I/O semaphore is only worth its risk if the refusals and the hold durations are real, so they are measured first.

Measured before this was written: the production panel had served **zero** archive uploads across every retained nginx log (all rotations, including the compressed ones), the upload store was empty, and there was not one 429 on an upload route. There was no contention to find, so the lock was deliberately left alone and this counter added in its place. If Busy stays empty in production, the lock is not the problem and should stay as it is.

type Finalizer

type Finalizer func(Session) (Result, error)

type Handler

type Handler struct {
	Store      *Store
	Finalizers map[Purpose]Finalizer
}

func NewHandler

func NewHandler(store *Store, finalizers map[Purpose]Finalizer) *Handler

func (*Handler) Cancel

func (h *Handler) Cancel(c *gin.Context)

func (*Handler) Chunk

func (h *Handler) Chunk(c *gin.Context)

func (*Handler) Init

func (h *Handler) Init(c *gin.Context)

func (*Handler) Merge

func (h *Handler) Merge(c *gin.Context)

type Metadata

type Metadata struct {
	Purpose  Purpose `json:"purpose"`
	Size     int64   `json:"size"`
	Filename string  `json:"filename"`
}

type OpTiming added in v0.1.20

type OpTiming struct {
	Count int           `json:"count"`
	Total time.Duration `json:"total"`
	Max   time.Duration `json:"max"`
}

OpTiming is how long one operation class held the store lock.

type Purpose

type Purpose string
const (
	PurposeBackup Purpose = "backup"
	PurposePlugin Purpose = "plugin"
	PurposeTheme  Purpose = "theme"
)

type Result

type Result struct {
	Message string
	Data    any
}

type Session

type Session struct {
	ID          string
	Metadata    Metadata
	Directory   string
	ArchivePath string
}

type Store

type Store struct {
	Root    string
	MaxSize int64
	// Zero values select 16 sessions, 8 GiB of reserved payload and 24 hours.
	MaxSessions  int
	MaxTotalSize int64
	TTL          time.Duration
	// FreeSpace reports the bytes available to this process on the filesystem
	// holding the store root. Nil selects the build-tagged platform query in
	// freespace_windows.go / freespace_unix.go / freespace_other.go; tests
	// inject a deterministic implementation so admission never depends on the
	// host's real disk.
	FreeSpace func(path string) (int64, error)
	// contains filtered or unexported fields
}

func (*Store) Cancel

func (s *Store) Cancel(uploadID string) error

func (*Store) CleanupExpired added in v0.1.17

func (s *Store) CleanupExpired() error

func (*Store) Complete added in v0.1.17

func (s *Store) Complete(uploadID string, finalize Finalizer) (Result, error)

Complete holds admission through finalization, so cancel/merge cannot remove an archive while an installer or backup restorer is reading it.

func (*Store) Contention added in v0.1.20

func (s *Store) Contention() ContentionStats

Contention returns a copy of the accumulated figures without clearing them.

func (*Store) Init

func (s *Store) Init(purpose Purpose, filename string, size int64) (Session, error)

func (*Store) Merge

func (s *Store) Merge(uploadID string) (Session, error)

func (*Store) ReconcileOrphans added in v0.1.20

func (s *Store) ReconcileOrphans() (int, int64, error)

ReconcileOrphans removes scratch files this package created and abandoned inside canonical upload directories, plus the empty reservation directories a process that died during Init left behind. It runs at store start, before the first admission: chunk.go holds the store lock for the whole of SaveChunk and merge, so a scratch file visible while the lock is held cannot belong to a live upload and must have been left by a process that died mid-write.

Only names this package writes are eligible (see isTemporaryStoreFile). Unknown files, unknown directories, symlinks and anything outside a canonical session directory are left in place: scan() reports them and admission fails, which is the documented behaviour for unrecognised data.

func (*Store) RunCleanup added in v0.1.17

func (s *Store) RunCleanup(ctx context.Context)

RunCleanup is owned by the server lifecycle, including first-run setup. The first pass reconciles scratch files left by a crashed process and expires sessions, then expiry repeats hourly. Both passes report through reportCleanup, so a failure is logged once per burst instead of being discarded.

func (*Store) SaveChunk

func (s *Store) SaveChunk(uploadID string, index int64, source io.Reader) error

func (*Store) Stats added in v0.1.20

func (s *Store) Stats() CleanupStats

Stats returns the cached maintenance view. It takes a dedicated lock and does no I/O, so it is safe to call while cleanup runs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL