Documentation
¶
Overview ¶
Package upload provides the shared chunked archive upload flow.
Index ¶
- Constants
- Variables
- func RequiredFreeSpace(size int64) int64
- type CleanupStats
- type ContentionStats
- type Finalizer
- type Handler
- type Metadata
- type OpTiming
- type Purpose
- type Result
- type Session
- type Store
- func (s *Store) Cancel(uploadID string) error
- func (s *Store) CleanupExpired() error
- func (s *Store) Complete(uploadID string, finalize Finalizer) (Result, error)
- func (s *Store) Contention() ContentionStats
- func (s *Store) Init(purpose Purpose, filename string, size int64) (Session, error)
- func (s *Store) Merge(uploadID string) (Session, error)
- func (s *Store) ReconcileOrphans() (int, int64, error)
- func (s *Store) RunCleanup(ctx context.Context)
- func (s *Store) SaveChunk(uploadID string, index int64, source io.Reader) error
- func (s *Store) Stats() CleanupStats
Constants ¶
const ( // UploadFootprintFactor is the number of declared-payload-sized files that // can exist on the store filesystem at once, read off the code paths in // chunk.go: every chunk is published under the session directory, merge // copies all of them into `.merged-*.zip` while they are still present // (two copies at the merge peak), and the backup finalizer stages a third // copy in ./data before the restore, while the chunks and archive.zip are // still there. Three is the measured peak rather than a round number: the // rename that publishes archive.zip replaces the merge scratch file, so // those two never coexist. UploadFootprintFactor int64 = 3 // StagingExpansionFactor bounds extracted or restored content as a // multiple of the declared payload. Extraction is bounded per package // rather than per byte: internal/plugin and web/api/admin cap a theme or // plugin at 512 MiB extracted, and web/backup accepts a backup whose // entries expand to at most backup.MaxArchiveSize (4 GiB), the same value // as the largest payload this store admits. StagingExpansionFactor int64 = 2 // StagingExpansionFloor keeps the budget honest for small payloads: a few // megabytes of ZIP can expand to the 512 MiB extraction cap, so a pure // factor would under-count them. The value matches maxPluginExtractedSize // and maxThemeExtractedSize. The one case it does not fully cover is a // small backup whose restore expands to the 4 GiB backup cap; that restore // runs on the next startup, after Complete has already released the // session's chunks and archive, so the reserve floor absorbs it instead. StagingExpansionFloor int64 = 512 << 20 // FreeSpaceReserve is the floor left untouched on the store filesystem for // everything else living beside the uploads: the database and its // journal/WAL, logs, runtime data directories and the staged restore that // runs on the next startup. 1 GiB covers those writes on the supported // deployments while staying small next to the 8 GiB reservation budget. FreeSpaceReserve int64 = 1 << 30 )
The store reservation counts the declared payload, but the flow holds several copies of those bytes before the session is released, and the store root shares its filesystem with the primary database, its journal, the logs and the extracted theme/plugin trees. Admission therefore has to reserve more than the payload itself.
const ChunkSize int64 = 5 * 1024 * 1024
Variables ¶
var DefaultStore = &Store{ Root: filepath.Join(".", "data", ".uploading"), MaxSize: backup.MaxArchiveSize, }
var ErrBusy = errors.New("upload store busy; retry later")
var ErrNoSpace = errors.New("insufficient free disk space for upload")
ErrNoSpace reports that the store refused an upload because the filesystem holding the store root cannot absorb the flow's worst-case footprint, or because it could not answer the free-space query at all.
var ErrNotFound = errors.New("upload not found")
var ErrQuota = errors.New("upload quota exceeded")
Functions ¶
func RequiredFreeSpace ¶ added in v0.1.20
RequiredFreeSpace is the free space a payload of size bytes needs before it can be admitted. It is exported so operators and tests can reason about the budget without duplicating the formula.
Types ¶
type CleanupStats ¶ added in v0.1.20
type CleanupStats struct {
// Sessions and ReservedBytes describe the declared reservations, not the
// bytes currently allocated; the free-space budget is what bounds the
// allocation those reservations imply.
Sessions int `json:"sessions"`
ReservedBytes int64 `json:"reserved_bytes"`
OldestSessionAge time.Duration `json:"oldest_session_age"`
LastScan time.Time `json:"last_scan"`
// LastScanDurationNS measures the most recent successful reservation scan,
// excluding time spent waiting for the store lock. Fast scans can measure
// zero on clocks with coarse resolution.
LastScanDurationNS int64 `json:"last_scan_duration_ns"`
// LastSuccess is the last cleanup pass that completed without error.
LastSuccess time.Time `json:"last_success"`
// LastError is the most recent cleanup failure, empty after a success.
LastError string `json:"last_error,omitempty"`
// ReclaimedFiles and ReclaimedBytes are running totals for the abandoned
// scratch files and empty reservation directories that reconciliation has
// removed since the process started.
ReclaimedFiles int `json:"reclaimed_files"`
ReclaimedBytes int64 `json:"reclaimed_bytes"`
// Contention reports how often the store lock refused a caller and how long
// it was held. It is the evidence for whether the single lock needs
// splitting, which is why it is measured before anything is changed; see
// contention.go.
Contention ContentionStats `json:"contention"`
}
CleanupStats is a point-in-time view of the store for operators. Session counts and reservations are refreshed by every successful scan, so reading the stats performs no disk I/O and never waits for an in-flight upload.
type ContentionStats ¶ added in v0.1.20
type ContentionStats struct {
// Busy counts refused lock acquisitions by operation: the direct evidence
// of one caller blocking another.
Busy map[string]int `json:"busy"`
// Hold counts completed lock holds by operation, with total and worst-case
// duration. Complete is the one to watch: it holds the lock across the whole
// installation, which is the longest hold by construction.
Hold map[string]OpTiming `json:"hold"`
}
ContentionStats answers one question with data instead of a guess: is the single store lock actually refusing unrelated work?
Every write path -- chunk saves, merge, plugin and theme installation, backup finalization -- shares one non-queuing lock, so a caller that finds it held is answered 429 and has to retry. Splitting that lock into per-session exclusion plus a small global I/O semaphore is only worth its risk if the refusals and the hold durations are real, so they are measured first.
Measured before this was written: the production panel had served **zero** archive uploads across every retained nginx log (all rotations, including the compressed ones), the upload store was empty, and there was not one 429 on an upload route. There was no contention to find, so the lock was deliberately left alone and this counter added in its place. If Busy stays empty in production, the lock is not the problem and should stay as it is.
type OpTiming ¶ added in v0.1.20
type OpTiming struct {
Count int `json:"count"`
Total time.Duration `json:"total"`
Max time.Duration `json:"max"`
}
OpTiming is how long one operation class held the store lock.
type Store ¶
type Store struct {
Root string
MaxSize int64
// Zero values select 16 sessions, 8 GiB of reserved payload and 24 hours.
MaxSessions int
MaxTotalSize int64
TTL time.Duration
// FreeSpace reports the bytes available to this process on the filesystem
// holding the store root. Nil selects the build-tagged platform query in
// freespace_windows.go / freespace_unix.go / freespace_other.go; tests
// inject a deterministic implementation so admission never depends on the
// host's real disk.
FreeSpace func(path string) (int64, error)
// contains filtered or unexported fields
}
func (*Store) CleanupExpired ¶ added in v0.1.17
func (*Store) Complete ¶ added in v0.1.17
Complete holds admission through finalization, so cancel/merge cannot remove an archive while an installer or backup restorer is reading it.
func (*Store) Contention ¶ added in v0.1.20
func (s *Store) Contention() ContentionStats
Contention returns a copy of the accumulated figures without clearing them.
func (*Store) ReconcileOrphans ¶ added in v0.1.20
ReconcileOrphans removes scratch files this package created and abandoned inside canonical upload directories, plus the empty reservation directories a process that died during Init left behind. It runs at store start, before the first admission: chunk.go holds the store lock for the whole of SaveChunk and merge, so a scratch file visible while the lock is held cannot belong to a live upload and must have been left by a process that died mid-write.
Only names this package writes are eligible (see isTemporaryStoreFile). Unknown files, unknown directories, symlinks and anything outside a canonical session directory are left in place: scan() reports them and admission fails, which is the documented behaviour for unrecognised data.
func (*Store) RunCleanup ¶ added in v0.1.17
RunCleanup is owned by the server lifecycle, including first-run setup. The first pass reconciles scratch files left by a crashed process and expires sessions, then expiry repeats hourly. Both passes report through reportCleanup, so a failure is logged once per burst instead of being discarded.
func (*Store) Stats ¶ added in v0.1.20
func (s *Store) Stats() CleanupStats
Stats returns the cached maintenance view. It takes a dedicated lock and does no I/O, so it is safe to call while cleanup runs.