Documentation
¶
Overview ¶
Package scratch owns vigolium's per-run temporary storage: where it is allocated, when this process's own is released, and when what an earlier run abandoned is collected.
Many subsystems need scratch on disk for the life of a scan - a dedup DiskSet per module that dedups, deparos' request cache and discovery dedup, the sitemap store, jstangle job directories, stateless mode's temporary SQLite database. Each owner removes its own on the clean path, but a SIGKILL, a panic, an OOM kill or a CI job torn down mid-scan skips every deferred cleanup, and nothing ever collected what those runs left behind.
Allocating directly in os.TempDir() made that leak expensive twice over. It is permanent and compounds - the workstation this was found on had 71,614 leftover diskset directories and 108 GB - and it sits directly in the path of the startup sweeps that fastdialer/hmap and nuclei each run over the WHOLE temp directory, lstat'ing every entry whose name contains the executable name. vigolium's own litter was making vigolium slower to start, in proportion to how much of it there was.
So everything lands under one root, and inside it under one directory per process:
$TMPDIR/vgl-scratch/p<pid>-<random>/...
That shape is what makes cleanup total rather than best-effort. Releasing this process's scratch is a single RemoveAll of its own directory, so an owner that forgets - or never gets the chance - costs nothing beyond the run. Collecting an abandoned run's is a read of a directory holding only vigolium's own process directories, not a walk of a shared temp directory with six figures in it.
Liveness is a lease, not an age. Each process directory holds a `.lease` file this process flocks for its lifetime, so the sweeper can ask the kernel whether an owner is still running — and get the right answer even after a SIGKILL, since the lock dies with the open file description.
mtime alone was never the liveness proxy it was described as: a directory's mtime changes when an entry is created or removed in it, NOT when files already inside it are written. A long scan that allocated its disksets up front and then wrote gigabytes into them for eight hours had a directory mtime fixed at its first minute, so DefaultMaxAge made its scratch a collection candidate while it was still in use. Age remains the backstop for directories with no lease (an older version's, or Windows).
Index ¶
- Constants
- Variables
- func Acquire() error
- func CreateTemp(pattern string) (*os.File, error)
- func MkdirTemp(pattern string) (string, error)
- func Release() bool
- func Root() string
- func SweepLegacy(maxAge time.Duration) (int, error)
- func SweepOnce(maxAge time.Duration) int
- func SweepRoot(maxAge time.Duration) (int, error)
Constants ¶
const DefaultMaxAge = 6 * time.Hour
DefaultMaxAge is how long a process directory must have gone untouched before it counts as abandoned. Generous on purpose: mtime is a liveness proxy, and scratch belonging to a very long scan must never be collected out from under it. The leak is permanent, so a slow collector still fixes it.
Variables ¶
var LegacyPrefixes = []string{
"vigolium-diskset-",
"vigolium-stateless-",
"vigolium-audit-stateless-",
"vigolium-autopilot-stateless-",
"vigolium-isolate-",
"vigolium-scratch-",
"vigolium-parallel-",
"vigolium-browserprobe-",
"vigolium-extract-",
"vigolium-pdf-",
"deparos-dedup-",
"reqcache-",
"sitemap-",
"jstangle-job-",
}
LegacyPrefixes are the names older vigolium versions allocated directly in os.TempDir(), before scratch had a root.
They are NOT part of the automatic sweep. Draining them is a one-time migration of a backlog that can run to six figures, and dripping it into every scan's teardown was measurably the wrong shape: it charged each run the full sweep budget and still needed a hundred-odd runs to finish. SweepLegacy does it in one pass, from `vigolium kit tmp-clean`.
Prefix match, not substring: every name here is a temp pattern plus the random suffix os.MkdirTemp appends, so matching loosely could only ever widen this to files vigolium did not create. Every prefix an allocator used BEFORE it was routed through this package belongs here. Note "vigolium-stateless-" does not cover "vigolium-audit-stateless-": these are prefixes, not substrings, so each variant has to be listed.
Functions ¶
func Acquire ¶
func Acquire() error
Acquire registers one holder of the scratch directory, creating it if this is the first. Every successful Acquire must be paired with exactly one Release.
pkg/cli.Execute acquires once for the whole process, so every command gets the cleanup and a server does not tear the directory down between scans. The subsystems that allocate scratch just call MkdirTemp/CreateTemp.
func CreateTemp ¶
CreateTemp allocates a scratch file for this process, with the same contract as os.CreateTemp(dir, pattern).
func MkdirTemp ¶
MkdirTemp allocates a scratch directory for this process, with the same contract as os.MkdirTemp(dir, pattern).
The caller should still remove it when done - scratch released early is disk the rest of the scan gets back - but it no longer has to: Release takes the whole process directory.
func Release ¶
func Release() bool
Release drops one reference. On the last one it removes everything allocated through MkdirTemp and CreateTemp since the first Acquire, and reports whether it did so.
That removal is the backstop that makes cleanup total: it does not matter which owner forgot its own deferred cleanup, or which of them never got the chance, because the whole directory goes. Extra Releases (more than Acquires) are ignored rather than deleting scratch still in use - in server mode two scans share the directory, and the first to finish must not take the second's with it.
func Root ¶
func Root() string
Root is the scratch root shared by every vigolium process on the machine. It is a path, not a promise the directory exists: the sweeps only read it, and a root that is absent has nothing to collect.
func SweepLegacy ¶
SweepLegacy removes the loose scratch that versions before the scratch root left directly in os.TempDir(), and reports how many entries it took.
Unbounded by design: this is the one-time migration `vigolium kit tmp-clean` drives, where the operator has asked for it and is watching it run. The automatic per-scan sweep deliberately does not do this - see LegacyPrefixes.
func SweepOnce ¶
SweepOnce collects scratch abandoned by earlier runs, at most once per process, and reports how many entries it removed.
It covers both the process directories under the root and the loose names older versions left directly in os.TempDir(). The second pass is why this runs at teardown rather than startup: a temp directory that has accumulated six figures of entries takes real time just to read, and that is time an operator waiting for scan results should not pay.
Types ¶
This section is empty.
Source Files