Documentation
¶
Overview ¶
Package diskwatch measures a sandbox's workspace and holds a running workload to its disk limit by sampling it (Task 20405).
The container driver's sandbox writes in a few places. The image's root filesystem is mounted read-only and /tmp is a size-capped tmpfs; the workspace — the project directory bind-mounted at /workspace, or a feature's staged checkout and its output directory — is the host's own disk, and is where the work goes. --storage-opt size= bounds the container's writable layer, which is not that, so the only way to bound the workspace on a host filesystem with no project quota is to measure it.
That makes this enforcement by measurement, not a quota, and the difference is the overshoot: a workload writing fast enough fills the write rate times the sampling interval past the limit before the next sample sees it. The free-space floor (orchestrator.min_free_disk_mb, Task 20381) is what stands between that burst and a full volume.
What a measurement counts ¶
Allocated size, the way du(1) counts it: st_blocks for every entry, so a sparse file costs what it occupies rather than what it claims. A file with several hard links is counted once. Symlinks are counted as themselves and never followed. Mount points below a root are neither counted nor crossed — whatever is mounted there is not the workspace's to spend — and a mount point is what the mount table lists, not any directory with a device number of its own: a btrfs subvolume has one, and any user can make one. A root may name entries directly beneath it to leave out, and may carry an allowance: a project's own .cloop/ is measured as a root of its own whose size at the start is the hub's bookkeeping, so only what the run adds to it counts.
On Linux the walk goes through directory descriptors, so no depth of nesting puts part of a tree out of its reach; see walk_linux.go.
What a measurement costs ¶
A walk is one goroutine per workload at a time — the Watchdog never overlaps two — run by LowPriority on a thread of its own at the lowest CPU priority and in the idle I/O class, and bounded by a deadline. A walk that misses its deadline is reported as unknown, never as under the limit: its count is a lower bound, which settles "over" when it is already over and nothing else.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultPolicy = Policy{ MinInterval: 5 * time.Second, MaxInterval: 2 * time.Minute, FirstInterval: 15 * time.Second, CostFactor: 10, Deadline: time.Minute, }
DefaultPolicy is the schedule the container driver samples on.
var ErrDeadline = errors.New("diskwatch: the walk did not finish within its deadline")
ErrDeadline reports a walk that did not finish within its deadline. The Usage returned beside it is a lower bound.
var ErrIncomplete = errors.New("diskwatch: part of the tree could not be read")
ErrIncomplete reports a walk that could not list every directory under its roots — one the walker may not read. The Usage returned beside it is a lower bound.
Functions ¶
func LowPriority ¶
func LowPriority(fn func()) error
LowPriority runs fn on an OS thread of its own whose CPU and I/O priority were lowered first — nice 19 and the idle I/O class on Linux — and waits for it to return. A panic in fn comes back as an error.
The thread is never unlocked. A goroutine that exits still locked to its thread takes the thread with it (runtime.LockOSThread), so the lowered priority ends with the walk instead of returning to the scheduler's pool and slowing whatever goroutine lands there next; and the runtime starts new threads from a template thread rather than cloning a locked one, so it does not spread to them either.
Lowering is best effort and its failure is not reported: a walk at normal priority is still bounded by its deadline, and refusing to measure would be the weaker outcome.
Types ¶
type Clock ¶
Clock is the walker's source of time. Tests inject one that advances on every read, which is how a deadline is exercised without a slow filesystem.
type Policy ¶
type Policy struct {
// MinInterval is the shortest wait between two samples.
MinInterval time.Duration
// MaxInterval is the longest wait when nothing calls for a sooner one. A
// walk costlier than MaxInterval/CostFactor still waits CostFactor times
// its cost: the cost bound wins.
MaxInterval time.Duration
// FirstInterval is the wait after a sample with no growth history behind
// it — the one taken at the start — when the tree is not near its limit.
// Shorter than MaxInterval because until a second sample there is no rate
// to schedule by, and the start of a run is when a workload that writes
// fast has the most room to overshoot.
FirstInterval time.Duration
// CostFactor is how many times the last walk's duration the next sample
// waits at least.
CostFactor int
// Deadline bounds one walk. A walk that misses it is unknown.
Deadline time.Duration
}
Policy is a Watchdog's sampling schedule.
The interval adapts in both directions and is bounded below by the cost of the last walk. A walk that took a second waits at least CostFactor seconds before the next, so a tree that is expensive to measure is measured less often rather than measured continuously — the sampler's share of one thread stays under 1/CostFactor however large the tree grows. Within that bound it samples sooner the closer the tree gets to its limit: at the observed growth rate, the next sample is due by half the time the tree would take to reach it.
func (Policy) AfterUnknown ¶
AfterUnknown returns how long to wait after a walk that did not complete. Its cost is what it spent before it gave up, which for a missed deadline is the whole deadline: the next attempt waits CostFactor deadlines, so a tree too large to measure is not measured back to back.
type Root ¶
type Root struct {
// Path is the directory measured. It is not resolved: a symlink here is a
// symlink, and is measured as one.
Path string
// Exclude names entries directly under Path that are not counted, such as
// ".cloop". Deeper entries with the same name are counted: a fixture
// directory called .cloop inside the workload's tree is the workload's.
Exclude []string
// Allowance is how many bytes of this root go uncounted. A tree whose
// size at the start is not the workload's — a project's own .cloop/ —
// counts only what it grows by; math.MaxInt64 counts none of it. Its own
// size is still reported, in Usage.RootBytes.
Allowance int64
// Optional makes a root that does not exist, or is not a directory,
// measure as empty rather than fail the measurement.
Optional bool
}
Root is one tree to measure.
type Sample ¶
type Sample struct {
Bytes int64
At time.Time
// Cost is how long the walk took.
Cost time.Duration
}
Sample is one completed measurement, as the schedule sees it.
type Usage ¶
type Usage struct {
// Bytes is what counts against a limit: each root's allocated size beyond
// its Allowance, summed.
Bytes int64
// RootBytes is each root's own allocated size, before any allowance, in
// the order the roots were given. A walk that stopped early leaves the
// roots it never reached at zero.
RootBytes []int64
// Entries is how many directory entries the walk visited.
Entries int
// Elapsed is how long the walk took, by the walker's clock.
Elapsed time.Duration
// MountsSkipped counts the mount points below a root that were left out.
MountsSkipped int
// Unreadable counts directories whose contents could not be listed, and
// UnreadablePaths names the first few.
Unreadable int
UnreadablePaths []string
}
Usage is one measurement.
type Walker ¶
type Walker struct {
// Clock reads the time for the deadline; nil means the real clock.
Clock Clock
// Deadline bounds one Measure call; 0 means none.
Deadline time.Duration
// MountPoints returns the mount points below the given roots, keyed by
// cleaned absolute path, and whether the mount table could be read at
// all. Nil reads the process's own table, /proc/self/mountinfo on Linux;
// tests inject theirs, because creating a mount needs a privilege CI does
// not have.
//
// A readable table is the whole answer: a directory is a mount point
// when the table lists it, whatever its st_dev. A device check would
// skip every btrfs subvolume — each has a device number of its own, and
// any user may create one in a directory they can write — so a workload
// could put its data where no walk counts it. Only when the table cannot
// be read does a change of device stand in for it.
MountPoints func(roots []string) (map[string]bool, bool)
// contains filtered or unexported fields
}
Walker measures trees. The zero value is usable: a real clock, no deadline, and the host's own mount table.
func (Walker) Measure ¶
Measure walks every root and returns their combined allocated size. A hard link shared between two roots is still counted once.
A non-nil error means the count is incomplete and therefore a lower bound: ErrDeadline, ErrIncomplete, ctx's error, or the reason a root could not be opened. The Usage is returned with every error, because a lower bound that is already over a limit settles the question it was taken to answer.
type Watchdog ¶
type Watchdog struct {
// Roots are the trees measured together.
Roots []Root
// LimitBytes is the limit; a non-positive one makes Run return at once.
LimitBytes int64
// Walker measures; its Deadline is set from Policy.Deadline when zero.
Walker Walker
// Policy is the schedule; its zero value is DefaultPolicy.
Policy Policy
// Initial is the measurement taken before the workload started, which
// schedules the first sample. Nil schedules it at MinInterval.
Initial *Sample
// Now and Sleep are the loop's clock. Sleep returns false when ctx ended
// first. Nil means the real ones.
Now func() time.Time
Sleep func(ctx context.Context, d time.Duration) bool
// Measure takes one measurement; nil walks Roots with Walker on a
// low-priority thread (LowPriority).
Measure func(ctx context.Context) (Usage, error)
// OnBreach is called with a measurement over the limit. It returns
// whether the workload was stopped: Run returns if so, and samples again
// at MinInterval if not — a stop that failed is a workload still writing.
OnBreach func(u Usage, at time.Time) bool
// OnUnknown is called for a measurement that did not complete and is not
// already over the limit, with the wait before the next attempt. It is
// the caller's to log: the sample is unknown, and never counts as under.
OnUnknown func(err error, u Usage, next time.Duration)
}
Watchdog samples a workload's workspace until it is cancelled or the workspace is over its limit.