Documentation
¶
Overview ¶
Package hostslot is a host-wide counting semaphore for heavy work: at most N heavy jobs — builds, test lanes, codegen, linters — run at once on one machine, however many agents, worktrees and terminals start them.
Why it exists. On a shared machine the cost of a heavy job is set by how many others run beside it. Measured on a 16 vCPU / 32 GiB workspace VM: `go build ./...` took 36–50 s beside 0–3 other heavy jobs and 101–155 s beside 8 or more, and a local CI gate went from ~40 min to ~2.3 h. The loss was memory-cgroup reclaim, not CPU, so past a handful of concurrent heavy jobs EVERY job slows down: running them all at once finishes later than queueing them.
Mechanism. A class ("heavy") has N slot files in one directory. Holding a slot is holding an exclusive OS file lock (flock(2); LockFileEx on Windows) on one of them, for as long as the holder runs. The OS drops the lock when the holder exits however it exits, so a crashed or killed holder never strands a slot: there is no staleness window, no pid probe and nothing for anyone to delete. After acquiring, a holder writes its pid and command into the slot file; that record exists only so a waiter can say whom it waits for. N comes from the machine (DefaultSlots) unless FORGE_<CLASS>_SLOTS overrides it (SlotsEnv).
Waiting is FIFO-ish across processes. A waiter that finds the queue non-empty or every slot taken files a ticket — itself a locked file, so a waiter that dies leaves no live ticket — and only the N oldest live tickets compete for a slot that frees, so a newcomer never overtakes a queue. A waiter prints one line naming the holders (pid, command, age) and repeats it periodically, and gives up when its context is cancelled.
Re-entrancy. A process that already runs inside a slot must not take another: a gate step holding a slot that runs `forge generate`, which wants a slot of its own, would otherwise deadlock as soon as every slot is held by parents waiting on children. Two signals mark a process as inside a slot, and either makes Acquire return at once without taking one:
- FORGE_SLOT_HELD in its environment (Slot.Env passes it to children), naming a slot whose record still carries the same nonce and which is still locked — a stale or copied token is ignored, not trusted;
- an ANCESTOR process holding a slot of the class with Options.Subtree set (a wrapper like `forge slot run`, or a scheduler), read from the kernel's parent links. This covers a child whose environment was scrubbed between the holder and it.
Index ¶
- Constants
- func DefaultSlots(m Machine) int
- func Inside(class string, opt Options) (string, bool)
- func SlotsEnv(class string) string
- type Capacity
- type Holder
- type Machine
- type Options
- type Report
- type Slot
- func (s *Slot) Capacity() Capacity
- func (s *Slot) Class() string
- func (s *Slot) Env() []string
- func (s *Slot) Index() int
- func (s *Slot) Reentrant() string
- func (s *Slot) Release() error
- func (s *Slot) SetLabel(label string)
- func (s *Slot) String() string
- func (s *Slot) Unlimited() bool
- func (s *Slot) Waited() time.Duration
Constants ¶
const ( CPUsPerSlot = 2 GiBPerSlot = 8 )
CPUsPerSlot and GiBPerSlot are the default budget of one heavy job.
Memory is the budget that protects the machine: with the race detector and a linker or two in flight a Go build or test lane takes several GiB, and past that the jobs slow each other down more than they gain by overlapping — measured on a 16 vCPU / 32 GiB VM, where `go build ./...` went from 36–50 s beside 0–3 other heavy jobs to 101–155 s beside 8 or more, the loss being memory-cgroup reclaim rather than CPU.
CPU is not, so its budget is two cores, not four. Sharing CPU degrades gracefully: on a 16-CPU / 128 GiB Mac four CPU-bound test lanes at once took 161 s against 44 s for one alone — the same throughput as running them in turn, not a collapse. And a heavy job is rarely CPU-bound for its whole run (a regenerate is mostly serial steps, a test lane waits on subprocesses): four concurrent control-plane merge gates on that Mac left the CPU ~33% idle at 4 slots while 304 s of slot wait piled up; at 8 slots the worst gate's wall fell 139 s -> 116 s and the summed wait to 60 s (a rerun on a busier box: 164 s -> 122 s, wait 309 s -> 137 s). Where memory is tight — the 16 vCPU / 32 GiB VM above — GiBPerSlot still decides, and the count is the same as before.
const EnvDir = "FORGE_SLOTS_DIR"
EnvDir relocates the slot directory. The default is the user cache dir, so every process of one user on a host shares it. Point every user of a multi-user host at one directory (mode 1777) to share slots across users.
const EnvHeld = "FORGE_SLOT_HELD"
EnvHeld carries the slots a process runs inside to its children, as comma-separated "<class>:<slot index>:<nonce>" entries. Set it by appending Slot.Env() to a child's environment; never by hand.
const Heavy = "heavy"
Heavy is the class for jobs that saturate several cores and gigabytes of memory: a whole-module build or test run, codegen, a full lint.
Variables ¶
This section is empty.
Functions ¶
func DefaultSlots ¶
DefaultSlots is the slot count the machine supports: min(CPUs / CPUsPerSlot, memory / GiBPerSlot), never less than 1. CPUs round down (a quota is exact); memory rounds to nearest, because a machine sold as 32 GiB reports a little under it once the kernel takes its share.
func Inside ¶
Inside reports whether this process already runs inside a slot of class (see the package doc), and why. Acquire would return at once, taking nothing; a scheduler that means to bound its own jobs by slots learns here that it cannot.
func SlotsEnv ¶
SlotsEnv is the variable that overrides a class's slot count: FORGE_HEAVY_SLOTS for "heavy". A positive integer sets the count; "off" (or "unlimited") turns the limit off. The machine-derived default is right for a developer machine; a workspace operator sets this per plan size, and a CI runner that runs one job per machine may set it to off.
Types ¶
type Capacity ¶
type Capacity struct {
Class string
// Slots is the number of jobs of this class that may run at once. Zero
// when Unlimited.
Slots int
// Unlimited means the limit is switched off (FORGE_<CLASS>_SLOTS=off):
// Acquire returns at once and takes nothing.
Unlimited bool
// Source says where Slots came from, for the waiting line and
// `forge slot status`.
Source string
Machine Machine
}
Capacity is how many slots a class has, and why.
func CapacityFor ¶
CapacityFor resolves a class's capacity: SlotsEnv(class) in env (normally os.Environ()) when set, otherwise DefaultSlots of this machine.
type Holder ¶
type Holder struct {
// Slot is the slot index; -1 for a waiter.
Slot int
PID int
Command string
Since time.Time
// Subtree is set when the holder's descendants count as inside its slot.
Subtree bool
}
Holder is one live slot holder or waiter, as its record describes it.
type Machine ¶
type Machine struct {
// CPUs is the usable CPU count (a cgroup quota may make it fractional).
CPUs float64
// CPUSource names where CPUs came from: "cgroup cpu.max" or "nproc".
CPUSource string
// MemBytes is usable memory; 0 when it could not be read.
MemBytes uint64
// MemSource names where MemBytes came from: "cgroup memory.max",
// "/proc/meminfo", "hw.memsize".
MemSource string
}
Machine is what the host gives this process to work with.
Inside a container or a VM sandbox the cgroup limit is the truth, not the kernel's totals: a kata VM sized at 32 GiB reports its own MemTotal, and a container's /proc/meminfo reports the HOST's. So a cgroup v2 cpu.max or memory.max on the path from this process's cgroup up to the root wins over nproc and MemTotal whenever it is lower.
func DetectMachine ¶
func DetectMachine() Machine
DetectMachine reads the CPU and memory this process may use.
type Options ¶
type Options struct {
// Dir is the slot directory; empty is $FORGE_SLOTS_DIR, else
// <user cache dir>/forge/slots. Each class gets a subdirectory.
Dir string
// Slots overrides the capacity (a --slots flag). Zero resolves it with
// CapacityFor: FORGE_<CLASS>_SLOTS, else the machine default.
Slots int
// Out receives the waiting lines. Nil waits silently.
Out io.Writer
// Label is the command recorded for this holder, shown to waiters. Empty
// is this process's own command line.
Label string
// Subtree marks the holder as running the slot's work in child
// processes, so every descendant counts as inside the slot even when its
// environment lost FORGE_SLOT_HELD. Set it for wrappers and schedulers
// that exec the work; leave it unset for a process that does the work
// itself, so an unrelated child of a long-lived host is not exempted.
Subtree bool
// NoticeEvery is how often the waiting line repeats. Zero is 30s.
NoticeEvery time.Duration
// Poll is the mean interval between attempts while waiting, jittered
// ±50%. Zero is 500ms.
Poll time.Duration
// Env is the environment read for FORGE_SLOT_HELD, FORGE_SLOTS_DIR and
// FORGE_<CLASS>_SLOTS — normally os.Environ(). The package reads no
// ambient variable itself, so a nil Env means none is set; in
// particular, a caller that leaves it nil cannot see the slot its parent
// passed it and will queue for one of its own.
Env []string
}
Options configure Acquire and Status. The zero value is the production behavior.
type Report ¶
type Report struct {
Capacity Capacity
// Dir is the class's slot directory.
Dir string
Holders []Holder
// Waiting is the queue, oldest first.
Waiting []Holder
}
Report is a class's state: its capacity, who holds a slot, who waits.
type Slot ¶
type Slot struct {
// contains filtered or unexported fields
}
Slot is one acquisition. Release it when the work is done; a process that exits releases it regardless.
func Acquire ¶
Acquire takes a slot of class, waiting while every slot is held. It returns at once, taking nothing, when the limit is off or this process already runs inside a slot of the class (see the package doc). It returns ctx's error if ctx ends first.
func (*Slot) Env ¶
Env returns what to append to a child's environment so the child — and everything it runs — knows it is inside this slot and takes no other of the class. Nil when there is nothing to pass.
func (*Slot) Reentrant ¶
Reentrant reports why Acquire took nothing because this process already runs inside a slot of the class ("" when it did take one, or the limit is off).
func (*Slot) Release ¶
Release gives the slot back. Safe to call more than once, and on a slot that took nothing.
func (*Slot) SetLabel ¶
SetLabel replaces the command recorded for this holder, which waiters and `forge slot status` show. A scheduler that takes a slot and only then decides which job gets it names the job here.