Documentation
¶
Overview ¶
Package procgroup inspects and drains POSIX process groups.
It exists to answer one question the orchestrator could not previously ask: when an agent harness exits, did it leave work running behind it?
The failure this addresses is subtle and silent. A harness told to "train the model" may start the training in the background and exit immediately — `nohup python train.py &` — at which point cloop reads TASK_DONE off its stdout, marks the task complete, and starts the next one. The next task then operates on a model file that does not exist yet. Nothing errors; the work is simply wrong, and the reason is invisible in every log cloop keeps.
The handle that makes detection possible is that the harness is started in its own process group (see pkg/provider/claudecode, which sets Setpgid). Everything it forks inherits that group ID, so after the harness itself is reaped, any process still reporting that PGID is work the task left running. That is a far better signal than watching the output pipes: a background child that redirects its own stdout — which nohup does by default — closes the inherited descriptors immediately and is invisible to a pipe-EOF check.
Known limits ¶
A child that calls setsid() leaves the group and becomes undetectable here. That is deliberate rather than a gap worth closing: setsid is the explicit gesture for "I am a daemon and I intend to outlive my parent", and the case this package exists to catch is the opposite one, where the work was meant to be waited for and simply was not.
Detection is Linux-only, because it reads procfs. On other systems Members reports Supported() == false and the callers degrade to their previous behaviour rather than guessing; see Drain for what that means in practice.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrUnsupported = fmt.Errorf("procgroup: process-group inspection unsupported on %s", runtime.GOOS)
ErrUnsupported is returned by Members on platforms without procfs.
Functions ¶
func Supported ¶
func Supported() bool
Supported reports whether process-group inspection works on this platform. Only Linux is implemented, via procfs.
func Terminate ¶
Terminate signals every member of process group pgid, escalating from SIGTERM to SIGKILL after grace, and reports how many processes were alive when it started.
Callers use this when background work outlived its budget. Leaving it running is not a neutral choice: the processes are orphaned, unattributable to any task, and a retry of the same task would start a second copy racing the first over the same output files.
It refuses to signal its own process group. That guard is the reason this is a function rather than an inline syscall.Kill: cloop's control plane runs as a long-lived service, a negative-pid kill is applied to every member of the group, and a pgid that had been defaulted or mis-parsed to the caller's own group would take down the service and every other project it is running.
func TerminateOrphaned ¶ added in v0.0.5
TerminateOrphaned is Terminate for a group whose leader the caller started and has already reaped — what a harness leaves behind after it exits.
A reaped leader's PID stays reserved only while the rest of its group lives: the kernel does not hand out a number any process still has as its group. Once the group empties, the number is free, and a new process can take it and lead a group of its own under the same id. So a process that holds the leader's PID now proves the group emptied and its id went to someone else, and signalling "the group" would kill unrelated work. That is refused.
Types ¶
type DrainOptions ¶
type DrainOptions struct {
// Grace is how long a survivor may live before it counts as real
// background activity. Harnesses routinely leave short-lived children
// mid-teardown, and reporting those as abandoned work would make the
// signal useless. Zero disables the grace window.
Grace time.Duration
// Budget bounds the total wait. Zero means do not wait at all: detect and
// report, but return immediately.
Budget time.Duration
// Poll is the interval between group scans. Zero selects a default.
Poll time.Duration
// OnDetect, if set, is called once when real background work is confirmed
// — after the grace window, before the wait begins. It exists so callers
// can surface the wait while it is happening: a task that blocks for
// twenty minutes on someone else's training run should not look identical
// to one that is merely slow.
OnDetect func(members []Process)
}
DrainOptions tunes how long Drain is willing to wait.
type Outcome ¶
type Outcome struct {
// Detected is how many processes were still running after the grace
// window. Zero means the harness left nothing behind.
Detected int
// Commands names what was running, for operator diagnosis. Bounded.
Commands []string
// Waited is how long Drain actually blocked.
Waited time.Duration
// Drained is true when the group emptied on its own. When Detected > 0
// and Drained is false, work outlived the budget and the task that
// started it cannot be considered finished.
Drained bool
// Survivors are the processes still alive when the budget expired.
Survivors []Process
}
Outcome describes what a Drain found and how it ended.
func Drain ¶
func Drain(ctx context.Context, pgid int, opts DrainOptions) Outcome
Drain waits for process group pgid to empty, and reports what it found.
The sequence is a scan, then — only if that scan finds something — the grace window and a second scan, and only if survivors remain after that, a polling wait until the group empties (Drained) or the budget expires (survivors reported).
Scanning before sleeping is what keeps this affordable. The overwhelmingly common case is a harness that left nothing behind, and it costs one procfs walk of a few milliseconds; paying the grace window up front instead would tax every task in every project for a condition almost none of them are in. An empty first scan is conclusive rather than merely likely, because a new background process can only be forked by an existing member of the group.
Drain never kills anything — that decision belongs to the caller, which knows whether the work is worth preserving. It also never returns an error: on a platform or a procfs it cannot read, it reports a zero Outcome, which callers correctly read as "nothing detected". Degrading to the previous behaviour is the right failure mode for a safety net; refusing to run a task because procfs was unreadable would not be.
type Process ¶
type Process struct {
PID int
// Command is the executable name from procfs (the "comm" field), which is
// truncated to 15 characters by the kernel. It is for operator diagnosis
// only — never for matching, since it is neither unique nor complete.
Command string
}
Process is one live member of a process group.
func Members ¶
Members returns the live processes in process group pgid, excluding the caller's own process.
Zombies are excluded: a process in state Z has finished its work and is waiting only to be reaped, so counting it would make a drain loop wait for something that will never change. Orphans are reparented to init, which reaps them promptly.
A nil error with an empty slice means the group is empty. On platforms without procfs it returns ErrUnsupported.
Source Files