Documentation
¶
Overview ¶
Package lifecycle implements workspace lifecycle automation: the Reaper loop finds RUNNING agent runs idle past their policy's AutoStopAfterSec threshold and stops them, emitting a "run.autostop" audit event for each.
Idleness is wall-clock age of agent_runs.updated_at, reset only by a write through the store (state transitions, sandbox_ref updates, TouchRun). The proxy's decision-ingest touch is coalesced to one UPDATE per TouchDebounce window, so thresholdFor adds TouchDebounce as slack. CPU work inside the sandbox reaches the clock through TouchRun too: internal/api's pause sweep reads each candidate's CPU from the substrate and touches a busy one. Where that signal is off (no metrics-server) and for file writes, activity that never leaves the sandbox does NOT reset the clock, so a busy run can still be reaped; operators needing an unbounded session use the never-reap escape hatch (AutoStopAfterSec <= 0).
Config.MaxAge (WARDYN_RUN_MAX_AGE) is a separate, absolute cap on a RUNNING run's age since creation. It is its own predicate: not ends_at (which a person may extend), not the idle compare-and-set, and it applies to a run whose policy never idle-reaps. It stops a run even with a request open, because bounding a hung run is its whole purpose. A run that carries a lifetime cap (RunSummary.HasLifetimeCap) is not stopped here: the lease sweep KILLS it at that cap, which already includes this age.
AutoStopAfterSec (policy auto_stop_after_sec): >0 idle timeout in seconds; 0 DISABLED/never reaped (default, matches docs/POLICIES.md); <0 also never reaped but kept distinct so an operator can express intent loudly.
Index ¶
Constants ¶
const TouchDebounce = 30 * time.Second
TouchDebounce covers the ingest touch's coalesced-UPDATE lag, added as slack in thresholdFor so debounce can never make an active run look idle.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Interval is how often the reaper scans. Default: 1 minute.
Interval time.Duration
// MaxAge, when positive, ends any RUNNING run created longer ago than this
// (WARDYN_RUN_MAX_AGE). Zero or negative is off.
MaxAge time.Duration
// Now overrides the wall clock. Nil means use real time.
Now func() time.Time
// TickLock, when non-nil, makes each tick single-flight across control
// planes: TRY a cluster-wide lock, return a release func, or (nil, false)
// if held elsewhere — then the tick is skipped, not queued. wardynd wires
// a Postgres try-advisory-lock; kept as a func so lifecycle has no DB
// dependency. Nil = ungated (single-process default; what tests use).
TickLock func(ctx context.Context) (release func(), ok bool)
// Sweeps, when non-nil, records each tick that does real work (the one that
// won TickLock) as the idle_reaper sweep: an attempt when it starts and a
// success only when it finishes without error. Nil records nothing.
Sweeps *sweephealth.Tracker
}
Config holds optional overrides for Reaper behaviour; zero value is valid.
type Reaper ¶
type Reaper struct {
// contains filtered or unexported fields
}
Reaper is the idle-workspace garbage collector: a periodic loop that finds RUNNING workspaces idle past their policy threshold and stops them. Reaper.Run blocks until ctx is cancelled; a stopper error is logged and skipped so one broken sandbox never blocks the rest of the tick.
func (*Reaper) Run ¶
Run starts the reap loop, returning when ctx is cancelled. The first tick fires after one full Interval so startup finishes before the first scan.
func (*Reaper) Tick ¶
Tick is one reap scan, gated by Config.TickLock when wired. Exported for integration callers/tests to drive directly; production code uses Run. The error is the tick's: the scan could not list runs, or a stop failed. A tick another control plane holds the lock for is skipped, which is not an error.
Deadline is Interval+defaultStopTimeout, not just Interval: a child context.WithTimeout can only shorten its parent's deadline, so budgeting at bare Interval would silently cap every per-stop deadline short of the full defaultStopTimeout the constant promises.
type Recorder ¶
type Recorder interface {
Record(ctx context.Context, ev types.AuditEvent) error
}
Recorder matches audit.Recorder exactly so a store.Recorder can be passed directly.
type RunSummary ¶
type RunSummary struct {
ID uuid.UUID
CreatedAt time.Time
// AdmittedAt is when the run left the pool queue; zero for a run that never waited. The max age
// counts from it, so a run that waited is not aged by its wait.
AdmittedAt time.Time
UpdatedAt time.Time
// PolicyAutoStopAfterSec: 0 or negative means never reap, positive is the
// idle timeout in seconds. Store must JOIN to the policy table; zero is
// intentional, not a missing join.
PolicyAutoStopAfterSec int
// Kept is a run kept after a control-plane outage whose agent still runs: the
// max age still ends it, and idle reaping does not apply (its lost mark governs it).
Kept bool
// HasLifetimeCap is a run that carries agent_runs.lifetime_cap_at: the lease
// sweep ends it at its cap (and the deployment's maximum age is one of the
// limits that cap is measured from), so the max-age stop skips it. A run
// from before that column keeps this reaper's max-age stop.
HasLifetimeCap bool
}
RunSummary is the minimal projection a Store must return for idle detection.
type StopOutcome ¶
type StopOutcome struct {
// Applied is false when a concurrent kill/complete already moved the run
// terminal, or the idleness guard no-op'd because updated_at advanced past
// the snapshot (an active attach). Either way, no spurious run.autostop.
Applied bool
// Errors, when non-empty, are post-transition teardown/revocation failures
// ("identity_error"/"broker_error": creds may still be live;
// "teardown_error": sandbox may still be routable) that leave the run not
// fully contained. Emitted as a distinct run.revoke/failure audit event,
// mirroring handleKillRun/revokeRunCascade. Nil on a clean stop.
Errors map[string]string
}
StopOutcome is what StopRun reports back to the Reaper beyond the raw error.
type Stopper ¶
type Stopper interface {
// StopRun must be idempotent: stopping an already-stopped run returns
// ({Applied:false}, nil). notAfter is the snapshot's updated_at; the
// transition is conditional on updated_at not having advanced past it, so
// a run touched by an active attach since the snapshot is NOT stopped. A
// non-nil error means the stop failed outright and the reaper logs/skips.
StopRun(ctx context.Context, runID uuid.UUID, notAfter time.Time) (StopOutcome, error)
// StopRunMaxAge stops a run that has outlived Config.MaxAge: the same
// RUNNING->STOPPED transition and teardown as StopRun, guarded on the run's
// created_at being at or before createdNotAfter instead of on idleness (and
// ignoring open requests). Idempotent like StopRun.
StopRunMaxAge(ctx context.Context, runID uuid.UUID, createdNotAfter time.Time) (StopOutcome, error)
}
Stopper stops a single run: transitions RUNNING->STOPPED guarded on the snapshot's idleness, then calls runner.StopSandbox + the revoke cascade. Hidden behind this interface so lifecycle stays target-agnostic.
type Store ¶
type Store interface {
// ListRunningWithPolicy returns RUNNING runs with their policy's
// auto_stop_after_sec (0 if unattached/unset), plus the store's own clock
// read once for the whole scan. The reaper measures idleness against that
// clock, not its own: UpdatedAt is Postgres-stamped, so wardynd clock skew
// vs. the database eats directly into TouchDebounce's 30s margin — enough
// skew stops an actively-attached run and revokes its credentials, or
// (skewed the other way) never reaps at all. A ZERO returned time means
// the store has no clock (in-memory doubles), and the reaper falls back
// to its own.
ListRunningWithPolicy(ctx context.Context) ([]RunSummary, time.Time, error)
}
Store is the narrow persistence interface the Reaper requires; the real adapter maps agent_runs JOIN run_policies, tests supply a fake.