lifecycle

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package lifecycle implements workspace lifecycle automation: the Reaper loop finds RUNNING agent runs idle past their policy's AutoStopAfterSec threshold and stops them, emitting a "run.autostop" audit event for each.

Idleness is wall-clock age of agent_runs.updated_at, reset only by a write through the store (state transitions, sandbox_ref updates, TouchRun). The proxy's decision-ingest touch is coalesced to one UPDATE per TouchDebounce window, so thresholdFor adds TouchDebounce as slack. CPU work inside the sandbox reaches the clock through TouchRun too: internal/api's pause sweep reads each candidate's CPU from the substrate and touches a busy one. Where that signal is off (no metrics-server) and for file writes, activity that never leaves the sandbox does NOT reset the clock, so a busy run can still be reaped; operators needing an unbounded session use the never-reap escape hatch (AutoStopAfterSec <= 0).

Config.MaxAge (WARDYN_RUN_MAX_AGE) is a separate, absolute cap on a RUNNING run's age since creation. It is its own predicate: not ends_at (which a person may extend), not the idle compare-and-set, and it applies to a run whose policy never idle-reaps. It stops a run even with a request open, because bounding a hung run is its whole purpose. A run that carries a lifetime cap (RunSummary.HasLifetimeCap) is not stopped here: the lease sweep KILLS it at that cap, which already includes this age.

AutoStopAfterSec (policy auto_stop_after_sec): >0 idle timeout in seconds; 0 DISABLED/never reaped (default, matches docs/POLICIES.md); <0 also never reaped but kept distinct so an operator can express intent loudly.

Index

Constants

View Source
const TouchDebounce = 30 * time.Second

TouchDebounce covers the ingest touch's coalesced-UPDATE lag, added as slack in thresholdFor so debounce can never make an active run look idle.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Interval is how often the reaper scans. Default: 1 minute.
	Interval time.Duration
	// MaxAge, when positive, ends any RUNNING run created longer ago than this
	// (WARDYN_RUN_MAX_AGE). Zero or negative is off.
	MaxAge time.Duration
	// Now overrides the wall clock. Nil means use real time.
	Now func() time.Time
	// TickLock, when non-nil, makes each tick single-flight across control
	// planes: TRY a cluster-wide lock, return a release func, or (nil, false)
	// if held elsewhere — then the tick is skipped, not queued. wardynd wires
	// a Postgres try-advisory-lock; kept as a func so lifecycle has no DB
	// dependency. Nil = ungated (single-process default; what tests use).
	TickLock func(ctx context.Context) (release func(), ok bool)
	// Sweeps, when non-nil, records each tick that does real work (the one that
	// won TickLock) as the idle_reaper sweep: an attempt when it starts and a
	// success only when it finishes without error. Nil records nothing.
	Sweeps *sweephealth.Tracker
}

Config holds optional overrides for Reaper behaviour; zero value is valid.

type Reaper

type Reaper struct {
	// contains filtered or unexported fields
}

Reaper is the idle-workspace garbage collector: a periodic loop that finds RUNNING workspaces idle past their policy threshold and stops them. Reaper.Run blocks until ctx is cancelled; a stopper error is logged and skipped so one broken sandbox never blocks the rest of the tick.

func New

func New(store Store, stopper Stopper, recorder Recorder, cfg Config) *Reaper

New constructs a Reaper. store, stopper, and recorder must be non-nil.

func (*Reaper) Run

func (r *Reaper) Run(ctx context.Context)

Run starts the reap loop, returning when ctx is cancelled. The first tick fires after one full Interval so startup finishes before the first scan.

func (*Reaper) Tick

func (r *Reaper) Tick(ctx context.Context) error

Tick is one reap scan, gated by Config.TickLock when wired. Exported for integration callers/tests to drive directly; production code uses Run. The error is the tick's: the scan could not list runs, or a stop failed. A tick another control plane holds the lock for is skipped, which is not an error.

Deadline is Interval+defaultStopTimeout, not just Interval: a child context.WithTimeout can only shorten its parent's deadline, so budgeting at bare Interval would silently cap every per-stop deadline short of the full defaultStopTimeout the constant promises.

type Recorder

type Recorder interface {
	Record(ctx context.Context, ev types.AuditEvent) error
}

Recorder matches audit.Recorder exactly so a store.Recorder can be passed directly.

type RunSummary

type RunSummary struct {
	ID        uuid.UUID
	CreatedAt time.Time
	// AdmittedAt is when the run left the pool queue; zero for a run that never waited. The max age
	// counts from it, so a run that waited is not aged by its wait.
	AdmittedAt time.Time
	UpdatedAt  time.Time
	// PolicyAutoStopAfterSec: 0 or negative means never reap, positive is the
	// idle timeout in seconds. Store must JOIN to the policy table; zero is
	// intentional, not a missing join.
	PolicyAutoStopAfterSec int
	// Kept is a run kept after a control-plane outage whose agent still runs: the
	// max age still ends it, and idle reaping does not apply (its lost mark governs it).
	Kept bool
	// HasLifetimeCap is a run that carries agent_runs.lifetime_cap_at: the lease
	// sweep ends it at its cap (and the deployment's maximum age is one of the
	// limits that cap is measured from), so the max-age stop skips it. A run
	// from before that column keeps this reaper's max-age stop.
	HasLifetimeCap bool
}

RunSummary is the minimal projection a Store must return for idle detection.

type StopOutcome

type StopOutcome struct {
	// Applied is false when a concurrent kill/complete already moved the run
	// terminal, or the idleness guard no-op'd because updated_at advanced past
	// the snapshot (an active attach). Either way, no spurious run.autostop.
	Applied bool
	// Errors, when non-empty, are post-transition teardown/revocation failures
	// ("identity_error"/"broker_error": creds may still be live;
	// "teardown_error": sandbox may still be routable) that leave the run not
	// fully contained. Emitted as a distinct run.revoke/failure audit event,
	// mirroring handleKillRun/revokeRunCascade. Nil on a clean stop.
	Errors map[string]string
}

StopOutcome is what StopRun reports back to the Reaper beyond the raw error.

type Stopper

type Stopper interface {
	// StopRun must be idempotent: stopping an already-stopped run returns
	// ({Applied:false}, nil). notAfter is the snapshot's updated_at; the
	// transition is conditional on updated_at not having advanced past it, so
	// a run touched by an active attach since the snapshot is NOT stopped. A
	// non-nil error means the stop failed outright and the reaper logs/skips.
	StopRun(ctx context.Context, runID uuid.UUID, notAfter time.Time) (StopOutcome, error)

	// StopRunMaxAge stops a run that has outlived Config.MaxAge: the same
	// RUNNING->STOPPED transition and teardown as StopRun, guarded on the run's
	// created_at being at or before createdNotAfter instead of on idleness (and
	// ignoring open requests). Idempotent like StopRun.
	StopRunMaxAge(ctx context.Context, runID uuid.UUID, createdNotAfter time.Time) (StopOutcome, error)
}

Stopper stops a single run: transitions RUNNING->STOPPED guarded on the snapshot's idleness, then calls runner.StopSandbox + the revoke cascade. Hidden behind this interface so lifecycle stays target-agnostic.

type Store

type Store interface {
	// ListRunningWithPolicy returns RUNNING runs with their policy's
	// auto_stop_after_sec (0 if unattached/unset), plus the store's own clock
	// read once for the whole scan. The reaper measures idleness against that
	// clock, not its own: UpdatedAt is Postgres-stamped, so wardynd clock skew
	// vs. the database eats directly into TouchDebounce's 30s margin — enough
	// skew stops an actively-attached run and revokes its credentials, or
	// (skewed the other way) never reaps at all. A ZERO returned time means
	// the store has no clock (in-memory doubles), and the reaper falls back
	// to its own.
	ListRunningWithPolicy(ctx context.Context) ([]RunSummary, time.Time, error)
}

Store is the narrow persistence interface the Reaper requires; the real adapter maps agent_runs JOIN run_policies, tests supply a fake.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL