lifecycle

package
v0.10.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package lifecycle implements the common child-launch machinery shared by every server type — process-group isolation, per-server environment overlay, and log-file management — plus the two launchers whose command shapes need no further logic (mlx, exec). See design/aa-server-status.md §4, §6.4, §9.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DeclaredPorts

func DeclaredPorts(s config.Server) []int

DeclaredPorts returns s's exhaustive declared port set ({port} ∪ listens, design/aa-server-status.md §6.2), deduplicated, for use as a Target's Ports.

func ExecCommand

func ExecCommand(s config.Server) (command string, args []string)

ExecCommand returns s.Command and s.Args verbatim, per design/aa-server-status.md §4: exec servers get no auto-appended flags — the server creates its own listeners (e.g. caddy).

func MLXCommand

func MLXCommand(s config.Server) (command string, args []string)

MLXCommand builds the mlx-serve invocation for s, per design/aa-server-status.md §4:

mlx-serve serve <model> --host <host> --port <port> [--drafter <path>]

host/port are auto-appended; --drafter is appended only when s.Drafter is set (Gemma-4-style assistant-checkpoint speculative decoding). aatoolkit adds no other flags of its own.

s.Args is appended last, after every built-in flag. The order is the contract: mlx-serve takes the last occurrence of a repeated flag, so an operator can override a built-in by naming it again in args. That is deliberate — args exists so the fleet config can reach mlx-serve knobs this package does not model (--prefix-cache-mem, --prefix-cache-disk, --kv-quant), and a precedence that let the built-ins win would make it unable to correct them.

The one hazard that buys: --host/--port are derived from s.Host/s.Port and feed the supervisor's own port observation, so an args entry contradicting them would break port observation rather than merely being ignored. Nothing rejects that today; see AATK-91.

Confirmed against a real mlx-serve 26.7.8 install: passing --model as a short "org/repo" name together with any --drafter value (short name or path) fails with a spurious FileNotFound — mlx-serve's own discovery registry excludes drafter/assistant checkpoints (their model_type isn't a servable chat model), so its short-name resolution can't validate the pairing. Passing BOTH as full local paths under mlx-serve's own models directory sidesteps that resolution path entirely and works. When s.Drafter is set, resolve both s.Model and s.Drafter to their cached local path if one exists; a name with no matching cached directory (or when s.Drafter is unset) is passed through unchanged, exactly as before.

func NewestLog

func NewestLog(logDir, name string) (path string, ok bool, err error)

NewestLog returns the path of the most-recently-modified log file for name under logDir. ok is false if no such file exists.

The glob "<name>-*.log" alone is not precise enough: a server literally named "chat" would also match "chat-llm-<ts>.log". Matches are filtered to the exact expected shape, "<name>-<logTimeLayout>.log", before comparing mtimes.

func PythonCommand

func PythonCommand(s config.Server) (command string, args []string)

PythonCommand builds the launch command for a python-type server, per design/aa-server-status.md §4, §10.

s.Entry's first whitespace-separated token is resolved against <venv>/bin (e.g. "supertonic serve" -> <venv>/bin/supertonic, with "serve" as the first arg; "python scripts/vad_server.py" -> <venv>/bin/python, with "scripts/vad_server.py" as the first arg). Any remaining tokens are appended verbatim, then --host/--port are auto-appended — no other flags are added.

func ResolveCommand

func ResolveCommand(s config.Server) (string, []string, error)

ResolveCommand returns the launch command and args for s, dispatching to the per-type *Command function. Used by both the engine's launch() and the REPL's "command" verb so the type-switch lives in one place.

func ResolveGracePeriod

func ResolveGracePeriod(s config.Server, supervisor config.Supervisor) time.Duration

ResolveGracePeriod returns s's own grace-period override when set, else falls back to supervisor's (already-defaulted) value. This is the only place the per-server-override-falls-back-to-supervisor-default rule (design/aa-server-status.md §7.1, config.Server.GracePeriod) is implemented — Teardown itself always takes an already-resolved time.Duration and never hard-codes or re-derives a default.

func RewriteBuildOutput

func RewriteBuildOutput(buildCmd, newPath string) (string, error)

RewriteBuildOutput replaces the "-o <path>" token pair in buildCmd with newPath, per design/aa-server-status.md §5: the canonical build command's output path is rewritten to a temp path for the staleness probe (and reused by the real rebuild).

Only a literal, space-separated "-o" token followed by a path token counts as the pair — "-o=path" (single token) and long-flag forms like "--output" are NOT rewritten; both fall through to the same "missing -o" hard error: only the explicit token pair counts.

func SourceCommand

func SourceCommand(s config.Server) (command string, args []string)

SourceCommand returns s.Binary and s.Args verbatim — source servers get no auto-appended flags (design/aa-server-status.md §4): run the binary with explicit args, ports come from the server's own args/config since a source server may be multi-port.

Types

type BuildLifecycle

type BuildLifecycle struct {
	Stop  func() error
	Start func() error
}

BuildLifecycle lets the caller supply stop/start callbacks so PerformBuild can mirror the prior lifecycle when replacing a stale binary: was running → stop → replace → start; was down → replace, stay down. Pass nil (or leave the func fields nil) when the server isn't running — PerformBuild skips the callback and just replaces the file.

type BuildResult

type BuildResult struct {
	// Replaced is true if the on-disk binary was rewritten because the
	// temp build differed from it (or it didn't exist). false means the
	// temp build was identical to what's already on disk — a no-op.
	Replaced bool
	// Restarted is true only when Replaced is true AND the caller supplied
	// a non-nil BuildLifecycle with both Stop and Start, and the full
	// stop → replace → start sequence completed without error.
	Restarted bool
}

BuildResult describes what PerformBuild did to a source server's on-disk binary.

func EnsureHealthProbeBuilt added in v0.3.0

func EnsureHealthProbeBuilt(s config.Server) (BuildResult, error)

EnsureHealthProbeBuilt builds s's Health.Source-declared probe program, if declared, so it exists (and is fresh) before the caller's first health probe — the source-exec form (design/aa-server-status.md §6.1, AATK-41). It is a no-op (zero BuildResult, nil error) when Health.Source isn't declared, so a Server{} written before this field existed is unaffected.

Building reuses PerformBuild's build-to-temp/hash-compare/replace machinery (source.go) via a synthetic TypeSource server value — building this probe program is exactly that operation, just for an artifact that is not the entry's own launch binary. A nil BuildLifecycle is passed: the probe has no running process of its own to stop/start around a replace.

s.Dir is deliberately NOT threaded into the synthetic server: s.Dir sets the entry's own launch cwd (a docker-compose Dir, say), an unrelated concern from where the probe's Go source lives. health.source carries no Dir of its own (its Build string is a complete, self-contained `go build` invocation), so reusing s.Dir here would silently point the probe's build at the wrong -C directory for any entry that sets one for its own launch.

Callers must invoke this once, during the start sequence, before the entry's first health probe — never from inside the probe itself. Building on every probe tick is the rejected design (design/aa-server-status.md §6.1: 0.13s warm, 4.72s cold against the 2s health_timeout cap); hoisting the build out of the probe and into the one-time start step is the whole point of this field.

func PerformBuild

func PerformBuild(s config.Server, lc *BuildLifecycle) (BuildResult, error)

PerformBuild runs the `build` verb's core operation for a source server s (design/aa-server-status.md §5): build to temp, hash-compare against the on-disk binary, and — if different — atomically replace it, mirroring the prior lifecycle around the replacement:

  • was running (lc.Stop != nil) → stop → replace → start
  • was down (lc nil or lc.Stop nil) → replace, stay down

build never starts a server that wasn't already running; starting from cold is up's job.

PerformBuild applies only to source-type servers; called with any other type ("build" on a non-source server) it returns a loud error instead of silently doing nothing, matching design/aa-server-status.md §2's `build` verb contract.

type KillSignal

type KillSignal string

KillSignal records which signal (if any) actually ended a teardown: the group either honored SIGTERM within the grace period, or needed a follow-up SIGKILL, or was already gone before any signal was needed.

const (
	// KillSignalNone means the group was already gone (no live process to
	// signal at all) — teardown is a no-op verify.
	KillSignalNone KillSignal = "none"
	// KillSignalTerm means SIGTERM alone was sufficient: the group exited
	// (and every declared port went free) within the grace period.
	KillSignalTerm KillSignal = "term"
	// KillSignalKill means SIGTERM was not enough — something in the group
	// survived the grace period, or a declared port was still listening —
	// so SIGKILL was sent to the group as well.
	KillSignalKill KillSignal = "kill"
)

type LaunchSpec

type LaunchSpec struct {
	LogDir  string
	Name    string
	Command string
	Args    []string
	Env     map[string]string

	// Dir, when non-empty, becomes the child's working directory
	// (cmd.Dir) — a relative venv/entry/binary on that server then
	// resolves against Dir, not against the supervisor's own launch cwd.
	// Empty Dir leaves cmd.Dir unset (inherits the supervisor's cwd),
	// matching today's behavior exactly. A leading "~/" is expanded
	// against the user's home directory, matching the same field's
	// existing expansion convention for source-type build sourcing
	// (see expandTilde in source.go).
	Dir string

	// Now is the launch time, which names this launch's log file
	// (design/aa-server-status.md §9). The zero value means "read the clock
	// here", which is the only reason Launch touches it at all; callers
	// that need a deterministic log name pass it explicitly.
	Now time.Time
}

LaunchSpec describes what to launch and how — a struct rather than a growing list of positional params, since every caller already has these values on hand from a config.Server.

type Process

type Process struct {
	Cmd     *exec.Cmd
	LogPath string
	// contains filtered or unexported fields
}

Process is a launched child: the running *exec.Cmd plus the log file path it's writing to. Reaping happens in a background goroutine (see Launch) — callers don't need to call Wait themselves.

func Launch

func Launch(spec LaunchSpec) (*Process, error)

Launch starts a child process per spec, per design/aa-server-status.md §6.4:

  • own process group (SysProcAttr{Setpgid: true}) — isolates the child from terminal signals and enables whole-tree group-kill later.
  • env is injected over the inherited environment: os.Environ() is the base, per-server keys in env win on collision.
  • stdout and stderr are both piped to the same resolved log file (see openLogForLaunch).
  • Wait() runs in a goroutine — Launch returns as soon as the process has started; reaping is fire-and-forget. That goroutine records the exit on the returned Process (see Exited), which is the only race-free way for a supervisor to learn a child died without being asked to kill it. Classifying that death is still out of scope for this package.

func LaunchExec

func LaunchExec(logDir string, s config.Server) (*Process, error)

LaunchExec launches s (an exec-type server) under logDir using the common launch core. command + args are passed through verbatim (ExecCommand).

func LaunchMLX

func LaunchMLX(logDir string, s config.Server) (*Process, error)

LaunchMLX launches s (an mlx-type server) under logDir using the common launch core.

func LaunchPython

func LaunchPython(logDir string, s config.Server) (*Process, error)

LaunchPython launches s (a python-type server) under logDir, after running the venv/package preflight described in design/aa-server-status.md §4, §10:

  • the venv directory and the resolved <venv>/bin/<entry-token> must exist (as a directory and a regular file, respectively);
  • each entry in s.Packages is import-checked individually via <venv>/bin/python -c "import <pkg>", so a failure names the exact missing package rather than a combined check.

A preflight failure returns a descriptive error naming what's missing and the venv path — it never launches, and it never crashes the caller; per §6.5, runtime command errors are the caller's job to surface loudly and return to the prompt.

func LaunchSource

func LaunchSource(logDir string, s config.Server) (*Process, error)

LaunchSource launches s (a source-type server) under logDir using the common launch core. binary + args are passed through verbatim (SourceCommand) — no auto-appended --host/--port flags.

s.Dir is still passed through as the child's cmd.Dir (per design/aa-server-status.md §7), but a relative s.Binary is resolved to an absolute path first: per config/types.go's Server.Dir doc, Binary always lands relative to aa-server-status's own launch cwd (that's where the build machinery in this file writes it), not relative to s.Dir — s.Dir's pre-existing role here is only to tell `go build` where to find source (insertGoDirFlag). Without this, a relative Binary would silently resolve against s.Dir at exec time instead of the directory it was actually built into.

func (*Process) Exited added in v0.3.0

func (p *Process) Exited() bool

Exited reports whether this child has terminated and been reaped. It is nil-safe, so callers holding an optional process handle can ask without guarding first.

False does not prove the child is alive: see the exited field's note on Processes constructed outside Launch. True does prove it is dead — the flag is only ever set after Wait has returned.

type Result

type Result struct {
	Name   string
	Signal KillSignal
	// VerifiedClean is true only when every declared port was confirmed
	// free and (if a health spec was given) the health probe no longer
	// responds. Teardown never returns a nil error with VerifiedClean
	// false — verification failure is always surfaced as an error (see
	// "never report a kill you didn't achieve").
	VerifiedClean bool
}

Result is the outcome of tearing down a single Target.

func Teardown

func Teardown(ctx context.Context, target Target, grace time.Duration) (Result, error)

Teardown tears down one server's process group: SIGTERM → wait the resolved grace period → if the group survives or any declared port still listens, SIGKILL the group → re-probe and verify every declared port is free and (if health is non-nil) the health probe is dead. Per design/aa-server-status.md §6.4, a surviving listener after SIGKILL is a loud error — Teardown never returns success alongside an unverified kill.

grace is resolved by the caller (ResolveGracePeriod) from the per-server override falling back to the supervisor default — Teardown itself takes the already-resolved duration and never hard-codes one.

func TeardownAll

func TeardownAll(ctx context.Context, servers []config.Server, supervisor config.Supervisor, pids map[string]int32) ([]Result, error)

TeardownAll tears down every entry in servers in reverse configured order — design/aa-server-status.md §6.4's "reverse config order" applies only to this multi-server path, never to Teardown itself. pids maps server name to the process-group leader PID to signal; a server with no entry in pids is skipped (nothing to tear down — e.g. it was never launched this session). Every server is attempted even if an earlier one errors — per §6.5, multi-server commands attempt all targets and report a loud aggregate rather than stopping at the first failure.

func TeardownForeign

func TeardownForeign(ctx context.Context, name string, pid int32, ports []int, grace time.Duration) (Result, error)

TeardownForeign tears down a process aa-server-status never launched — matched only by PID, exactly the "foreign stray kill" path used by the `dead` verb (design/aa-server-status.md §6.4: "Foreign strays killed by `dead` are group-killed by PID via gopsutil"). It shares the same TERM→grace→KILL→verify mechanism as Teardown; the only difference is that the caller supplies a bare PID discovered via observation instead of a Process handle this package itself launched.

type StalenessResult

type StalenessResult struct {
	// Stale is true when the on-disk binary differs from (or is missing
	// relative to) a fresh build of current source.
	Stale bool
	// TempBinary is the path to the freshly-built temp binary that
	// produced this result. Callers that need to act on staleness (e.g.
	// the build verb) can reuse it instead of building again — call
	// Cleanup when done with it.
	TempBinary string
	// Cleanup removes the temp build directory. Always call it (even on
	// Stale == false) once TempBinary is no longer needed.
	Cleanup func()
}

StalenessResult is the outcome of a staleness probe for a source server.

func ProbeStaleness

func ProbeStaleness(s config.Server) (StalenessResult, error)

ProbeStaleness runs the canonical build for s to a temp path (never touching the on-disk binary) and hashes the result against s.Binary, per design/aa-server-status.md §5. A missing on-disk binary counts as stale. Computed on `status` and on `<source> up` (callers, not this function).

ProbeStaleness only applies to source-type servers; called with any other type it returns a loud error rather than silently probing nothing.

type Target

type Target struct {
	// Name identifies the target in errors and TeardownResult (a server
	// name for the down/dead/TeardownAll paths, or a synthetic label like
	// "foreign pid 1234" for a stray with no configured name).
	Name string
	// PID is the process-group leader to signal. Negated internally to
	// address the whole group (syscall.Kill(-PID, sig)).
	PID int32
	// Ports is the exhaustive declared port set to verify free after kill.
	Ports []int
	// Health is an optional post-kill probe target — when non-nil, the
	// verify step also confirms the health endpoint is no longer answering
	// (a listener some other way still alive would otherwise slip through
	// if it isn't itself one of Ports, though in practice health.Port is
	// always a member of Ports per config validation).
	Health *health.Spec
}

Target names the process group teardown operates on: a PID (the group leader, since every launched child is its own process-group leader per Launch's Setpgid) plus the declared ports and optional health spec used to verify the kill actually landed. Ports is the server's exhaustive {port} ∪ listens set (design/aa-server-status.md §6.2) — every one of them must be free (not merely TIME_WAIT — see observe.listeningPorts, which only counts LISTEN) for teardown to report success.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL