Documentation
¶
Overview ¶
Package selfheal is the bounded self-healing supervisor (plan §8.5): it restarts crashed/stuck services and ESCALATES to a Mooring-originated infra alert (§8.4) when it gives up — designed so that on a constrained box it can only REDUCE pressure or hold steady, NEVER manufacture an OOM (worst case: it declines to act and pages you).
This file is the PURE decision core: a per-(app,service) finite-state machine driven entirely by already-polled snapshot data. It performs no I/O — the watcher (watcher.go) supplies observations, applies the four safety gates, executes the chosen rung through the write-plane runner, and persists the result. Keeping the decision pure makes the safety properties exhaustively testable.
Index ¶
- type Act
- type Actioner
- type Config
- type Decision
- type FSM
- type GateInput
- type GateOutcome
- type Key
- type Observation
- type Phase
- type Policy
- type Rung
- type Store
- func (s *Store) AcquireExpectedDown(ctx context.Context, app string, until int64) error
- func (s *Store) ActiveExpectedDown(now int64) (map[string]bool, error)
- func (s *Store) ClearAllExpectedDown(ctx context.Context) error
- func (s *Store) ClearCircuit(ctx context.Context, k Key, now int64) error
- func (s *Store) Delete(ctx context.Context, k Key) error
- func (s *Store) DeleteApp(ctx context.Context, app string) error
- func (s *Store) DeletePolicy(ctx context.Context, project string) error
- func (s *Store) LoadAll() (map[Key]FSM, error)
- func (s *Store) PolicyFor(project string) (Policy, bool, error)
- func (s *Store) ReleaseExpectedDown(ctx context.Context, app string) error
- func (s *Store) Save(ctx context.Context, k Key, f FSM, now int64) error
- func (s *Store) SavePolicy(ctx context.Context, project string, p Policy, now int64) error
- type Watcher
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Actioner ¶
type Actioner interface {
Remediate(ctx context.Context, app monitor.App, service string, rung Rung) error
}
Actioner executes a remediation rung for one service. The watcher calls it ONLY after the four safety gates pass, and while it HOLDS the one-docker-child semaphore (acquired non-blocking by the gate).
func NewRunnerActioner ¶
func NewRunnerActioner(runner *dockerexec.Runner, jobFor func(app monitor.App, service string, action []string) dockerexec.Job) Actioner
NewRunnerActioner builds the production Actioner. jobFor lets the caller supply the env-file / config-file details for an app (reusing the write-plane builder).
type Config ¶
type Config struct {
Store *Store
Alerts *alertstore.Store // nil → pages are logged only (alerting disabled)
Snap func() *monitor.Snapshot
Sem *dockerexec.Semaphore
Act Actioner
Policy Policy // the built-in/global default
PolicyFor func(app string) Policy // per-app override (nil → always Policy); see spec.self_healing
Log *slog.Logger
Interval time.Duration
FloorBytes uint64 // memory-headroom floor (0 = gate disabled, e.g. no host metrics)
WritePlaneOK bool // the §0 write-plane gate result
Protected map[string]bool // project names that are the edge/control plane — never targets
Now func() int64 // injectable clock; defaults to time.Now().Unix
}
Config configures a Watcher. The function/clock fields are injectable for tests.
type Decision ¶
type Decision struct {
Next FSM // the FSM to persist IF the action is taken (or is a no-op/page)
Act Act // what the watcher should do
Rung Rung // the rung to run when Act==ActRemediate
Kind string // the can't-fix taxonomy kind when Act==ActPage
Reason string // human-readable, for the event/audit
}
Decision is the pure outcome of stepping the FSM for one service.
func Decide ¶
func Decide(prev FSM, o Observation, p Policy, now int64) Decision
Decide steps the FSM for one service. It is a pure function of the prior state, the observation, the policy, and the current time. It NEVER performs an action; it only decides what should happen. The watcher applies the safety gates to a returned ActRemediate before executing, and only then commits Decision.Next.
type FSM ¶
type FSM struct {
Phase Phase
UnhealthyStreak int // consecutive failing ticks (anti-flap sustain)
HealthyStreak int // consecutive healthy ticks (recovery stabilization)
Attempts int // remediation attempts in the current window
LastRung Rung // highest rung attempted this window
BackoffUntil int64 // unix sec; no remediation before this deadline
WindowStart int64 // unix sec; start of the current attempt window
OOMStrikes int // consecutive OOM-classified failures
DegradedSince int64 // unix sec; first failing tick of the current episode
Open bool // an infra alert is currently open for this service
}
FSM is the persisted per-(app,service) state.
func CommitRemediation ¶
CommitRemediation advances the FSM after a rung has actually been executed: it consumes an attempt, records the rung, and arms the backoff. The watcher calls it ONLY when the safety gates passed and the action ran.
type GateInput ¶
type GateInput struct {
Rung Rung
// Gate 1 — §0 resource gate for the action's plane. WritePlaneOK is the global
// ≥1 GB write-plane gate; RedeployEnabled gates the redeploy rung specifically.
WritePlaneOK bool
RedeployEnabled bool
// Gate 2 — the global one-docker-child semaphore. AcquireSemaphore is a
// NON-BLOCKING TryAcquire supplied by the watcher; nil means "treat as busy".
// When it returns true the caller now HOLDS the semaphore and must Release it.
AcquireSemaphore func() bool
// Gate 3 — memory-headroom floor. A restart momentarily runs old+new, so below
// the floor we must not restart. HeadroomBytes is current free memory (host),
// FloorBytes the configured minimum that must remain available.
HeadroomBytes uint64
FloorBytes uint64
// Gate 4 — edge protection. The edge slice and control plane are never targets.
IsEdgeOrControlPlane bool
}
GateInput is the environment for the gate checks at action time.
type GateOutcome ¶
type GateOutcome string
GateOutcome is the result of evaluating the gates for a proposed action.
const ( GateProceed GateOutcome = "proceed" // all gates pass: execute the rung GateDefer GateOutcome = "defer" // a transient gate failed: retry next tick, no attempt consumed GatePage GateOutcome = "page" // headroom too low to safely restart: page instead GateSkip GateOutcome = "skip" // edge / control-plane target: never a remediation target )
func Gates ¶
func Gates(in GateInput) (GateOutcome, string)
Gates evaluates the four gates IN ORDER for a proposed remediation. On GateProceed the caller holds the docker-child semaphore and MUST release it after running the action; on every other outcome no semaphore is held.
type Observation ¶
type Observation struct {
Running bool
Health string // none|healthy|unhealthy|starting
RestartCount int
OOMKilled bool
ExitCode int
WaitingOnEdge bool // a service still waiting on its edge-issued cert
ExpectedDown bool // a VALID write-plane lease is held for this app
}
Observation is the per-tick view of one service, derived from the latest snapshot (no extra I/O). ExpectedDown / WaitingOnEdge are computed by the watcher.
type Phase ¶
type Phase string
Phase is the supervisor state for one (app,service). The happy path is HEALTHY → SUSPECT → DEGRADED → REMEDIATING → (RECOVERED → HEALTHY) and the giving- up path is → CIRCUIT_OPEN. WAITING_ON_EDGE / EXPECTED_DOWN are suspensions where the supervisor deliberately does NOT act.
type Policy ¶
type Policy struct {
SustainTicks int // failing ticks before the first remediation (anti-flap)
AttemptCap int // remediations per window before the circuit opens
StabilizeTicks int // healthy ticks required to declare RECOVERED
OOMStrikeCap int // OOM-classified failures before short-circuiting the ladder
WindowSeconds int64 // attempt-window length; attempts reset after it elapses
BackoffBaseSecs int64 // exponential backoff base between attempts
BackoffMaxSecs int64 // backoff ceiling
RedeployEnabled bool // rung-3 redeploy (≥1 GB host AND operator opt-in)
}
Policy holds the tunables (plan §8.5 / Tier-1 selfheal.* config).
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store persists the per-(app,service) FSM and the expected_down leases. Alert and FSM state are recovered from SQLite on restart so a bounce neither re-fires remediation nor loses an open circuit.
func (*Store) AcquireExpectedDown ¶
AcquireExpectedDown opens/extends a bounded lease for an app (the write plane holds it while it intentionally touches the app). until is the auto-expiry.
func (*Store) ActiveExpectedDown ¶
ActiveExpectedDown returns the set of apps with a non-expired lease.
func (*Store) ClearAllExpectedDown ¶
ClearAllExpectedDown wipes every lease — called fail-closed on boot, so a deploy that crashed without releasing its lease can't suppress a crash-loop alert forever.
func (*Store) ClearCircuit ¶
ClearCircuit resets a latched CIRCUIT_OPEN service to HEALTHY so the supervisor will act on it again (the operator's "I fixed the underlying problem" button).
func (*Store) DeleteApp ¶
DeleteApp removes ALL self-healing state for an app: the per-service FSM rows, the tuned policy, and any expected-down lease. Used by the app-delete teardown.
func (*Store) DeletePolicy ¶
DeletePolicy drops an app's tuned policy (reverting it to the built-in default).
func (*Store) PolicyFor ¶
PolicyFor returns an app's tuned policy. ok=false (and the built-in default should be used) when the app has no row.
func (*Store) ReleaseExpectedDown ¶
ReleaseExpectedDown clears an app's lease (the action finished).
func (*Store) SavePolicy ¶
SavePolicy upserts one app's tuned self-healing policy. The whole-app policy is the mooring.yaml source of truth; the supervisor reads it per tick via PolicyFor and falls back to the built-in default for an app with no row.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher is the bounded self-healing supervisor loop (plan §8.5).
func (*Watcher) ClearCircuit ¶
ClearCircuit requests that a latched CIRCUIT_OPEN service be reset to HEALTHY (the operator's "I fixed the root cause, try again" button). It is safe to call from another goroutine (the web handler): it only records the request under a short lock; the watcher applies it at the start of the next tick, so it never races the fsm map and never blocks on tick I/O.