selfheal

package
v0.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 1, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package selfheal is the bounded self-healing supervisor (plan §8.5): it restarts crashed/stuck services and ESCALATES to a Mooring-originated infra alert (§8.4) when it gives up — designed so that on a constrained box it can only REDUCE pressure or hold steady, NEVER manufacture an OOM (worst case: it declines to act and pages you).

This file is the PURE decision core: a per-(app,service) finite-state machine driven entirely by already-polled snapshot data. It performs no I/O — the watcher (watcher.go) supplies observations, applies the four safety gates, executes the chosen rung through the write-plane runner, and persists the result. Keeping the decision pure makes the safety properties exhaustively testable.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Act

type Act string

Act is what the watcher should do this tick.

const (
	ActNone      Act = "none"      // nothing to do (healthy, or suspended)
	ActRemediate Act = "remediate" // run Decision.Rung (subject to the safety gates)
	ActPage      Act = "page"      // give up: emit an infra alert (Decision.Kind)
	ActResolve   Act = "resolve"   // recovered: clear any open infra alert
)

type Actioner

type Actioner interface {
	Remediate(ctx context.Context, app monitor.App, service string, rung Rung) error
}

Actioner executes a remediation rung for one service. The watcher calls it ONLY after the four safety gates pass, and while it HOLDS the one-docker-child semaphore (acquired non-blocking by the gate).

func NewRunnerActioner

func NewRunnerActioner(runner *dockerexec.Runner, jobFor func(app monitor.App, service string, action []string) dockerexec.Job) Actioner

NewRunnerActioner builds the production Actioner. jobFor lets the caller supply the env-file / config-file details for an app (reusing the write-plane builder).

type Config

type Config struct {
	Store        *Store
	Alerts       *alertstore.Store // nil → pages are logged only (alerting disabled)
	Snap         func() *monitor.Snapshot
	Sem          *dockerexec.Semaphore
	Act          Actioner
	Policy       Policy                  // the built-in/global default
	PolicyFor    func(app string) Policy // per-app override (nil → always Policy); see spec.self_healing
	Log          *slog.Logger
	Interval     time.Duration
	FloorBytes   uint64          // memory-headroom floor (0 = gate disabled, e.g. no host metrics)
	WritePlaneOK bool            // the §0 write-plane gate result
	Protected    map[string]bool // project names that are the edge/control plane — never targets
	Now          func() int64    // injectable clock; defaults to time.Now().Unix
}

Config configures a Watcher. The function/clock fields are injectable for tests.

type Decision

type Decision struct {
	Next   FSM    // the FSM to persist IF the action is taken (or is a no-op/page)
	Act    Act    // what the watcher should do
	Rung   Rung   // the rung to run when Act==ActRemediate
	Kind   string // the can't-fix taxonomy kind when Act==ActPage
	Reason string // human-readable, for the event/audit
}

Decision is the pure outcome of stepping the FSM for one service.

func Decide

func Decide(prev FSM, o Observation, p Policy, now int64) Decision

Decide steps the FSM for one service. It is a pure function of the prior state, the observation, the policy, and the current time. It NEVER performs an action; it only decides what should happen. The watcher applies the safety gates to a returned ActRemediate before executing, and only then commits Decision.Next.

type FSM

type FSM struct {
	Phase           Phase
	UnhealthyStreak int   // consecutive failing ticks (anti-flap sustain)
	HealthyStreak   int   // consecutive healthy ticks (recovery stabilization)
	Attempts        int   // remediation attempts in the current window
	LastRung        Rung  // highest rung attempted this window
	BackoffUntil    int64 // unix sec; no remediation before this deadline
	WindowStart     int64 // unix sec; start of the current attempt window
	OOMStrikes      int   // consecutive OOM-classified failures
	DegradedSince   int64 // unix sec; first failing tick of the current episode
	Open            bool  // an infra alert is currently open for this service
}

FSM is the persisted per-(app,service) state.

func CommitRemediation

func CommitRemediation(f FSM, rung Rung, p Policy, now int64) FSM

CommitRemediation advances the FSM after a rung has actually been executed: it consumes an attempt, records the rung, and arms the backoff. The watcher calls it ONLY when the safety gates passed and the action ran.

type GateInput

type GateInput struct {
	Rung Rung

	// Gate 1 — §0 resource gate for the action's plane. WritePlaneOK is the global
	// ≥1 GB write-plane gate; RedeployEnabled gates the redeploy rung specifically.
	WritePlaneOK    bool
	RedeployEnabled bool

	// Gate 2 — the global one-docker-child semaphore. AcquireSemaphore is a
	// NON-BLOCKING TryAcquire supplied by the watcher; nil means "treat as busy".
	// When it returns true the caller now HOLDS the semaphore and must Release it.
	AcquireSemaphore func() bool

	// Gate 3 — memory-headroom floor. A restart momentarily runs old+new, so below
	// the floor we must not restart. HeadroomBytes is current free memory (host),
	// FloorBytes the configured minimum that must remain available.
	HeadroomBytes uint64
	FloorBytes    uint64

	// Gate 4 — edge protection. The edge slice and control plane are never targets.
	IsEdgeOrControlPlane bool
}

GateInput is the environment for the gate checks at action time.

type GateOutcome

type GateOutcome string

GateOutcome is the result of evaluating the gates for a proposed action.

const (
	GateProceed GateOutcome = "proceed" // all gates pass: execute the rung
	GateDefer   GateOutcome = "defer"   // a transient gate failed: retry next tick, no attempt consumed
	GatePage    GateOutcome = "page"    // headroom too low to safely restart: page instead
	GateSkip    GateOutcome = "skip"    // edge / control-plane target: never a remediation target
)

func Gates

func Gates(in GateInput) (GateOutcome, string)

Gates evaluates the four gates IN ORDER for a proposed remediation. On GateProceed the caller holds the docker-child semaphore and MUST release it after running the action; on every other outcome no semaphore is held.

type Key

type Key struct{ App, Service string }

Key identifies one supervised service.

type Observation

type Observation struct {
	Running       bool
	Health        string // none|healthy|unhealthy|starting
	RestartCount  int
	OOMKilled     bool
	ExitCode      int
	WaitingOnEdge bool // a service still waiting on its edge-issued cert
	ExpectedDown  bool // a VALID write-plane lease is held for this app
}

Observation is the per-tick view of one service, derived from the latest snapshot (no extra I/O). ExpectedDown / WaitingOnEdge are computed by the watcher.

type Phase

type Phase string

Phase is the supervisor state for one (app,service). The happy path is HEALTHY → SUSPECT → DEGRADED → REMEDIATING → (RECOVERED → HEALTHY) and the giving- up path is → CIRCUIT_OPEN. WAITING_ON_EDGE / EXPECTED_DOWN are suspensions where the supervisor deliberately does NOT act.

const (
	Healthy       Phase = "HEALTHY"
	Suspect       Phase = "SUSPECT"
	Degraded      Phase = "DEGRADED"
	Remediating   Phase = "REMEDIATING"
	CircuitOpen   Phase = "CIRCUIT_OPEN"
	Recovered     Phase = "RECOVERED"
	WaitingOnEdge Phase = "WAITING_ON_EDGE"
	ExpectedDown  Phase = "EXPECTED_DOWN"
)

type Policy

type Policy struct {
	SustainTicks    int   // failing ticks before the first remediation (anti-flap)
	AttemptCap      int   // remediations per window before the circuit opens
	StabilizeTicks  int   // healthy ticks required to declare RECOVERED
	OOMStrikeCap    int   // OOM-classified failures before short-circuiting the ladder
	WindowSeconds   int64 // attempt-window length; attempts reset after it elapses
	BackoffBaseSecs int64 // exponential backoff base between attempts
	BackoffMaxSecs  int64 // backoff ceiling
	RedeployEnabled bool  // rung-3 redeploy (≥1 GB host AND operator opt-in)
}

Policy holds the tunables (plan §8.5 / Tier-1 selfheal.* config).

func DefaultPolicy

func DefaultPolicy() Policy

DefaultPolicy is the conservative built-in policy.

type Rung

type Rung string

Rung is one step of the remediation ladder, in escalating order.

const (
	RungNone     Rung = ""
	RungRestart  Rung = "restart"
	RungRecreate Rung = "recreate" // --force-recreate: re-runs host-side template render + cert-sync
	RungRedeploy Rung = "redeploy" // off by default, ≥1 GB only
)

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store persists the per-(app,service) FSM and the expected_down leases. Alert and FSM state are recovered from SQLite on restart so a bounce neither re-fires remediation nor loses an open circuit.

func NewStore

func NewStore(db *store.DB) *Store

NewStore builds a Store.

func (*Store) AcquireExpectedDown

func (s *Store) AcquireExpectedDown(ctx context.Context, app string, until int64) error

AcquireExpectedDown opens/extends a bounded lease for an app (the write plane holds it while it intentionally touches the app). until is the auto-expiry.

func (*Store) ActiveExpectedDown

func (s *Store) ActiveExpectedDown(now int64) (map[string]bool, error)

ActiveExpectedDown returns the set of apps with a non-expired lease.

func (*Store) ClearAllExpectedDown

func (s *Store) ClearAllExpectedDown(ctx context.Context) error

ClearAllExpectedDown wipes every lease — called fail-closed on boot, so a deploy that crashed without releasing its lease can't suppress a crash-loop alert forever.

func (*Store) ClearCircuit

func (s *Store) ClearCircuit(ctx context.Context, k Key, now int64) error

ClearCircuit resets a latched CIRCUIT_OPEN service to HEALTHY so the supervisor will act on it again (the operator's "I fixed the underlying problem" button).

func (*Store) Delete

func (s *Store) Delete(ctx context.Context, k Key) error

Delete drops an FSM (a service that no longer exists in any app).

func (*Store) DeleteApp

func (s *Store) DeleteApp(ctx context.Context, app string) error

DeleteApp removes ALL self-healing state for an app: the per-service FSM rows, the tuned policy, and any expected-down lease. Used by the app-delete teardown.

func (*Store) DeletePolicy

func (s *Store) DeletePolicy(ctx context.Context, project string) error

DeletePolicy drops an app's tuned policy (reverting it to the built-in default).

func (*Store) LoadAll

func (s *Store) LoadAll() (map[Key]FSM, error)

LoadAll returns every persisted FSM, keyed by (app,service).

func (*Store) PolicyFor

func (s *Store) PolicyFor(project string) (Policy, bool, error)

PolicyFor returns an app's tuned policy. ok=false (and the built-in default should be used) when the app has no row.

func (*Store) ReleaseExpectedDown

func (s *Store) ReleaseExpectedDown(ctx context.Context, app string) error

ReleaseExpectedDown clears an app's lease (the action finished).

func (*Store) Save

func (s *Store) Save(ctx context.Context, k Key, f FSM, now int64) error

Save upserts one FSM.

func (*Store) SavePolicy

func (s *Store) SavePolicy(ctx context.Context, project string, p Policy, now int64) error

SavePolicy upserts one app's tuned self-healing policy. The whole-app policy is the mooring.yaml source of truth; the supervisor reads it per tick via PolicyFor and falls back to the built-in default for an app with no row.

type Watcher

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher is the bounded self-healing supervisor loop (plan §8.5).

func New

func New(cfg Config) *Watcher

New builds a Watcher.

func (*Watcher) ClearCircuit

func (w *Watcher) ClearCircuit(k Key)

ClearCircuit requests that a latched CIRCUIT_OPEN service be reset to HEALTHY (the operator's "I fixed the root cause, try again" button). It is safe to call from another goroutine (the web handler): it only records the request under a short lock; the watcher applies it at the start of the next tick, so it never races the fsm map and never blocks on tick I/O.

func (*Watcher) Run

func (w *Watcher) Run(ctx context.Context)

Run boots (clearing stale expected_down leases fail-closed, then recovering FSM state from SQLite) and ticks the supervisor until ctx is cancelled.

func (*Watcher) Tick

func (w *Watcher) Tick(ctx context.Context)

Tick runs one supervision pass over the latest snapshot. Exported for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL