Documentation
¶
Overview ¶
Package selfheal is the deterministic "Go floor" of the self-heal supervision design (SELF-HEAL-DESIGN.md §4.3). It detects a TRULY stuck session, classifies the cause, and — gated by a release stage — decides on a bounded, idempotent, fully-logged recovery.
STAGE 1 (this release, v1.9.67) is OBSERVE-ONLY. The engine evaluates the §1.3 stuck predicate, exercises the caps / backoff / circuit-breaker / flicker state-machine, and LOGS what it WOULD do — but takes ZERO action. Observe mode holds no action executor at all, so "no recovery primitive is ever called" is a structural property, not a runtime check (see Engine.Observe and the no-action test). Modes single_action / full are DEFINED but GUARDED: they refuse to act until Stages 2-3 are re-approved (the three §9 gap-fixes).
The module is pure: it consumes a Candidate snapshot (already-read data) and a monotonic clock, and emits a structured Event to an EventSink. It never reaches into tmux, never calls SaveInstances, never touches the rebind path. That keeps the predicate and the safety machine fully unit-testable and guarantees the observe-only invariant.
Index ¶
- Variables
- func DwellThreshold(s tmux.Substate) (time.Duration, bool)
- func IsStuckSubstate(s tmux.Substate) bool
- type Action
- type ActionExecutor
- type Candidate
- type Caps
- type CapsState
- type Config
- type Decision
- type Engine
- type Event
- type EventSink
- type MemorySink
- type Mode
- type NDJSONSink
- type PolicyMachine
- func (p *PolicyMachine) ClearQuarantine(sessionID string)
- func (p *PolicyMachine) Gate(c Candidate, now time.Time) (Decision, CapsState)
- func (p *PolicyMachine) IsQuarantined(sessionID string) bool
- func (p *PolicyMachine) RecordAttempt(c Candidate, now time.Time)
- func (p *PolicyMachine) RecordOutcome(c Candidate, healthy bool)
- func (p *PolicyMachine) SetFlickering(sessionID string, flickering bool)
- type PredicateResult
- type ReadSig
Constants ¶
This section is empty.
Variables ¶
var ErrActionInGuardedMode = errors.New("selfheal: actions are HELD (Stages 2-3 not re-approved)")
ErrActionInGuardedMode is returned if an action executor is ever invoked while the engine is in a guarded mode (single_action / full). Stages 2-3 are HELD pending Ashesh re-approval + the three §9 gap-fixes, so those modes refuse to act. observe never reaches an executor at all.
Functions ¶
func DwellThreshold ¶
DwellThreshold returns the cause-specific dwell window for a stuck substate, or 0 (and ok=false) for a non-stuck class.
func IsStuckSubstate ¶
IsStuckSubstate reports whether a substate is a self-heal-actionable stuck class (§1.3 condition 1). SubstateRunning / SubstateNone are never stuck.
Types ¶
type Action ¶
type Action string
Action is what self-heal would do for a candidate. It is recorded in the audit event; in observe mode it is the would_have value and is never executed.
const ( ActionNone Action = "" // ActionRestartModelSwitch — restart pinned to a safe model + re-issue last // intent once (model_unavailable, §2.1). ActionRestartModelSwitch Action = "restart_model_switch" // ActionRestart — one bounded restart via the resume path (wedged_pane, §2.3). ActionRestart Action = "restart" // ActionResend — re-send the last intent once before any restart // (idle_at_empty_prompt, §2.3). ActionResend Action = "resend" // ActionRestartReassertCreds — one restart that reasserts the credential // symlink, the known fix for the scratch-clobber 401 class (auth_401, §2.2). ActionRestartReassertCreds Action = "restart_reassert_creds" // ActionEscalate — surface one consolidated alert, take no recovery // (cap hit, breaker open, 401 that survived a restart). ActionEscalate Action = "escalate" )
func WouldHaveAction ¶
WouldHaveAction returns the action self-heal would take for a confirmed stuck candidate (used to populate would_have in observe mode).
type ActionExecutor ¶
type ActionExecutor interface {
// Execute applies the action to the candidate and reports the immediate
// outcome string. Only ever called in single_action/full AFTER the §9 gaps
// close and Ashesh re-approves.
Execute(c Candidate, a Action) (outcome string, err error)
}
ActionExecutor performs a real recovery. STAGE 1 NEVER CONSTRUCTS ONE — the observe-mode engine has a nil executor and a guarded mode refuses to call it. The interface exists so the wiring is in place for Stage 2 without reshaping the engine, but every method is unreachable in v1.9.67.
type Candidate ¶
type Candidate struct {
// Identity (carried into the audit event).
SessionID string
Title string
Group string
Profile string
Account string
// Coarse status + additive substate (Honest Status v2, v1.9.66).
Status string
Substate tmux.Substate
// Busy is the canonical busy-first signal (tmux.go busy-first ordering). A
// live busy indicator is AUTHORITATIVE and disqualifying (§1.3 #2 / §3.1).
Busy bool
// HookRunningFresh is true when a hook-status "running" was seen within the
// freshness window (sessionstatus freshnessFor). Mid-turn → never act
// (§1.3 #3 / §3.2).
HookRunningFresh bool
// OutputSig is a stable signature (hash) of the recent pane/output content
// THIS read. OutputMoved is true when it differs from the previous read's
// signature — token/output movement means mid-turn, disqualifying (§1.3 #3).
OutputSig string
OutputMoved bool
// Stopped marks a user-intentional stopped session (highest precedence,
// §1.3 #5). Such a session is never a candidate.
Stopped bool
// OptedOut is the per-session or group-level "never self-heal me" flag,
// checked as a quick disqualifier in the predicate (§3.7).
OptedOut bool
// StatusChangedAt anchors the dwell clock: when the current status/substate
// was entered. Zero means unknown (treated as not-yet-dwelled).
StatusChangedAt time.Time
// LastSentAt is when self-heal/the keysender last delivered input to this
// session. idle_at_empty_prompt only counts as stuck AFTER a send (§1.3 #4,
// §1.4): the dwell for that class is measured from LastSentAt. Zero means
// "we never sent it anything" → a long-waiting deliberate-idle session,
// never a candidate.
LastSentAt time.Time
}
Candidate is a pure snapshot of one session's self-heal-relevant state for a single evaluation cycle. It is assembled by the daemon adapter from the data the poll loop already read (status, substate, hook freshness, output signatures, the dwell anchors). The policy never reaches back into tmux or the DB; everything it needs to decide is in here. This is what makes the predicate deterministic and the observe-only invariant structural.
type Caps ¶
type Caps struct {
// PerSession6h is the max recoveries per session per rolling 6h window
// (default 2). auth_401 is special-cased to 1 (PerSessionAuth401).
PerSession6h int
// PerSessionAuth401 is the per-session cap for the auth_401 class (default 1):
// a 401 that survives one restart trips the breaker immediately (§2.2/§3.4).
PerSessionAuth401 int
// GlobalPerHour is the max recoveries across the WHOLE fleet per hour
// (default 5 = TriageMaxPerHour). On hitting it → escalate-only, one
// consolidated incident (§3.3).
GlobalPerHour int
// BreakerK is the consecutive-failed-recovery count that opens the per-session
// circuit breaker (default 2; 1 for auth_401, BreakerKAuth401) (§3.4).
BreakerK int
BreakerKAuth401 int
}
Caps holds the §3.3 rate limits. Defaults mirror the rails agent-deck already trusts: the global hourly cap is the TriageMaxPerHour=5 precedent.
func DefaultCaps ¶
func DefaultCaps() Caps
DefaultCaps returns the §3.3/§7 starting dials (tuned later from observe data).
type CapsState ¶
type CapsState struct {
Session6h int `json:"session_6h"`
GlobalHour int `json:"global_hour"`
BreakerFails int `json:"breaker_fails"`
BreakerOpen bool `json:"breaker_open"`
}
CapsState is a read-only view of the cap counters for one decision, recorded in the audit event's "caps" field.
type Decision ¶
type Decision string
Decision is the verdict the policy reached for a candidate this cycle.
const ( DecisionAct Decision = "act" // would act (observe: would_have set) DecisionSkipHealthy Decision = "skip_healthy" // substate not a stuck class DecisionSkipBusy Decision = "skip_busy" // live busy indicator (authoritative) DecisionSkipMidTurn Decision = "skip_midturn" // fresh hook-running / output moved DecisionSkipDwell Decision = "skip_dwell" // not dwelled past threshold yet DecisionSkipConfirm Decision = "skip_confirm" // second read disagreed (2-read drop) DecisionSkipStopped Decision = "skip_stopped" // session stopped (user-intentional) DecisionSkipOptOut Decision = "skip_optout" // per-session/group opt-out DecisionCapHit Decision = "cap_hit" // per-session or global cap exceeded DecisionBreakerOpen Decision = "breaker_open" // circuit breaker / flicker quarantine )
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is the per-process self-heal policy. It owns the safety machine, the audit sink, and (for non-observe stages only) the action executor. In observe mode exec is nil and stage gating refuses any execution path, so "no recovery primitive is ever called" is structural.
func NewObserveEngine ¶
NewObserveEngine builds the Stage-1 observe-only engine. It has NO action executor — by construction it cannot take a recovery action. This is the only constructor the daemon uses in v1.9.67.
func (*Engine) Policy ¶
func (e *Engine) Policy() *PolicyMachine
Policy exposes the safety machine (for the flicker subscription wiring).
func (*Engine) ProcessRead ¶
ProcessRead evaluates ONE read of one candidate, advances the two-read confirm state-machine, exercises the safety machine, and emits exactly one audit event describing what self-heal would do. In observe mode it returns having taken NO action — guaranteed because:
- the executor is nil, and
- executeIfAuthorized refuses any non-observe mode (Stages 2-3 HELD).
now must be a date-u-anchored monotonic-ish UTC time (the daemon anchors it). outputMoved/hookRunningFresh come from the caller's existing reads; the engine also folds in its own prevSig comparison so a caller that doesn't compute movement still gets it.
type Event ¶
type Event struct {
TS string `json:"ts"`
SessionID string `json:"session_id"`
Title string `json:"title"`
Conductor string `json:"conductor,omitempty"`
Group string `json:"group,omitempty"`
Profile string `json:"profile"`
Account string `json:"account,omitempty"`
Substate string `json:"substate"`
Dwell float64 `json:"dwell_seconds"`
Reads []ReadSig `json:"reads"`
Decision Decision `json:"decision"`
// Action is the action TAKEN (null/empty on skip and in observe mode — observe
// never takes an action). WouldHave carries the observe-mode would-be action.
Action Action `json:"action,omitempty"`
ActionParams map[string]any `json:"action_params,omitempty"`
Caps CapsState `json:"caps"`
Outcome string `json:"outcome,omitempty"`
Stage Mode `json:"stage"`
// WouldHave is present in observe mode: the action self-heal WOULD have taken
// had it been authorized. Empty when the decision was a skip.
WouldHave Action `json:"would_have,omitempty"`
}
Event is the single structured record emitted per detection cycle, whether or not it acts (§5). It is the observability hook for the future OGDB operational-KG (we emit the record; we do NOT build the sink). In observe mode Stage is "observe" and WouldHave carries the action that WOULD have run.
type EventSink ¶
EventSink is the durable audit destination. The default is an append-only NDJSON file; the operational-KG ingestion path can subscribe to the same records later (§5). Append must be safe for concurrent callers.
type MemorySink ¶
type MemorySink struct {
Events []Event
// contains filtered or unexported fields
}
MemorySink collects events in memory (tests, and any in-process subscriber).
func (*MemorySink) Snapshot ¶
func (m *MemorySink) Snapshot() []Event
Snapshot returns a copy of the recorded events.
type Mode ¶
type Mode string
Mode is the global self-heal authority level (config selfheal mode).
const ( // ModeObserve logs would_have for every candidate and takes NO action. The // default and the only mode that does anything in v1.9.67. ModeObserve Mode = "observe" // ModeSingleAction is Stage 2 (model_unavailable restart+Opus only). DEFINED // but GUARDED — refuses to act until Ashesh re-approves + the §9 gaps close. ModeSingleAction Mode = "single_action" // ModeFull is Stage 3 (all classes, auto within caps). DEFINED but GUARDED. ModeFull Mode = "full" )
type NDJSONSink ¶
type NDJSONSink struct {
// contains filtered or unexported fields
}
NDJSONSink is an append-only newline-delimited-JSON file sink. It uses a targeted append (O_APPEND create), NEVER a full-file rewrite and NEVER SaveInstances — it cannot wipe or truncate existing audit history (§3.5: no destructive write primitive). The parent directory is created 0o700.
func NewNDJSONSink ¶
func NewNDJSONSink(path string) (*NDJSONSink, error)
NewNDJSONSink returns a sink appending to path. The directory is created if absent. The file itself is created lazily on first Append.
func (*NDJSONSink) Append ¶
func (s *NDJSONSink) Append(e Event) error
Append writes one event as a single JSON line. O_APPEND guarantees the write only ever extends the file; it never truncates or overwrites prior records.
func (*NDJSONSink) Path ¶
func (s *NDJSONSink) Path() string
Path returns the audit file path (so callers/tests can report where it lands).
type PolicyMachine ¶
type PolicyMachine struct {
// contains filtered or unexported fields
}
PolicyMachine is the deterministic safety state-machine (§3): caps, backoff, circuit breaker, and FlickerDetector subscription. It is the SAME machine in every stage; in observe mode it is exercised and logged but never gates a real action (because no action runs). Safe for concurrent use.
func NewPolicyMachine ¶
func NewPolicyMachine(caps Caps) *PolicyMachine
NewPolicyMachine builds a machine with the given caps (use DefaultCaps()).
func (*PolicyMachine) ClearQuarantine ¶
func (p *PolicyMachine) ClearQuarantine(sessionID string)
ClearQuarantine releases a session's breaker (human/Maestro action, §3.4).
func (*PolicyMachine) Gate ¶
Gate decides whether the safety machine would ALLOW a recovery for a confirmed candidate right now, and returns the cap snapshot + the gating decision. It is pure w.r.t. recorded state (no side effects); RecordAttempt commits a would-be attempt. The returned Decision is DecisionAct when allowed, or the specific guard that blocked it (cap_hit / breaker_open).
Ordering (most-protective first): breaker / flicker quarantine, then global fleet cap, then per-session cap. Backoff is folded into the per-session cap check via the most-recent attempt time.
func (*PolicyMachine) IsQuarantined ¶
func (p *PolicyMachine) IsQuarantined(sessionID string) bool
IsQuarantined reports whether the session's breaker is open (manual clear or a return to healthy is required). flicker is treated as quarantine-equivalent.
func (*PolicyMachine) RecordAttempt ¶
func (p *PolicyMachine) RecordAttempt(c Candidate, now time.Time)
RecordAttempt commits a would-be recovery attempt to the rolling windows. In observe mode this is what exercises the caps/backoff machine without any real action: the engine records the attempt it WOULD have made so the next cycle's Gate reflects it and the audit shows caps advancing.
func (*PolicyMachine) RecordOutcome ¶
func (p *PolicyMachine) RecordOutcome(c Candidate, healthy bool)
RecordOutcome updates the circuit breaker after a recovery's outcome is known. healthy=true resets the consecutive-fail counter; healthy=false increments it and, on reaching K, opens the breaker (quarantine). Used by Stages 2-3; in observe mode there is no real outcome, so it is exercised only in tests.
func (*PolicyMachine) SetFlickering ¶
func (p *PolicyMachine) SetFlickering(sessionID string, flickering bool)
SetFlickering records the FlickerDetector verdict for a session. A flapping session is quarantine-equivalent: self-heal must never restart into a flap (the #1349 / duplicate-killer family, §3.4/§3.5).
type PredicateResult ¶
type PredicateResult struct {
// Candidate is true only when ALL §1.3 conditions hold for this read. The
// two-read confirm is layered ON TOP by the Engine (one read is never enough,
// §1.3 #4 / PLAYBOOK F5).
Candidate bool
// Decision is the most-precise reason the predicate reached its verdict. For
// a true candidate it is DecisionAct (pending the 2-read confirm); for a
// false one it names the disqualifier.
Decision Decision
// Dwell is the measured dwell at evaluation time (0 if no anchor).
Dwell time.Duration
}
PredicateResult is the outcome of evaluating the §1.3 stuck predicate against a single read of a candidate. It is intentionally verbose so the audit event can record exactly which condition decided the verdict.
func Evaluate ¶
func Evaluate(c Candidate, now time.Time) PredicateResult
Evaluate runs the §1.3 stuck predicate for ONE read. It is pure: same inputs → same verdict. The disqualifier ordering mirrors the design's precedence — stopped (user intent) and opt-out first as cheap exits, then the authoritative busy / mid-turn signals, then substate class, then dwell.
A true result means "candidate for THIS read". The caller (Engine) must require two independent confirming reads before acting (§1.3 #4).