hubdoctor

package
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: MIT Imports: 71 Imported by: 0

Documentation

Overview

Package hubdoctor diagnoses a cloop *control plane* — the thing an operator deploys — as distinct from pkg/doctor, which diagnoses a developer's checkout.

Why it is a separate package

pkg/doctor answers "can this working copy run cloop": is there a .cloop directory, is `go` on PATH, is the API key set, is state.json valid JSON. Every one of its checks predates the hub, and none of them can fail on a deployment that is thoroughly broken as a hub: an issuer that does not resolve, a certificate that expired last week, an RBAC policy under which nobody is an admin, strict mode with nothing to dispatch to. That hub passes `cloop doctor` with fourteen green lines and cannot serve a single request.

So this is a second diagnosis with a different subject, not more checks bolted onto the first. Its checks are the ones whose answers only exist once cloop is hosted — identity, transport, authorization, image trust, executors, storage and admission — and each one is written to be actionable rather than merely true: every non-pass finding carries a one-line remediation naming the file, field or command that fixes it.

The contract every check keeps

  • A check never fails the whole run. A hub whose issuer is unreachable must still be told about its expiring certificate, so each check contributes findings and the report is the sum of them.
  • Severity means something specific. Fail: the hub is broken or unsafe in a way an operator must fix. Warn: it works, but a foreseeable event breaks it or a control is weaker than the deployment implies. Pass: verified, not merely absent.
  • Absence is not a pass. A check that could not run — no network, no database — reports that as a warning, because "we did not look" and "we looked and it was fine" are different answers and only one of them is worth acting on.

Exit codes and --json

Run returns a Report; the caller decides how loudly to complain. ExitCode is 0 with only passes and warnings, 1 with any failure, so `cloop hub doctor` is usable as a CI gate and as a pre-flight in a deployment pipeline. --json emits the same Report verbatim, which is why every field has a stable JSON tag and every finding has a stable Check id: those ids are the thing a pipeline greps for, and renaming one is a breaking change.

Index

Constants

View Source
const (
	// CheckUnfinished: a step that did not finish by its deadline, or is
	// still running from an earlier report.
	CheckUnfinished = "doctor.unfinished"
	// CheckPanicked: a step that panicked.
	CheckPanicked = "doctor.panic"
)

The check ids of the findings a live run adds about its own steps.

View Source
const (
	// SmokeExitOK: every target smoked clean.
	SmokeExitOK = 0
	// SmokeExitConfig: a non-smoke check failed, so the hub is misconfigured
	// independently of whether it can dispatch. Matches Report.ExitCode.
	SmokeExitConfig = 1
	// SmokeExitNoTargets: there was nothing to smoke. Not the same as healthy:
	// a hub with no executor cannot run anything.
	SmokeExitNoTargets = 3
	// SmokeExitFailed: at least one executor failed a stage.
	SmokeExitFailed = 4
	// SmokeExitLeaked: the run could not clean up after itself. Highest
	// precedence — it is the only outcome that needs a human on a machine.
	SmokeExitLeaked = 5
)

Smoke exit codes. Distinct so the command is usable both as a Kubernetes readiness gate (which only cares about zero) and as a post-deploy CI step (which wants to tell a broken executor from a misconfigured hub from a diagnostic that left rubbish behind).

2 is deliberately unused: Cobra and `cloop executor test` already spend it.

View Source
const DefaultCheckTimeout = 20 * time.Second

DefaultCheckTimeout bounds one step of a live run. A step's own probes are bounded by Options.Timeout each, and the slowest steps make two or three of them; this is the backstop for a step that waits on something with no timeout of its own — a lock, a socket nobody answers.

View Source
const DefaultDialTimeout = 3 * time.Second

DefaultDialTimeout bounds one reachability dial.

Shorter than DefaultProbeTimeout because the outcomes differ in cost: an HTTP probe against a cold IdP is worth waiting ten seconds for, while a TCP dial that has not completed in three seconds is reported as inconclusive anyway — so waiting longer buys a slower command and the same finding.

View Source
const DefaultNetworkPolicyProbeTimeout = 3 * time.Minute

DefaultNetworkPolicyProbeTimeout bounds one probe. Three pods have to be scheduled, pull an image and run, and the policed one spends six seconds waiting for a connection that never completes — on a cold node with a cold image cache that adds up, and a probe that times out reports nothing rather than reporting the wrong thing.

View Source
const DefaultProbeTimeout = 10 * time.Second

DefaultProbeTimeout bounds one network probe. Ten seconds is long enough for a slow IdP behind a cold proxy and short enough that a hub with an unreachable issuer, an unreachable registry and three unreachable executors still finishes in under a minute.

View Source
const DefaultSmokeTimeout = 90 * time.Second

DefaultSmokeTimeout bounds one executor's smoke run.

Ninety seconds is generous against the observed cost of starting a container from a warm image and stingy against a cold image pull, which is the right trade: a smoke test that waits five minutes for a registry is being used as a deployment tool, and the pull belongs in `cloop executor test` or in the image pre-staging reconciliation already does at startup.

View Source
const EdgeLagWarnSequences = 20

EdgeLagWarnSequences is how many commits a device's edge build may trail this hub's before hub doctor warns.

Variables

This section is empty.

Functions

func HubCAConfigNames added in v0.0.7

func HubCAConfigNames(cfg *config.Config) []string

HubCAConfigNames are the names the configuration says sandboxes reach the monitors under tls: auto at: each one's advertise URL's host and the host its listen address names. A monitor advertised at no URL is reached at the address it binds, which only the running hub knows; the leaf it was issued names it, and a stage reads that too.

func LongestMonitorSession added in v0.0.7

func LongestMonitorSession(cfg *config.Config) time.Duration

LongestMonitorSession is the longest session a credential monitor that issues from the hub CA grants: the git proxy's or the Kubernetes monitor's session_minutes, for whichever runs under tls: auto. Zero when neither does.

Types

type Check added in v0.0.7

type Check struct {
	ID, Title string
	Run       func(ctx context.Context, add func(Finding))
	// Timeout bounds this check. Zero uses the run's CheckTimeout.
	Timeout time.Duration
	// Every, when set, is how often a live run makes this check at most, as
	// liveEvery says for the hub's own.
	Every time.Duration
}

Check is a step a caller adds to a live run: a test's check that hangs or panics, or an embedder's own. Its ID must not be one of the hub's steps'.

type Finding

type Finding struct {
	Check       string         `json:"check"`
	Title       string         `json:"title"`
	Severity    Severity       `json:"severity"`
	Message     string         `json:"message"`
	Remediation string         `json:"remediation,omitempty"`
	Details     map[string]any `json:"details,omitempty"`
	// AsOf is when the check was made, for a finding a live run carried over
	// from an earlier one (liveEvery); zero means now. Run never sets it.
	AsOf time.Time `json:"as_of,omitzero"`
}

Finding is one diagnostic result.

Check is the stable machine-readable identifier ("oidc.discovery"), Title is what a human reads, Message says what was observed, and Remediation says what to do about it. Remediation is required for anything that is not a pass — a diagnostic that reports a problem without naming its fix is the failure mode this command exists to remove, and TestEveryNonPassCarriesRemediation enforces it.

type Live added in v0.0.7

type Live struct {
	// contains filtered or unexported fields
}

Live runs live diagnoses. The zero value is ready. A hub keeps one for its lifetime, which is what lets it notice a step that never returned.

func (*Live) Run added in v0.0.7

func (l *Live) Run(ctx context.Context, dir string, cfg *config.Config, opts LiveOptions) *LiveReport

Run diagnoses the control plane rooted at dir with cfg, the configuration the hub runs under. Like Run it never fails: what could not be checked is a finding.

type LiveOptions added in v0.0.7

type LiveOptions struct {
	// Options are the checks' own. Smoke and ProbeNetworkPolicy are ignored:
	// a live run has no step that starts work.
	Options

	// CheckTimeout bounds each step. Zero uses DefaultCheckTimeout.
	CheckTimeout time.Duration

	// Extra steps run after the hub's own.
	Extra []Check

	// OnPanic, when set, is told of every step that panics — also one that
	// panics after its deadline, which no report will mention — with where.
	// It runs on the panicking step's goroutine.
	OnPanic func(step string, value any, stack []byte)

	// Generation names the configuration the run is made under. A step made
	// less often than reports are (liveEvery) is reused only by a run of the
	// same generation, so a changed configuration is diagnosed afresh.
	Generation string

	// Fresh asks for a run of everything recent enough not to be: a step
	// made less often than reports are is reused only if its last run is
	// under a minute old.
	Fresh bool
}

LiveOptions tunes a live run.

type LiveReport added in v0.0.7

type LiveReport struct {
	Report
	Steps []StepResult `json:"steps"`
	// NotRun names what this diagnosis leaves to `cloop hub doctor`.
	NotRun []NotRun `json:"not_run"`
	// Elapsed is how long the run took.
	Elapsed time.Duration `json:"-"`
}

LiveReport is a live run's result: the findings, as Run reports them, and what each step did.

type NotRun added in v0.0.7

type NotRun struct {
	Check string `json:"check"`
	Flag  string `json:"flag"`
	Why   string `json:"why"`
}

NotRun is a part of `cloop hub doctor` a live run never makes, and why.

type Options

type Options struct {
	// Offline skips every check that would make a network request: OIDC
	// discovery, the JWKS fetch, registry reachability, and executor health
	// probes that dial. Those checks report a warning saying they were
	// skipped rather than silently vanishing.
	Offline bool

	// Timeout bounds each individual network probe. Zero uses
	// DefaultProbeTimeout.
	Timeout time.Duration

	// HTTPClient overrides the client used for probes. Tests inject one
	// pointed at an httptest server; production leaves it nil so the system
	// trust store (and SSL_CERT_DIR, which the compose stack sets) applies.
	HTTPClient *http.Client

	// Now overrides the clock, so certificate-expiry findings are testable
	// without generating a certificate per case.
	Now func() time.Time

	// LookPath overrides executable lookup, for the cosign check. Tests
	// substitute it; nil means exec.LookPath.
	LookPath func(string) (string, error)

	// DialContext overrides the dialer used by the advertised-address
	// reachability probes. Tests substitute one so a case can produce a
	// refusal or a resolution failure without depending on what the machine
	// running the suite can reach; nil means a plain net.Dialer.
	DialContext func(ctx context.Context, network, addr string) (net.Conn, error)

	// ProbeNetworkPolicy runs the NetworkPolicy enforcement experiment against
	// the configured Kubernetes executors.
	//
	// Opt-in because it is the only check here that *writes to the operator's
	// cluster*: two throwaway Pods and a default-deny policy, removed on every
	// exit path. A diagnostic that mutated infrastructure by default would be
	// one nobody could safely run against production. See netpol.go.
	ProbeNetworkPolicy bool

	// ProbeExecutorID narrows a probe to one executor. Empty probes all of
	// them, which on the usual single-cluster hub is the same thing.
	//
	// Shared by --probe-network-policy and --smoke because it means the same
	// thing to both ("restrict this to one executor") and because an operator
	// should not have to remember two spellings of --executor.
	ProbeExecutorID string

	// Smoke runs the dispatch smoke test: a trivial hermetic workload sent
	// through the same primitives a real task takes. See smoke.go.
	//
	// Opt-in for the same reason ProbeNetworkPolicy is — it starts a real
	// workload and mints a real, if throwaway, credential — though it is far
	// cheaper and safe against production.
	Smoke bool

	// SmokeTimeout bounds one executor's smoke run. Zero uses
	// DefaultSmokeTimeout. Not Timeout, which bounds a single HTTP request
	// and is nowhere near long enough to start a container.
	SmokeTimeout time.Duration

	// SmokeLeaseDir is the directory the smoke stages its throwaway
	// credential file under, for a driver that reads it from the hub's
	// filesystem. Empty takes the tmpfs a real lease uses (/dev/shm, else
	// $TMPDIR). Tests point it somewhere private: the default is shared by
	// every hub and every test binary on the machine, so what a test found
	// there afterwards could be anybody's.
	SmokeLeaseDir string

	// ProbeImage overrides the busybox image the probe Pods run, for a cluster
	// that mirrors its own registry or enforces an image policy.
	ProbeImage string

	// ProbeTimeout bounds one probe. Zero uses
	// DefaultNetworkPolicyProbeTimeout — deliberately not Timeout, which bounds
	// a single HTTP request and is far too short for scheduling three Pods.
	ProbeTimeout time.Duration

	// ProbeLogf narrates the probe's steps. `cloop hub doctor` wires it to the
	// terminal, because the alternative is a command that prints nothing for
	// three minutes and looks hung.
	ProbeLogf func(format string, args ...any)

	// DiskProbe overrides how the free-space check measures the hub's state
	// volume (Task 20381). Tests substitute one so a volume can sit below its
	// floor without anything filling a disk; nil means diskusage.Volumes.
	DiskProbe func(paths ...string) ([]diskusage.Volume, error)

	// Port is the port of the hub being diagnosed (`cloop hub doctor
	// --port`). The exposure check looks for that hub's listener on it and
	// asks it whether it requires sign-in; zero judges the configuration
	// alone (Task 20393).
	Port int

	// Listeners overrides how the exposure check reads the sockets listening
	// on a port. Tests substitute one so a hub can appear to listen on every
	// interface without one binding there; nil reads the kernel's socket
	// tables (listeners.go).
	Listeners func(port int) ([]netip.AddrPort, error)

	// GlobalBudget overrides how the host-wide spend caps are read, which
	// budget.Enforce reads from the running user's config directory. Tests
	// substitute one so a machine's own caps cannot decide them; nil means
	// globalbudget.Load.
	GlobalBudget func() (globalbudget.GlobalBudgetConfig, error)

	// ClaudeHomesRoot overrides where the doctor's own process keeps
	// per-user Claude homes, for the leftover-homes check (Task 20422).
	// Empty is claudecodeauth.HomeRootPath, from this process's environment.
	ClaudeHomesRoot string

	// Self is what the hub being diagnosed knows about itself, when it is the
	// process diagnosing (Live, Task 20434). Nil is an operator's shell, which
	// knows the configuration and its own environment and has to ask the rest.
	Self *Self
}

Options tunes a run.

type Report

type Report struct {
	// Dir is the control plane's working directory (where .cloop lives).
	Dir string `json:"dir"`
	// CheckedAt is when the run started.
	CheckedAt time.Time `json:"checked_at"`
	// StrictMode reports executors.allow_host_process=false — recorded
	// because it changes what several findings mean.
	StrictMode bool `json:"strict_mode"`
	// Offline reports that network probes were skipped.
	Offline bool `json:"offline"`
	// Findings, in the order the checks produced them.
	Findings []Finding `json:"findings"`

	// SmokeRan reports that --smoke was asked for and the check ran, whether
	// or not it found anything to dispatch to. It is what lets SmokeExitCode
	// tell "no executor to smoke" from "smoke was never requested", which are
	// the same empty Smoke array but opposite answers for a readiness gate.
	SmokeRan bool `json:"smoke_ran,omitempty"`
	// Smoke carries the per-stage dispatch results, one entry per executor.
	// Empty unless Options.Smoke was set.
	Smoke []SmokeResult `json:"smoke,omitempty"`
}

Report is the whole diagnosis.

func Run

func Run(ctx context.Context, dir string, cfg *config.Config, opts Options) *Report

Run executes every hub check against the control plane rooted at dir and returns the report.

It never returns an error: a hub is diagnosed by *reporting* what is wrong with it, and a doctor that aborts on the first unreadable file is a doctor that cannot diagnose the case it is most needed for. cfg may be nil, which is itself a finding.

func (*Report) Counts

func (r *Report) Counts() (pass, warn, fail int)

Counts returns the number of findings at each severity.

func (*Report) ExitCode

func (r *Report) ExitCode() int

ExitCode is 1 when any check failed, 0 otherwise.

Warnings deliberately do not fail the command. A warning is frequently a deployment choice rather than a defect — a hub behind a proxy has no certificate of its own — and a gate that goes red on those gets switched off, taking the failures with it.

func (*Report) JSON

func (r *Report) JSON() ([]byte, error)

JSON renders the report for --json, with a trailing newline.

func (*Report) SmokeExitCode added in v0.0.2

func (r *Report) SmokeExitCode() int

SmokeExitCode is the exit status for a run that included --smoke.

It is deliberately more granular than ExitCode. A readiness gate only asks whether the value is zero, but a post-deploy CI step wants to tell three different situations apart without parsing JSON: the hub is misconfigured, the hub is fine but an executor cannot run work, and the diagnostic itself left something behind. Those have different owners and different urgencies.

Precedence runs from most to least urgent, because a run can be several of these at once and the exit code has room for one. A leak outranks everything — it is the only outcome that needs a human on a machine rather than a change to a config file.

func (*Report) Worst

func (r *Report) Worst() Severity

Worst returns the highest severity present, or SeverityPass when empty.

type Self added in v0.0.7

type Self struct {
	// SSO, StaticToken and TLS say how the hub's listener admits a request as
	// the process runs now: single sign-on in force, a static admin token
	// configured, TLS served. The exposure check judges from them instead of
	// asking the listener without a credential — which a hub with a static
	// token and no SSO counts as a failed sign-in, five of which lock the
	// asking address out.
	SSO, StaticToken, TLS bool

	// KMS returns the roots the hub resolved at startup, which every keyring
	// it opens uses, and whether it pinned any. The sealing-key checks use
	// them instead of resolving secrets.kms again, which would open a new KMS
	// client — for Vault, a new login and a new token — on every report.
	KMS func() (kms.Roots, bool)

	// Probe answers an executor's liveness from what the hub already knows:
	// what its supervisor last observed, or the hub member a device is
	// connected to. err nil is reachable; source says where the answer came
	// from. It replaces a probe of every executor on every member — leases a
	// Kubernetes credential, asks a device another member holds — per report.
	Probe func(ctx context.Context, ex executor.Executor) (source string, err error)
}

Self is a serving hub's own answers to questions the doctor otherwise asks from outside. Each one replaces a probe that, made by the hub on itself every minute, would cost more than it tells.

type Severity

type Severity string

Severity is how much a finding matters. The three values are ordered, and Report.ExitCode only distinguishes Fail from the rest.

const (
	// SeverityPass: verified working. Not "nothing was configured".
	SeverityPass Severity = "pass"
	// SeverityWarn: works now, but is fragile, weaker than the deployment
	// implies, or could not be verified.
	SeverityWarn Severity = "warn"
	// SeverityFail: broken or unsafe. An operator must act.
	SeverityFail Severity = "fail"
)

func (Severity) Symbol

func (s Severity) Symbol() string

Symbol is the single character the text renderer prefixes a finding with.

type SmokeResult added in v0.0.2

type SmokeResult struct {
	ExecutorID string `json:"executor_id"`
	Kind       string `json:"kind,omitempty"`
	Isolation  string `json:"isolation,omitempty"`
	// Stages in smokeStageOrder.
	Stages []StageResult `json:"stages"`
	// FirstFailure is the stage that broke, or "" when none did. It is the
	// single most useful field in the report: the stages after a failure are
	// consequences, and an operator reading a wall of red needs to know which
	// line to act on.
	FirstFailure SmokeStage `json:"first_failure,omitempty"`
	// Leaked names resources cleanup could not remove. Non-empty is an
	// operator action item, not a warning.
	Leaked     []string `json:"leaked,omitempty"`
	DurationMS int64    `json:"duration_ms"`
}

SmokeResult is one executor's whole run.

func (SmokeResult) OK added in v0.0.2

func (r SmokeResult) OK() bool

OK reports whether every stage that ran either passed or was skipped.

type SmokeStage added in v0.0.2

type SmokeStage string

SmokeStage names one leg of the circuit. The values are stable: they appear in --json and are what a pipeline greps for.

const (
	// StagePlacement: would the scheduler actually choose this executor for
	// this workload? Run first because a failure here means nothing was
	// dispatched, which is a different problem from a dispatch that broke.
	StagePlacement SmokeStage = "placement"
	// StageWorkspace: did the workload get a usable working directory.
	StageWorkspace SmokeStage = "workspace"
	// StageLease: was a throwaway credential file minted, granted and
	// delivered to the sandbox.
	StageLease SmokeStage = "lease"
	// StageDispatch: did the workload start and exit zero.
	StageDispatch SmokeStage = "dispatch"
	// StageLogs: did its output come back over the stream.
	StageLogs SmokeStage = "logs"
	// StageWriteBack: did file changes return, for a backend that advertises
	// the capability.
	StageWriteBack SmokeStage = "write_back"
	// StageRevocation: is the credential material gone, verified rather than
	// assumed.
	StageRevocation SmokeStage = "revocation"
	// StageCleanup: did everything this run created get removed.
	StageCleanup SmokeStage = "cleanup"
)

type StageOutcome added in v0.0.2

type StageOutcome string

StageOutcome is what happened to one stage.

const (
	// StagePass: verified.
	StagePass StageOutcome = "pass"
	// StageSkip: not applicable to this backend, or not provable hermetically.
	// Deliberately not a pass — see the package comment.
	StageSkip StageOutcome = "skip"
	// StageFail: this leg of the circuit is broken.
	StageFail StageOutcome = "fail"
)

type StageResult added in v0.0.2

type StageResult struct {
	Stage   SmokeStage   `json:"stage"`
	Outcome StageOutcome `json:"outcome"`
	Message string       `json:"message"`
	// Remediation names the fix. Required for anything that is not a pass,
	// for the same reason Finding.Remediation is.
	Remediation string         `json:"remediation,omitempty"`
	DurationMS  int64          `json:"duration_ms,omitempty"`
	Details     map[string]any `json:"details,omitempty"`
}

StageResult is one stage's verdict.

type StepOutcome added in v0.0.7

type StepOutcome string

StepOutcome is how one step of a live run ended.

const (
	// StepFinished: it returned in time; its findings are in the report.
	StepFinished StepOutcome = "finished"
	// StepTimedOut: it was still running at its deadline. What it added
	// before then is in the report; the rest of its findings are not.
	StepTimedOut StepOutcome = "timed_out"
	// StepPanicked: it panicked. What it added first is in the report.
	StepPanicked StepOutcome = "panicked"
	// StepStillRunning: an earlier report's run of it has not returned, so
	// it was not started again.
	StepStillRunning StepOutcome = "still_running"
	// StepReused: the step is made less often than reports are (liveEvery),
	// and this report carries the findings of its last run, at RanAt.
	StepReused StepOutcome = "reused"
)

type StepResult added in v0.0.7

type StepResult struct {
	ID         string      `json:"id"`
	Title      string      `json:"title"`
	Outcome    StepOutcome `json:"outcome"`
	DurationMS int64       `json:"duration_ms"`
	// Panic is the recovered value of a step that panicked, rendered.
	Panic string `json:"panic,omitempty"`
	// RanAt is when a reused step's findings were made.
	RanAt time.Time `json:"ran_at,omitzero"`
}

StepResult is what one step of a live run did.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL