Documentation
¶
Overview ¶
Package hubdoctor diagnoses a cloop *control plane* — the thing an operator deploys — as distinct from pkg/doctor, which diagnoses a developer's checkout.
Why it is a separate package ¶
pkg/doctor answers "can this working copy run cloop": is there a .cloop directory, is `go` on PATH, is the API key set, is state.json valid JSON. Every one of its checks predates the hub, and none of them can fail on a deployment that is thoroughly broken as a hub: an issuer that does not resolve, a certificate that expired last week, an RBAC policy under which nobody is an admin, strict mode with nothing to dispatch to. That hub passes `cloop doctor` with fourteen green lines and cannot serve a single request.
So this is a second diagnosis with a different subject, not more checks bolted onto the first. Its checks are the ones whose answers only exist once cloop is hosted — identity, transport, authorization, image trust, executors, storage and admission — and each one is written to be actionable rather than merely true: every non-pass finding carries a one-line remediation naming the file, field or command that fixes it.
The contract every check keeps ¶
- A check never fails the whole run. A hub whose issuer is unreachable must still be told about its expiring certificate, so each check contributes findings and the report is the sum of them.
- Severity means something specific. Fail: the hub is broken or unsafe in a way an operator must fix. Warn: it works, but a foreseeable event breaks it or a control is weaker than the deployment implies. Pass: verified, not merely absent.
- Absence is not a pass. A check that could not run — no network, no database — reports that as a warning, because "we did not look" and "we looked and it was fine" are different answers and only one of them is worth acting on.
Exit codes and --json ¶
Run returns a Report; the caller decides how loudly to complain. ExitCode is 0 with only passes and warnings, 1 with any failure, so `cloop hub doctor` is usable as a CI gate and as a pre-flight in a deployment pipeline. --json emits the same Report verbatim, which is why every field has a stable JSON tag and every finding has a stable Check id: those ids are the thing a pipeline greps for, and renaming one is a breaking change.
Index ¶
Constants ¶
const ( // SmokeExitOK: every target smoked clean. SmokeExitOK = 0 // SmokeExitConfig: a non-smoke check failed, so the hub is misconfigured // independently of whether it can dispatch. Matches Report.ExitCode. SmokeExitConfig = 1 // SmokeExitNoTargets: there was nothing to smoke. Not the same as healthy: // a hub with no executor cannot run anything. SmokeExitNoTargets = 3 // SmokeExitFailed: at least one executor failed a stage. SmokeExitFailed = 4 // SmokeExitLeaked: the run could not clean up after itself. Highest // precedence — it is the only outcome that needs a human on a machine. SmokeExitLeaked = 5 )
Smoke exit codes. Distinct so the command is usable both as a Kubernetes readiness gate (which only cares about zero) and as a post-deploy CI step (which wants to tell a broken executor from a misconfigured hub from a diagnostic that left rubbish behind).
2 is deliberately unused: Cobra and `cloop executor test` already spend it.
const DefaultDialTimeout = 3 * time.Second
DefaultDialTimeout bounds one reachability dial.
Shorter than DefaultProbeTimeout because the outcomes differ in cost: an HTTP probe against a cold IdP is worth waiting ten seconds for, while a TCP dial that has not completed in three seconds is reported as inconclusive anyway — so waiting longer buys a slower command and the same finding.
const DefaultNetworkPolicyProbeTimeout = 3 * time.Minute
DefaultNetworkPolicyProbeTimeout bounds one probe. Three pods have to be scheduled, pull an image and run, and the policed one spends six seconds waiting for a connection that never completes — on a cold node with a cold image cache that adds up, and a probe that times out reports nothing rather than reporting the wrong thing.
const DefaultProbeTimeout = 10 * time.Second
DefaultProbeTimeout bounds one network probe. Ten seconds is long enough for a slow IdP behind a cold proxy and short enough that a hub with an unreachable issuer, an unreachable registry and three unreachable executors still finishes in under a minute.
const DefaultSmokeTimeout = 90 * time.Second
DefaultSmokeTimeout bounds one executor's smoke run.
Ninety seconds is generous against the observed cost of starting a container from a warm image and stingy against a cold image pull, which is the right trade: a smoke test that waits five minutes for a registry is being used as a deployment tool, and the pull belongs in `cloop executor test` or in the image pre-staging reconciliation already does at startup.
const EdgeLagWarnSequences = 20
EdgeLagWarnSequences is how many commits a device's edge build may trail this hub's before hub doctor warns.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Finding ¶
type Finding struct {
Check string `json:"check"`
Title string `json:"title"`
Severity Severity `json:"severity"`
Message string `json:"message"`
Remediation string `json:"remediation,omitempty"`
Details map[string]any `json:"details,omitempty"`
}
Finding is one diagnostic result.
Check is the stable machine-readable identifier ("oidc.discovery"), Title is what a human reads, Message says what was observed, and Remediation says what to do about it. Remediation is required for anything that is not a pass — a diagnostic that reports a problem without naming its fix is the failure mode this command exists to remove, and TestEveryNonPassCarriesRemediation enforces it.
type Options ¶
type Options struct {
// Offline skips every check that would make a network request: OIDC
// discovery, the JWKS fetch, registry reachability, and executor health
// probes that dial. Those checks report a warning saying they were
// skipped rather than silently vanishing.
Offline bool
// Timeout bounds each individual network probe. Zero uses
// DefaultProbeTimeout.
Timeout time.Duration
// HTTPClient overrides the client used for probes. Tests inject one
// pointed at an httptest server; production leaves it nil so the system
// trust store (and SSL_CERT_DIR, which the compose stack sets) applies.
HTTPClient *http.Client
// Now overrides the clock, so certificate-expiry findings are testable
// without generating a certificate per case.
Now func() time.Time
// LookPath overrides executable lookup, for the cosign check. Tests
// substitute it; nil means exec.LookPath.
LookPath func(string) (string, error)
// DialContext overrides the dialer used by the advertised-address
// reachability probes. Tests substitute one so a case can produce a
// refusal or a resolution failure without depending on what the machine
// running the suite can reach; nil means a plain net.Dialer.
DialContext func(ctx context.Context, network, addr string) (net.Conn, error)
// ProbeNetworkPolicy runs the NetworkPolicy enforcement experiment against
// the configured Kubernetes executors.
//
// Opt-in because it is the only check here that *writes to the operator's
// cluster*: two throwaway Pods and a default-deny policy, removed on every
// exit path. A diagnostic that mutated infrastructure by default would be
// one nobody could safely run against production. See netpol.go.
ProbeNetworkPolicy bool
// ProbeExecutorID narrows a probe to one executor. Empty probes all of
// them, which on the usual single-cluster hub is the same thing.
//
// Shared by --probe-network-policy and --smoke because it means the same
// thing to both ("restrict this to one executor") and because an operator
// should not have to remember two spellings of --executor.
ProbeExecutorID string
// Smoke runs the dispatch smoke test: a trivial hermetic workload sent
// through the same primitives a real task takes. See smoke.go.
//
// Opt-in for the same reason ProbeNetworkPolicy is — it starts a real
// workload and mints a real, if throwaway, credential — though it is far
// cheaper and safe against production.
Smoke bool
// SmokeTimeout bounds one executor's smoke run. Zero uses
// DefaultSmokeTimeout. Not Timeout, which bounds a single HTTP request
// and is nowhere near long enough to start a container.
SmokeTimeout time.Duration
// SmokeLeaseDir is the directory the smoke stages its throwaway
// credential file under, for a driver that reads it from the hub's
// filesystem. Empty takes the tmpfs a real lease uses (/dev/shm, else
// $TMPDIR). Tests point it somewhere private: the default is shared by
// every hub and every test binary on the machine, so what a test found
// there afterwards could be anybody's.
SmokeLeaseDir string
// ProbeImage overrides the busybox image the probe Pods run, for a cluster
// that mirrors its own registry or enforces an image policy.
ProbeImage string
// ProbeTimeout bounds one probe. Zero uses
// DefaultNetworkPolicyProbeTimeout — deliberately not Timeout, which bounds
// a single HTTP request and is far too short for scheduling three Pods.
ProbeTimeout time.Duration
// ProbeLogf narrates the probe's steps. `cloop hub doctor` wires it to the
// terminal, because the alternative is a command that prints nothing for
// three minutes and looks hung.
ProbeLogf func(format string, args ...any)
// DiskProbe overrides how the free-space check measures the hub's state
// volume (Task 20381). Tests substitute one so a volume can sit below its
// floor without anything filling a disk; nil means diskusage.Volumes.
DiskProbe func(paths ...string) ([]diskusage.Volume, error)
// Port is the port of the hub being diagnosed (`cloop hub doctor
// --port`). The exposure check looks for that hub's listener on it and
// asks it whether it requires sign-in; zero judges the configuration
// alone (Task 20393).
Port int
// Listeners overrides how the exposure check reads the sockets listening
// on a port. Tests substitute one so a hub can appear to listen on every
// interface without one binding there; nil reads the kernel's socket
// tables (listeners.go).
Listeners func(port int) ([]netip.AddrPort, error)
// GlobalBudget overrides how the host-wide spend caps are read, which
// budget.Enforce reads from the running user's config directory. Tests
// substitute one so a machine's own caps cannot decide them; nil means
// globalbudget.Load.
GlobalBudget func() (globalbudget.GlobalBudgetConfig, error)
}
Options tunes a run.
type Report ¶
type Report struct {
// Dir is the control plane's working directory (where .cloop lives).
Dir string `json:"dir"`
// CheckedAt is when the run started.
CheckedAt time.Time `json:"checked_at"`
// StrictMode reports executors.allow_host_process=false — recorded
// because it changes what several findings mean.
StrictMode bool `json:"strict_mode"`
// Offline reports that network probes were skipped.
Offline bool `json:"offline"`
// Findings, in the order the checks produced them.
Findings []Finding `json:"findings"`
// SmokeRan reports that --smoke was asked for and the check ran, whether
// or not it found anything to dispatch to. It is what lets SmokeExitCode
// tell "no executor to smoke" from "smoke was never requested", which are
// the same empty Smoke array but opposite answers for a readiness gate.
SmokeRan bool `json:"smoke_ran,omitempty"`
// Smoke carries the per-stage dispatch results, one entry per executor.
// Empty unless Options.Smoke was set.
Smoke []SmokeResult `json:"smoke,omitempty"`
}
Report is the whole diagnosis.
func Run ¶
Run executes every hub check against the control plane rooted at dir and returns the report.
It never returns an error: a hub is diagnosed by *reporting* what is wrong with it, and a doctor that aborts on the first unreadable file is a doctor that cannot diagnose the case it is most needed for. cfg may be nil, which is itself a finding.
func (*Report) ExitCode ¶
ExitCode is 1 when any check failed, 0 otherwise.
Warnings deliberately do not fail the command. A warning is frequently a deployment choice rather than a defect — a hub behind a proxy has no certificate of its own — and a gate that goes red on those gets switched off, taking the failures with it.
func (*Report) SmokeExitCode ¶ added in v0.0.2
SmokeExitCode is the exit status for a run that included --smoke.
It is deliberately more granular than ExitCode. A readiness gate only asks whether the value is zero, but a post-deploy CI step wants to tell three different situations apart without parsing JSON: the hub is misconfigured, the hub is fine but an executor cannot run work, and the diagnostic itself left something behind. Those have different owners and different urgencies.
Precedence runs from most to least urgent, because a run can be several of these at once and the exit code has room for one. A leak outranks everything — it is the only outcome that needs a human on a machine rather than a change to a config file.
type Severity ¶
type Severity string
Severity is how much a finding matters. The three values are ordered, and Report.ExitCode only distinguishes Fail from the rest.
const ( // SeverityPass: verified working. Not "nothing was configured". SeverityPass Severity = "pass" // SeverityWarn: works now, but is fragile, weaker than the deployment // implies, or could not be verified. SeverityWarn Severity = "warn" // SeverityFail: broken or unsafe. An operator must act. SeverityFail Severity = "fail" )
type SmokeResult ¶ added in v0.0.2
type SmokeResult struct {
ExecutorID string `json:"executor_id"`
Kind string `json:"kind,omitempty"`
Isolation string `json:"isolation,omitempty"`
// Stages in smokeStageOrder.
Stages []StageResult `json:"stages"`
// FirstFailure is the stage that broke, or "" when none did. It is the
// single most useful field in the report: the stages after a failure are
// consequences, and an operator reading a wall of red needs to know which
// line to act on.
FirstFailure SmokeStage `json:"first_failure,omitempty"`
// Leaked names resources cleanup could not remove. Non-empty is an
// operator action item, not a warning.
Leaked []string `json:"leaked,omitempty"`
DurationMS int64 `json:"duration_ms"`
}
SmokeResult is one executor's whole run.
func (SmokeResult) OK ¶ added in v0.0.2
func (r SmokeResult) OK() bool
OK reports whether every stage that ran either passed or was skipped.
type SmokeStage ¶ added in v0.0.2
type SmokeStage string
SmokeStage names one leg of the circuit. The values are stable: they appear in --json and are what a pipeline greps for.
const ( // StagePlacement: would the scheduler actually choose this executor for // this workload? Run first because a failure here means nothing was // dispatched, which is a different problem from a dispatch that broke. StagePlacement SmokeStage = "placement" // StageWorkspace: did the workload get a usable working directory. StageWorkspace SmokeStage = "workspace" // StageLease: was a throwaway credential file minted, granted and // delivered to the sandbox. StageLease SmokeStage = "lease" // StageDispatch: did the workload start and exit zero. StageDispatch SmokeStage = "dispatch" // StageLogs: did its output come back over the stream. StageLogs SmokeStage = "logs" // StageWriteBack: did file changes return, for a backend that advertises // the capability. StageWriteBack SmokeStage = "write_back" // StageRevocation: is the credential material gone, verified rather than // assumed. StageRevocation SmokeStage = "revocation" // StageCleanup: did everything this run created get removed. StageCleanup SmokeStage = "cleanup" )
type StageOutcome ¶ added in v0.0.2
type StageOutcome string
StageOutcome is what happened to one stage.
const ( // StagePass: verified. StagePass StageOutcome = "pass" // StageSkip: not applicable to this backend, or not provable hermetically. // Deliberately not a pass — see the package comment. StageSkip StageOutcome = "skip" // StageFail: this leg of the circuit is broken. StageFail StageOutcome = "fail" )
type StageResult ¶ added in v0.0.2
type StageResult struct {
Stage SmokeStage `json:"stage"`
Outcome StageOutcome `json:"outcome"`
Message string `json:"message"`
// Remediation names the fix. Required for anything that is not a pass,
// for the same reason Finding.Remediation is.
Remediation string `json:"remediation,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
Details map[string]any `json:"details,omitempty"`
}
StageResult is one stage's verdict.
Source Files
¶
- admission.go
- audittrail.go
- configdrift.go
- diskfree.go
- edgelag.go
- egress.go
- executors.go
- exposure.go
- gitproxy.go
- hubdoctor.go
- images.go
- kubeguard.go
- listeners.go
- loadrepairs.go
- metadata.go
- monitortls.go
- netpol.go
- oidc.go
- projects.go
- proxyendpoint.go
- rbac.go
- reachability.go
- retention.go
- runbuild.go
- scim.go
- secrets.go
- smoke.go
- statictoken.go
- storage.go
- tls.go
- wal.go