Documentation
¶
Overview ¶
Package probe implements readiness probes for cluster services.
Index ¶
- Constants
- func ContainerRunning(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func MungeReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func ParseSlurmctldPing(stdout string) map[string]bool
- func SSHDReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func SackdReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func SlurmctldReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func SlurmdReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func SlurmdbdReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func SlurmrestdReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func Snapshot(ctx context.Context, client *docker.Client, name docker.ContainerName, ...) (map[Service]bool, error)
- func SystemdReady(ctx context.Context, client *docker.Client, name docker.ContainerName) error
- func UnitActive(ctx context.Context, client *docker.Client, name docker.ContainerName, ...) error
- func UnitJournal(ctx context.Context, client *docker.Client, name docker.ContainerName, ...) string
- func UntilReady(ctx context.Context, client *docker.Client, name docker.ContainerName, ...) error
- func UntilReadyWithEvents(ctx context.Context, client *docker.Client, name docker.ContainerName, ...) error
- type Func
- type Probe
- type Service
- type TerminalError
Constants ¶
const DefaultInterval = 500 * time.Millisecond
DefaultInterval is the delay between readiness probe rounds that UntilReady and UntilReadyWithEvents use for an interval of zero or less.
const SlurmrestdPort = "6820"
SlurmrestdPort is the TCP port slurmrestd listens on: SLURMRESTD_LISTEN of the unit Slurm ships, Slurm's default port for it.
Variables ¶
This section is empty.
Functions ¶
func ContainerRunning ¶
ContainerRunning verifies that the container is in the "running" state. Returns a TerminalError for states that cannot recover (exited, dead).
func MungeReady ¶
MungeReady verifies that the munge authentication service is active. A failed unit is a TerminalError (see UnitActive).
func ParseSlurmctldPing ¶ added in v0.10.0
ParseSlurmctldPing parses scontrol ping output into a map from controller hostname to whether it answered. Lines not in the "Slurmctld(<mode>) at <host> is <UP|DOWN>" format are ignored.
func SSHDReady ¶
SSHDReady verifies that sshd is accepting connections and responding with the SSH protocol banner on port 22.
func SackdReady ¶ added in v0.11.0
SackdReady verifies that the sackd service is active. A failed unit is a TerminalError (see UnitActive).
func SlurmctldReady ¶
SlurmctldReady verifies that the slurmctld on the given controller node is responding to RPC requests. While it does not, a failed slurmctld unit (e.g. a slurm.conf line slurmctld rejects) is a TerminalError carrying the tail of the unit's journal.
scontrol ping exits zero as soon as any configured controller answers, so with a backup controller the exit code alone cannot tell which one is up. The per-controller lines ("Slurmctld(primary) at controller is UP") are matched against the container: sind names containers <realm>-<cluster>-<hostname>, so the line whose host is the container name's suffix belongs to this node. When no line matches (e.g. a user-supplied slurm.conf with other hostnames) the exit code decides.
func SlurmdReady ¶
SlurmdReady verifies that the slurmd service is active. A failed unit is a TerminalError (see UnitActive).
func SlurmdbdReady ¶ added in v0.11.0
SlurmdbdReady verifies that the slurmdbd service is active. A failed unit is a TerminalError (see UnitActive).
func SlurmrestdReady ¶ added in v0.11.0
SlurmrestdReady verifies that the slurmrestd service is active and accepts connections on SlurmrestdPort. The unit is of type simple, so it is active as soon as slurmrestd started, before it listens. A failed unit is a TerminalError (see UnitActive).
func Snapshot ¶ added in v0.9.0
func Snapshot(ctx context.Context, client *docker.Client, name docker.ContainerName, services []Service) (map[Service]bool, error)
Snapshot returns a one-shot readiness snapshot of the given services on a running node, fusing the systemd-based checks (munge, sshd, slurmd, mariadb, slurmdbd) into a single docker exec. slurmctld is checked with scontrol ping instead, because "slurmctld is active" is weaker than "slurmctld answers RPCs" — the unit can be active during startup while RPCs still fail.
Snapshot is intended for status-query call sites such as cluster.GetStatus. Unlike the individual *Ready probes, it does not surface per-probe errors; a failing check simply maps to false. Callers that need retry granularity (e.g. cluster-create readiness polling) should keep using the individual probes and UntilReady.
The container must be running. Non-exit errors (daemon unreachable, etc.) are propagated; a non-zero exit from systemctl (at least one unit inactive) is expected and parsed normally.
func SystemdReady ¶
SystemdReady verifies that systemd has finished booting. Both "running" and "degraded" are considered ready, since degraded means systemd completed startup but some units failed (which is expected when Slurm daemons haven't been configured yet).
func UnitActive ¶ added in v0.11.0
func UnitActive(ctx context.Context, client *docker.Client, name docker.ContainerName, unit string) error
UnitActive verifies that a systemd unit on the node is active. A failed unit does not recover on its own (a unit that systemd restarts is "activating" meanwhile), so it is reported as a TerminalError carrying the tail of the unit's journal to show why the unit stopped.
func UnitJournal ¶ added in v0.11.0
func UnitJournal(ctx context.Context, client *docker.Client, name docker.ContainerName, unit string) string
UnitJournal returns the tail of a systemd unit's journal on the node, to show why the unit failed. It is best effort: on error it returns what journalctl printed, possibly nothing.
func UntilReady ¶
func UntilReady(ctx context.Context, client *docker.Client, name docker.ContainerName, probes []Probe, interval time.Duration) error
UntilReady polls the given probes until they all pass or the context expires. The caller controls the deadline via the context. The interval controls the delay between polling attempts; zero or less means DefaultInterval. Each round runs the probes in order and stops at the first that fails; as nothing tells UntilReady when what a passed probe checks changes, every round starts again from the first probe. On timeout, the error includes the name and message of the last failing probe.
func UntilReadyWithEvents ¶ added in v0.9.0
func UntilReadyWithEvents(ctx context.Context, client *docker.Client, name docker.ContainerName, probes []Probe, interval time.Duration, events <-chan monitor.Event) error
UntilReadyWithEvents is like UntilReady but also listens for events from a monitor. An event of the container triggers immediate probe re-evaluation instead of waiting for the next poll interval, reducing detection latency for event-backed state transitions. The events already queued then are taken with it, so that a burst (systemd activates dozens of units while it boots) costs one probe round, not one per event. Container die events are treated as terminal errors.
A probe that has passed is not run again in later rounds while the events can tell that what it checks has changed:
- an event of its unit (Probe.Unit) runs it again: a unit that became active again, or one that failed, which a unit probe reports as a TerminalError;
- another event of the container (start, oom, pause, unpause) runs every passed probe again;
- a full events buffer runs every passed probe again, as the watcher drops the events a full subscriber has no room for;
- from a monitor error on (monitor.EventMonitorError of the container or of the whole watcher), once events is closed, and for an unbuffered events channel, every round runs every probe, as in UntilReady.
Once events is closed, the wait goes on polling.
Types ¶
type Func ¶
Func is a probe function that checks a single readiness condition.
func ClusterRegistered ¶ added in v0.11.0
ClusterRegistered returns a check that passes once slurmdbd lists the cluster, which slurmctld registers when it first starts with accounting. Until then, sacctmgr refuses to add users. The registration can land after slurmctld answers pings.
type Probe ¶
type Probe struct {
Name string
Check Func
// Unit is the systemd unit whose state the check follows, as busctl
// names it ("munge.service"), or empty. A wait with events does not run
// a check that has passed again until an event of its unit arrives (see
// UntilReadyWithEvents).
Unit string
}
Probe is a named readiness check.
func ForService ¶ added in v0.7.0
ForService returns the readiness probe for a service, which follows the service's systemd unit. A service sind does not wait for (mariadb) gets a probe without a check.
func NodeProbes ¶
NodeProbes returns the probes applicable to a node with the given role.
type Service ¶ added in v0.9.0
type Service string
Service identifies a per-node readiness check. The string value is the systemd unit name for munge/sshd/slurmd/slurmdbd/mariadb/sackd/slurmrestd and the slurm RPC endpoint name for slurmctld, so it also doubles as the user-facing label for each check in status output.
const ( ServiceMunge Service = "munge" ServiceSSHD Service = "sshd" ServiceSlurmctld Service = "slurmctld" ServiceSlurmd Service = "slurmd" ServiceSlurmdbd Service = "slurmdbd" ServiceMariadb Service = "mariadb" // ServiceSackd is the auth/slurm token daemon of a login node. ServiceSackd Service = "sackd" // ServiceSlurmrestd is the REST API daemon of the api node. ServiceSlurmrestd Service = "slurmrestd" )
Per-node readiness services managed by sind.
type TerminalError ¶ added in v0.7.1
type TerminalError struct {
Msg string
}
TerminalError indicates a probe failure that cannot be recovered by retrying. For example, a container in "exited" or "dead" state will never become "running" on its own.
func (*TerminalError) Error ¶ added in v0.7.1
func (e *TerminalError) Error() string
Source Files