Documentation
¶
Overview ¶
Package monitor provides event-driven monitoring of Docker containers and systemd services inside them. It complements the poll-based probes in pkg/probe by pushing state transitions through channels.
Index ¶
- func DockerEventsArgs(clusterName string) []string
- func SystemdMonitorArgs(container docker.ContainerName) []string
- type DockerMonitor
- type Event
- type EventKind
- type NodeTarget
- type SystemdMonitor
- type Watcher
- func (w *Watcher) AddNodes(nodes []NodeTarget)
- func (w *Watcher) Start(ctx context.Context, nodes []NodeTarget) error
- func (w *Watcher) Subscribe() <-chan Event
- func (w *Watcher) SubscribeTo(container docker.ContainerName) <-chan Event
- func (w *Watcher) Unsubscribe(ch <-chan Event)
- func (w *Watcher) Wait()
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DockerEventsArgs ¶
DockerEventsArgs returns the docker CLI arguments for streaming container events for the given cluster. Only the actions in dockerActionMap are asked for: every docker exec of the readiness probes would otherwise send exec_create, exec_start and exec_die events to parse and drop.
func SystemdMonitorArgs ¶
func SystemdMonitorArgs(container docker.ContainerName) []string
SystemdMonitorArgs returns the docker CLI arguments for streaming systemd unit state changes from inside a container.
Types ¶
type DockerMonitor ¶
type DockerMonitor struct {
// contains filtered or unexported fields
}
DockerMonitor reads a docker events JSON stream and emits Events.
func NewDockerMonitor ¶
func NewDockerMonitor(containerPrefix string) *DockerMonitor
NewDockerMonitor creates a monitor for the given cluster. The container prefix is used to extract short node names from full container names.
func (*DockerMonitor) Run ¶
Run reads from r (the stdout of docker events --format '{{json .}}') and sends parsed events to ch. It blocks on scanner.Scan(), which is not interruptible by context cancellation alone — to unblock Run on ctx.Done(), the caller must also close r (e.g. by killing the underlying process or closing the stdout pipe). The caller is responsible for closing ch after Run returns.
type Event ¶
type Event struct {
Kind EventKind
Node string // short name: "controller", "worker-0"
Container docker.ContainerName // full container name
Unit string // systemd unit name (empty for container events)
Time time.Time
Detail string // human-readable detail (exit code, error message, etc.)
Err error // non-nil for EventMonitorError
}
Event represents a single state change observed by a monitor.
type EventKind ¶
type EventKind string
EventKind identifies the category of a monitored event.
const EventContainerDie EventKind = "container.die"
EventContainerDie indicates a container has exited.
const EventContainerOOM EventKind = "container.oom"
EventContainerOOM indicates a container was killed by the OOM killer.
const EventContainerPause EventKind = "container.pause"
EventContainerPause indicates a container was paused.
const EventContainerStart EventKind = "container.start"
EventContainerStart indicates a container has started.
const EventContainerUnpause EventKind = "container.unpause"
EventContainerUnpause indicates a container was unpaused.
const EventMonitorError EventKind = "monitor.error"
EventMonitorError indicates that a monitor stopped while the watcher runs: it failed to start, its stream failed, or its stream ended. Its events are missing from then on: those of the container it names, or those of every container when it names none (the docker events stream).
const EventUnitActive EventKind = "unit.active"
EventUnitActive indicates a systemd unit reached ActiveState=active.
const EventUnitFailed EventKind = "unit.failed"
EventUnitFailed indicates a systemd unit reached ActiveState=failed.
type NodeTarget ¶
type NodeTarget struct {
ShortName string
Container docker.ContainerName
}
NodeTarget identifies a node to monitor.
type SystemdMonitor ¶
type SystemdMonitor struct {
// contains filtered or unexported fields
}
SystemdMonitor reads busctl monitor output from inside a container and emits Events for systemd unit state changes.
func NewSystemdMonitor ¶
func NewSystemdMonitor(node string, container docker.ContainerName) *SystemdMonitor
NewSystemdMonitor creates a monitor for a single node container.
func (*SystemdMonitor) Run ¶
Run reads from r (the stdout of docker exec CONTAINER busctl monitor ...) and sends parsed events to ch. It blocks on scanner.Scan(), which is not interruptible by context cancellation alone — to unblock Run on ctx.Done(), the caller must also close r (e.g. by killing the underlying process or closing the stdout pipe). The caller is responsible for closing ch after Run returns.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher monitors a sind cluster for state changes across all nodes. It manages the lifecycle of docker events and per-node systemd monitors, and broadcasts events to subscribers.
func NewWatcher ¶
NewWatcher creates a Watcher for the given cluster. The executor is used to start long-lived streaming processes (docker events, busctl monitor).
func (*Watcher) AddNodes ¶
func (w *Watcher) AddNodes(nodes []NodeTarget)
AddNodes starts systemd monitors for the given nodes. Call this after Start and after the node containers have been created so that systemd state changes can accelerate readiness probing. The monitors run on the context Start was given, until the watcher stops, so that they also serve the waits that follow the caller's own (e.g. for the Slurm daemons). Before a successful Start, AddNodes does nothing.
func (*Watcher) Start ¶
func (w *Watcher) Start(ctx context.Context, nodes []NodeTarget) error
Start begins monitoring. It starts the docker events stream, spawns systemd monitors for each given node, and starts the broadcast loop. Cancel the context to stop all monitors, those AddNodes starts later included.
func (*Watcher) Subscribe ¶
Subscribe returns a channel that receives a copy of every event. The channel is closed when the Watcher stops. The caller should drain the channel to avoid blocking the broadcast loop.
func (*Watcher) SubscribeTo ¶ added in v0.11.0
func (w *Watcher) SubscribeTo(container docker.ContainerName) <-chan Event
SubscribeTo is Subscribe for the events of one container, or of every container when it is empty. A readiness wait for one node subscribes to its container only: while many nodes boot, the other nodes' events would otherwise fill its buffer and crowd out its own. A subscription to one container also receives the monitor errors of the whole watcher (those without a container), as its container's events are missing after one.
func (*Watcher) Unsubscribe ¶
Unsubscribe removes a previously subscribed channel and closes it.
func (*Watcher) Wait ¶
func (w *Watcher) Wait()
Wait blocks until all monitor goroutines have exited and closes all remaining subscriber channels. Any Unsubscribe calls must complete before Wait runs; otherwise a concurrent Unsubscribe would double-close the channel Wait is iterating. Callers using the usual defer watcher.Wait() / defer watcher.Unsubscribe(ch) pattern get this ordering for free (deferred calls run LIFO).
Source Files