observability

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2026 License: GPL-3.0, LGPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package observability manages a realm's monitoring stack: Grafana Alloy, which collects the Slurm metrics, the container resource use and the journals of the monitored clusters' nodes, Prometheus and Loki, which store them, and Grafana, which shows them. The stack runs as four containers on the realm's mesh network. The package generates their configuration from the nodes it is given; pkg/cluster decides which nodes those are and when the stack is created, synced and removed.

It is not called monitor: pkg/monitor is the Docker and systemd event watcher of the readiness probes.

Index

Constants

View Source 
const (

	// DashboardsDir is where the sind dashboards are provisioned from.
	DashboardsDir = grafanaProvisioning + "/dashboards/sind"
	// PrometheusUID and LokiUID are the fixed UIDs of the datasources,
	// which the dashboards refer to.
	PrometheusUID = "sind-prometheus"
	LokiUID       = "sind-loki"
)

Grafana's provisioning.

View Source 
const (
	AlloyImage      = "grafana/alloy:v1.20.1"
	PrometheusImage = "prom/prometheus:v3.15.0"
	LokiImage       = "grafana/loki:3.7.8"
	GrafanaImage    = "grafana/grafana:13.2.3"
)

The stack's images, each pinned and bumped on its own. Grafana must be the standard image: its Prometheus and Loki datasources ship only in the standard image's bundled plugins, and the -slim image would fetch them at start. The standard image is also the only one of the four with curl, which sind runs there to reach the others.

View Source 
const (
	GrafanaPort    = "3000"
	PrometheusPort = "9090"
	LokiPort       = "3100"
	AlloyPort      = "12345"
)

Ports the services listen on, on the mesh network.

View Source 
const AlloyConfigPath = "/etc/alloy/sind.alloy"

AlloyConfigPath is the configuration file sind writes into the Alloy container, before its first start and at every sync.

View Source 
const DefaultReadyTimeout = 2 * time.Minute

DefaultReadyTimeout is how long EnsureStack waits for the stack to be ready, when the context has no earlier deadline: Loki reports ready only some seconds after it starts.

View Source 
const JournalDir = "/var/log/sind"

JournalDir is where Alloy mounts the journal volume.

View Source 
const LabelVersion = "sind.observability.version"

LabelVersion is the label of the stack's containers that records the fingerprint of the stack sind created them with (see Fingerprint): its images, configuration and dashboards. A create that finds another fingerprint recreates the containers, keeping their volumes.

View Source 
const NodeJournalDir = "/var/log/journal"

NodeJournalDir is where a monitored node mounts its directory of the journal volume: journald writes a persistent journal there.

Variables

View Source 
var Services = []Service{ServiceGrafana, ServicePrometheus, ServiceLoki, ServiceAlloy}

Services lists the stack's services in the order sind get monitoring shows them.

Functions

func CgroupPath 

func CgroupPath(driver string, id docker.ContainerID) string

CgroupPath returns the cgroup of a container under a cgroup driver: /system.slice/docker-<id>.scope with systemd, /docker/<id> with cgroupfs.

func ContainerName 

func ContainerName(realm string, s Service) docker.ContainerName

ContainerName returns the container of a service in a realm: <realm>-<service>. Two-part names never equal a cluster's <realm>-<cluster>-<type> resources.

func DNSName 

func DNSName(realm string, s Service) string

DNSName returns the mesh DNS name of a service: <service>.<realm>.sind.

func Dashboards 

func Dashboards() map[string][]byte

Dashboards returns the sind dashboards' JSON models by file name.

func GenerateAlloyConfig 

func GenerateAlloyConfig(realm, cgroupDriver string, targets []Target) string

GenerateAlloyConfig returns Alloy's configuration for a realm's monitored nodes, under the daemon's cgroup driver (see CgroupPath):

  • prometheus.scrape of slurmctld's metrics endpoints on every managed controller, by container name on the mesh network, without following redirects: a standby controller answers 303, so up is 1 only on the controller in control;
  • prometheus.exporter.blackbox with the Slurm daemon probes of Collect;
  • prometheus.exporter.cadvisor on the nodes' cgroups, read from the host's cgroup tree, and a relabeling of its series to the cluster, node and role, and the cgroup below the node, from which the job_id and unit labels come;
  • a loki.source.journal per node, on its directory of the journal volume, with the unit and level of each entry, the level parsed from the message for the Slurm daemons and MariaDB, which log all their lines at priority info;
  • remote writes to Prometheus and Loki by container name.

Every series and log line carries realm, cluster and node, so a dashboard can link a panel to the matching logs.

func GrafanaProvisioning 

func GrafanaProvisioning(realm string) map[string][]byte

GrafanaProvisioning returns the files sind copies into /etc/grafana/provisioning before Grafana's first start, by path below it: the datasources, the dashboard provider and the dashboards.

func Images 

func Images() []string

Images returns the stack's images, in the order of its services.

func JournalSubpath 

func JournalSubpath(cluster, node string) string

JournalSubpath returns a node's directory in the journal volume.

func JournalVolume 

func JournalVolume(realm string) docker.VolumeName

JournalVolume returns the realm's journal volume, where each monitored node writes its journal into a directory of its own, <cluster>/<node>, and Alloy reads them.

func VolumeName 

func VolumeName(realm string, s Service) docker.VolumeName

VolumeName returns the data volume of a service in a realm.

Types

type Check 

type Check struct {
	Service string // slurmctld, slurmd or slurmdbd
	Check   string // live, ready or tcp
	Module  string // the blackbox module: http or tcp
	Port    string
	Path    string // the HTTP path, empty for tcp
}

Check is a health probe of a Slurm daemon on a node.

type Collection 

type Collection struct {
	// Journal and Cgroups are collected from every monitored node: its
	// journal and its container's and units' resource use.
	Journal, Cgroups bool
	// SlurmMetrics scrapes slurmctld's /metrics endpoints.
	SlurmMetrics bool
	// Checks are the node's Slurm daemon probes.
	Checks []Check
}

Collection is what the stack collects from a node.

func Collect 

func Collect(role config.Role, managed bool) Collection

Collect returns what the stack collects from a node of a role, managed or not: the journal and the cgroups of every node, slurmctld's metrics and probes on managed controllers, slurmd's on managed workers and slurmdbd's on a managed db node. Unmanaged nodes run no Slurm of sind's, so there are no Slurm probes that would only show sind's daemons as down. slurmdbd serves no HTTP, so its probe is a TCP connect; sackd and slurmrestd get none.

type Manager 

type Manager struct {
	Docker *docker.Client
	Realm  string
	Pull   bool // pull the stack's images afresh (--pull always)
	// Interval is the delay between readiness probe rounds, zero for
	// probe.DefaultInterval; ReadyTimeout bounds the wait, zero for
	// DefaultReadyTimeout.
	Interval     time.Duration
	ReadyTimeout time.Duration
}

Manager creates, configures and removes a realm's monitoring stack.

func NewManager 

func NewManager(dc *docker.Client, realm string) *Manager

NewManager returns a Manager of a realm's stack.

func (*Manager) EnsureStack 

func (m *Manager) EnsureStack(ctx context.Context, dnsIP, alloyConfig string) (changed bool, err error)

EnsureStack brings up the realm's stack on the mesh network: it creates the data volumes and containers that do not exist yet, with alloyConfig as Alloy's first configuration, recreates those of another fingerprint (another sind version's) and keeps their volumes, starts those that are stopped, and waits until all four are ready. dnsIP is the mesh DNS container's pinned address, from which Grafana, Prometheus and Loki take theirs (see mesh.ServiceIP); empty for a mesh without one, whose stack gets dynamic addresses. It reports whether it created or started a container.

func (*Manager) Fingerprint 

func (m *Manager) Fingerprint() string

Fingerprint returns the fingerprint of the stack a realm gets from this sind: of its services' container arguments and of the files sind provisions them with, except Alloy's configuration, which each sync rewrites, and of --pull. A stack created by another sind version, with other images, arguments, configuration or dashboards, has another one.

func (*Manager) Remove 

func (m *Manager) Remove(ctx context.Context) error

Remove removes the stack's containers, keeping their volumes. It finds them by their labels first, so that a realm without a stack costs one docker call.

func (*Manager) RemoveJournals 

func (m *Manager) RemoveJournals(ctx context.Context, cluster string, nodes ...string) error

RemoveJournals removes the journal directories of nodes of a cluster, or the cluster's whole directory when no node is given.

func (*Manager) RemoveVolumes 

func (m *Manager) RemoveVolumes(ctx context.Context) error

RemoveVolumes removes the stack's data volumes and the journal volume, once nothing mounts them. It finds them by their Compose project first.

func (*Manager) ResetJournals 

func (m *Manager) ResetJournals(ctx context.Context, cluster string, nodes ...string) error

ResetJournals empties the journal directories of nodes of a cluster, creating them, so that a node's container can mount its directory and shows no journal of an earlier node of its name.

func (*Manager) ServiceReady 

func (m *Manager) ServiceReady(ctx context.Context, s Service) error

ServiceReady checks that a service's container runs and that its readiness endpoint answers with success, through Grafana's curl. A stopped or dead container is a probe.TerminalError.

func (*Manager) Start 

func (m *Manager) Start(ctx context.Context) (found bool, err error)

Start starts the stack's containers that are stopped, as after a host reboot, and waits until the stack is ready. It creates nothing: a realm without any of the containers has no stack, and found is false then. A stack that lacks some of its containers is an error, as it cannot get ready; EnsureStack creates them again.

func (*Manager) Status 

func (m *Manager) Status(ctx context.Context) ([]ServiceStatus, bool, error)

Status returns the state of each of the stack's services, in the order of Services, and whether any of their containers exists. A running service is ready when ServiceReady passes; that check runs in Grafana, so without Grafana running none is checked. Like Remove, it lists the stack's containers first, so that a realm without a stack costs one docker call, and inspects those that exist in one more.

func (*Manager) Sync 

func (m *Manager) Sync(ctx context.Context, alloyConfig string) error

Sync writes Alloy's configuration and has Alloy reload it.

func (*Manager) URL 

func (m *Manager) URL(s Service, p string) string

URL returns the address of a service's endpoint path on the mesh network, by container name.

func (*Manager) WaitReady 

func (m *Manager) WaitReady(ctx context.Context) error

WaitReady waits until every service of the stack is ready (see ServiceReady), Grafana first, as the others are probed through it.

type Service 

type Service string

Service is one of the stack's containers. 

const (
	ServiceGrafana    Service = "grafana"
	ServicePrometheus Service = "prometheus"
	ServiceLoki       Service = "loki"
	ServiceAlloy      Service = "alloy"
)

The stack's services, which name their containers and volumes (<realm>-<service>) and their Compose services.

func (Service) Image 

func (s Service) Image() string

Image returns the image a service runs.

func (Service) Port 

func (s Service) Port() string

Port returns the port a service listens on.

type ServiceStatus 

type ServiceStatus struct {
	Service   Service               `json:"service"`
	Container docker.ContainerName  `json:"container"`
	State     docker.ContainerState `json:"status"` // empty when the container does not exist
	Image     string                `json:"image"`
	IP        string                `json:"ip"`
	DNSName   string                `json:"dns_name,omitempty"` // empty for Alloy, which has none
	Port      string                `json:"port"`
	Ready     bool                  `json:"ready"`
	Checked   bool                  `json:"checked"` // whether Ready was checked, which needs Grafana running
}

ServiceStatus is the state of one of the stack's services.

type Target 

type Target struct {
	Cluster     string
	Node        string // the node's hostname, e.g. worker-0
	Role        config.Role
	Managed     bool // sind manages Slurm on the node (the sind.managed label)
	Container   docker.ContainerName
	ContainerID docker.ContainerID // the full ID, which names the node's cgroup
}

Target is a monitored node: a node container of a monitored cluster, which mounts its directory of the realm's journal volume.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.