cluster

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 7, 2026 License: GPL-3.0, LGPL-3.0 Imports: 31 Imported by: 0

Documentation

Overview

Package cluster provides types and operations for sind cluster management.

Create, Delete, DeleteAll, WorkerAdd, WorkerRemove, PowerOn, PowerReboot and PowerCycle change state that every cluster of a realm shares and take no lock themselves: their caller holds the realm lock, state.LockRealm with the docker client as LockOptions.Client, for the whole operation.

Index

Constants

View Source 
const (
	// CVMFSPath is where the nodes mount CVMFS, and where the Docker host
	// has it for the bind mount.
	CVMFSPath = "/cvmfs"
	// CVMFSVolumeDriver is the name of the Docker volume plugin that
	// provides CVMFS.
	CVMFSVolumeDriver = "cvmfs"
	// CVMFSVolume is the plugin volume the nodes mount. All clusters share
	// it, so sind never removes it.
	CVMFSVolume docker.VolumeName = "cvmfs"
)

CVMFS on the nodes (storage.cvmfs).

View Source 
const (
	PositionPrimary = "primary"
	PositionBackup  = "backup"
)

Controller positions in a primary/backup pair, matching the order of the SlurmctldHost lines and the labels scontrol ping uses.

View Source 
const (
	LabelRealm        = "sind.realm"
	LabelCluster      = "sind.cluster"
	LabelRole         = "sind.role"
	LabelManaged      = "sind.managed"
	LabelSlurmVersion = "sind.slurm.version"
	// LabelDataHostPath records the host directory the node bind-mounts as
	// its data (storage.dataStorage.hostPath), empty for the cluster's data
	// volume.
	LabelDataHostPath = "sind.data.hostpath"
	// LabelDataMountPath records the data mount point
	// (storage.dataStorage.mountPath, DefaultDataMountPath by default).
	LabelDataMountPath = "sind.data.mountpath"
	// LabelCVMFS records how the node mounts CVMFS (storage.cvmfs): "volume"
	// for the cvmfs volume plugin, "hostPath" for the Docker host's /cvmfs,
	// empty for none.
	LabelCVMFS = "sind.cvmfs"
	// LabelUsers and LabelGroups record the cluster users and groups
	// (users, groups), as space-separated entries (see LinuxUsers.Labels).
	LabelUsers  = "sind.users"
	LabelGroups = "sind.groups"
	// LabelIdentity records the identity mode (identity) of the cluster:
	// local, nssSlurm or clientIds.
	LabelIdentity = "sind.identity"
)

Label keys used on sind containers.

Docker merges the image's labels into a container's, so a label sind leaves out can come from the image. Every node container therefore gets each of these labels, with an empty value where there is nothing to record, and readers treat an empty or missing label alike, as nodes created by earlier sind versions lack some of them.

View Source 
const ControllerBackupShortName = slurm.BackupControllerHost

ControllerBackupShortName is the hostname of the backup controller created when the controller node spec sets backupController: true. It matches the second SlurmctldHost line in the generated slurm.conf.

View Source 
const DNSSuffix = "sind"

DNSSuffix is the base domain for all sind DNS names.

View Source 
const DefaultDataMountPath = "/data"

DefaultDataMountPath is the default mount path for the shared data volume.

View Source 
const DefaultReadinessInterval = probe.DefaultInterval

DefaultReadinessInterval is the delay between readiness probe rounds that Create and WorkerAdd use for a readinessInterval of zero or less.

View Source 
const HomeMountPath = "/home"

HomeMountPath is where the nodes of a cluster with users mount the cluster's home volume, which holds the users' home directories.

View Source 
const MaskMunge = "ln -sf /dev/null /etc/systemd/system/munge.service\n"

MaskMunge is the line the node entrypoint starts with under identity clientIds: it masks munge.service before systemd starts, as auth/slurm replaces munge and the node has no munge key.

View Source 
const NodeEntrypoint = `` /* 197-byte string literal not displayed */

NodeEntrypoint is the shell script every node container starts with, as PID 1, before it execs systemd. It moves itself into init.scope and enables all available controllers in the root cgroup's cgroup.subtree_control.

docker exec places its process in the container's root cgroup unless that cgroup has controllers enabled; runc then falls back to init's cgroup. A process in the root cgroup makes every later write to its cgroup.subtree_control fail with EBUSY (cgroup v2's no internal processes rule). Without this script, a docker exec of sind's that lands before systemd has enabled controllers keeps systemd from ever doing so, and daemons with Delegate=yes get none: slurmd then cannot use the memory or cpu controller, which jobacct_gather/cgroup needs. Each controller is enabled on its own, as one write of all of them fails as a whole if the kernel refuses one. Failures are ignored so the node still boots.

View Source 
const RunTmpfsSize = "64m"

RunTmpfsSize is the size of every node's /run tmpfs. /run holds systemd's runtime state, the daemons' sockets and pid files, and journald's volatile journal, which journald keeps to a tenth of the file system; without a size it could grow to half the host's memory, all of it charged to the node. systemd wants 16M free on /run to reload.

View Source 
const TaskAffinityCapability = "SYS_NICE"

TaskAffinityCapability is the capability managed workers get when the cluster's TaskPlugin includes task/affinity: slurmstepd, as root, sets each task's CPU affinity after the task has become the job's user, and changing the affinity of another user's process needs CAP_SYS_NICE, which Docker drops by default. Without it, every job of a user other than root fails with task_g_set_affinity.

Variables

View Source 
var (
	// ErrClusterExists reports that resources with the names of the
	// cluster to create exist already (PreflightCheck, Create).
	ErrClusterExists = errors.New("cluster resources exist")
	// ErrClusterNotFound reports that a cluster, or its controller, does
	// not exist in the realm (GetStatus, WorkerAdd).
	ErrClusterNotFound = errors.New("cluster not found")
	// ErrNodeNotFound reports that a named node does not exist in its
	// cluster (power operations, WorkerRemove).
	ErrNodeNotFound = errors.New("node not found")
	// ErrNodesConfMissing reports that the controller has no
	// sind-nodes.conf, so sind cannot add managed workers (WorkerAdd).
	ErrNodesConfMissing = errors.New("sind-nodes.conf not found on controller: managed workers require sind-generated Slurm configuration; use --unmanaged to add nodes without modifying Slurm config")
	// ErrNotReady reports that the nodes or Slurm of a Create or WorkerAdd
	// did not become ready within its wait limit (config.Cluster.Wait,
	// WorkerAddOptions.Wait).
	ErrNotReady = errors.New("not ready")
	// ErrNetworkFull reports that the nodes of a Create or WorkerAdd do
	// not fit on the realm's mesh or the cluster network, Docker bridge
	// networks of at most docker.MaxBridgeEndpoints containers
	// (PreflightCheck, Create, WorkerAdd).
	ErrNetworkFull = errors.New("network full")
)

Errors that library callers can test for with errors.Is. The errors pkg/cluster returns wrap them and carry their own message.

View Source 
var (
	// ErrRootlessDaemon is a daemon in rootless mode.
	ErrRootlessDaemon = errors.New("the Docker daemon runs in rootless mode")
	// ErrUsernsRemap is a daemon that remaps user namespaces.
	ErrUsernsRemap = errors.New("the Docker daemon runs with userns-remap")
	// ErrCgroupV1 is a daemon that runs containers on cgroup v1.
	ErrCgroupV1 = errors.New("the Docker daemon runs containers on cgroup v1")
	// ErrDockerTooOld is a Docker Engine older than doctor.MinDockerMajor.
	ErrDockerTooOld = errors.New("the Docker Engine is too old")
	// ErrNoNsdelegate is a Docker host whose cgroup2 hierarchy is mounted
	// without nsdelegate (CheckNsdelegate).
	ErrNoNsdelegate = errors.New("the Docker host mounts cgroup2 without nsdelegate")
)

The errors DaemonSupport wraps for a Docker daemon that cannot run sind nodes.

View Source 
var AllVolumeTypes = []VolumeType{VolumeConfig, VolumeMunge, VolumeData, VolumeState, VolumeHome}

AllVolumeTypes lists the cluster volume types in creation order.

View Source 
var AuthTypes = []AuthType{AuthMunge, AuthSlurm, AuthJWT}

AuthTypes are the authentication types GetAuthKey takes.

Functions

func BuildContainerExecArgs added in v0.3.0 

func BuildContainerExecArgs(container docker.ContainerName, user, workDir string, isTTY bool, command []string) []string

BuildContainerExecArgs builds docker CLI arguments for running a command directly inside a cluster container via docker exec. The command runs as root in workDir, the node's data mount point (see EnterTarget), or, when user names a cluster user, as that user in its home directory. When command is nil, an interactive login shell is started.

func BuildRunArgs 

func BuildRunArgs(cfg RunConfig) []string

BuildRunArgs returns the docker arguments for creating a node container. The returned slice does not include "create" or "run -d" — the caller passes these args to Client.CreateContainer or Client.RunContainer.

func BuildSSHArgs 

func BuildSSHArgs(sshContainer docker.ContainerName, node, cluster, realm string, isTTY bool, sshOptions, command []string) []string

BuildSSHArgs builds the docker CLI arguments for running SSH through the sind-ssh relay container. The returned args are suitable for passing to docker directly (e.g. "docker exec -i -t sind-ssh ssh ...").

node is the short name (e.g. "worker-0"), cluster is the cluster name, isTTY controls whether -t is added to docker exec, sshOptions are passed through to SSH before the target, and command is the optional remote command.

func CheckDaemon added in v0.11.0 

func CheckDaemon(ctx context.Context, client *docker.Client) error

CheckDaemon asks the Docker daemon what it is and returns DaemonSupport's verdict. Call it before mesh.Manager.EnsureMesh and Create, so that an unsupported daemon fails before anything is pulled or created.

func CheckNsdelegate added in v0.11.0 

func CheckNsdelegate(ctx context.Context, client *docker.Client, image string) error

CheckNsdelegate reads the cgroup2 mount of the Docker daemon's kernel in a throwaway container of image (doctor.ProbeCgroupInfo), and returns an error that wraps ErrNoNsdelegate, with the commands that enable it, when the mount lacks nsdelegate, or ErrCgroupV1 when the container has no cgroup2 at doctor.CgroupRoot. Unlike `sind doctor` for a daemon on this machine, it looks at the daemon's kernel wherever that runs. Create and WorkerAdd call it before they create a node container: without nsdelegate the failure would show only later, as a node or Slurm daemon that does not become ready.

func ComposeProject 

func ComposeProject(realm, clusterName string) string

ComposeProject returns the Docker Compose project name for a cluster.

func ContainerLogArgs 

func ContainerLogArgs(realm, node, cluster string, follow bool) []string

ContainerLogArgs builds docker CLI arguments for streaming container logs. node is the short name (e.g. "controller", "worker-0"), cluster is the cluster name, and follow controls whether --follow is added.

func ContainerName 

func ContainerName(realm, cluster, shortName string) docker.ContainerName

ContainerName returns the Docker container name for a node. shortName is the node's hostname, e.g. "controller", "db", "api", "submitter", "worker-0".

func ContainerPrefix 

func ContainerPrefix(realm, cluster string) string

ContainerPrefix returns the container name prefix for a cluster, used to extract short names from full container names.

func CreateClusterNetwork 

func CreateClusterNetwork(ctx context.Context, client *docker.Client, realm, clusterName string) error

CreateClusterNetwork creates the cluster-specific Docker bridge network.

func CreateClusterVolume added in v0.9.0 

func CreateClusterVolume(ctx context.Context, client *docker.Client, realm, clusterName string, vtype VolumeType) error

CreateClusterVolume creates a single cluster volume.

func CreateNode 

func CreateNode(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, cfg RunConfig) (docker.ContainerID, error)

CreateNode creates a node container on its cluster network and the mesh network, and starts it. Returns the container ID.

func DNSName 

func DNSName(shortName, cluster, realm string) string

DNSName returns the fully qualified DNS name for a node. shortName is the node's hostname, e.g. "controller", "worker-0".

func DNSSearchDomain 

func DNSSearchDomain(cluster, realm string) string

DNSSearchDomain returns the DNS search domain for a cluster.

func DaemonSupport added in v0.11.0 

func DaemonSupport(info *docker.DaemonInfo) error

DaemonSupport returns nil when the daemon that info describes can run sind nodes. Every node starts with --security-opt writable-cgroups=true, which Docker refuses on a daemon in rootless mode or with userns-remap, and only at `docker start`: after the images are pulled and the cluster's network and volumes created. A node's entrypoint and systemd need cgroup v2; on cgroup v1 the node never becomes ready. Docker Engines before doctor.MinDockerMajor know neither writable-cgroups nor the gw-priority of the nodes' network endpoints, and refuse every node at `docker create`. A version that does not parse passes, as `sind doctor` reports it.

func DataMountPath added in v0.10.0 

func DataMountPath(labels docker.Labels) string

DataMountPath returns where a node container mounts the cluster's data: its LabelDataMountPath, or DefaultDataMountPath without one.

func DataPathWarning added in v0.11.0 

func DataPathWarning(hostPath string) string

DataPathWarning returns a warning when a cluster's data host path is the host's root directory or the user's home directory ($HOME), which every node then mounts read-write: typically by accident, from sind create cluster run there with the default --data . It returns "" for any other path.

func Delete 

func Delete(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, clusterName string) error

Delete orchestrates the full cluster deletion flow.

The caller holds the realm lock (state.LockRealm) until Delete returns.

Deleting a non-existent cluster is not an error. The function handles partial clusters (e.g., from a failed creation) by removing whatever resources exist. 

HasOtherClusters?
      │
deleteClusterResources   deregisters from the mesh only if other clusters remain
      │
    yes → done
    no  → CleanupMesh

Whether other clusters remain does not depend on this cluster's containers, so Delete settles it first: when the mesh goes too, removing the nodes' DNS records and known_hosts entries would be wasted work.

func DeleteAll added in v0.11.0 

func DeleteAll(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager) error

DeleteAll deletes every cluster of the realm, in parallel, and then the realm's mesh. It finds the clusters by the labels of their containers, networks and volumes, and removes the mesh even when it finds no cluster: a create that was killed can leave a mesh behind without one.

A cluster that fails to delete does not stop the others. The mesh then stays for what is left, the clusters that were deleted are deregistered from it, and DeleteAll returns the failures, joined.

func DeleteContainers 

func DeleteContainers(ctx context.Context, client *docker.Client, containers []docker.ContainerListEntry) error

DeleteContainers force-removes the given containers in parallel (docker rm -f). A container that is already gone is logged and treated as success — the caller asked for the resource removed, not for a particular state transition. Like DeleteVolumes, it tries every container even when one fails, so that one failure does not cancel the removals still waiting for a docker call slot, and returns every failure.

func DeleteNetwork 

func DeleteNetwork(ctx context.Context, client *docker.Client, name docker.NetworkName) error

DeleteNetwork removes the cluster network. A network that is already gone is logged and treated as success.

func DeleteVolumes 

func DeleteVolumes(ctx context.Context, client *docker.Client, volumes []docker.VolumeName) error

DeleteVolumes removes the given cluster volumes in parallel. Each removal retries past the dockerd async-cleanup race that follows `docker rm -f`. A volume that is already gone is logged and treated as success. Every volume is attempted; the errors of those that could not be removed are joined in argument order.

func DeregisterMesh 

func DeregisterMesh(ctx context.Context, meshMgr *mesh.Manager, clusterName string, containers []docker.ContainerListEntry) error

DeregisterMesh removes DNS records and known_hosts entries for all containers in batch, starting a stopped mesh first (see deregisterHostnames). This is the inverse of registerMesh during cluster creation. Failures are logged and swallowed: the cluster is being torn down, and stale entries in a mesh helper that's already unreachable will be overwritten on next register or cleared when the mesh itself is torn down.

func DetectCVMFS added in v0.11.0 

func DetectCVMFS(ctx context.Context, client *docker.Client, image string) (config.StorageType, error)

DetectCVMFS picks how the nodes mount CVMFS: config.StorageVolume when the cvmfs Docker volume plugin is installed and enabled, otherwise config.StorageHostPath, a bind mount of the Docker host's /cvmfs. The bind mount is tried in a throwaway container of image first, so a host without /cvmfs, or one whose /cvmfs Docker cannot propagate into containers, fails before any node is created.

func DiscoverClusterNames added in v0.3.0 

func DiscoverClusterNames(ctx context.Context, client *docker.Client, realm string) ([]string, error)

DiscoverClusterNames finds cluster names from orphaned networks and volumes that may not have containers. This supplements GetClusters (which only finds clusters with running containers) for cleanup operations.

Filters on both the realm and cluster labels so mesh resources (which carry only the realm label) are skipped, and resources from other realms can't match even when names collide. The network and volume listings run concurrently.

func EnterTarget 

func EnterTarget(ctx context.Context, client *docker.Client, realm, clusterName string) (target, workDir string, err error)

EnterTarget determines the target node for sind enter and sind exec, and the directory they start in: the node's data mount point. The target is "submitter" if present in the cluster, otherwise the controller: in a primary/backup pair the one in control, falling back to "controller" when that cannot be determined. In an unmanaged cluster sind does not know which controller is in control, so the running one is picked, preferring "controller".

func GetNodeNames added in v0.11.0 

func GetNodeNames(ctx context.Context, client *docker.Client, realm string) ([]string, error)

GetNodeNames returns the nodes of every cluster in the realm as NODE.CLUSTER, ordered by cluster name, then like GetNodes. It lists the realm's containers once and inspects none, for shell completion.

func HasOtherClusters 

func HasOtherClusters(ctx context.Context, client *docker.Client, realm, clusterName string) (bool, error)

HasOtherClusters checks whether any sind cluster containers exist besides the named cluster. This is used to decide whether to clean up mesh infrastructure after deleting a cluster.

A container belongs to clusterName only if its sind.cluster label says so. The name prefix "<realm>-<clusterName>-" cannot tell: it also matches the containers of cluster "<clusterName>-2". A realm container without a cluster label counts as another cluster, which keeps the mesh.

func HomeDir added in v0.11.0 

func HomeDir(user string) string

HomeDir returns the home directory of a cluster user.

func IdentityFromLabels added in v0.11.0 

func IdentityFromLabels(labels docker.Labels) config.IdentityMode

IdentityFromLabels returns the identity mode a node container's LabelIdentity records, local without one.

func IsManaged added in v0.10.0 

func IsManaged(labels docker.Labels) bool

IsManaged reports whether a node container's labels mark it as managed by sind: its Slurm configuration is sind's and sind runs its Slurm daemon. On a controller it tells whether the whole cluster is managed. Containers created before sind recorded LabelManaged count as managed.

func NetworkName 

func NetworkName(realm, cluster string) docker.NetworkName

NetworkName returns the Docker network name for a cluster.

func NodeLabels 

func NodeLabels(realm, clusterName string, role config.Role, managed bool, slurmVersion, dataHostPath string, containerNumber int) docker.Labels

NodeLabels returns the standard labels for a node container. managed records whether sind manages Slurm on the node (see IsManaged). containerNumber is the 1-based instance number for compose compatibility. The slurm version label is always set, empty when sind does not know the version (unmanaged clusters), and so is the data host path label, empty in Docker volume mode: the image's labels would otherwise show through on the container, and the sind-node image carries a slurm version label.

func NodeShortNames 

func NodeShortNames(nodes []config.Node) []string

NodeShortNames returns the short hostname for each node defined in the config, including the backup controller when enabled. Worker nodes are indexed sequentially across all worker groups, matching the indexing used in slurm.ManagedWorkers.

func OtherRealms added in v0.11.0 

func OtherRealms(ctx context.Context, client *docker.Client, realm, clusterName string) ([]string, error)

OtherRealms returns, sorted, the realms other than realm that hold a node container of a cluster named clusterName. Labels that are not valid realm names are left out.

func PowerCut 

func PowerCut(ctx context.Context, client *docker.Client, realm, clusterName string, shortNames []string) error

PowerCut immediately kills the specified nodes (docker kill).

func PowerCycle 

func PowerCycle(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, clusterName string, shortNames []string) error

PowerCycle hard-restarts the specified nodes: it kills them all (docker kill), then starts them as PowerOn does.

func PowerFreeze 

func PowerFreeze(ctx context.Context, client *docker.Client, realm, clusterName string, shortNames []string) error

PowerFreeze suspends all processes in the specified nodes (docker pause). The containers remain running but are completely unresponsive.

func PowerOn 

func PowerOn(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, clusterName string, shortNames []string) error

PowerOn starts the specified stopped nodes (docker start). It first starts the realm's mesh DNS and SSH relay if they are stopped, as after a host reboot, and afterwards points the started nodes' mesh DNS records at their addresses: Docker can give a node another address each time it starts.

func PowerReboot 

func PowerReboot(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, clusterName string, shortNames []string) error

PowerReboot gracefully restarts the specified nodes: it stops them all (docker stop), then starts them as PowerOn does.

func PowerShutdown 

func PowerShutdown(ctx context.Context, client *docker.Client, realm, clusterName string, shortNames []string) error

PowerShutdown gracefully stops the specified nodes (docker stop).

func PowerUnfreeze 

func PowerUnfreeze(ctx context.Context, client *docker.Client, realm, clusterName string, shortNames []string) error

PowerUnfreeze resumes the specified frozen nodes (docker unpause).

func PreflightCheck 

func PreflightCheck(ctx context.Context, client *docker.Client, realm string, cfg *config.Cluster) error

PreflightCheck verifies that no Docker resources conflict with the cluster that would be created from the given configuration. It checks for existing networks, volumes, and containers with matching names, and that the realm's mesh and the cluster network have room for the nodes (checkBridgePorts).

Container existence is checked with a single `docker ps` filtered by the cluster's realm + name labels; this keeps the call count constant regardless of node count.

func ServiceLogArgs 

func ServiceLogArgs(realm, node, cluster, service string, follow bool) []string

ServiceLogArgs builds docker CLI arguments for streaming service journal logs. node is the short name, cluster is the cluster name, service is the systemd unit name (e.g. "slurmctld", "slurmd"), and follow controls whether --follow is added.

func VolumeName 

func VolumeName(realm, cluster string, volumeType VolumeType) docker.VolumeName

VolumeName returns the Docker volume name for a cluster resource.

func WorkerRemove 

func WorkerRemove(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, clusterName string, shortNames []string) error

WorkerRemove removes worker nodes from a cluster.

The caller holds the realm lock (state.LockRealm) until WorkerRemove returns.

For managed nodes (those present in sind-nodes.conf), the flow is:

  1. Read sind-nodes.conf through the controller, which has to run
  2. Update sind-nodes.conf to remove the node definitions and reconfigure slurmctld, while DNS + known_hosts are deregistered
  3. Stop + remove containers

For unmanaged nodes, and every node of an unmanaged cluster, step 1 and the Slurm part of step 2 are skipped: the Slurm configuration of an unmanaged cluster is the user's, even if it contains a sind-nodes.conf. So is a configuration without sind-nodes.conf.

func WriteClusterConfig 

func WriteClusterConfig(ctx context.Context, client *docker.Client, realm string, cfg *config.Cluster, image string) error

WriteClusterConfig generates and writes slurm.conf, sind-nodes.conf, and cgroup.conf to the config volume. Uses a temporary container to access the volume.

With a managed db node it also writes slurmdbd.conf, and the slurmdbd.conf.d fragments of a map-form slurmdbd section, and turns on accounting in slurm.conf; an unmanaged db node gets neither. With identity clientIds it writes a new slurm.key, the key of auth/slurm, which every node reads from the config volume, and with a managed api node a new jwt_hs256.key, the key of auth/jwt, and the JWT parameters in slurm.conf and slurmdbd.conf. These files can hold secrets, such as a StoragePass, and every node mounts the config volume: they are written with mode 0600, so that others can never read them, and slurmdbd refuses a slurmdbd.conf, auth/slurm a slurm.key and auth/jwt a jwt_hs256.key, that others may read. docker cp writes files as root, so the helper then runs (like the munge helper) to give them to SlurmUser with docker exec, and the fragments' directory too, with mode 0700.

func WriteMungeKey 

func WriteMungeKey(ctx context.Context, client *docker.Client, realm, clusterName string, key []byte, image string) error

WriteMungeKey writes the given munge key to the munge volume. Uses a temporary container to access the volume.

Types

type AuthKey added in v0.11.0 

type AuthKey struct {
	Type AuthType
	Key  []byte
}

AuthKey is a key that authenticates a cluster's Slurm traffic.

func GetAuthKey added in v0.11.0 

func GetAuthKey(ctx context.Context, client *docker.Client, realm, clusterName string, authType AuthType) (*AuthKey, error)

GetAuthKey reads a key that authenticates a cluster's Slurm traffic from one of its node containers: for authType munge the munge key, for slurm slurm.key, with identity clientIds, and for jwt jwt_hs256.key, with a managed api node. An empty authType is the cluster's AuthType: slurm with identity clientIds, munge otherwise. A type the cluster does not use is an error that says why.

type AuthType added in v0.11.0 

type AuthType string

AuthType is a Slurm authentication plugin a cluster uses, auth/<type>, and so which key GetAuthKey returns. 

const (
	// AuthMunge is auth/munge: munged on every node, with the munge key.
	AuthMunge AuthType = "munge"
	// AuthSlurm is auth/slurm, with slurm.key, under identity clientIds.
	AuthSlurm AuthType = "slurm"
	// AuthJWT is auth/jwt, with jwt_hs256.key, the alternative
	// authentication of slurmrestd's tokens on a cluster with a managed api
	// node.
	AuthJWT AuthType = "jwt"
)

Authentication types.

type Cluster 

type Cluster struct {
	Name         string
	SlurmVersion string
	State        State
	Nodes        []*Node
}

Cluster represents a live sind cluster as it exists in Docker. This is distinct from config.Cluster, which represents the configuration input.

func Create 

func Create(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, cfg *config.Cluster, readinessInterval time.Duration) (result *Cluster, retErr error)

Create orchestrates the full cluster creation flow.

The caller holds the realm lock (state.LockRealm) from before mesh.Manager.EnsureMesh until Create returns, and checks the daemon with CheckDaemon before either.

The caller must ensure mesh infrastructure exists (via mesh.Manager.EnsureMesh) before calling Create, and apply the config's defaults (config.Cluster.ApplyDefaults). Create validates the config first, and refuses a config realm other than meshMgr.Realm, the realm it creates the cluster in. The context deadline controls the overall timeout; readinessInterval controls the polling interval for readiness probes, and zero or less means DefaultReadinessInterval. cfg.Wait, when positive, limits how long Create waits for the nodes and Slurm to become ready, counted for each node from when its container has started, so that image pulls do not count; the steps after the nodes are ready end cfg.Wait after the last node container started. When the limit is reached, Create rolls back and returns an error that wraps ErrNotReady. 

┌ PreflightCheck → createResources ─────────────────┐
├ resolveInfra (DNS IP ║ SSH key ║ Slurm version) ──┤
├ DetectCVMFS (storage.cvmfs only) ─────────────────┼→ setupNodes
├ CheckNsdelegate ──────────────────────────────────┤
└ pullImages (missing images; all with cfg.Pull) ───┘
                        │
registerMesh ║ enableSlurm → createSlurmAccounts (accounts only) ║ createHomes (users only)
                        │
                    *Cluster

Each distinct image the daemon does not have, or with cfg.Pull each distinct image, is pulled once, concurrently, and the steps that run one (the helpers of createResources, the Slurm version, DetectCVMFS, CheckNsdelegate) wait for the pull; nothing is created with --pull always, and no node container's docker create pulls (see startPull). A failed check rolls back what the other branches have created by then; no node container exists yet.

An unmanaged cluster (managed: false on the controller) gets the same containers, volumes and munge key, but sind writes no Slurm configuration, does not discover the Slurm version and enables no Slurm daemon.

type HAStatus added in v0.10.0 

type HAStatus struct {
	Position  string `json:"position"`   // "primary" or "backup"
	InControl bool   `json:"in_control"` // this controller currently runs the cluster
}

HAStatus describes a controller's place in a primary/backup pair.

type LinuxGroup added in v0.11.0 

type LinuxGroup struct {
	Name string
	GID  int
	// Members are the users that have the group as a supplementary group.
	// As in /etc/group, users with it as their primary group are not
	// listed.
	Members []string
}

LinuxGroup is a group sind creates on the nodes that get the cluster users.

type LinuxUser added in v0.11.0 

type LinuxUser struct {
	Name string
	UID  int
	GID  int // primary group
}

LinuxUser is a user account sind creates on the nodes that get the cluster users.

type LinuxUsers added in v0.11.0 

type LinuxUsers struct {
	Users  []LinuxUser
	Groups []LinuxGroup
}

LinuxUsers are the user accounts and groups of a cluster (users and groups), with every ID resolved: each user without a primary group has a private group of its own name with gid = uid.

func LinuxUsersFromLabels added in v0.11.0 

func LinuxUsersFromLabels(labels docker.Labels) (LinuxUsers, error)

LinuxUsersFromLabels returns the users and groups a node container's LabelUsers and LabelGroups record (see LinuxUsers.Labels), empty when it has neither.

func NewLinuxUsers added in v0.11.0 

func NewLinuxUsers(cfg *config.Cluster) LinuxUsers

NewLinuxUsers resolves the users and groups of a cluster config, after ApplyDefaults has assigned their IDs: the private groups, in user order, then the declared groups with their members.

func (LinuxUsers) IsEmpty added in v0.11.0 

func (l LinuxUsers) IsEmpty() bool

IsEmpty reports whether the cluster has neither users nor groups.

func (LinuxUsers) Labels added in v0.11.0 

func (l LinuxUsers) Labels() docker.Labels

Labels returns the labels that record the users and groups on every node container (LabelUsers and LabelGroups), so that workers added later get the same ones. Both are space-separated entries, not comma-separated: docker ps joins all labels of a container with commas. Both are set, empty without users or groups, so that no image label of that name shows through. 

sind.users   name:uid:gid ...          alice:1000:1000 bob:2002:3000
sind.groups  name:gid[:member+...] ... alice:1000 hpc:3000:alice

type MountPoint added in v0.3.0 

type MountPoint struct {
	Path   string             `json:"path"`   // mount path inside the container (e.g. "/etc/slurm")
	Source string             `json:"source"` // volume name or host path
	Type   config.StorageType `json:"type"`   // "volume" or "hostPath"
	OK     bool               `json:"ok"`     // true if the Docker volume exists (always true for hostPath)
}

MountPoint describes a volume or bind mount on cluster containers.

func GetMountPoints added in v0.3.0 

func GetMountPoints(ctx context.Context, client *docker.Client, realm, clusterName string, containers []*docker.ContainerInfo) ([]MountPoint, error)

GetMountPoints returns the mount points for a cluster, checking volume existence for Docker volumes. containers are the cluster's node containers as docker inspect reports them, whose labels, unlike those docker ps lists, keep values with commas (see docker.ContainerInfo).

The data mount is what Docker mounts at a node's data mount point (the sind.data.mountpath label, or /data): a host-path bind mount or a Docker volume, the cluster's data volume when no node has one. The sind.users label adds the home volume at /home, and the sind.cvmfs label /cvmfs from the cvmfs plugin volume or the host's /cvmfs. With identity clientIds (the sind.identity label) there is no munge volume.

type NetworkHealth 

type NetworkHealth struct {
	Mesh           bool   `json:"mesh_ok"`         // sind-mesh network exists
	MeshName       string `json:"mesh_name"`       // mesh network name (e.g. "sind-mesh")
	MeshDriver     string `json:"mesh_driver"`     // mesh network driver (e.g. "bridge")
	MeshSubnet     string `json:"mesh_subnet"`     // mesh network subnet
	MeshGateway    string `json:"mesh_gateway"`    // mesh network gateway
	DNS            bool   `json:"dns_ok"`          // sind-dns container running
	DNSName        string `json:"dns_name"`        // DNS container name (e.g. "sind-dns")
	SSH            bool   `json:"ssh_ok"`          // sind-ssh relay container running
	SSHName        string `json:"ssh_name"`        // SSH relay container name (e.g. "sind-ssh")
	Cluster        bool   `json:"cluster_ok"`      // cluster network exists
	ClusterName    string `json:"cluster_name"`    // cluster network name (e.g. "sind-dev-net")
	ClusterDriver  string `json:"cluster_driver"`  // cluster network driver (e.g. "bridge")
	ClusterSubnet  string `json:"cluster_subnet"`  // cluster network subnet
	ClusterGateway string `json:"cluster_gateway"` // cluster network gateway
}

NetworkHealth holds the health and IPAM details of cluster networking.

func GetNetworkHealth 

func GetNetworkHealth(ctx context.Context, client *docker.Client, realm, clusterName string) (*NetworkHealth, error)

GetNetworkHealth checks the health of mesh, DNS, SSH relay, and cluster networking. The DNS and SSH relay containers count as healthy only while they run: after a host reboot they exist but are stopped.

type NetworkSummary 

type NetworkSummary struct {
	Name    string `json:"name"`
	Driver  string `json:"driver"`
	Subnet  string `json:"subnet"`
	Gateway string `json:"gateway"`
}

NetworkSummary holds summary information about a sind network.

func GetNetworks 

func GetNetworks(ctx context.Context, client *docker.Client, realm string) ([]*NetworkSummary, error)

GetNetworks lists all sind-related Docker networks with IPAM details. This includes per-cluster networks (sind-<cluster>-net) and the mesh network (sind-mesh), but not the realm lock (state.LockNetworkName), which carries the realm label while a command holds it.

type Node 

type Node struct {
	Name        string             // short name: "controller", "worker-0"
	Role        config.Role        // "controller", "db", "api", "submitter", "worker"
	ContainerID docker.ContainerID // Docker container ID
	IP          string             // container IP address
	State       State
}

Node represents a running node in a sind cluster.

func WorkerAdd 

func WorkerAdd(ctx context.Context, client *docker.Client, meshMgr *mesh.Manager, opts WorkerAddOptions, readinessInterval time.Duration) (result []*Node, retErr error)

WorkerAdd adds worker nodes to an existing cluster.

The caller holds the realm lock (state.LockRealm) until WorkerAdd returns.

For managed workers (default), the flow is:

  1. Validate: options, controller exists, room on the realm's mesh and the cluster network (see checkBridgePorts), sind-nodes.conf present
  2. Inspect the controller and the newest worker, for the cluster's settings and the new workers' shape
  3. Check an explicit image: its Slurm version, its identity support; meanwhile check the Docker host for nsdelegate (CheckNsdelegate) in a container of the controller's image
  4. Create worker container(s)
  5. Wait for readiness, inject SSH keys, collect host keys
  6. Register DNS + known_hosts, while sind-nodes.conf gets the new nodes, slurmctld is reconfigured and slurmd starts on the new nodes

For unmanaged workers (Unmanaged=true), the Slurm steps are skipped. Workers added to an unmanaged cluster are always unmanaged. A failure after the first container exists removes the new containers, mesh entries and NodeName lines again. readinessInterval is the delay between readiness probe rounds, as for Create.

type NodeDetail added in v0.9.0 

type NodeDetail struct {
	Container string                `json:"container"`
	Cluster   string                `json:"cluster"`
	Role      config.Role           `json:"role"`
	Managed   bool                  `json:"managed"` // sind manages Slurm on the node (see IsManaged)
	FQDN      string                `json:"fqdn"`
	IP        string                `json:"ip"`
	Status    docker.ContainerState `json:"status"`
	Services  ServiceHealth         `json:"services"`
	HA        *HAStatus             `json:"ha,omitempty"` // controllers of a primary/backup pair only
}

NodeDetail holds the full identity and health information for a single node as reported by 'sind get node'. It extends NodeSummary with a per-service health map. All readiness-checked services (munge, sshd, and on managed nodes the role's Slurm service, or mariadb and slurmdbd on a db node) are reported under Services.

type NodeHealth 

type NodeHealth struct {
	State    docker.ContainerState `json:"status"`       // container state from Docker (e.g. "running", "exited")
	IP       string                `json:"ip"`           // container IP address
	Services ServiceHealth         `json:"services"`     // all readiness-checked services (munge unless identity clientIds, sshd, and on managed nodes the role's slurmctld/slurmd, mariadb and slurmdbd, slurmrestd, or sackd)
	HA       *HAStatus             `json:"ha,omitempty"` // controllers of a primary/backup pair only
}

NodeHealth holds the health status of a single node.

func GetNodeHealth 

func GetNodeHealth(ctx context.Context, client *docker.Client, info *docker.ContainerInfo, role config.Role, realm, clusterName string) (*NodeHealth, error)

GetNodeHealth checks the health of a single node container, from its docker inspect result, which the caller already holds. If the container is not running, remaining checks are skipped and default to false. The role determines which Slurm service is checked; unmanaged nodes (see IsManaged) get none. clusterName is used to select the cluster network IP.

type NodeStatus 

type NodeStatus struct {
	Name    string      `json:"name"`    // DNS-style name: "controller.dev"
	Role    config.Role `json:"role"`    // "controller", "db", "api", "submitter", "worker"
	Managed bool        `json:"managed"` // sind manages Slurm on the node (see IsManaged)
	Health  *NodeHealth `json:"health"`  //nolint:revive // nested health is intentional
}

NodeStatus combines node identity with health information.

type NodeSummary 

type NodeSummary struct {
	Container string      `json:"container"` // Docker container name
	Cluster   string      `json:"cluster"`   // cluster name
	Role      config.Role `json:"role"`
	FQDN      string      `json:"fqdn"` // DNS name
	IP        string      `json:"ip"`   // container IP on cluster network
	State     State       `json:"status"`
}

NodeSummary holds summary information about a node in a sind cluster.

func GetAllNodes added in v0.9.0 

func GetAllNodes(ctx context.Context, client *docker.Client, realm string) ([]*NodeSummary, error)

GetAllNodes lists all nodes across all clusters in the realm.

func GetNodes 

func GetNodes(ctx context.Context, client *docker.Client, realm, clusterName string) ([]*NodeSummary, error)

GetNodes lists all nodes in the named cluster.

type RealmSummary added in v0.9.0 

type RealmSummary struct {
	Name     string `json:"name"`
	Clusters int    `json:"clusters"`
}

RealmSummary holds summary information about a sind realm.

func GetRealms added in v0.9.0 

func GetRealms(ctx context.Context, client *docker.Client) ([]*RealmSummary, error)

GetRealms lists all sind realms by querying Docker for containers with the sind.realm label. Containers are grouped by realm, and clusters are counted per realm.

type Resources 

type Resources struct {
	Containers    []docker.ContainerListEntry
	Network       docker.NetworkName
	NetworkExists bool
	Volumes       []docker.VolumeName
}

Resources holds the Docker resources belonging to a cluster.

func ListClusterResources 

func ListClusterResources(ctx context.Context, client *docker.Client, realm, clusterName string) (*Resources, error)

ListClusterResources discovers all Docker resources belonging to the named cluster. Containers are found by label filter; network and volumes are checked by name convention and skipped when their labels name another realm or cluster. The three lookups run concurrently.

type RunConfig 

type RunConfig struct {
	Realm           string      // realm name (e.g. "sind")
	ClusterName     string      // cluster name
	ShortName       string      // node hostname: "controller", "worker-0"
	Role            config.Role // "controller", "db", "api", "submitter", "worker"
	Image           string      // container image
	CPUs            int         // CPU limit
	Memory          string      // memory limit (e.g. "2g")
	TmpSize         string      // /tmp tmpfs size (e.g. "1g")
	SlurmVersion    string      // slurm version for labels (optional)
	DNSIP           string      // mesh DNS container IP (optional)
	DataHostPath    string      // host path for data volume (empty = use docker volume)
	DataMountPath   string      // mount point for data (default: /data)
	Managed         bool        // sind manages Slurm on the node: its configuration and daemon (see IsManaged)
	SharedState     bool        // mount the shared slurmctld state volume (controllers of a backup pair)
	ContainerNumber int         // 1-based compose container instance number
	CapAdd          []string    // extra Linux capabilities (e.g. "SYS_ADMIN")
	CapDrop         []string    // dropped Linux capabilities
	Devices         []string    // host devices to expose (e.g. "/dev/fuse")
	SecurityOpt     []string    // extra security options

	// CVMFS is how the node mounts /cvmfs: config.StorageVolume for the
	// plugin volume, config.StorageHostPath for the host's /cvmfs, or empty
	// for none (see DetectCVMFS).
	CVMFS config.StorageType

	// MeshNetwork is the realm's mesh network, which the node joins next to
	// its cluster network; CreateNode sets it from its mesh.Manager.
	MeshNetwork docker.NetworkName

	// Users are the cluster users and groups. The node records them in its
	// labels, and with users it mounts the home volume at HomeMountPath.
	Users LinuxUsers
	// AddUsers creates the users and groups on the node (see
	// nodeGetsUsers).
	AddUsers bool

	// Identity is the cluster's identity mode. With clientIds, munge is
	// masked and the node mounts no munge volume.
	Identity config.IdentityMode
	// NSSSlurm switches the node's passwd and group lookups to nss_slurm
	// first: managed workers with identity nssSlurm or clientIds.
	NSSSlurm bool

	// StoragePass is, on a managed db node, the StoragePass the slurmdbd
	// section sets: the password of slurmdbd's MariaDB account, which
	// authenticates with unix_socket without one (see accountingSQL).
	StoragePass string

	// TaskAffinity is set on managed workers of a cluster whose slurm.conf
	// enables task/affinity (see slurm.TaskAffinity): they get
	// TaskAffinityCapability.
	TaskAffinity bool
}

RunConfig holds the parameters needed to build docker run arguments for creating a node container.

func NodeRunConfigs 

func NodeRunConfigs(cfg *config.Cluster, realm, dnsIP, slurmVersion string, cvmfs config.StorageType) []RunConfig

NodeRunConfigs builds RunConfig entries for all nodes in the cluster config. Worker nodes are indexed sequentially across all worker groups. In an unmanaged cluster every node is unmanaged; otherwise only workers, db and api nodes with managed: false are. cvmfs is the backend every node mounts CVMFS with (see DetectCVMFS), empty for none. The identity mode decides which nodes get the cluster users and groups (see nodeGetsUsers) and which resolve them with nss_slurm. Managed workers get TaskAffinity when the slurm.conf sind generates enables task/affinity.

type ServiceHealth added in v0.7.0 

type ServiceHealth map[probe.Service]bool

ServiceHealth maps a readiness-check service to its health status.

type State added in v0.2.0 

type State string

State represents the state of a cluster or node. 

const (
	StateRunning State = "running"
	StateStopped State = "stopped"
	StatePaused  State = "paused"
	StateMixed   State = "mixed"   // cluster: nodes in different states
	StateEmpty   State = "empty"   // cluster: no nodes exist
	StateUnknown State = "unknown" // node: unrecognised container state
)

Possible cluster/node states.

type Status 

type Status struct {
	Name         string         `json:"name"`
	SlurmVersion string         `json:"slurm_version"`
	State        State          `json:"status"`
	Nodes        []*NodeStatus  `json:"nodes"`
	Network      *NetworkHealth `json:"network"`
	Mounts       []MountPoint   `json:"mounts"`
}

Status holds the full status of a sind cluster.

func GetStatus 

func GetStatus(ctx context.Context, client *docker.Client, realm, clusterName string) (*Status, error)

GetStatus returns the full status of a cluster, aggregating node, network, and volume health information.

type Summary added in v0.2.0 

type Summary struct {
	Name         string `json:"name"`
	SlurmVersion string `json:"slurm_version"`
	State        State  `json:"status"`
	NodeCount    int    `json:"nodes"`
	Submitters   int    `json:"submitters"`
	Controllers  int    `json:"controllers"`
	DBs          int    `json:"dbs"`
	APIs         int    `json:"apis"`
	Workers      int    `json:"workers"`
}

Summary holds summary information about a sind cluster.

func GetClusters 

func GetClusters(ctx context.Context, client *docker.Client, realm string) ([]*Summary, error)

GetClusters lists all sind clusters by querying Docker for containers with the sind.cluster label. Containers are grouped by cluster name.

type VolumeSummary 

type VolumeSummary struct {
	Name   string `json:"name"`
	Driver string `json:"driver"`
}

VolumeSummary holds summary information about a sind volume.

func GetVolumes 

func GetVolumes(ctx context.Context, client *docker.Client, realm string) ([]*VolumeSummary, error)

GetVolumes lists all sind-related Docker volumes.

type VolumeType added in v0.7.0 

type VolumeType string

VolumeType identifies a cluster volume kind. 

const (
	VolumeConfig VolumeType = "config"
	VolumeMunge  VolumeType = "munge"
	VolumeData   VolumeType = "data"
	// VolumeState holds the slurmctld StateSaveLocation shared by the
	// primary and backup controller. Only clusters with a backup controller
	// have it.
	VolumeState VolumeType = "state"
	// VolumeHome holds the home directories of the cluster users. Only
	// clusters with users have it.
	VolumeHome VolumeType = "home"
)

Cluster volume types.

type WorkerAddOptions 

type WorkerAddOptions struct {
	ClusterName string
	Count       int // 0 or less adds one worker
	Image       string
	CPUs        int
	Memory      string
	TmpSize     string
	Unmanaged   bool
	Pull        bool // pull Image first; needs Image
	// Wait, when positive, limits how long WorkerAdd waits for the new
	// workers to become ready, counted as for Create's config.Cluster.Wait.
	Wait        time.Duration
	CapAdd      []string
	CapDrop     []string
	Devices     []string
	SecurityOpt []string
}

WorkerAddOptions holds the parameters for adding worker nodes to a cluster. The image, resources and privileges are the new workers' shape: each one left at its zero value is taken from the cluster's newest worker (see workerShape). For the lists CapAdd, CapDrop, Devices and SecurityOpt, the zero value is nil: an empty list that is not nil replaces the newest worker's list too, so the new workers get none.

func (WorkerAddOptions) Check added in v0.11.0 

func (o WorkerAddOptions) Check() error

Check reports the first option WorkerAdd cannot act on: a negative CPU count, a memory or /tmp size sind cannot read, Pull without Image, an unknown capability, a device path that is not absolute or an unknown security option. It calls docker for none of them, so that the CLI can reject them as usage errors before it takes the realm lock.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.