fleet

package
v1.2.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package fleet holds the machines registry: the one file that says what each machine in the fleet IS, and therefore what may be placed on it.

THE LOCK (the coordinator, 2026-09-18): runner hosts are CI-only. No card, no probe and no load may be placed on a machine that serves the merge group's shards. A card and a CI shard on one host make the shard slow, the gate red and the queue stop -- this was measured all through 2026-09-17, when the merge group's darwin legs starved behind the coordination bench's own children and nothing landed for twenty minutes at a time.

The registry exists because a bench name reached a machine as a bare string: `--bench example-host` was a hostname the fill loop would happily ssh to, and nothing in the tools knew that example-host is six CI runners and not a card bench. Now a machine name is RESOLVED: a verb asks the registry for the row, and a name the registry does not carry is refused by name, with the reason and the remedy on the line.

The file is data, tab separated, kept in git beside the lanes file:

name<TAB>ssh<TAB>os/arch<TAB>roles<TAB>seat<TAB>cores<TAB>notes

A line whose fields do not parse refuses the file: a truncated or unknown-role row is not a machine, and guessing the rest would let a card through. The runner/bench lock is per row: a machine that is both runner and bench without `allow-shared=<YYYY-MM-DD> <why>` is recorded as lock-failed, not as a bench, and does not stop its neighbours.

Index

Constants

View Source
const (
	VerdictOK   = "OK"
	VerdictFail = "FAIL"
	VerdictWarn = "WARN"
)

The verdicts a certificate may carry. They are tokens, not prose.

WARN is a NOTE ON A PASS, not a failure: a workload marked `report: yes` measures something worth watching that has never stopped a card -- the runners' `_diag` logs can reach 15.7 GB across the fleet and nothing is broken by it. A WARN is written, counted and printed, and it neither fails the run nor withholds the certificate, because a check that cries wolf is a check people learn to pass over.

View Source
const (
	ForgeRunners  = "runners"
	ForgeRegistry = "registry"
)

The two workloads that are not questions for a machine at all. Whether the forge says a runner host's runners are online, and whether the registry's roles match what the forge is actually running, are questions for the forge and the registry; asking the machine would only tell us what the machine believes. A host serving merge-group runners while machines.tsv still calls it `bench,services` is the failure mode: the machine knows, the forge knows, and the file that decides where cards go does not.

View Source
const (
	// RoleBench says cards, probes and load may be placed here. It is the only role that
	// permits work, and every other role is silent about work.
	RoleBench = "bench"
	// RoleRunner says the machine serves the merge group's CI shards.
	RoleRunner = "runner"
	// RoleCoordination says a friend's own window lives here.
	RoleCoordination = "coordination"
	// RoleServices says the stack lives here: Loki, Grafana, Redis.
	RoleServices = "services"
)

The roles a machine may carry. They are a SET, not a rank: a machine is every one of the things its line says it is.

View Source
const BuildScript = "# nova-certify workload build\nnova-merge version 2>&1 || true\n"

BuildScript is the one question every certification asks first: what build is installed here? It is exported because the fill asks the same question, of the same machines, and two spellings of it would be two answers.

View Source
const DefaultCertifyTimeout = 10 * time.Minute

DefaultCertifyTimeout bounds ONE workload on ONE machine. A build inside a cold wall on a small bench is minutes, not seconds; the whole run is bounded per workload rather than once, so one slow machine cannot eat the fleet's budget.

View Source
const DefaultGo = "go1.26.5"

DefaultGo is the toolchain a bench's Go workloads ask for. It tracks go.mod's `go` line; when go.mod moves, this moves with it and every certificate written under the superseded workloads expires on its own, because the workload bytes are in the standard hash.

View Source
const DefaultMaxAge = 24 * time.Hour

DefaultMaxAge is how long a certificate stands before it is stale even though nothing moved. Twenty-four hours: a machine drifts by the hand of whoever last logged into it, and an entire fleet can be found drifted with nothing in the tools having changed at all.

View Source
const EvidenceCap = 240

EvidenceCap is the most of a machine's answer that reaches a line or a row. A workload that writes a screenful must not be able to fill the certificates file.

View Source
const ReasonUnknown = "unknown-machine"

ReasonUnknown is the reason token a refusal carries when the registry does not carry the name. It is a token, not prose, so a loop reading the line can branch on it and a person reading it learns the same thing.

Variables

This section is empty.

Functions

func AppendCertificate

func AppendCertificate(path string, c Certificate) error

AppendCertificate adds one row. Certify appends and never rewrites: the file is the record of what was run, and a machine that failed and was repaired is current on its newest row while the failure it had stays readable.

func BuildVersion

func BuildVersion(out string) string

BuildVersion reads the version token out of a `nova-merge version` output. It extracts ONLY from a recognized version-line shape, and ONLY when that shape's tool field names buildTool ("nova-merge"): the full buildinfo.Parse line (`<tool> <version> <goos>/<goarch> <go version> [key=value ...]`), or one of the two shorter standalone forms nova-merge falls back to when it has no platform line to give (`<tool> <version>` or `<tool> <version> <goos>/<goarch>`). It never scans an arbitrary line for any token IsValidBuildVersion happens to accept -- a diagnostic line can carry a 12-to-40 character hex substring that looks like a revision but names no build (`fatal: bad object deadbeef1234` is not a build, it is a git error that happens to contain a hex-shaped word, and it is four fields, not two or three, so none of the recognized shapes match it), so a line that is not one of the three recognized shapes is rejected outright, regardless of what its words spell. Requiring the tool field closes the shorter shapes too: a two-field diagnostic like `fatal deadbeef1234` has no tool field naming nova-merge at all, and a three-field one like `fatal deadbeef1234 x` is not accepted on "the third field contains a slash" alone -- `x` has none, and a diagnostic that prints `error: bad/ref` is held to the full goos/goarch shape, not the slash alone. SSH banners, diagnostic messages, and error text are rejected the same way. If no line matches a recognized shape naming buildTool, it returns "".

func Certified

func Certified(certs []Certificate, machine, class, build, hash string) bool

Certified is the whole currency rule, and it is the one pulse.Fill asks before a card reaches a machine: the named machine has a row for this class whose verdict is OK, whose build is the build installed there NOW, and whose standard hash is the standard NOW. The newest matching row wins, so a bench that failed and was repaired is certified and a bench that passed and then drifted is not.

func Certify

func Certify(in CertifyInput) int

Certify runs every workload of every named machine's roles, writes one certificate row each, and answers 0 when all pass, 1 when any fail, 2 when it refuses to start.

func IsValidBuildVersion

func IsValidBuildVersion(tok string) bool

IsValidBuildVersion reports whether tok is a recognized build version format (a semantic version tag like v0.17.0, a vcs timestamp-revision like 20260921145725-c1670c8884cd[-dirty], a 12-to-40 character hex revision, or devel). Diagnostic strings, error messages, and SSH banners are rejected.

func Stale

func Stale(certs []Certificate, machine, class, build, hash string, now time.Time, maxAge time.Duration) bool

Stale says whether the newest certificate for one machine and class is missing, failed, written under another build or standard, or simply older than maxAge. It is the one place "current" is decided for the trigger, and it agrees with Certified by construction: a certificate Certified accepts is stale only when it has aged out.

func StandardHash

func StandardHash(standard string, loads []Workload) (string, error)

StandardHash is what makes a certificate expire: the sha256 over the provisioning standard file and over every workload's bytes, in class order. Either half moving is a new hash and so a fleet with no current certificates, which is the honest state.

Types

type Certificate

type Certificate struct {
	Machine  string
	Build    string
	Hash     string
	Class    string
	Verdict  string
	Evidence string
	At       time.Time
}

Certificate is one row: what machine did what work, under what build and what standard, with what it said, and when.

func ReadCertificates

func ReadCertificates(path string) ([]Certificate, error)

ReadCertificates reads the file whole. A file nobody has written yet is no certificates, which is not an error: the first run of certify creates it.

func (Certificate) Row

func (c Certificate) Row() string

Row renders one certificate as the tab-separated line the file holds. Evidence goes through oneline.Escape, so a tab or a newline in what a machine said cannot become a column or a row.

type CertifyInput

type CertifyInput struct {
	Machines  string     // the machines registry
	Only      string     // one machine name; empty with All false is a refusal
	All       bool       // every machine in the registry, under its own roles
	Workloads []Workload // the set to run; nil takes StandardWorkloads
	Certs     string     // the certificates file appended to
	Hash      string     // the standard hash every row carries
	Build     string     // the build every row carries; empty asks each machine its own
	Bin       string     // the install directory the release puts the tools in; "" is $HOME/.local/bin
	Repo      string     // the repository the forge is asked about
	IfStale   bool       // true skips a machine whose every class is current
	MaxAge    time.Duration
	Log       io.Writer // the structured event stream; nil writes none
	Timeout   time.Duration
	DryRun    bool
	Remote    Remote
	Forge     Forge
	Now       func() time.Time
	Stdout    io.Writer
	Stderr    io.Writer
}

CertifyInput is the verb apart from flag parsing, so a test drives a whole run against fakes and the release verb drives the same engine through its own ssh seam.

type Clock

type Clock = log.Clock

Clock is pkg/log's clock, re-exported here only so a caller need not import both.

type Forge

type Forge interface {
	Runners(repo string) ([]RunnerStatus, error)
}

Forge answers the runner list for one repository.

type Machine

type Machine struct {
	Name  string   // the bench name every verb's --bench takes
	SSH   string   // the ssh target; an alias in ~/.ssh/config or a host
	OS    string   // linux, darwin, windows
	Arch  string   // x64, amd64, arm64
	Roles []string // sorted, unique, every one from knownRoles
	Seat  string   // the nova-secrets seat on the machine; "" when it carries none
	Cores int      // whole cores, as the machine counts them
	Notes string   // free text; "" when the line said `-`
	Line  int      // the line of the file this came from, for a refusal that can be found
}

Machine is one line of the registry.

func (Machine) AllowShared

func (m Machine) AllowShared() (date, why string, ok bool)

AllowShared reads the dated exception out of the notes: `allow-shared=<YYYY-MM-DD> <why>`. It returns the date, the reason and whether one is there at all.

func (Machine) HasRole

func (m Machine) HasRole(role string) bool

HasRole says whether the machine carries one role.

func (Machine) RoleList

func (m Machine) RoleList() string

RoleList is the roles as the file writes them: comma separated, sorted.

type Refusal

type Refusal struct {
	Name   string
	Reason string
	Remedy string
}

Refusal is what a verb prints when a machine may not take the work it was handed. It carries the machine, the reason as a token, and the remedy written the way it would be typed.

func (*Refusal) Line

func (e *Refusal) Line(token string) string

Line is the one line a verb prints, under its own event token:

CERTIFY REFUSED bench=nobody reason=unknown-machine remedy="..."

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry is the whole machines file, read and validated.

func ReadRegistry

func ReadRegistry(path string) (*Registry, error)

ReadRegistry reads the machines file. A line whose fields do not parse refuses the whole file -- that is not a machine, and the half that parsed is the half that would let a card through. The runner/bench lock is per row: a shared machine without the dated note is recorded as lock-failed and its neighbours still load.

func (*Registry) Lookup

func (r *Registry) Lookup(name string) (Machine, bool)

Lookup finds one machine by name.

func (*Registry) Machines

func (r *Registry) Machines() []Machine

Machines is every machine, in file order.

func (*Registry) Path

func (r *Registry) Path() string

Path is the file this registry was read from, so a refusal can name it.

type Remote

type Remote interface {
	Run(ctx context.Context, target, script string) (string, error)
}

Remote runs one script on one machine and answers with its combined output. The production one is `ssh <target> bash -s` with the script on stdin -- never a command line pasted together, because the script carries heredocs and quoting nobody could check.

type RunnerStatus

type RunnerStatus struct {
	Name   string `json:"name"`
	Status string `json:"status"` // online, offline
}

RunnerStatus is one self-hosted runner as the forge reports it.

type Workload

type Workload struct {
	Class  string         // the name on every line and every row; the file's name
	Roles  []string       // the roles this workload applies to, sorted
	Expect *regexp.Regexp // a pass is this matching the machine's output
	Wall   bool           // true runs the body inside nova-sandbox
	Reads  []string       // the wall's readable roots; `$HOME` is the machine's own
	Forge  string         // ForgeRunners or ForgeRegistry, or "" for a workload the machine runs
	Report bool           // true makes a failure a WARN: measured and never a refusal
	Body   string         // the command run ON the machine
	Source string         // where this workload was read from, for a refusal that can be found
	// contains filtered or unexported fields
}

Workload is one representative piece of work: what roles it applies to, what a pass looks like, whether it runs inside the wall, and the body the machine runs.

func ParseWorkload

func ParseWorkload(source string, raw []byte) (Workload, error)

ParseWorkload reads one card: front matter of `key: value` lines, a blank line, then the body. The CLASS is the file's name and never a key, so two cards cannot claim one class and a person looking for a class knows which file to open.

func StandardWorkloads

func StandardWorkloads() ([]Workload, error)

StandardWorkloads is the set that ships with the tool: the one a run takes when `--workloads` names no directory.

func (Workload) AppliesTo

func (w Workload) AppliesTo(role string) bool

AppliesTo says whether this workload is run on a machine carrying one role.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL