Documentation
¶
Overview ¶
Package fleet holds the machines registry: the one file that says what each machine in the fleet IS, and therefore what may be placed on it.
THE LOCK (Glenn, 2026-09-18): runner hosts are CI-only. No card, no probe and no load may be placed on a machine that serves the merge group's shards. A card and a CI shard on one host make the shard slow, the gate red and the queue stop -- this was measured all through 2026-09-17, when the merge group's darwin legs starved behind the coordination bench's own children and nothing landed for twenty minutes at a time.
The registry exists because a bench name reached a machine as a bare string: `--bench batman` was a hostname the fill loop would happily ssh to, and nothing in the tools knew that batman is six CI runners and not a card bench. Now a bench name is RESOLVED -- every verb that puts work on a machine asks the registry first, and a name whose roles lack `bench` is refused by name, with the reason and the remedy on the line.
The file is data, tab separated, kept in git beside the lanes file:
name<TAB>ssh<TAB>os/arch<TAB>roles<TAB>seat<TAB>cores<TAB>notes
A line whose fields do not parse refuses the file: a truncated or unknown-role row is not a machine, and guessing the rest would let a card through. The runner/bench lock is per row (#2031): a machine that is both runner and bench without `allow-shared=<YYYY-MM-DD> <why>` is recorded as lock-failed, not as a bench, and does not stop its neighbours.
Index ¶
- Constants
- func AppendCertificate(path string, c Certificate) error
- func BuildVersion(out string) string
- func Certified(certs []Certificate, machine, class, build, hash string) bool
- func Certify(in CertifyInput) int
- func IsValidBuildVersion(tok string) bool
- func RunBenchesConcurrently(names []string, fn func(name string) error) error
- func Stale(certs []Certificate, machine, class, build, hash string, now time.Time, ...) bool
- func StandardHash(standard string, loads []Workload) (string, error)
- func Status(in StatusInput) int
- type Certificate
- type CertifyInput
- type Clock
- type Forge
- type Machine
- type Refusal
- type Registry
- func (r *Registry) BenchNames() []string
- func (r *Registry) CertifiedBenchNames() []string
- func (r *Registry) LockFailed() []Machine
- func (r *Registry) Lookup(name string) (Machine, bool)
- func (r *Registry) Machines() []Machine
- func (r *Registry) Path() string
- func (r *Registry) RequireBench(name string) error
- func (r *Registry) SharedLockRefusal(m Machine) *Refusal
- func (r *Registry) WithRole(role string) []Machine
- type Remote
- type RunnerStatus
- type StatusInput
- type Workload
Constants ¶
const ( VerdictOK = "OK" VerdictFail = "FAIL" VerdictWarn = "WARN" )
The verdicts a certificate may carry. They are tokens, not prose.
WARN is a NOTE ON A PASS, not a failure: a workload marked `report: yes` measures something worth watching that has never stopped a card -- the runners' `_diag` logs were 15.7 GB across the fleet on 2026-09-18 and nothing was broken by it. A WARN is written, counted and printed, and it neither fails the run nor withholds the certificate, because a check that cries wolf is a check people learn to pass over.
const ( ForgeRunners = "runners" ForgeRegistry = "registry" )
The two workloads that are not questions for a machine at all. Whether the forge says a runner host's runners are online, and whether the registry's roles match what the forge is actually running, are questions for the forge and the registry; asking the machine would only tell us what the machine believes. On 2026-09-18 space was serving sixteen merge-group runners while machines.tsv said it was `bench,services` -- the machine knew, the forge knew, and the file that decides where cards go did not.
const ( // RoleBench says cards, probes and load may be placed here. It is the only role that // permits work, and every other role is silent about work. RoleBench = "bench" // RoleRunner says the machine serves the merge group's CI shards. RoleRunner = "runner" // RoleCoordination says a friend's own window lives here. RoleCoordination = "coordination" // RoleServices says the stack lives here: Loki, Grafana, Redis. RoleServices = "services" )
The roles a machine may carry. They are a SET, not a rank: a machine is every one of the things its line says it is.
const ( ReasonRunnerHost = "runner-host" // it serves the merge group's shards ReasonCoordinationHost = "coordination-host" // a friend's window lives there ReasonServicesHost = "services-host" // the stack lives there and nothing else may ReasonNotABench = "not-a-bench" // it carries no role that permits work ReasonUnknown = "unknown-machine" // the registry does not carry the name )
The reasons a machine may not take work. They are tokens, not prose, so a loop reading the line can branch on one and a person reading it learns the same thing.
const BuildScript = "# nova-certify workload build\nnova-merge version 2>&1 || true\n"
BuildScript is the one question every certification asks first: what build is installed here? It is exported because the fill asks the same question, of the same machines, and two spellings of it would be two answers.
const DefaultCertifyTimeout = 10 * time.Minute
DefaultCertifyTimeout bounds ONE workload on ONE machine. A build inside a cold wall on a small bench is minutes, not seconds; the whole run is bounded per workload rather than once, so one slow machine cannot eat the fleet's budget.
const DefaultGo = "go1.26.5"
DefaultGo is the toolchain a bench's Go workloads ask for. It tracks go.mod's `go` line; when go.mod moves, this moves with it and every certificate written under the old workloads expires on its own, because the workload bytes are in the standard hash.
const DefaultMaxAge = 24 * time.Hour
DefaultMaxAge is how long a certificate stands before it is stale even though nothing moved. Twenty-four hours: a machine drifts by the hand of whoever last logged into it, and the whole fleet was found drifted on the morning of 2026-09-18 with nothing in the tools having changed at all.
const EvidenceCap = 240
EvidenceCap is the most of a machine's answer that reaches a line or a row. A workload that writes a screenful must not be able to fill the certificates file.
Variables ¶
This section is empty.
Functions ¶
func AppendCertificate ¶
func AppendCertificate(path string, c Certificate) error
AppendCertificate adds one row. Certify appends and never rewrites: the file is the record of what was run, and a machine that failed and was repaired is current on its newest row while the failure it had stays readable.
func BuildVersion ¶
BuildVersion reads the version token out of a `nova-merge version` output. It extracts ONLY from a recognized version-line shape, and ONLY when that shape's tool field names buildTool ("nova-merge"): the full buildinfo.Parse line (`<tool> <version> <goos>/<goarch> <go version> [key=value ...]`), or one of the two shorter standalone forms nova-merge falls back to when it has no platform line to give (`<tool> <version>` or `<tool> <version> <goos>/<goarch>`). It never scans an arbitrary line for any token IsValidBuildVersion happens to accept -- a diagnostic line can carry a 12-to-40 character hex substring that looks like a revision but names no build (`fatal: bad object deadbeef1234` is not a build, it is a git error that happens to contain a hex-shaped word, and it is four fields, not two or three, so none of the recognized shapes match it), so a line that is not one of the three recognized shapes is rejected outright, regardless of what its words spell. Requiring the tool field closes the shorter shapes too: a two-field diagnostic like `fatal deadbeef1234` has no tool field naming nova-merge at all, and a three-field one like `fatal deadbeef1234 x` was previously accepted on nothing more than "the third field contains a slash" -- `x` has none, but a diagnostic that happened to print `error: bad/ref` would have (Stella's HOLD 6 on #2478). SSH banners, diagnostic messages, and error text are rejected the same way. If no line matches a recognized shape naming buildTool, it returns "".
func Certified ¶
func Certified(certs []Certificate, machine, class, build, hash string) bool
Certified is the whole currency rule, and it is the one pulse.Fill asks before a card reaches a machine: the named machine has a row for this class whose verdict is OK, whose build is the build installed there NOW, and whose standard hash is the standard NOW. The newest matching row wins, so a bench that failed and was repaired is certified and a bench that passed and then drifted is not.
func Certify ¶
func Certify(in CertifyInput) int
Certify runs every workload of every named machine's roles, writes one certificate row each, and answers 0 when all passed, 1 when any failed, 2 when it refused to start.
func IsValidBuildVersion ¶
IsValidBuildVersion reports whether tok is a recognized build version format (a semantic version tag like v0.17.0, a vcs timestamp-revision like 20260921145725-c1670c8884cd[-dirty], a 12-to-40 character hex revision, or devel). Diagnostic strings, error messages, and SSH banners are rejected.
func RunBenchesConcurrently ¶
RunBenchesConcurrently runs fn concurrently for each bench name in names. Wall time is under 1.5x the slowest fn call. Returns one error naming every bench whose fn returned an error.
func Stale ¶
func Stale(certs []Certificate, machine, class, build, hash string, now time.Time, maxAge time.Duration) bool
Stale says whether the newest certificate for one machine and class is missing, failed, written under another build or standard, or simply older than maxAge. It is the one place "current" is decided for the trigger, and it agrees with Certified by construction: a certificate Certified accepts is stale only when it has aged out.
func StandardHash ¶
StandardHash is what makes a certificate expire: the sha256 over the provisioning standard file and over every workload's bytes, in class order. Either half moving is a new hash and so a fleet with no current certificates, which is the honest state.
func Status ¶
func Status(in StatusInput) int
Status prints the current certificate per machine and class and answers 1 when any machine carries a stale or failed class, 0 when the fleet is certified, 2 on a refusal.
The build is NOT read from the machines here: --status is the record, and reading twenty machines to print a file is how a status verb becomes something nobody runs. A row whose build no longer matches the fleet shows as stale on the next real run.
Types ¶
type Certificate ¶
type Certificate struct {
Machine string
Build string
Hash string
Class string
Verdict string
Evidence string
At time.Time
}
Certificate is one row: what machine did what work, under what build and what standard, with what it said, and when.
func ReadCertificates ¶
func ReadCertificates(path string) ([]Certificate, error)
ReadCertificates reads the file whole. A file nobody has written yet is no certificates, which is not an error: the first run of certify creates it.
func (Certificate) Row ¶
func (c Certificate) Row() string
Row renders one certificate as the tab-separated line the file holds. Evidence goes through oneline.Escape, so a tab or a newline in what a machine said cannot become a column or a row.
type CertifyInput ¶
type CertifyInput struct {
Machines string // the machines registry
Only string // one machine name; empty with All false is a refusal
All bool // every machine in the registry, under its own roles
Workloads []Workload // the set to run; nil takes StandardWorkloads
Certs string // the certificates file appended to
Hash string // the standard hash every row carries
Build string // the build every row carries; empty asks each machine its own
Bin string // the install directory the release puts the tools in; "" is $HOME/.local/bin
Repo string // the repository the forge is asked about
IfStale bool // true skips a machine whose every class is current
MaxAge time.Duration
Log io.Writer // the structured event stream; nil writes none
Timeout time.Duration
DryRun bool
Remote Remote
Forge Forge
Now func() time.Time
Stdout io.Writer
Stderr io.Writer
}
CertifyInput is the verb apart from flag parsing, so a test drives a whole run against fakes and the release verb drives the same engine through its own ssh seam.
type Forge ¶
type Forge interface {
Runners(repo string) ([]RunnerStatus, error)
}
Forge answers the runner list for one repository.
type Machine ¶
type Machine struct {
Name string // the bench name every verb's --bench takes
SSH string // the ssh target; an alias in ~/.ssh/config or a host
OS string // linux, darwin, windows
Arch string // x64, amd64, arm64
Roles []string // sorted, unique, every one from knownRoles
Seat string // the nova-secrets seat on the machine; "" when it carries none
Cores int // whole cores, as the machine counts them
Notes string // free text; "" when the line said `-`
Line int // the line of the file this came from, for a refusal that can be found
}
Machine is one line of the registry.
func (Machine) AllowShared ¶
AllowShared reads the dated exception out of the notes: `allow-shared=<YYYY-MM-DD> <why>`. It returns the date, the reason and whether one is there at all.
func (Machine) Certified ¶
Certified reads the certification out of the notes: `certified=<YYYY-MM-DD>`, with anything after the date free text (the report it was written from, usually). It answers the date and whether the machine carries one at all.
type Refusal ¶
Refusal is what a verb prints when a machine may not take the work it was handed. It carries the machine, the reason as a token, and the remedy written the way it would be typed.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry is the whole machines file, read and validated.
func ReadRegistry ¶
ReadRegistry reads the machines file. A line whose fields do not parse refuses the whole file -- that is not a machine, and the half that parsed is the half that would let a card through. The runner/bench lock is per row: a shared machine without the dated note is recorded as lock-failed and its neighbours still load (#2031).
func (*Registry) BenchNames ¶
BenchNames is every name a verb may put work on, in file order.
func (*Registry) CertifiedBenchNames ¶
CertifiedBenchNames is THE POOL: every machine that may take work AND has been certified, in file order. It is what a verb fills when the caller names no bench, so a bench joins the fleet's work by its registry row and by nothing else -- no list in any Go file, no edit to any tool, no release.
func (*Registry) LockFailed ¶
LockFailed is every well-formed row that fails the runner/bench lock, in file order. Those machines are not benches: fill disables them by name and their neighbours keep dealing.
func (*Registry) RequireBench ¶
RequireBench is THE guard. It answers nil when the named machine may take work, and a *Refusal naming the reason and the remedy when it may not -- an unknown name, a CI runner host, the coordination bench, a services host, or a shared machine without the dated note.
Every verb that reaches a machine calls this before it reaches one. The check is on the NAME, before any ssh, so a refused machine is never even connected to.
func (*Registry) SharedLockRefusal ¶
SharedLockRefusal is the named refusal for a row that is both runner and bench with no dated allow-shared note.
type Remote ¶
Remote runs one script on one machine and answers with its combined output. The production one is `ssh <target> bash -s` with the script on stdin -- never a command line pasted together, because the script carries heredocs and quoting nobody could check.
type RunnerStatus ¶
type RunnerStatus struct {
Name string `json:"name"`
Status string `json:"status"` // online, offline
}
RunnerStatus is one self-hosted runner as the forge reports it.
type StatusInput ¶
type StatusInput struct {
Machines string
Certs string
Workloads []Workload
Hash string
MaxAge time.Duration
Now func() time.Time
Stdout io.Writer
Stderr io.Writer
}
StatusInput is `fleet certify --status`: the reading verb. It touches no machine and takes no forge -- it is the record, read back, one line per machine and class.
type Workload ¶
type Workload struct {
Class string // the name on every line and every row; the file's name
Roles []string // the roles this workload applies to, sorted
Expect *regexp.Regexp // a pass is this matching the machine's output
Wall bool // true runs the body inside nova-sandbox
Reads []string // the wall's readable roots; `$HOME` is the machine's own
Forge string // ForgeRunners or ForgeRegistry, or "" for a workload the machine runs
Report bool // true makes a failure a WARN: measured and never a refusal
Body string // the command run ON the machine
Source string // where this workload was read from, for a refusal that can be found
// contains filtered or unexported fields
}
Workload is one representative piece of work: what roles it applies to, what a pass looks like, whether it runs inside the wall, and the body the machine runs.
func ParseWorkload ¶
ParseWorkload reads one card: front matter of `key: value` lines, a blank line, then the body. The CLASS is the file's name and never a key, so two cards cannot claim one class and a person looking for a class knows which file to open.
func ReadWorkloads ¶
ReadWorkloads reads an override directory: every `<class>.card` in it, whole and validated. A directory holding one broken card is refused entire, for the registry's own reason -- the half that reads is the half that lets a machine through.
func StandardWorkloads ¶
StandardWorkloads is the set that ships with the tool: the one a run takes when `--workloads` names no directory.
Source Files
Directories