nodeplane

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 11, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package nodeplane is the control plane's half of the node wire.

It holds what the server knows about each registered node, hands out commands to nodes that are long-polling for them, and collects the results. The Runner it exposes is the same server.Runner the in-process path implements, so the listener cannot tell whether the compute it is driving is a goroutine away or a continent away.

Index

Constants

This section is empty.

Variables

View Source 
var ErrNoNode = errors.New("nodeplane: no registered node can run this lease")

ErrNoNode means no registered node can run a lease.

Distinct from a launch failure: nothing was attempted, nothing is running, and the caller may release the lease without ambiguity. That certainty is the whole reason it is its own error.

View Source 
var ErrNotEntitled = errors.New("nodeplane: this node holds no command that entitles it to that")

ErrNotEntitled means a node asked for something no command it holds allows.

View Source 
var ErrRefused = errors.New("nodeplane: registration refused")

ErrRefused means a registration was understood and rejected on its merits.

PERMANENT, which is the distinction that matters: a node told this stops rather than retrying, so it must never carry a failure that could heal. A version mismatch and a foreign deployment qualify; a ledger that could not write the node row does not — that is an outage.

View Source 
var ErrSuperseded = errors.New("nodeplane: another process is registered as this node")

ErrSuperseded means the request came from a node process that is no longer the registered one.

TWO HOSTS UNDER ONE NAME is what this catches, and it is the shape a copied certificate bundle produces. Both authenticate — the certificate is genuine — and both claim the same node. Without this the control plane's answer to "whose compute is this" is whichever host polled last, and each host's reconciliation reasons about leases the other one owns.

View Source 
var ErrTakeCustody = errors.New("nodeplane: this launch's lease is now the node's to hold")

ErrTakeCustody answers a result for a launch the plane stopped waiting for.

THE REPORT IS NOT REJECTED, IT IS REDIRECTED. The node did the work and told the truth, just later than the command timeout — by which point the plane had told the listener the lease was the node's. Answering 204 would leave the container running under a lease NOBODY renews, and the reaper resells that capacity a TTL later.

View Source 
var ErrUnregistered = errors.New("nodeplane: node is not registered")

ErrUnregistered means the node making a request is not known.

Functions

func BootstrapHandler 

func BootstrapHandler(log *slog.Logger, p *Plane, ca *wirecert.CA, opts ...HandlerOption) http.Handler

BootstrapHandler serves the two routes a machine that has never enrolled needs, and nothing else.

IT IS A SEPARATE HANDLER BECAUSE IT NEEDS A SEPARATE LISTENER. Neither route can require a certificate — a node deciding whether to trust this control plane must be able to read its authority first, and a machine asking to join has nothing to present — so the listener in front of this one admits strangers. Put that on the wire nodes work over and the two share a connection budget, which is a fleet an anonymous caller can take offline at a few requests a second. Here the blast radius of saturating it is that enrollment waits.

THE AUTHORITY IS A PARAMETER RATHER THAN AN OPTION. Without one, both routes can only answer 404, so a bootstrap handler that has none should not be constructible at all.

Nothing here grants anything. Reading the authority reveals what every handshake already presents; asking to join needs a join token, records a pending request, and waits for an operator to compare the fingerprint the node printed on its own console.

func Handler 

func Handler(log *slog.Logger, p *Plane, store LeaseStore, jit JITSource, opts ...HandlerOption) http.Handler

Handler serves the OPERATIONAL node wire: every route that acts for a node, and not one that a machine without a certificate can use.

THERE IS NO UNAUTHENTICATED ROUTE HERE, and that absence is what lets the listener in front of this demand a certificate in the handshake (wirecert.ServerTLS). /v1/ca and /v1/enroll used to be registered below; they are BootstrapHandler's now, on a listener of their own, because a connection budget shared with callers who need not prove anything is a budget an anonymous caller can take from the fleet.

Every route that acts for a node is wrapped in forNode, and that is deliberate rather than tidy: the enforcement is visible in the routing table, so a route added without it is missing something a reader can SEE, instead of missing a check buried in a handler nobody re-reads.

func LoopbackOnly 

func LoopbackOnly(addr string) bool

LoopbackOnly reports whether an address is safe to serve the node wire on without TLS.

config.LoopbackAddr answers the same question for validation, and the two must agree: whether a node needs a certificate is decided by whether the listener will ask for one.

Until mTLS lands the node names itself in the path and nothing verifies that claim, so a listener reachable beyond this host would let anything on the network bind leases and take commands. Refusing is the only honest option; the alternative is a deployment that looks like it works and has no boundary at all.

Types

type BarrierStore 

type BarrierStore interface {
	// ComputeBarrierInForce reports the durable request, if a waiter made one.
	ComputeBarrierInForce(ctx context.Context) (alloc.ComputeBarrier, bool, error)
	// DropComputeBarrier removes a request that can no longer mean anything.
	DropComputeBarrier(ctx context.Context, id string) error
	// AdmissionGeneration is what the barrier's own generation is compared
	// against, so a resume voids it, and whether the deployment is sealed at it.
	AdmissionGeneration(ctx context.Context) (int64, bool, error)
	// Quiescence is the LEDGER barrier. The compute barrier only means something
	// once this holds nothing: until then a legitimate in-flight launch moves a
	// host's dispatch fence on every round and no run can ever complete.
	Quiescence(ctx context.Context) (alloc.Quiescence, error)
	// NodeFenceOf reads the epoch and dispatch generation an observation must be
	// taken against, captured BEFORE the host is asked.
	NodeFenceOf(ctx context.Context, node string) (alloc.NodeFence, bool, error)
	// RecordBarrierObservation stores one fenced answer, or ends that host's run.
	RecordBarrierObservation(ctx context.Context, obs alloc.BarrierObservation) error
	// InvalidateBarrierRun discards whatever a host had proved, because a new
	// incarnation is arriving and the ledger cannot see that yet.
	InvalidateBarrierRun(ctx context.Context, node string) error
	// InvalidateEveryBarrierRun does the same for an arrival that names no host
	// billet can identify — a loopback registration whose body would not decode.
	InvalidateEveryBarrierRun(ctx context.Context) error
	// ResolveQuarantineFor is reused rather than reimplemented: a barrier's
	// inventory is a real inventory, and feeding it through the one path that
	// already fences one frees quarantined capacity while the drain waits.
	ResolveQuarantineFor(ctx context.Context, node string, running []string, epoch int64) (int, error)
}

BarrierStore is the durable half of a compute barrier.

SEPARATE FROM Registrar because it is asked on a timer rather than on the command path, and because a plane without one simply never proves anything — which is the correct behaviour for the in-process and test wirings, not a degraded one.

type CachePolicy 

type CachePolicy interface {
	ActionsCacheAllowed(ctx context.Context, owner, repository string) (bool, error)
}

CachePolicy answers the kill switch for transparent Actions caching.

type Enrollments 

type Enrollments interface {
	// RequestEnrollmentWithToken records the request AND spends the credential
	// that authorised it in one transaction. Two calls would let a crash between
	// them burn a single-use token with no request to show for it, stranding the
	// machine it was minted for.
	RequestEnrollmentWithToken(
		ctx context.Context, name, fingerprint, csrPEM, token string,
	) (alloc.Enrollment, error)
	LookupEnrollment(ctx context.Context, name string) (alloc.Enrollment, bool, error)
}

Enrollments records machines asking to join and what was decided about them, and checks the credential that lets one ask at all.

type HandlerOption 

type HandlerOption func(*handler)

HandlerOption configures the wire.

func RequireClientCert 

func RequireClientCert() HandlerOption

RequireClientCert makes a verified certificate the source of a node's name.

WITHOUT IT THE PATH IS THE ONLY AUTHORITY, which is not authentication: any process that can reach the listener claims to be any node, binds its leases, takes its commands, and asks for a JIT registration — a credential that registers a runner against the organisation. A wire served without this refuses to bind anywhere but loopback.

With it the certificate decides, and a request whose path disagrees is rejected rather than reconciled.

func WithCachePolicy 

func WithCachePolicy(policy CachePolicy) HandlerOption

WithCachePolicy gives nodes the central interception kill switch.

func WithEnrollment 

func WithEnrollment(e Enrollments) HandlerOption

WithEnrollment lets a machine ask to join without already holding a certificate.

The alternative — and what billet did before — is that admission happens entirely out of band: an operator runs `billet ca issue` and copies a bundle to the machine. That works, and it is not discoverable: a node that is powered on and pointed at a control plane appears nowhere until somebody already knows it exists.

func WithRenewal 

func WithRenewal(ca *wirecert.CA) HandlerOption

WithRenewal lets a node replace its own certificate before it expires.

AUTHENTICATED BY THE CERTIFICATE BEING REPLACED, so this grants nothing: a host that can already act as a node asks to keep doing so. What it prevents is the cliff — a fleet enrolled on one afternoon whose certificates all expire on the same day a year later, with no warning louder than a log line and no way back except re-enrolling every machine by hand.

func WithRevocations 

func WithRevocations(r Revocations) HandlerOption

WithRevocations lets the wire refuse a credential an operator has taken back.

Without it a certificate is good until it expires, which for a decommissioned machine or a leaked key means up to a year of a host that can still be handed work — including a JIT credential that registers a runner against the organisation.

func WithTargetJIT added in v0.10.0 

func WithTargetJIT(sources map[string]JITSource) HandlerOption

WithTargetJIT gives the handler one credential-holding source per GitHub target, keyed by the target's config name.

A REGISTRATION IS MINTED WITH THE CREDENTIAL OF THE TIER'S TARGET, and the tier is what every route here already knows — from the lease it acts for, or the label it is asked about — so the target is resolved from the catalogue rather than taken from the request. With this set, the constructor's source serves only tiers the catalogue does not know; a tier whose target is not among these is refused rather than minted through some other owner's App.

func WithTrustBundle 

func WithTrustBundle(pem []byte) HandlerOption

WithTrustBundle sets every authority a node should accept, which during a rotation is more than one.

type JITRegistration 

type JITRegistration interface {
	Config() string
	RunnerName() string
	ID() int64
}

JITRegistration is a minted registration whose config is a credential.

type JITRunnerRecovery 

type JITRunnerRecovery struct {
	RunnerID int64
	Present  bool
	Busy     bool
}

JITRunnerRecovery reports whether an exact legacy registration is busy.

type JITSet 

type JITSet struct {
	ID   int
	Name string
}

JITSet is a scale set, as the wire needs it.

type JITSource 

type JITSource interface {
	Describe(ctx context.Context, name, group string) (*JITSet, []string, error)
	JITConfig(ctx context.Context, scaleSetID int, runnerName, workFolder string) (JITRegistration, error)
	RemoveRunner(ctx context.Context, runnerID int64, runnerName string) error
	RecoverRunner(ctx context.Context, runnerName string) (JITRunnerRecovery, error)
}

JITSource mints runner registrations. Held by the control plane alone.

The same shape internal/node.JITSource has, declared separately for the same reason LeaseStore is: the transport must not depend on the runtime it serves.

type LeaseStore 

type LeaseStore interface {
	Bind(ctx context.Context, leaseID string, epoch int64, node string) error
	Advance(ctx context.Context, leaseID string, epoch int64, to alloc.Phase) error
	Heartbeat(ctx context.Context, leaseID string, epoch int64) error
	MarkFailure(ctx context.Context, leaseID string, epoch int64, reason string) error
	Resize(ctx context.Context, leaseID string, epoch int64, instanceType string,
		vcpu int, memory config.ByteSize) error
	Release(ctx context.Context, leaseID string, epoch int64, outcome alloc.Phase) error
	Lease(ctx context.Context, leaseID string) (*alloc.Lease, error)
	// RecordCacheObservation writes what the node saw the cache do for a lease's
	// job, from alloc's closed vocabularies, fenced on the epoch.
	RecordCacheObservation(ctx context.Context, leaseID string, epoch int64,
		obs alloc.CacheObservation) error
	// MarkDeregistered records that a lease's GitHub runner registration has been
	// removed, so ActiveRunnerLeases stops counting it as a live runner. It is
	// monotonic and unfenced; deregistration is a fact about GitHub, not about who
	// holds the lease.
	MarkDeregistered(ctx context.Context, leaseID string) error
	LaunchedLeaseIDs(ctx context.Context, node string) (map[string]bool, error)
	// EndedLeaseNode is the host a lease's job was attributed to, from the
	// history that outlives the lease row; ErrLeaseNotFound when there is none.
	EndedLeaseNode(ctx context.Context, leaseID string) (string, error)
	// QuarantinedLeaseIDs are leases holding capacity for compute nobody has
	// accounted for. A node needs them to tell an orphan from a job whose
	// listener died while it was still running.
	QuarantinedLeaseIDs(ctx context.Context, node string) (map[string]bool, error)
}

LeaseStore is the ledger, as the node wire needs it.

Declared here rather than imported from internal/node so the transport does not depend on the runtime it serves — the two are on opposite sides of a process boundary. The shapes match because both describe the same allocator.

type Option 

type Option func(*Plane)

Option configures a Plane.

func WithBarrierStore 

func WithBarrierStore(s BarrierStore) Option

WithBarrierStore lets this plane answer a compute barrier.

func WithClock 

func WithClock(now func() time.Time) Option

WithClock replaces the clock, for tests.

func WithCommandTimeout 

func WithCommandTimeout(d time.Duration) Option

WithCommandTimeout bounds how long a launch waits for its result.

func WithPollTimeout 

func WithPollTimeout(d time.Duration) Option

WithPollTimeout sets the long-poll window a node is told to wait out.

It also sets how long silence has to last before a node is forgotten, and therefore how often Watch looks — see staleAfter. Shortening it in a test is what makes expiry observable without waiting out the real window.

func WithRegistrar 

func WithRegistrar(r Registrar) Option

WithRegistrar makes registration durable in the ledger as well as in memory.

func WithSites 

func WithSites(sites []config.SiteConfig) Option

WithSites declares the places this deployment has compute in and each site's authoritative storage backend.

THE CONTROL PLANE IS THE AUTHORITY, and this is the only place the rule can be enforced: a node names a site in ITS OWN config, on another machine, in a file with no reason to list the deployment's places, so it cannot check itself. Every remote claim arrives at Register.

Empty means the deployment has declared no sites, in which case a node claiming one is refused: there is nothing it could correctly mean.

A NAME IS KEYED EXACTLY AS DECLARED, and nothing here normalizes one. That is the whole rule rather than an omission: config validation refuses a site name with surrounding whitespace (see checkIdentityPadding), so the bytes in this map are the bytes an operator wrote and the bytes Register compares req.Site against. Trimming on one side of that comparison and not the other is what made a site declared as " home " authorise `tiers[].site: home` at load and then refuse every node reporting `home`, forever.

func WithTierCatalog 

func WithTierCatalog(tiers []config.Tier) Option

WithTierCatalog gives the plane the deployment's tiers.

HELD HERE RATHER THAN ON THE NODE, because a node with its own copy needs that copy to agree with the server's and nothing checks: a missing tier refuses the launch loudly, but a drifted `image:` runs the wrong image silently. The plane puts the shape on the launch command instead.

type Plane 

type Plane struct {
	// contains filtered or unexported fields
}

Plane tracks nodes and routes commands to them.

func New 

func New(log *slog.Logger, deployment string, leaseTTL time.Duration, opts ...Option) *Plane

New builds a Plane.

func (*Plane) AdoptOwnership 

func (p *Plane) AdoptOwnership(node, incarnation string, leaseIDs []string)

AdoptOwnership records that this process is responsible for leases the ledger already places on its node.

A CONTROL PLANE RESTART FORGETS EVERYTHING, and a superseded process then cannot finish. The sequence: a node is holding compute, the plane restarts, the node re-registers and adopts what it finds, a second host supersedes it — and the new plane never saw the launch, so it has no owner for that lease. The draining process is refused its own release, custody is never given up, and the drain runs forever.

The ledger knows what it forgot: a lease bound to this node and still open is this node's, and the process registering now is the one holding it.

func (*Plane) AdoptOwnershipWithInventory 

func (p *Plane) AdoptOwnershipWithInventory(
	node, incarnation string,
	leaseIDs []string,
	inventoryKnown bool,
)

AdoptOwnershipWithInventory atomically restores owners and records whether the registering process supplied a complete provider inventory.

func (*Plane) ArrivalWasUnreadable 

func (p *Plane) ArrivalWasUnreadable(ctx context.Context) error

ArrivalWasUnreadable discards what EVERY host had proved, for an arrival whose identity could not be established.

THE LOOPBACK CASE, AND ONLY IT. That wire requires no certificate, so a registration whose body will not decode — an unknown field from a node rolled ahead of this control plane, an oversized body — names nobody. Those are permanent refusals a node does not retry, so leaving every proof standing on one is the same defect as leaving one standing, minus the ability to say whose.

Over-invalidation is the point, and it costs one barrier round on what is by definition a single-machine deployment.

func (*Plane) ArrivingForRegistration 

func (p *Plane) ArrivingForRegistration(ctx context.Context, node string) error

ArrivingForRegistration discards what a host had PROVED to a compute barrier, and is what every registration path must call FIRST.

BEFORE THE SEMANTIC CHECKS, AND THAT ORDERING IS THE WHOLE POINT. A registration carries the host's own live inventory, and a PERMANENTLY refused one — zero capacity, a protocol range this build no longer overlaps, a site nobody declared — is still an authenticated host saying what it is running. Refusing the request does not make that claim untrue, and the node stops retrying, so a proof left standing there can stay standing forever. Rolling a control plane past an old node's protocol range is exactly that shape, and it is the operation during which somebody is most likely to run `local down`.

IT IS DELIBERATELY NOT SUBJECT TO "A REFUSAL CHANGES NOTHING", which governs the IN-MEMORY inventory beginRegistration clears. The two are restored differently: that one comes back only from a SUCCESSFUL registration, so clearing it on a doomed request leaves a live host's inventory unknown indefinitely, and a completion needs it. A barrier run comes back from the barrier loop asking again on its own cadence, with nobody involved — so discarding one that did not need discarding costs a single round.

SHARED BECAUSE THERE ARE TWO REGISTRATION PATHS AND ONLY ONE IS THE REAL ONE. `Plane.Register` is the in-process and test entry point; every node on the wire arrives through `handler.register`, which assembles the sequence itself so it can read the launched leases in between. An earlier version of this ran on `Plane.Register` alone and therefore never ran in production, and its test passed because the test called the wrong one.

func (*Plane) AskNodeForTest 

func (p *Plane) AskNodeForTest(ctx context.Context, name, barrierID string) error

AskNodeForTest runs one barrier round against one host.

Exported for tests only. The loop's own cadence is minutes wide by design, and a test that waited it out would be testing time.NewTimer.

func (*Plane) AuthorizeLease 

func (p *Plane) AuthorizeLease(node, incarnation, leaseID string) (bool, error)

AuthorizeLease is MayMutateLease that also says WHY it admitted: true when the process is the recorded owner of the lease, false when it is merely the node's current process with nothing recorded against the lease.

ONE SNAPSHOT, BECAUSE THE TWO ANSWERS DECIDE DIFFERENT THINGS. An admission by ownership needs no further check; an admission by membership alone must be followed by the ledger's word on where the lease is placed (or, for an ended lease, whose job it was). Answering the second question from a second acquisition of the lock let an owner recorded in between — a replacement adopting the lease — turn "current but ownerless" into "recorded", and the ledger check was skipped for a lease another host's compute was still on.

func (*Plane) BarrierLoop 

func (p *Plane) BarrierLoop(ctx context.Context)

BarrierLoop asks the fleet what it is running, for as long as somebody is waiting for an answer.

A TIMER FOR THE SAME REASON Watch IS ONE. `billet drain` runs in a separate process with no handle to this plane, so its request is a durable row and this is what observes it. Nothing else would: an idle sealed deployment dispatches no commands at all, which is exactly the state a drain is waiting out.

IT PROVES NOTHING BY ITSELF. Every answer is recorded against a fence captured before the question was asked, and the predicate that reads those records is alloc's. This loop only makes sure the question keeps being asked.

func (*Plane) BarrierTargetsForTest 

func (p *Plane) BarrierTargetsForTest() []string

BarrierTargetsForTest names the hosts a barrier round would ask, sorted.

SORTED RATHER THAN IN askFleet's ORDER, which is a map iteration and therefore no order at all — an assertion against it would pass or fail by chance.

Exported for tests only, and it exists because the skip it reveals is half of one rule: a host below the barrier version must not be SENT the command, and must be reported as unprovable. The two halves live in different packages, so without this an end-to-end test can only see the second and a mutant that deletes the first survives it.

func (*Plane) CheckIncarnation 

func (p *Plane) CheckIncarnation(name, claimed string) error

CheckIncarnation reports whether a request came from the current node process.

COMPATIBILITY IS SCOPED TO NODES THAT HAVE NOT CLAIMED ONE, not to the REQUEST: scoping it to the request would let an absent header bypass the check entirely, so an older node beside a current one would never send the header and both would take work as the same node forever.

So absence is accepted only while the registered node is also absent — a fleet mid-upgrade. Once a process has claimed an incarnation, every later request must carry it.

func (*Plane) CurrentIncarnationForTest 

func (p *Plane) CurrentIncarnationForTest(name string) string

CurrentIncarnationForTest reports the process a node's name currently resolves to on this plane, or "" for a node it does not know.

func (*Plane) EntitledToLaunch 

func (p *Plane) EntitledToLaunch(node, incarnation, leaseID string) (string, error)

EntitledToLaunch reports whether a node is currently executing a launch for this lease.

A REGISTERED NODE IS NOT AN ENTITLED ONE, and conflating the two left the JIT endpoint open to anything holding a node certificate. A registration proves which host you are; it says nothing about what work you were given. Without this, a compromised host could ask for runner registrations in a loop — for any scale set, under any name — and start runners that billet never escrowed capacity for, never tracked, and never tears down. That contradicts the one containment property the design claims: that compromising a compute host does not let it mint runners.

The lease id carries the entitlement because it is already in the runner name billet chooses (see provider.InstanceName), so a node can only ask for the registration belonging to the launch it was actually told to perform.

func (*Plane) ForgetForTest 

func (p *Plane) ForgetForTest(name string)

ForgetForTest drops a node, as a control-plane restart would.

Exported for tests only. It stages the one state a node cannot produce for itself: being unknown to a server that is still answering, which is what makes its next write fail with "register again" rather than with a transport error.

func (*Plane) ForgetLease 

func (p *Plane) ForgetLease(node, leaseID string)

ForgetLease drops the ownership record for a lease that has ended.

BOUNDED, because the alternative is one map entry per job for the life of the installation. A node that never goes quiet is never expired, so nothing else would ever remove them.

func (*Plane) MayMutateLease 

func (p *Plane) MayMutateLease(node, incarnation, leaseID string) error

MayMutateLease reports whether this process may change a lease's fate.

THE CURRENT PROCESS, OR THE ONE THAT WAS GIVEN THE LAUNCH. Anything else is a host acting on work it was never handed — which, between a superseded incarnation and its replacement, means releasing capacity that another host's container is still using.

func (*Plane) NewRunner 

func (p *Plane) NewRunner() *Runner

NewRunner returns the plane's server.Runner.

func (*Plane) Nodes 

func (p *Plane) Nodes() []string

Nodes reports the registered node names, for diagnostics.

func (*Plane) OwnerOfLease 

func (p *Plane) OwnerOfLease(leaseID string) (RequestOwner, bool)

OwnerOfLease reports the process that adopted or launched one fenced lease.

func (*Plane) OwnerOfRequest 

func (p *Plane) OwnerOfRequest(requestID int64) (RequestOwner, bool)

OwnerOfRequest reports which process holds the compute a request was launched under, and whether that process is still the node's current one.

A DESTROY IS ONLY CONFIRMED BY THE PROCESS THAT HAS THE CONTAINER. The wire broadcasts to whoever is polling, which is the CURRENT incarnation — so a superseded process draining its custody is never asked, answers nothing, and its replacement reports the destroy as done because it genuinely has nothing to remove. Believing that answer releases the lease under a live job.

func (*Plane) OwnsForTest 

func (p *Plane) OwnsForTest(leaseID, node, incarnation string) bool

OwnsForTest reports whether a process is recorded as a lease's owner.

func (*Plane) PickForTest 

func (p *Plane) PickForTest(lease *alloc.Lease) (string, error)

PickForTest reports which node a lease would be aimed at.

func (*Plane) Poll 

func (p *Plane) Poll(ctx context.Context, nodeName, incarnation string) (nodeapi.Command, bool, error)

Poll blocks until a command is available for this node, the context ends, or the poll window closes.

THE "NOTHING TO DO" CASE IS A BOOL, NOT A NIL. Returning a nil command with a nil error would make the ordinary outcome of an idle fleet indistinguishable from a bug at every call site, and this codebase has already rejected that shape twice — in the provider's Find and in the deployment lock — for the same reason: it is the return value that gets logged as a shrug.

An empty command with ok=false is how a quiet poll ends. The node re-polls immediately; that is not an error and must not be treated as one.

func (*Plane) QueuedForTest 

func (p *Plane) QueuedForTest(name string) int

QueuedForTest reports how many commands are waiting to be taken.

Exported for tests only. Expiry and dispatch race unless a test can see that the command it queued has actually arrived, and polling for that is the only way to make the ordering deterministic without inventing a hook production would never use.

func (*Plane) ReconcileInventory 

func (p *Plane) ReconcileInventory(
	ctx context.Context, node, incarnation string, running []string,
) (int, error)

ReconcileInventory frees capacity held for compute this host says it is not running.

THE SAME PROOF AS REGISTRATION, ON A CADENCE THAT ACTUALLY MEETS IT. A lease is quarantined by the reaper, whose clock is the lease TTL — and a node that reconnects after a control-plane restart does so within seconds, long before the leases it was holding expire. So the inventory that arrives with a registration is almost always taken BEFORE the quarantine it would resolve, and nothing looked again.

Fenced by the wire rather than by an epoch: this route refuses a superseded incarnation, which is a stronger statement than the registration epoch — it is about the process, not the registration.

func (*Plane) ReconciledForTest 

func (p *Plane) ReconciledForTest(name string) bool

ReconciledForTest reports whether the live incarnation has vouched for a complete inventory.

Exported for tests because the state it exposes is otherwise invisible from outside the package and is what a refusal must not disturb: only a SUCCESSFUL registration restores it, and a completion may settle a lease from absence only while it holds.

func (*Plane) Register 

func (p *Plane) Register(
	ctx context.Context, req nodeapi.RegisterRequest,
) (nodeapi.RegisterResponse, error)

Register records a node's claim about itself.

EVERY FIELD IS A CLAIM. What a node says about its provider and guest-OS support decides only what the server will ASK it to do; it can never widen the capacity ledger, whose limits come from the server's own configuration. A node that lies gets commands it cannot execute and fails them, which is a bad node rather than an over-committed host.

func (*Plane) Result 

func (p *Plane) Result(nodeName, incarnation string, res nodeapi.CommandResult) error

Result records what a node made of a command.

func (*Plane) Seen 

func (p *Plane) Seen(name, incarnation string)

Seen records that a node just spoke, whatever it said.

EVERY REQUEST IS EVIDENCE OF LIFE. The node's command loop is synchronous, so if Recover, Sweep or Tend wedges it never reaches Poll again — while its custody janitor keeps heartbeating perfectly well. Taking liveness only from Poll and Result would let each heartbeat run expiry, so the same call that proved the node alive would declare it dead, and every later heartbeat would be refused as unregistered while its leases expired.

Command eligibility is bounded by the command timeout, which is the right instrument for a node that takes work and never answers. Membership is bounded by silence, which is the right instrument for a node that has gone.

func (*Plane) SetPollWindowForTest 

func (p *Plane) SetPollWindowForTest(d time.Duration)

SetPollWindowForTest shortens the long-poll window.

Exported for tests only, and named so nobody mistakes it for configuration: the window is part of the contract a node is told at registration, so a deployment that wants a different one changes it in one place rather than having each side pick.

func (*Plane) WaitersForTest 

func (p *Plane) WaitersForTest(name string) int

WaitersForTest reports how many pollers are parked on this node.

Exported for tests so they can synchronise on a poll that is genuinely WAITING rather than sleeping and hoping — the difference between testing what happens to a woken poll and testing what happens to a poll that never blocked.

func (*Plane) Watch 

func (p *Plane) Watch(ctx context.Context)

Watch expires silent nodes until the context ends.

A TIMER, BECAUSE NOTHING ELSE ASKS. A node's liveness decides what its tier ADVERTISES, and an idle deployment never picks a node, lists the fleet or destroys anything — so expiry driven only by those would leave a host that crashed on a quiet afternoon advertising its capacity until somebody happened to launch something.

Half the silence window, so a node is noticed within about one and a half of them rather than up to two.

func (*Plane) Withdraw 

func (p *Plane) Withdraw(ctx context.Context, name, incarnation string) error

Withdraw takes a node out of placement because its current process said it will not poll again.

SILENCE STILL MEANS NOTHING, AND THIS IS THE ONE OTHER WAY OUT. A node that drains and exits used to say nothing, and nothing here could tell a clean exit from a partition — so the plane kept choosing the stopped host until expireStaleLocked forgot it, and every job aimed there in that window waited the window out before being placed elsewhere. A partition has to stay conservative, because the compute barrier and custody both rest on it; what this adds is a deliberate message from the authority on the node's own intent, which is the node, and only from the process currently registered under the name — a superseded process withdrawing would take its replacement out of the fleet.

LEDGER FIRST, MEMORY SECOND. Placement escrows against the ledger's live set long before this map is consulted, so the durable half is the one that stops the next job being aimed here; and a ledger that cannot record the withdrawal leaves the node exactly where it was — placeable until the ledger answers or the host goes silent — which is today's behaviour rather than a new one. The write happens outside the mutex, for recordGone's reason, and the in-memory removal re-checks the fence afterwards because a registration can land in between. The ledger's own fence would already have refused that one; this is the belt to that brace.

ONE RACE IS LEFT OPEN, AND ITS OUTCOME IS TODAY'S. Between the two lock sections the node is still in the map, so a Watch tick can expire it if it has been silent for the whole window — four poll windows — during a ledger write the request deadline bounds to a fraction of that. When it happens the node is forgotten by silence, NodeGone records it and this returns ErrSuperseded, which is exactly what a stop used to produce. A marker that made expiry skip a withdrawing node would carve an exception into the silence rule to close a window a healthy ledger cannot open; it is documented here rather than written.

WHAT IT DOES NOT DO. It releases no lease and appends nothing to pendingGone — the ledger already knows — and it touches neither the owners map nor the barrier run, exactly as expiry does not: a lease this process adopted stays attributed to it, and the next registration's ArrivingForRegistration discards the run. Commands the node never took are answered "nothing started", as after silence; a command still in flight is tombstoned and answered as a re-registration does it — unknown, custody for a launch — and the tombstones outlive the node record (lateResults), because a result cut on the client side by the stop signal can still be inside the handler when this runs.

type Registrar 

type Registrar interface {
	// Lease supplies durable placement before a reported inventory grants ownership.
	Lease(ctx context.Context, leaseID string) (*alloc.Lease, error)

	// RegisterNode records the host and returns the row's new fencing epoch, which
	// NodeGone must present to prove which incarnation it is talking about.
	RegisterNode(ctx context.Context, reg alloc.NodeRegistration) (int64, error)

	// NodeGone records that this control plane has given up on a host. Fenced on
	// the epoch, so an expiry that lands after the node has already come back
	// matches nothing.
	NodeGone(ctx context.Context, name string, epoch int64) error

	// NodeWithdrawn records that a host said it is leaving, fenced on the epoch
	// AND the incarnation, and answers alloc.ErrWithdrawalStale when the fence
	// has moved. It releases nothing and marks no disruption; see Plane.Withdraw.
	NodeWithdrawn(ctx context.Context, name string, epoch int64, incarnation string) error

	// ForgetEveryNode marks the whole fleet unreachable, for a plane that has just
	// started and has no judgement about anything yet.
	ForgetEveryNode(ctx context.Context) error

	// BumpDispatch advances a host's durable launch-dispatch fence.
	//
	// ON THIS INTERFACE RATHER THAN A SEPARATE ONE, because it is not optional:
	// the plane must be unable to hand a host a launch without advancing it. A
	// compute barrier's acknowledgement is accepted only while that number is
	// where it was when the barrier was issued, so a launch that skipped this
	// would be running behind an answer somebody is about to call proof.
	BumpDispatch(ctx context.Context, node string) (int64, error)

	// ResolveQuarantineFor frees capacity held for compute a returning host says
	// it is not running, and reports how many leases it released.
	// The epoch fences it: an overtaken registration must not terminalize a lease
	// a newer one has just vouched for, using a listing taken before that
	// container was visible.
	ResolveQuarantineFor(ctx context.Context, node string, running []string, epoch int64) (int, error)

	// SettleCompletionOnTerminalLease records GitHub's outcome against a lease
	// something else has already settled, and reports whether it was. It never
	// terminalizes an open lease: every completion path that cannot reach the
	// holder waits for the host's inventory, or an operator, to settle the
	// lease and only corrects the verdict.
	SettleCompletionOnTerminalLease(
		ctx context.Context, leaseID string, leaseEpoch int64, outcome alloc.Phase,
	) (bool, error)
}

Registrar is the ledger's node table.

A NODE IS NOT REGISTERED UNTIL THE LEDGER SAYS SO. The plane's own map decides where commands go; the allocator's node row is what Bind checks before it will place a lease. Registering in one and not the other produced a node that took commands and then had every Bind refused — which looked like a broken node rather than a missing row.

type RequestOwner 

type RequestOwner struct {
	Node string
	// Incarnation is the process itself, which is what a destroy's confirmation
	// must be compared against.
	//
	// CURRENCY IS A SNAPSHOT AND CANNOT BE TRUSTED LATER. It is read before the
	// command is dispatched and can change while the command is in flight: a
	// replacement registers, TAKES the destroy, truthfully reports it has nothing
	// to remove, and a decision made on the earlier reading treats that as the
	// owner confirming. The lease is released under a live container.
	Incarnation string
	// Current is false for a superseded process that is draining: it does not
	// poll, so it never sees a destroy and cannot confirm one. Useful for
	// deciding whether to bother asking, never for deciding who answered.
	Current bool
}

RequestOwner is the process holding a request's compute, and whether it is still the one its node's commands reach.

type Revocations 

type Revocations interface {
	// CertRevokedFor answers by serial AND by the cutoff the node carries, so a
	// credential billet never recorded — one from before it tracked serials, or
	// an earlier certificate for a name that was issued twice — is still refused.
	CertRevokedFor(ctx context.Context, node, serial string, issuedAt time.Time) (bool, error)
	CertRevoked(ctx context.Context, serial string) (bool, error)
	RecordIssuedCert(ctx context.Context, cert alloc.IssuedCert) error

	// RecordRenewedCert records a renewal and refuses one whose parent has been
	// revoked since this request began, in one transaction. Two calls would let a
	// revocation commit between the check and the record and take back a
	// credential the machine had already stopped presenting.
	RecordRenewedCert(
		ctx context.Context, cert alloc.IssuedCert, parent string, parentIssuedAt time.Time,
	) error
}

Revocations answers whether a certificate has been withdrawn, and records the ones this wire hands out.

AN ISSUE IS PART OF REVOCATION, not a separate concern. A credential billet never wrote down cannot be taken back: renewal mints a fresh key and serial over this wire, and without recording it the only serials an operator can name are the ones from bundles they issued by hand — which a node that has renewed is no longer presenting.

An interface rather than the allocator, so the wire depends on the two questions it asks rather than on the ledger.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner drives compute on a remote node.

It implements server.Runner, so the listener cannot tell a remote node from an in-process one — which is the point of the split, and also why the ambiguity below has to be mapped onto the SAME errors the in-process runner returns rather than onto new ones the listener has never heard of.

func (*Runner) Destroy 

func (r *Runner) Destroy(ctx context.Context, requestID int64) error

Destroy asks whichever node holds a request's compute to remove it.

BROADCAST, because the server does not track which node holds which request and inventing that map here would be a second authority for a fact the ledger already owns. Destroy is idempotent by contract and a node with nothing for the request answers immediately, so asking all of them is correct rather than merely convenient.

The result is the FIRST failure, if any: a destroy that only partly succeeded has left compute running somewhere and must not report success.

func (*Runner) DestroyCompleted 

func (r *Runner) DestroyCompleted(ctx context.Context, requestID int64, result string) error

DestroyCompleted sends GitHub's authoritative result with a completion-triggered destroy.

func (*Runner) DestroyCompletedBound 

func (r *Runner) DestroyCompletedBound(
	ctx context.Context,
	requestID int64,
	result, leaseID, nodeName string,
	leaseEpoch int64,
	outcome alloc.Phase,
) error

DestroyCompletedBound reconciles a restored completion with its durable host and lease rather than treating an unrelated live fleet as proof of absence.

func (*Runner) KeepAlive 

func (r *Runner) KeepAlive(ctx context.Context)

KeepAlive does nothing here, and the nothing is the design.

A NO-OP THAT SAYS WHY, because a silent one is indistinguishable from forgetting to implement it — and this interface exists precisely so the server can tell whether renewal is happening at all.

Renewal is for leases a RUNNER is holding: compute it could not confirm gone, tracked in its custody map. When the runner is remote that map lives on the node, and the control plane cannot enumerate it. The node runs the same janitor for its own custody on its own clock, which is where it has to be — this interface's own comment says renewal must not share a schedule with anything that talks to a compute backend, and the only party talking to that backend is the node.

So there is genuinely nothing here to renew, and pretending otherwise would mean guessing at leases this side cannot see. Blocking until the context ends matches the contract the caller relies on: it runs this in a goroutine and expects it to live as long as the process.

func (*Runner) Launch 

func (r *Runner) Launch(ctx context.Context, lease *alloc.Lease, job server.Job) error

Launch asks a node to start compute for a lease.

THE THREE OUTCOMES ARE NOT TWO. A local launch either worked or failed; a remote one can also be UNKNOWN, and conflating unknown with failed is how capacity gets released while a container is still running.

  • Nothing was sent: ErrNoNode. Nothing started, and the caller may release the lease with a clear conscience.
  • The node answered: its answer, translated. A clean failure releases; a failure the node marked custody keeps the lease, because the node has taken it into its own janitor.
  • The node took the command and said nothing: server.ErrCustody. This is the one that matters. The command may be executing right now, so the lease is kept and the node's own recovery is what finds whatever started. Treating silence as failure would re-advertise capacity that is in use.

func (*Runner) Sweep 

func (r *Runner) Sweep(ctx context.Context) error

Sweep asks every node to destroy compute whose lease is no longer open.

THIS IS WHAT KEEPS ORPHAN DETECTION ALIVE ACROSS THE SPLIT. The control plane sweeps after each reap, because the lease it has just terminalised is exactly what leaves a container unaccounted for — and it cannot enumerate a remote host. Without implementing this the server logs that its runner "cannot enumerate its compute" and quietly loses the ability to notice a leak at the only moment it reliably could.

The node also sweeps on a timer of its own. That is a backstop, not a substitute: a timer notices minutes later, and only the server knows when an orphan was actually created.

func (*Runner) Tend 

func (r *Runner) Tend(ctx context.Context) error

Tend asks every node to advance the compute it is holding capacity for.

func (*Runner) Upgrade 

func (r *Runner) Upgrade(ctx context.Context, node string, spec nodeapi.UpgradeSpec) error

Upgrade tells one node to replace its own billet, and reports whether the updater started.

ONE NODE, NOT A BROADCAST, and that is the whole difference from Sweep and Tend. Those are whole-fleet housekeeping where doing it everywhere at once is the point; an upgrade drains a host and takes its capacity out of the deployment, so a rollout does them in a cohort it chose. Telling every node at once is the shape that empties a fleet.

IT RETURNS WHEN THE UPDATER HAS STARTED, not when the upgrade is done. The node execs a detached transaction that outlives the process this command was delivered to — so there is nothing here to wait for, and waiting would hold the node's single command slot for the length of a drain.

THE WIRE GATE IS THE CALLER'S, and deliberately not here. Whether a node negotiated a version that has this command is recorded on its LEDGER ROW, which the plane does not read — the rollout coordinator does, because it is already reading every host's release to decide who needs upgrading at all. Checking it here would mean a second source for the same fact.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.