Documentation
¶
Overview ¶
Package nodeclient is a compute host's half of the node wire.
It dials the control plane, registers, and then does two things forever: long- polls for commands, and answers the ledger questions the runner asks. The LeaseStore it implements is the same interface internal/node.Runner already takes, which is why the runner needs no knowledge that its ledger is now on the other side of a network.
Index ¶
- Variables
- func Enroll(ctx context.Context, base, name, joinToken string, caPEM, csrPEM []byte) ([]byte, []byte, error)
- func ExecuteForTest(ctx context.Context, compute Compute, cmd nodeapi.Command) nodeapi.CommandResult
- func FetchCA(ctx context.Context, base, wantFingerprint string) ([]byte, string, error)
- func Run(ctx context.Context, c *Client, compute Compute, opts LoopOptions) error
- type Client
- func (c *Client) ActionsCacheAllowed(ctx context.Context, owner, repository string) (bool, error)
- func (c *Client) Advance(ctx context.Context, leaseID string, epoch int64, to alloc.Phase) error
- func (c *Client) BaseForTest() string
- func (c *Client) Bind(ctx context.Context, leaseID string, epoch int64, nodeName string) error
- func (c *Client) Describe(ctx context.Context, name, group string) (*node.Set, []string, error)
- func (c *Client) EnsureRunnerRemoved(ctx context.Context, leaseID string) error
- func (c *Client) Heartbeat(ctx context.Context, leaseID string, epoch int64) error
- func (c *Client) Incarnation() string
- func (c *Client) JITConfig(ctx context.Context, scaleSetID int, runnerName, workFolder string) (node.Registration, error)
- func (c *Client) LaunchedLeaseIDs(ctx context.Context, nodeName string) (map[string]bool, error)
- func (c *Client) Lease(ctx context.Context, leaseID string) (*alloc.Lease, error)
- func (c *Client) LeaseTTL() time.Duration
- func (c *Client) MarkFailure(ctx context.Context, leaseID string, epoch int64, reason string) error
- func (c *Client) Poll(ctx context.Context) (nodeapi.Command, bool, error)
- func (c *Client) PollWindow() time.Duration
- func (c *Client) QuarantinedLeaseIDs(ctx context.Context, nodeName string) (map[string]bool, error)
- func (c *Client) Reconcile(ctx context.Context, nodeName string, running []string) (int, error)
- func (c *Client) RecordCacheObservation(ctx context.Context, leaseID string, epoch int64, obs alloc.CacheObservation) error
- func (c *Client) RecoverRunner(ctx context.Context, leaseID, _ string, _ int64, runnerName string) (node.RunnerRecovery, error)
- func (c *Client) Register(ctx context.Context, reg Registration) error
- func (c *Client) Release(ctx context.Context, leaseID string, epoch int64, outcome alloc.Phase) error
- func (c *Client) RemoveRunner(ctx context.Context, leaseID string, runnerID int64, runnerName string) error
- func (c *Client) Renew(ctx context.Context, name string) ([]byte, []byte, []byte, error)
- func (c *Client) Report(ctx context.Context, res nodeapi.CommandResult) error
- func (c *Client) Resize(ctx context.Context, leaseID string, epoch int64, instanceType string, ...) error
- func (c *Client) ValidateTrustedRunnerGroup(ctx context.Context, tier, group string, workflows []string) error
- func (c *Client) WireVersion() int
- func (c *Client) Withdraw(ctx context.Context) error
- type Compute
- type LoopOptions
- type Options
- type Registration
Constants ¶
This section is empty.
Variables ¶
var ErrDenied = errors.New("nodeclient: an operator denied this node")
ErrDenied means an operator refused this machine. Terminal: asking again cannot change it, and the node should stop rather than poll forever.
ErrEnrollUnavailable means the control plane could not answer right now — a busy ledger, or more enrollments in flight than it will hold at once.
SEPARATE FROM A REFUSAL BECAUSE THE CALLER'S MOVE IS OPPOSITE. Every 503 on this route came back as an unadorned error, so a caller waiting for a human to approve it abandoned the wait over a condition whose own message said to try again — with the pending request still recorded and the join token already spent, so the operator's approval landed on a node that had stopped asking.
var ErrNotApproved = errors.New("nodeclient: this node is waiting to be approved")
ErrNotApproved means the control plane has the request and nobody has decided yet. The caller waits and asks again.
var ErrRefused = errors.New("nodeclient: the control plane refused this node")
ErrRefused means the control plane understood the request and rejected it.
RETRYING CANNOT HELP, which is the whole reason it is a distinct error. A protocol version mismatch or a foreign deployment identity will be refused identically forever, and a node that retried every five seconds would sit there looking alive while never being able to work — the failure nobody notices because nothing is crashing.
var ErrSuperseded = errors.New("nodeclient: another process is registered as this node")
ErrSuperseded means another process has registered under this node's name.
The node STOPS on this, and stopping is the point. Re-registering would take the name back, the other host would take it back in turn, and the control plane's accounting would follow neither while both ran containers against the same leases. It is a configuration mistake — one certificate bundle copied to two machines, or one name written into two config files — and an operator has to fix it.
var ErrUnauthenticated = errors.New("nodeclient: this node did not prove who it is")
ErrUnauthenticated means the connection proved nothing about who it is.
NOT A VERDICT, and that distinction is the point. ErrRefused is permanent and the node stops on it. This is an expired, missing, or replaced certificate — something an operator fixes on this host, after which the same node connects fine. A node that gave up here would have to be restarted by hand once they had.
var ErrUnregistered = errors.New("nodeclient: the control plane does not know this node")
ErrUnregistered means the control plane does not know this node.
The node's answer is to register again, not to retry the same call: this is what a restarted control plane looks like from here, and retrying a lease write it will keep refusing is an infinite loop that fixes nothing.
Functions ¶
func Enroll ¶
func Enroll( ctx context.Context, base, name, joinToken string, caPEM, csrPEM []byte, ) ([]byte, []byte, error)
Enroll asks a control plane to admit this node, and reports what it said: the certificate, and the authority to install beside it.
Returns ErrNotApproved while an operator has not decided, which is the ordinary case on the first call: the caller prints the fingerprint, waits, and asks again.
THE AUTHORITY COMES BACK TOO, AND IT NEED NOT BE THE ONE WE BOOTSTRAPPED WITH. The wait for a human is unbounded, so an operator can rotate the deployment's CA in the middle of it and approval then signs with the new one. Returning only the certificate left the caller writing the authority it had been holding since before the rotation, and a node whose own certificate does not chain to its own ca.crt cannot start.
Safe to adopt because of what has already happened: this connection was verified against the fingerprint the operator compared out of band, so the bundle is coming from the control plane billet meant, not from whatever answered.
func ExecuteForTest ¶
func ExecuteForTest(ctx context.Context, compute Compute, cmd nodeapi.Command) nodeapi.CommandResult
ExecuteForTest runs one command, for tests that need the command semantics without a control plane to deliver them.
Exported for tests only. The refusal paths — an unknown command kind, a launch with no lease — are reachable only from a server that is newer or broken, and staging either over a real wire would mean building a deliberately wrong control plane to test a node.
func FetchCA ¶
FetchCA reads a control plane's authority and checks it against a fingerprint the operator supplied.
THE HANDSHAKE IS DELIBERATELY UNVERIFIED HERE, and the fingerprint is what replaces it. A node enrolling for the first time has no authority to verify against — that is what it is fetching — so the connection cannot be trusted and nothing from it is believed until the fingerprint matches a value that travelled by a channel the network does not control.
WITHOUT A FINGERPRINT THIS REFUSES. Accepting whatever answered would be trust-on-first-use with no verification step, which on a network an attacker can reach is just trust: they answer first, the node enrolls with them, and every job it runs afterwards is theirs. The operator reads the value off the control plane with `billet ca show`.
func Run ¶
Run registers this node and serves commands until the context ends.
RECOVERY HAPPENS BEFORE THE FIRST POLL, and the order is not incidental. A node that starts taking new work while its previous incarnation's containers are unaccounted for will double-count the host: the ledger believes that capacity is free, and it is not. Recover adopts what is still running so the leases behind it stay alive.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client talks to a control plane.
func (*Client) ActionsCacheAllowed ¶
func (c *Client) ActionsCacheAllowed( ctx context.Context, owner, repository string, ) (bool, error)
ActionsCacheAllowed reads the control plane's current interception kill switch.
func (*Client) BaseForTest ¶
BaseForTest reports the URL requests are built from.
Exported for tests because the failure it guards is invisible from outside: a base without a scheme builds a request that never leaves the process.
func (*Client) Describe ¶
Describe finds a tier's scale set, via the control plane.
The node cannot ask GitHub itself: it holds no App key, by design. See the JIT half of internal/nodeapi for why that is a security property rather than an inconvenience.
func (*Client) EnsureRunnerRemoved ¶
EnsureRunnerRemoved resolves a restart-surviving registration by its lease and withdraws it before recovered custody touches compute.
func (*Client) Incarnation ¶
Incarnation identifies this node process.
func (*Client) JITConfig ¶
func (c *Client) JITConfig( ctx context.Context, scaleSetID int, runnerName, workFolder string, ) (node.Registration, error)
JITConfig asks the control plane to mint one runner registration.
func (*Client) LaunchedLeaseIDs ¶
LaunchedLeaseIDs reports which leases this node is believed to have launched.
func (*Client) LeaseTTL ¶
LeaseTTL is how long a lease survives without a heartbeat.
Zero until the first registration answers, because the server is what decides it. A caller that renews on this must not start before then.
func (*Client) MarkFailure ¶
MarkFailure records why a running lease is destined to fail before teardown.
func (*Client) Poll ¶
Poll waits for one command.
Returns ok=false when the window closed with nothing to do, which is the ordinary outcome on an idle fleet and not an error — the same shape the plane uses on the other side, for the same reason.
func (*Client) PollWindow ¶
PollWindow is how long a command poll may block.
func (*Client) QuarantinedLeaseIDs ¶
QuarantinedLeaseIDs reports the leases holding capacity for compute the control plane cannot account for on this node.
func (*Client) Reconcile ¶
Reconcile tells the control plane what this host is running, so it can free capacity held for compute that is gone.
func (*Client) RecordCacheObservation ¶ added in v0.8.0
func (c *Client) RecordCacheObservation( ctx context.Context, leaseID string, epoch int64, obs alloc.CacheObservation, ) error
RecordCacheObservation tells the control plane what the cache did for a lease's job.
CHECKED WHERE IT IS EMITTED. An older control plane has no route for this and answers a bare 404, which would read here as a decode failure on every cache request of every job on the node. A pairing below the version loses one diagnostic column and nothing else, so the observation is dropped as success rather than reported as an error the caller would resend at teardown; the registration log line already names the negotiated version.
func (*Client) RecoverRunner ¶
func (c *Client) RecoverRunner(ctx context.Context, leaseID, _ string, _ int64, runnerName string, ) (node.RunnerRecovery, error)
RecoverRunner asks the control plane to preserve a proven-busy legacy registration or retire it before quarantined compute is touched.
func (*Client) Register ¶
func (c *Client) Register(ctx context.Context, reg Registration) error
Register introduces this node and learns the timings it must respect.
func (*Client) Release ¶
func (c *Client) Release(ctx context.Context, leaseID string, epoch int64, outcome alloc.Phase) error
Release ends a lease with a terminal outcome.
func (*Client) RemoveRunner ¶
func (c *Client) RemoveRunner( ctx context.Context, leaseID string, runnerID int64, runnerName string, ) error
RemoveRunner asks the credential-holding control plane to withdraw routing.
func (*Client) Renew ¶
Renew asks the control plane to sign a new certificate for this node.
The key is generated HERE and stays here; only the request and the signature cross the wire. Authenticated by the certificate being replaced, so it grants nothing new — a host that can already act as this node asks to keep doing so.
func (*Client) Resize ¶
func (c *Client) Resize( ctx context.Context, leaseID string, epoch int64, instanceType string, vcpu int, memory config.ByteSize, ) error
Resize changes an EC2 lease's charged shape before the provider attempts it.
func (*Client) ValidateTrustedRunnerGroup ¶
func (c *Client) ValidateTrustedRunnerGroup( ctx context.Context, tier, group string, workflows []string, ) error
ValidateTrustedRunnerGroup asks the credential-holding control plane to revalidate the workflow boundary immediately before this node requests a JIT registration. The control plane repeats the check against the entitled tier while minting, so a compromised node cannot substitute this request.
func (*Client) WireVersion ¶
WireVersion is the protocol version this node's registration settled on.
Zero until the first registration answers, for the same reason LeaseTTL is: the control plane is what chooses it — and zero again once the control plane has said it does not know this node, see forgetRegistration.
func (*Client) Withdraw ¶
Withdraw tells the control plane this node will not poll again, so nothing more is placed on it.
SENT ONLY BY THE LOOP, once the node holds nothing: a withdrawal removes the host from placement and nothing else, and the plane accepts it only from the process currently registered under the name. ErrSuperseded and ErrUnregistered both mean there is nothing of this process's to withdraw.
type Compute ¶
type Compute interface {
Launch(ctx context.Context, lease *alloc.Lease, tier *nodeapi.TierSpec, job server.Job) error
Destroy(ctx context.Context, requestID int64) error
Recover(ctx context.Context) error
// Instances are the lease ids this host is actually running, read from the
// provider rather than from anything the control plane said.
//
// SENT AT REGISTRATION AS PROOF. A lease whose holder stopped heartbeating is
// quarantined rather than terminalized, so its capacity stays charged until
// the compute is confirmed gone — and the sweep that would confirm it only
// fires for a container that still exists. A host that rebooted has none, so
// without this its quarantined capacity would never come back.
Instances(ctx context.Context) ([]string, error)
Sweep(ctx context.Context) error
// KeepAlive renews held leases on its own clock until the context ends.
KeepAlive(ctx context.Context)
// Tend advances custody entries: adopted work that finished, discarded work
// whose cleanup is now confirmed.
Tend(ctx context.Context) error
// AssumeCustody takes a lease whose launch succeeded but whose result could
// not be delivered, because the control plane will assume custody too.
AssumeCustody(ctx context.Context, lease *alloc.Lease, requestID int64) error
// Holding reports whether this node is still responsible for compute, which
// is what decides whether a superseded process may stop.
Holding() bool
// Superseded moves running work into custody, because after supersession the
// control plane routes its completions to somebody else.
Superseded()
}
Compute is the part of internal/node.Runner the loop drives.
An interface so the loop can be tested without a container runtime, and so the loop cannot quietly grow a dependency on anything else the runner exposes.
KEEPALIVE AND TEND ARE HERE BECAUSE CUSTODY NEEDS AN OWNER, and leaving them out silently broke the whole custody design across the split. In one process the runner's janitor renews the leases of compute it could not confirm gone; a node that never runs it hands the server ErrCustody — "I am holding this" — while holding nothing. The server stops heartbeating, the reaper releases the capacity a TTL later, and a container keeps running on capacity that has been sold to somebody else. That is precisely the failure custody was built to prevent, reintroduced by moving the runner behind a network.
type LoopOptions ¶
type LoopOptions struct {
Provider config.ProviderKind
GuestOS []config.GuestOS
Deployment string
// Site is where this machine is, or empty in a deployment with one place.
Site string
// VCPU and Memory are what this host CONTRIBUTES, which is what it detected
// unless its own config said otherwise.
//
// Required. The control plane refuses a registration that offers nothing,
// because a node contributing zero joins the fleet, is never chosen, and
// produces no error for anyone to find.
VCPU int
Memory config.ByteSize
// EC2Shapes are the ordered purchasable shapes of whichever REMOTE backend this
// node runs. Empty for a host-backed provider.
EC2Shapes []config.RemoteShape
// CodeBuildFleet is the reserved-capacity CodeBuild fleet this node draws on,
// or empty for on-demand compute.
CodeBuildFleet string
// CodeBuildJITParameterPath and CodeBuildRegion are where a codebuild node
// stages runner registrations, reported so the control plane can sweep the
// ones a dead node left behind. Empty for every other backend.
CodeBuildJITParameterPath string
CodeBuildRegion string
Log *slog.Logger
// SweepEvery bounds how often the node looks for compute nothing is asking
// about. Zero disables it.
SweepEvery time.Duration
// Identity is this node's rotating certificate, when it has one. Nil on a
// loopback wire, where there are no certificates to renew.
//
// GIVEN TO THE LOOP RATHER THAN RENEWED BY THE CALLER, because renewal has to
// happen while the node is RUNNING. A check at startup would leave a host that
// is up for a year to expire in place, and the failure is total: an expired
// certificate cannot renew — renewal is authenticated by the certificate being
// renewed — so the node has to be re-enrolled by hand.
Identity *wirecert.Rotating
// Hurry, when closed, ends the drain's wait early. It is the operator's
// second signal: stop waiting, but still stop properly.
Hurry <-chan struct{}
// DrainTimeout is when a stopping node starts REPORTING that it is still
// waiting for the compute it holds. It is not a deadline — nothing bounds
// that wait but the work finishing or a second signal. Zero uses a default of
// six hours, which is how long GitHub lets a job run.
DrainTimeout time.Duration
// Backoff is how long to wait after a failed registration or poll. Zero uses
// a default.
//
// It governs BOTH registration and poll failures, so a caller lengthening it
// to calm a flapping link is not left hammering the poll endpoint.
Backoff time.Duration
}
LoopOptions configures Run.
type Options ¶
type Options struct {
// Base is the control plane's address, as a URL or a bare host:port.
//
// A bare address takes its scheme from whether TLS is configured, which is the
// only place that answer is known. Config carries host:port because that is
// what an operator writes and what the address can be validated as; a URL
// there would let someone write http:// beside a certificate and get a node
// that quietly never used it.
Base string
// Node is this host's name, which must match its entry in the server's
// fleet configuration.
Node string
// HTTP is the transport. A caller supplies one so timeouts are decided once
// rather than here.
HTTP *http.Client
// TLS is the node's side of the wire: its certificate, and the deployment
// authority it verifies the control plane against.
//
// Nil serves loopback, and nothing else — a control plane bound to a network
// address refuses to start without a certificate, so a node reaching one over
// the network without this fails at the handshake rather than half-connecting.
TLS *tls.Config
// RequestTimeout bounds an ordinary request. Zero uses requestTimeout.
//
// Configuration rather than a test hook: a node on a slow or distant link
// legitimately needs longer, and the alternative is every such deployment
// discovering the constant by hitting it.
RequestTimeout time.Duration
}
Options configures a Client.
type Registration ¶
type Registration struct {
Provider config.ProviderKind
GuestOS []config.GuestOS
Deployment string
Site string
// VCPU and Memory are what this host contributes.
VCPU int
Memory config.ByteSize
EC2Shapes []config.RemoteShape
// CodeBuildFleet is the reserved-capacity fleet this node's builds run on, or
// empty for on-demand compute. The control plane refuses a second live node
// naming the same one, because its capacity is shared.
CodeBuildFleet string
// CodeBuildJITParameterPath and CodeBuildRegion are where a codebuild node
// stages runner registrations, so the control plane can sweep the ones a dead
// node left behind. Empty for every other backend.
CodeBuildJITParameterPath string
CodeBuildRegion string
// Instances are the lease ids this host is actually running, and
// InventoryKnown says the list is complete rather than absent. See
// nodeapi.RegisterRequest.
Instances []string
InventoryKnown bool
}
Registration is what this node says about itself when it introduces itself.
A STRUCT RATHER THAN A GROWING PARAMETER LIST. Two of these are strings that mean unrelated things — a deployment identity and a place — and transposing them compiles, runs, and produces a node in the wrong deployment or the wrong site rather than an error.