nodeclient

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package nodeclient is a compute host's half of the node wire.

It dials the control plane, registers, and then does two things forever: long- polls for commands, and answers the ledger questions the runner asks. The LeaseStore it implements is the same interface internal/node.Runner already takes, which is why the runner needs no knowledge that its ledger is now on the other side of a network.

Index

Constants

This section is empty.

Variables

View Source
var ErrDenied = errors.New("nodeclient: an operator denied this node")

ErrDenied means an operator refused this machine. Terminal: asking again cannot change it, and the node should stop rather than poll forever.

View Source
var ErrEnrollUnavailable = errors.New("nodeclient: the control plane cannot take this enrollment right now")

ErrEnrollUnavailable means the control plane could not answer right now — a busy ledger, or more enrollments in flight than it will hold at once.

SEPARATE FROM A REFUSAL BECAUSE THE CALLER'S MOVE IS OPPOSITE. Every 503 on this route came back as an unadorned error, so a caller waiting for a human to approve it abandoned the wait over a condition whose own message said to try again — with the pending request still recorded and the join token already spent, so the operator's approval landed on a node that had stopped asking.

View Source
var ErrNotApproved = errors.New("nodeclient: this node is waiting to be approved")

ErrNotApproved means the control plane has the request and nobody has decided yet. The caller waits and asks again.

View Source
var ErrRefused = errors.New("nodeclient: the control plane refused this node")

ErrRefused means the control plane understood the request and rejected it.

RETRYING CANNOT HELP, which is the whole reason it is a distinct error. A protocol version mismatch or a foreign deployment identity will be refused identically forever, and a node that retried every five seconds would sit there looking alive while never being able to work — the failure nobody notices because nothing is crashing.

View Source
var ErrSuperseded = errors.New("nodeclient: another process is registered as this node")

ErrSuperseded means another process has registered under this node's name.

The node STOPS on this, and stopping is the point. Re-registering would take the name back, the other host would take it back in turn, and the control plane's accounting would follow neither while both ran containers against the same leases. It is a configuration mistake — one certificate bundle copied to two machines, or one name written into two config files — and an operator has to fix it.

View Source
var ErrUnauthenticated = errors.New("nodeclient: this node did not prove who it is")

ErrUnauthenticated means the connection proved nothing about who it is.

NOT A VERDICT, and that distinction is the point. ErrRefused is permanent and the node stops on it. This is an expired, missing, or replaced certificate — something an operator fixes on this host, after which the same node connects fine. A node that gave up here would have to be restarted by hand once they had.

View Source
var ErrUnregistered = errors.New("nodeclient: the control plane does not know this node")

ErrUnregistered means the control plane does not know this node.

The node's answer is to register again, not to retry the same call: this is what a restarted control plane looks like from here, and retrying a lease write it will keep refusing is an infinite loop that fixes nothing.

Functions

func Enroll

func Enroll(
	ctx context.Context, base, name, joinToken string, caPEM, csrPEM []byte,
) ([]byte, []byte, error)

Enroll asks a control plane to admit this node, and reports what it said: the certificate, and the authority to install beside it.

Returns ErrNotApproved while an operator has not decided, which is the ordinary case on the first call: the caller prints the fingerprint, waits, and asks again.

THE AUTHORITY COMES BACK TOO, AND IT NEED NOT BE THE ONE WE BOOTSTRAPPED WITH. The wait for a human is unbounded, so an operator can rotate the deployment's CA in the middle of it and approval then signs with the new one. Returning only the certificate left the caller writing the authority it had been holding since before the rotation, and a node whose own certificate does not chain to its own ca.crt cannot start.

Safe to adopt because of what has already happened: this connection was verified against the fingerprint the operator compared out of band, so the bundle is coming from the control plane billet meant, not from whatever answered.

func ExecuteForTest

func ExecuteForTest(ctx context.Context, compute Compute, cmd nodeapi.Command) nodeapi.CommandResult

ExecuteForTest runs one command, for tests that need the command semantics without a control plane to deliver them.

Exported for tests only. The refusal paths — an unknown command kind, a launch with no lease — are reachable only from a server that is newer or broken, and staging either over a real wire would mean building a deliberately wrong control plane to test a node.

func FetchCA

func FetchCA(ctx context.Context, base, wantFingerprint string) ([]byte, string, error)

FetchCA reads a control plane's authority and checks it against a fingerprint the operator supplied.

THE HANDSHAKE IS DELIBERATELY UNVERIFIED HERE, and the fingerprint is what replaces it. A node enrolling for the first time has no authority to verify against — that is what it is fetching — so the connection cannot be trusted and nothing from it is believed until the fingerprint matches a value that travelled by a channel the network does not control.

WITHOUT A FINGERPRINT THIS REFUSES. Accepting whatever answered would be trust-on-first-use with no verification step, which on a network an attacker can reach is just trust: they answer first, the node enrolls with them, and every job it runs afterwards is theirs. The operator reads the value off the control plane with `billet ca show`.

func Run

func Run(ctx context.Context, c *Client, compute Compute, opts LoopOptions) error

Run registers this node and serves commands until the context ends.

RECOVERY HAPPENS BEFORE THE FIRST POLL, and the order is not incidental. A node that starts taking new work while its previous incarnation's containers are unaccounted for will double-count the host: the ledger believes that capacity is free, and it is not. Recover adopts what is still running so the leases behind it stay alive.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client talks to a control plane.

func New

func New(opts Options) (*Client, error)

New builds a Client. It does not dial; Register does.

func (*Client) ActionsCacheAllowed

func (c *Client) ActionsCacheAllowed(
	ctx context.Context,
	owner, repository string,
) (bool, error)

ActionsCacheAllowed reads the control plane's current interception kill switch.

func (*Client) Advance

func (c *Client) Advance(ctx context.Context, leaseID string, epoch int64, to alloc.Phase) error

Advance moves a lease to a new phase.

func (*Client) BaseForTest

func (c *Client) BaseForTest() string

BaseForTest reports the URL requests are built from.

Exported for tests because the failure it guards is invisible from outside: a base without a scheme builds a request that never leaves the process.

func (*Client) Bind

func (c *Client) Bind(ctx context.Context, leaseID string, epoch int64, nodeName string) error

Bind claims a lease for this node.

func (*Client) Describe

func (c *Client) Describe(ctx context.Context, name, group string) (*node.Set, []string, error)

Describe finds a tier's scale set, via the control plane.

The node cannot ask GitHub itself: it holds no App key, by design. See the JIT half of internal/nodeapi for why that is a security property rather than an inconvenience.

func (*Client) EnsureRunnerRemoved

func (c *Client) EnsureRunnerRemoved(ctx context.Context, leaseID string) error

EnsureRunnerRemoved resolves a restart-surviving registration by its lease and withdraws it before recovered custody touches compute.

func (*Client) Heartbeat

func (c *Client) Heartbeat(ctx context.Context, leaseID string, epoch int64) error

Heartbeat renews a lease.

func (*Client) Incarnation

func (c *Client) Incarnation() string

Incarnation identifies this node process.

func (*Client) JITConfig

func (c *Client) JITConfig(
	ctx context.Context, scaleSetID int, runnerName, workFolder string,
) (node.Registration, error)

JITConfig asks the control plane to mint one runner registration.

func (*Client) LaunchedLeaseIDs

func (c *Client) LaunchedLeaseIDs(ctx context.Context, nodeName string) (map[string]bool, error)

LaunchedLeaseIDs reports which leases this node is believed to have launched.

func (*Client) Lease

func (c *Client) Lease(ctx context.Context, leaseID string) (*alloc.Lease, error)

Lease reads a lease.

func (*Client) LeaseTTL

func (c *Client) LeaseTTL() time.Duration

LeaseTTL is how long a lease survives without a heartbeat.

Zero until the first registration answers, because the server is what decides it. A caller that renews on this must not start before then.

func (*Client) MarkFailure

func (c *Client) MarkFailure(ctx context.Context, leaseID string, epoch int64, reason string) error

MarkFailure records why a running lease is destined to fail before teardown.

func (*Client) Poll

func (c *Client) Poll(ctx context.Context) (nodeapi.Command, bool, error)

Poll waits for one command.

Returns ok=false when the window closed with nothing to do, which is the ordinary outcome on an idle fleet and not an error — the same shape the plane uses on the other side, for the same reason.

func (*Client) PollWindow

func (c *Client) PollWindow() time.Duration

PollWindow is how long a command poll may block.

func (*Client) QuarantinedLeaseIDs

func (c *Client) QuarantinedLeaseIDs(ctx context.Context, nodeName string) (map[string]bool, error)

QuarantinedLeaseIDs reports the leases holding capacity for compute the control plane cannot account for on this node.

func (*Client) Reconcile

func (c *Client) Reconcile(ctx context.Context, nodeName string, running []string) (int, error)

Reconcile tells the control plane what this host is running, so it can free capacity held for compute that is gone.

func (*Client) RecordCacheObservation added in v0.8.0

func (c *Client) RecordCacheObservation(
	ctx context.Context, leaseID string, epoch int64, obs alloc.CacheObservation,
) error

RecordCacheObservation tells the control plane what the cache did for a lease's job.

CHECKED WHERE IT IS EMITTED. An older control plane has no route for this and answers a bare 404, which would read here as a decode failure on every cache request of every job on the node. A pairing below the version loses one diagnostic column and nothing else, so the observation is dropped as success rather than reported as an error the caller would resend at teardown; the registration log line already names the negotiated version.

func (*Client) RecoverRunner

func (c *Client) RecoverRunner(ctx context.Context, leaseID, _ string, _ int64,
	runnerName string,
) (node.RunnerRecovery, error)

RecoverRunner asks the control plane to preserve a proven-busy legacy registration or retire it before quarantined compute is touched.

func (*Client) Register

func (c *Client) Register(ctx context.Context, reg Registration) error

Register introduces this node and learns the timings it must respect.

func (*Client) Release

func (c *Client) Release(ctx context.Context, leaseID string, epoch int64, outcome alloc.Phase) error

Release ends a lease with a terminal outcome.

func (*Client) RemoveRunner

func (c *Client) RemoveRunner(
	ctx context.Context, leaseID string, runnerID int64, runnerName string,
) error

RemoveRunner asks the credential-holding control plane to withdraw routing.

func (*Client) Renew

func (c *Client) Renew(ctx context.Context, name string) ([]byte, []byte, []byte, error)

Renew asks the control plane to sign a new certificate for this node.

The key is generated HERE and stays here; only the request and the signature cross the wire. Authenticated by the certificate being replaced, so it grants nothing new — a host that can already act as this node asks to keep doing so.

func (*Client) Report

func (c *Client) Report(ctx context.Context, res nodeapi.CommandResult) error

Report tells the control plane what happened to a command.

func (*Client) Resize

func (c *Client) Resize(
	ctx context.Context, leaseID string, epoch int64, instanceType string,
	vcpu int, memory config.ByteSize,
) error

Resize changes an EC2 lease's charged shape before the provider attempts it.

func (*Client) ValidateTrustedRunnerGroup

func (c *Client) ValidateTrustedRunnerGroup(
	ctx context.Context, tier, group string, workflows []string,
) error

ValidateTrustedRunnerGroup asks the credential-holding control plane to revalidate the workflow boundary immediately before this node requests a JIT registration. The control plane repeats the check against the entitled tier while minting, so a compromised node cannot substitute this request.

func (*Client) WireVersion

func (c *Client) WireVersion() int

WireVersion is the protocol version this node's registration settled on.

Zero until the first registration answers, for the same reason LeaseTTL is: the control plane is what chooses it — and zero again once the control plane has said it does not know this node, see forgetRegistration.

func (*Client) Withdraw

func (c *Client) Withdraw(ctx context.Context) error

Withdraw tells the control plane this node will not poll again, so nothing more is placed on it.

SENT ONLY BY THE LOOP, once the node holds nothing: a withdrawal removes the host from placement and nothing else, and the plane accepts it only from the process currently registered under the name. ErrSuperseded and ErrUnregistered both mean there is nothing of this process's to withdraw.

type Compute

type Compute interface {
	Launch(ctx context.Context, lease *alloc.Lease, tier *nodeapi.TierSpec, job server.Job) error
	Destroy(ctx context.Context, requestID int64) error
	Recover(ctx context.Context) error

	// Instances are the lease ids this host is actually running, read from the
	// provider rather than from anything the control plane said.
	//
	// SENT AT REGISTRATION AS PROOF. A lease whose holder stopped heartbeating is
	// quarantined rather than terminalized, so its capacity stays charged until
	// the compute is confirmed gone — and the sweep that would confirm it only
	// fires for a container that still exists. A host that rebooted has none, so
	// without this its quarantined capacity would never come back.
	Instances(ctx context.Context) ([]string, error)
	Sweep(ctx context.Context) error

	// KeepAlive renews held leases on its own clock until the context ends.
	KeepAlive(ctx context.Context)
	// Tend advances custody entries: adopted work that finished, discarded work
	// whose cleanup is now confirmed.
	Tend(ctx context.Context) error

	// AssumeCustody takes a lease whose launch succeeded but whose result could
	// not be delivered, because the control plane will assume custody too.
	AssumeCustody(ctx context.Context, lease *alloc.Lease, requestID int64) error

	// Holding reports whether this node is still responsible for compute, which
	// is what decides whether a superseded process may stop.
	Holding() bool

	// Superseded moves running work into custody, because after supersession the
	// control plane routes its completions to somebody else.
	Superseded()
}

Compute is the part of internal/node.Runner the loop drives.

An interface so the loop can be tested without a container runtime, and so the loop cannot quietly grow a dependency on anything else the runner exposes.

KEEPALIVE AND TEND ARE HERE BECAUSE CUSTODY NEEDS AN OWNER, and leaving them out silently broke the whole custody design across the split. In one process the runner's janitor renews the leases of compute it could not confirm gone; a node that never runs it hands the server ErrCustody — "I am holding this" — while holding nothing. The server stops heartbeating, the reaper releases the capacity a TTL later, and a container keeps running on capacity that has been sold to somebody else. That is precisely the failure custody was built to prevent, reintroduced by moving the runner behind a network.

type LoopOptions

type LoopOptions struct {
	Provider   config.ProviderKind
	GuestOS    []config.GuestOS
	Deployment string
	// Site is where this machine is, or empty in a deployment with one place.
	Site string
	// VCPU and Memory are what this host CONTRIBUTES, which is what it detected
	// unless its own config said otherwise.
	//
	// Required. The control plane refuses a registration that offers nothing,
	// because a node contributing zero joins the fleet, is never chosen, and
	// produces no error for anyone to find.
	VCPU   int
	Memory config.ByteSize
	// EC2Shapes are the ordered purchasable shapes of whichever REMOTE backend this
	// node runs. Empty for a host-backed provider.
	EC2Shapes []config.RemoteShape
	// CodeBuildFleet is the reserved-capacity CodeBuild fleet this node draws on,
	// or empty for on-demand compute.
	CodeBuildFleet string
	// CodeBuildJITParameterPath and CodeBuildRegion are where a codebuild node
	// stages runner registrations, reported so the control plane can sweep the
	// ones a dead node left behind. Empty for every other backend.
	CodeBuildJITParameterPath string
	CodeBuildRegion           string
	Log                       *slog.Logger
	// SweepEvery bounds how often the node looks for compute nothing is asking
	// about. Zero disables it.
	SweepEvery time.Duration
	// Identity is this node's rotating certificate, when it has one. Nil on a
	// loopback wire, where there are no certificates to renew.
	//
	// GIVEN TO THE LOOP RATHER THAN RENEWED BY THE CALLER, because renewal has to
	// happen while the node is RUNNING. A check at startup would leave a host that
	// is up for a year to expire in place, and the failure is total: an expired
	// certificate cannot renew — renewal is authenticated by the certificate being
	// renewed — so the node has to be re-enrolled by hand.
	Identity *wirecert.Rotating

	// Hurry, when closed, ends the drain's wait early. It is the operator's
	// second signal: stop waiting, but still stop properly.
	Hurry <-chan struct{}
	// DrainTimeout is when a stopping node starts REPORTING that it is still
	// waiting for the compute it holds. It is not a deadline — nothing bounds
	// that wait but the work finishing or a second signal. Zero uses a default of
	// six hours, which is how long GitHub lets a job run.
	DrainTimeout time.Duration
	// Backoff is how long to wait after a failed registration or poll. Zero uses
	// a default.
	//
	// It governs BOTH registration and poll failures, so a caller lengthening it
	// to calm a flapping link is not left hammering the poll endpoint.
	Backoff time.Duration
}

LoopOptions configures Run.

type Options

type Options struct {
	// Base is the control plane's address, as a URL or a bare host:port.
	//
	// A bare address takes its scheme from whether TLS is configured, which is the
	// only place that answer is known. Config carries host:port because that is
	// what an operator writes and what the address can be validated as; a URL
	// there would let someone write http:// beside a certificate and get a node
	// that quietly never used it.
	Base string
	// Node is this host's name, which must match its entry in the server's
	// fleet configuration.
	Node string
	// HTTP is the transport. A caller supplies one so timeouts are decided once
	// rather than here.
	HTTP *http.Client
	// TLS is the node's side of the wire: its certificate, and the deployment
	// authority it verifies the control plane against.
	//
	// Nil serves loopback, and nothing else — a control plane bound to a network
	// address refuses to start without a certificate, so a node reaching one over
	// the network without this fails at the handshake rather than half-connecting.
	TLS *tls.Config
	// RequestTimeout bounds an ordinary request. Zero uses requestTimeout.
	//
	// Configuration rather than a test hook: a node on a slow or distant link
	// legitimately needs longer, and the alternative is every such deployment
	// discovering the constant by hitting it.
	RequestTimeout time.Duration
}

Options configures a Client.

type Registration

type Registration struct {
	Provider   config.ProviderKind
	GuestOS    []config.GuestOS
	Deployment string
	Site       string
	// VCPU and Memory are what this host contributes.
	VCPU      int
	Memory    config.ByteSize
	EC2Shapes []config.RemoteShape
	// CodeBuildFleet is the reserved-capacity fleet this node's builds run on, or
	// empty for on-demand compute. The control plane refuses a second live node
	// naming the same one, because its capacity is shared.
	CodeBuildFleet string
	// CodeBuildJITParameterPath and CodeBuildRegion are where a codebuild node
	// stages runner registrations, so the control plane can sweep the ones a dead
	// node left behind. Empty for every other backend.
	CodeBuildJITParameterPath string
	CodeBuildRegion           string
	// Instances are the lease ids this host is actually running, and
	// InventoryKnown says the list is complete rather than absent. See
	// nodeapi.RegisterRequest.
	Instances      []string
	InventoryKnown bool
}

Registration is what this node says about itself when it introduces itself.

A STRUCT RATHER THAN A GROWING PARAMETER LIST. Two of these are strings that mean unrelated things — a deployment identity and a place — and transposing them compiles, runs, and produces a node in the wrong deployment or the wrong site rather than an error.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL