node

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 5, 2026 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Overview

Package node turns assigned leases into running compute.

It is the in-process half of what will become `billet node`: the control plane hands it a lease, it mints a single-use runner registration, and it asks a provider to start something that consumes it. When the node splits out over mTLS, the remote side implements the same two methods and this becomes the local case rather than the only one.

Index

Constants

View Source 
const (
	// AckAccepted is what an updater writes once it has taken responsibility.
	AckAccepted = "accepted"
	// AckRefused prefixes the reason when it has not.
	AckRefused = "refused: "
	// MaxAckBytes bounds what either end will write or read. An acknowledgement is
	// one short line.
	MaxAckBytes = 4 << 10
	// AckFD is the child descriptor the answer travels on: the first after the
	// standard three, which is what ExtraFiles[0] becomes in the child.
	AckFD = 3
)

The updater's vocabulary, defined ONCE, here.

IN THE PACKAGE THAT READS IT, and used by the command that writes it — which is the way round that works, since cmd/billet imports this package and nothing imports cmd/billet. Two copies of a protocol whose whole job is to be recognised is the two-pins problem: the writer would go on writing a word the reader had stopped accepting, and the failure would be a rollout waiting forever on an updater that had answered.

View Source 
const DefaultMaxCustody = 0

DefaultMaxCustody is OFF, deliberately.

Elapsed time is not evidence that a job has stopped making progress: billet imposes no job limit of its own, self-hosted runners are routinely configured past GitHub's six-hour default, and a legitimately long job would be killed for no reason anyone could see in the logs.

Killing live work must be authorised by something that actually knows — a completion from GitHub, an observed process exit, or an operator. Time drives the WARNING instead, which is the honest signal: "billet is holding capacity it cannot account for, and it has been doing so for two hours."

An operator who does know their longest job can set a bound. Zero means none.

Variables

View Source 
var ErrNoUpgrader = errors.New("node: this node has no transactional updater")

ErrNoUpgrader means this node cannot replace its own billet.

View Source 
var ErrUpgradeRefused = errors.New("node: the updater refused this upgrade")

ErrUpgradeRefused means the updater started, decided against the instruction, and changed nothing.

Functions

This section is empty.

Types

type ActionsPolicy 

type ActionsPolicy interface {
	ActionsCacheAllowed(ctx context.Context, owner, repository string) (bool, error)
}

ActionsPolicy reads the control plane's current interception kill switch.

type CacheCredentials 

type CacheCredentials struct {
	Token        string
	ActionsProxy string
	ActionsCAPEM string
}

CacheCredentials identifies one managed guest's cache session.

type CacheObserver added in v0.8.0 

type CacheObserver interface {
	ObserveCache(ctx context.Context, instance, leaseID string, epoch int64,
		obs alloc.CacheObservation) error
}

CacheObserver is told what the cache did for one guest, so the lease's history can carry it. The runner implements it.

THE LEASE TRAVELS WITH THE CALL, from the session's durable record, so a process that never launched the guest can still attribute what it saw. The runner prefers its own mapping while it holds one, because that carries the epoch in force now.

type CacheService 

type CacheService struct {
	// contains filtered or unexported fields
}

CacheService lets one authenticated microVM replace its reserved drive slots.

func NewCacheService 

func NewCacheService(
	endpoint string,
	namespace string,
	stateDir string,
	storage storecontract.Store,
	attacher provider.VolumeAttacher,
	log *slog.Logger,
) (*CacheService, error)

NewCacheService constructs a node-local cache endpoint.

func (*CacheService) Cleanup 

func (s *CacheService) Cleanup(ctx context.Context, instance string) error

Cleanup discards every volume after its compute is proved gone.

func (*CacheService) Endpoint 

func (s *CacheService) Endpoint() string

Endpoint is the origin placed in a microVM's metadata.

func (*CacheService) Prepare 

func (s *CacheService) Prepare(instance string, trust provider.TrustClass) (CacheCredentials, error)

Prepare creates one unguessable session credential before compute starts.

func (*CacheService) PrepareScoped 

func (s *CacheService) PrepareScoped(
	instance string,
	scope CacheSessionScope,
) (CacheCredentials, error)

PrepareScoped creates a session bound to the pool's configured cache scope.

func (*CacheService) ReconcileInventory 

func (s *CacheService) ReconcileInventory(ctx context.Context, instances []*provider.Instance) error

ReconcileInventory closes cache sessions whose compute is absent from a successful provider inventory. Only a successful list is evidence: an empty inventory is meaningful, while a list error must leave every session fenced.

func (*CacheService) RenewActive 

func (s *CacheService) RenewActive(ctx context.Context, until time.Time) error

RenewActive keeps every mounted generation out of eviction. A job may outlive the store's initial lease, so the node refreshes these for as long as it owns the corresponding compute session.

func (*CacheService) RetryClosed 

func (s *CacheService) RetryClosed(ctx context.Context) error

RetryClosed releases cache volumes whose earlier cleanup was interrupted.

func (*CacheService) ServeHTTP 

func (s *CacheService) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP serves the exact API understood by the sticky-disk action.

func (*CacheService) SetActionsPolicy 

func (s *CacheService) SetActionsPolicy(policy ActionsPolicy)

SetActionsPolicy installs the control-plane policy reader before the listener starts.

func (*CacheService) SetCacheObserver added in v0.8.0 

func (s *CacheService) SetCacheObserver(observer CacheObserver)

SetCacheObserver installs where observations are reported, before the listener starts. Without one they are kept on the session and reported to nobody.

func (*CacheService) SettleDocker 

func (s *CacheService) SettleDocker(ctx context.Context, instance string, succeeded bool) error

SettleDocker publishes a prepared store only when GitHub reported success.

func (*CacheService) SettleObservation added in v0.8.0 

func (s *CacheService) SettleObservation(ctx context.Context, instance string)

SettleObservation settles what the cache did for one instance while the caller still holds its lease.

CALLED BY THE RUNNER BEFORE IT FORGETS A REQUEST, because the report needs the lease the instance belongs to and Cleanup runs after that mapping is gone. Cleanup settles too, for the paths where the mapping survives it.

type CacheSessionScope 

type CacheSessionScope struct {
	Trust       provider.TrustClass
	Intercept   bool
	Owner       string
	Repository  string
	WorkflowRef string
	// LeaseID and Epoch name the lease the guest runs under, so an observation
	// can be attributed after the runner has forgotten the instance. Recorded
	// on the durable session; they authorise nothing, and the runner's own
	// mapping is preferred while it exists because a re-adoption moves the
	// epoch.
	LeaseID string
	Epoch   int64
}

CacheSessionScope is the static authority configured for one runner pool.

type ExecUpgrader 

type ExecUpgrader struct {
	// Binary is the billet to exec. Empty means the running executable.
	Binary string
	// ConfigPath is the config the updater reads.
	ConfigPath string
}

ExecUpgrader runs billet's own updater as a detached process.

func (ExecUpgrader) StartUpgrade 

func (e ExecUpgrader) StartUpgrade(ctx context.Context, spec nodeapi.UpgradeSpec) error

StartUpgrade execs `billet host-upgrade` and does not wait for it.

DETACHED FROM THIS PROCESS'S CONTEXT, ON PURPOSE. The updater's whole job is to stop the very service that started it, so inheriting a context this node cancels on shutdown would kill the updater at the exact moment it succeeded — leaving a machine with both services stopped and nothing running that knows what to do about it. context.WithoutCancel is what breaks that link.

Setpgid puts it in its own process group for the same reason: a signal sent to the node's group must not reach a transaction midway through replacing a binary.

type JITSource 

type JITSource interface {
	// Describe finds a tier's scale set. Returns a nil set when there is none.
	Describe(ctx context.Context, name, group string) (*Set, []string, error)
	// JITConfig mints a registration for one runner against one scale set.
	JITConfig(ctx context.Context, scaleSetID int, runnerName, workFolder string) (Registration, error)
	// RemoveRunner removes routing before failed-launch compute is touched.
	RemoveRunner(ctx context.Context, leaseID string, runnerID int64, runnerName string) error
	// EnsureRunnerRemoved resolves a restart-surviving registration by lease.
	EnsureRunnerRemoved(ctx context.Context, leaseID string) error
	// RecoverRunner preserves an exact busy legacy registration or removes it
	// while idle before quarantined compute is adopted.
	RecoverRunner(ctx context.Context, leaseID, tier string, requestID int64,
		runnerName string) (RunnerRecovery, error)
}

JITSource mints single-use credentials for ephemeral pool registrations and finds the scale set to mint them against.

An interface rather than the concrete client so this package does not depend on the preview scale-set API, and so a test can drive the whole launch path without a GitHub organization.

type Job 

type Job = server.Job

Job re-exports the listener's job identity so this package's signature matches server.Runner without importing anything else from it at call sites.

type LeaseStore 

type LeaseStore interface {
	Bind(ctx context.Context, leaseID string, epoch int64, node string) error
	Advance(ctx context.Context, leaseID string, epoch int64, to alloc.Phase) error
	Heartbeat(ctx context.Context, leaseID string, epoch int64) error
	MarkFailure(ctx context.Context, leaseID string, epoch int64, reason string) error
	Resize(ctx context.Context, leaseID string, epoch int64, instanceType string,
		vcpu int, memory config.ByteSize) error
	Release(ctx context.Context, leaseID string, epoch int64, outcome alloc.Phase) error
	// RecordCacheObservation writes what this host saw the cache do for a
	// lease's job, fenced on the epoch; the ledger keeps the first observation.
	RecordCacheObservation(ctx context.Context, leaseID string, epoch int64,
		obs alloc.CacheObservation) error
	Lease(ctx context.Context, leaseID string) (*alloc.Lease, error)
	LaunchedLeaseIDs(ctx context.Context, node string) (map[string]bool, error)
	QuarantinedLeaseIDs(ctx context.Context, node string) (map[string]bool, error)
	Reconcile(ctx context.Context, node string, running []string) (int, error)

	// LeaseTTL is how long a lease survives without a heartbeat, and it sets the
	// cadence for renewing held ones. READ FROM THE LEDGER, never assumed: the
	// last time a heartbeat interval was derived from a constant that no longer
	// matched, advertised capacity climbed to six times the configured budget
	// before anyone noticed.
	LeaseTTL() time.Duration
}

LeaseStore is the part of the capacity ledger the runner uses.

An interface rather than *alloc.Allocator because the runner's hardest paths are the ones where the ledger REFUSES, and a concrete allocator gives a test no way to produce that. Custody exists to keep a lease held when compute is unaccounted for; the branch that keeps holding it when the release itself fails was untestable and therefore untested, which in this codebase has reliably meant wrong.

It is deliberately the whole set the runner calls and nothing more, so the dependency is legible at a glance.

type Option 

type Option func(*Runner)

Option configures a Runner.

func WithCacheService 

func WithCacheService(cache *CacheService) Option

WithCacheService gives Firecracker guests the node-local sticky-disk endpoint.

THE RUNNER IS THE CACHE'S OBSERVER, because only the runner knows which lease an instance name belongs to, and that is what a cache observation is recorded against.

func WithMaxCustody 

func WithMaxCustody(d time.Duration) Option

WithMaxCustody bounds how long compute may be held before billet destroys it and reclaims the capacity.

Zero, the default, means no bound — see DefaultMaxCustody for why. This exists because the previous commit's message claimed an operator could configure the limit while nothing in the package let them: the field was private and New took no option, so the only configuration was a same-package test writing to it directly. A capability described in a comment and absent from the API is worse than one that was never claimed.

func WithRegistryMirrors 

func WithRegistryMirrors(mirrors config.RegistryMirrors) Option

WithRegistryMirrors gives managed guests their site's public pull-through caches.

func WithUpgrader 

func WithUpgrader(u Upgrader) Option

WithUpgrader gives a node the ability to replace its own billet.

ABSENT BY DEFAULT, and deliberately. A node running out of a working directory, or under a test harness, has no packaged binary to replace and no units to restart; making the capability opt-in means those refuse the command with a sentence an operator can act on instead of attempting a transaction against a machine that is not shaped for one.

type Registration 

type Registration interface {
	// Config returns the encoded JIT configuration.
	Config() string
	// RunnerName is what GitHub registered, which is what teardown needs.
	RunnerName() string
	// ID is GitHub's durable runner identity.
	ID() int64
}

Registration is a minted runner registration. The config inside is a CREDENTIAL until the runner consumes it.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner starts and stops the compute for assigned leases.

func New 

func New(
	a LeaseStore, node string, jit JITSource, p provider.Provider,
	log *slog.Logger, opts ...Option,
) *Runner

New builds a runner over a provider.

func (*Runner) AssumeCustody 

func (r *Runner) AssumeCustody(ctx context.Context, lease *alloc.Lease, requestID int64) error

AssumeCustody takes responsibility for a lease whose launch SUCCEEDED but whose outcome the caller could not deliver.

EXISTS BECAUSE THE SPLIT CREATED A GAP IN-PROCESS OPERATION CANNOT HAVE. When the runner and the listener share a process, a successful launch is reported by returning nil and the listener keeps heartbeating the lease. Over a wire the report can be lost: the control plane times the command out and must assume custody — the only safe reading of silence — and stops heartbeating, because custody means the node is holding it. Meanwhile the node believes it merely launched something, so nothing renews the lease and the reaper releases capacity that a container is still using.

The handoff has to be CAUSAL rather than hopeful, so the party that failed to report is the party that takes custody. Idempotent: adopting a lease already held simply refreshes it.

func (*Runner) Destroy 

func (r *Runner) Destroy(ctx context.Context, requestID int64) error

Destroy removes whatever Launch started for a request.

Idempotent: a request nothing was started for is success, because this runs on redelivered completions, on shutdown, and after a failure.

func (*Runner) DestroyCompleted 

func (r *Runner) DestroyCompleted(ctx context.Context, requestID int64, result string) error

DestroyCompleted settles result-dependent cache state before removing compute.

func (*Runner) Holding 

func (r *Runner) Holding() bool

Holding reports whether this node is still responsible for compute.

ASKED WHEN THE NODE HAS BEEN SUPERSEDED and is deciding whether it may stop. Exiting while custody remains is how a container ends up with a lease nobody renews: the replacement cannot see it — it is on a different machine — so nothing else will ever hold it.

func (*Runner) Instances 

func (r *Runner) Instances(ctx context.Context) ([]string, error)

Instances are the lease ids this host is actually running.

FROM THE PROVIDER, never from the ledger: the point of sending it is to tell the control plane something it cannot see for itself.

func (*Runner) KeepAlive 

func (r *Runner) KeepAlive(ctx context.Context)

KeepAlive renews held leases on their own clock until the context ends.

SEPARATE FROM Tend, AND THAT SEPARATION IS THE WHOLE POINT. Tend runs after Reap on a shared tick and makes unbounded provider calls, so renewal inside it would leave the interval from a successful heartbeat to the following Reap unbounded — a slow `docker ps` delays the next renewal without delaying the next reap. Anything longer than the lease TTL and the reaper terminalizes a lease held on purpose, hands its capacity back, and lets a listener advertise it while the container is still running.

So this does exactly one thing, touches only the ledger, and ticks at a third of the TTL — the same cadence and the same reasoning as the listener's own heartbeats. Two renewals may be missed entirely before anything expires.

func (*Runner) Launch 

func (r *Runner) Launch(
	ctx context.Context, lease *alloc.Lease, tier *nodeapi.TierSpec, job Job,
) error

Launch mints a registration and starts something that will consume it.

func (*Runner) ObserveCache added in v0.8.0 

func (r *Runner) ObserveCache(
	ctx context.Context, instance, leaseID string, epoch int64, obs alloc.CacheObservation,
) error

ObserveCache records what the cache did for one instance against its lease.

THIS PROCESS'S OWN MAPPING FIRST, wherever it holds the instance: a request that is running, a launch still in progress, or compute in custody. That carries the epoch in force now, which the session's record does not once a quarantine has moved it. The session's lease is what a process that never launched the guest, or has already forgotten it, attributes with; a session from before the record carried one names nothing, and the observation is kept against the next attempt.

func (*Runner) Recover 

func (r *Runner) Recover(ctx context.Context) error

Recover decides what to do with compute an earlier run left behind, once, at startup and before any listener opens a session.

IT DOES NOT DESTROY EVERYTHING. GitHub does not transparently retry a job a runner has already started: the scale-set documentation describes reassignment only for a job "not acquired by a runner in time". Force-killing a container running a twenty-minute job is therefore a deliberate job failure, not a recovery.

So a surviving container whose lease is still open is ADOPTED. Billet cannot manage it — the request-id mapping and completion handling died with the last process — but the runner inside is talking to GitHub on its own and may well finish. What billet can do is keep the lease alive so the capacity is not resold underneath it, and clean up once it stops. That is what custody is.

A container whose lease is NOT open is a genuine orphan: nothing is waiting for its result and its capacity has already gone back to the budget. Those are destroyed here rather than left for the first sweep, because until they are gone the host is over-committed by exactly their size.

func (*Runner) StartUpgrade 

func (r *Runner) StartUpgrade(ctx context.Context, spec nodeapi.UpgradeSpec) error

StartUpgrade launches the updater and returns as soon as it is running.

IT DOES NOT WAIT, AND THAT IS THE WHOLE CONTRACT. A node executes commands one at a time and each command's timeout starts when it is QUEUED, so an upgrade carried out inline would hold the node's single slot for as long as the drain takes — which is as long as the longest job, with no bound on it. Every other command to this host would expire in the queue behind it, including the destroys that let the drain finish.

func (*Runner) Superseded 

func (r *Runner) Superseded()

Superseded moves everything this node is running into custody.

AFTER SUPERSESSION, EVERYTHING HERE IS UNACCOUNTED FOR — which is the definition of custody. The control plane routes a job's completion to whichever process currently owns the name, so the destroy for a container running HERE now goes to the replacement, which cannot see it, reports success, and lets the lease be released.

Nothing else would ever finish this work. Tend is what confirms compute gone and releases its lease, and Tend only looks at custody — so a running entry left where it was made Holding() true forever and the drain unable to end.

The lease's outcome is recorded as done rather than failed: the job was launched cleanly and is running: what changed is who is allowed to talk about it, not whether it worked.

func (*Runner) Sweep 

func (r *Runner) Sweep(ctx context.Context) error

Sweep destroys instances whose lease is no longer open on this node.

The steady-state counterpart to Recover, and the reason a failed cleanup is survivable rather than permanent. Three things leak compute while the process is alive and none of them is reachable by a startup-only pass: a stray that Find could not confirm, a Destroy the daemon refused, and a lease reaped out from under a container that is still running.

Safe to run concurrently with live launches, because of an ordering the launch path guarantees: Bind and Advance(launching) both commit BEFORE the provider is asked to create anything. So an instance that appears in the list already has a lease at launching or beyond, and a sweep cannot see compute whose lease has not yet been written. It cannot race a starting job either — the list is taken first, so anything in it predates the query that judges it.

func (*Runner) Tend 

func (r *Runner) Tend(ctx context.Context) error

Tend advances everything in custody by one step, and is called on the same tick as Sweep.

The heartbeat comes FIRST and its failure is the most informative outcome here. A lease that will not heartbeat is one the reaper already terminalized, or one whose epoch moved because somebody else took it — either way this process no longer holds the capacity, so there is nothing left to protect and the instance becomes something to destroy rather than something to preserve. That is also the ordinary way an adopted container ends: GitHub reports the job complete, the listener releases the lease, and the next Tend finds the heartbeat refused and cleans up.

func (*Runner) WatchInterruptions 

func (r *Runner) WatchInterruptions(ctx context.Context)

type RunnerRecovery 

type RunnerRecovery string

RunnerRecovery is the safe disposition of a quarantined runner registration. 

const (
	RunnerRecoveryTracked RunnerRecovery = "tracked"
	RunnerRecoveryBusy    RunnerRecovery = "busy"
	RunnerRecoveryRetired RunnerRecovery = "retired"
)

type Set 

type Set struct {
	ID   int
	Name string
}

Set is the part of a scale set this package needs.

type Upgrader 

type Upgrader interface {
	StartUpgrade(ctx context.Context, spec nodeapi.UpgradeSpec) error
}

Upgrader starts the transactional updater that replaces this node's billet.

A SEAM RATHER THAN A DIRECT exec, for two reasons that are both about honesty. A node with no updater configured must REFUSE the command rather than silently do nothing — a rollout that recorded it as instructed would wait forever for a convergence that cannot come — and the argv this produces is the only part of the mechanism a test can see, since everything downstream stops services and replaces binaries.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.