Documentation
¶
Overview ¶
Package server is billet's control plane: the per-tier scale-set listeners and the scheduler that turns assigned jobs into launched instances.
Index ¶
- Variables
- type BoundCompletionAwareRunner
- type CompletionAwareRunner
- type ControlPlaneOption
- func AdvertiseNothing() ControlPlaneOption
- func OptionsFromConfig(cfg *config.Config) ([]ControlPlaneOption, error)
- func WithCleanupRetry(first, ceiling time.Duration) ControlPlaneOption
- func WithCompletionLedger(db *state.DB) ControlPlaneOption
- func WithDrainTimeout(d time.Duration) ControlPlaneOption
- func WithHurry(c <-chan struct{}) ControlPlaneOption
- func WithLeadershipLost(fn func() bool) ControlPlaneOption
- func WithNodeRunner(r Runner) ControlPlaneOption
- func WithReapInterval(d time.Duration) ControlPlaneOption
- func WithRolloutCoordinator(c *rollout.Coordinator, every time.Duration) ControlPlaneOption
- func WithRolloutStarter(s *rollout.Starter, every time.Duration) ControlPlaneOption
- func WithStagedCredentialSweeper(s StagedCredentialSweeper) ControlPlaneOption
- func WithTargets(targets ...Target) ControlPlaneOption
- type Job
- type Listener
- type Message
- type Option
- func WithCleanupRetryPacing(first, ceiling time.Duration) Option
- func WithCompletionStore(db *state.DB) Option
- func WithDrainGrace(d time.Duration) Option
- func WithFinishGraces(closing, releasing time.Duration) Option
- func WithHurrySignal(c <-chan struct{}) Option
- func WithLeadershipLostCheck(fn func() bool) Option
- func WithLogger(log *slog.Logger) Option
- func WithMaxCapacity(ceiling int) Option
- func WithRunner(r Runner) Option
- func WithRunnerRegistry(registry RunnerRegistry) Option
- func WithShutdownGrace(d time.Duration) Option
- func WithStalePromiseAfter(d time.Duration) Option
- type Provisioner
- type Runner
- type RunnerRegistry
- type ScaleSet
- type Server
- type Session
- type StagedCredentialSweeper
- type Statistics
- type Sweeper
- type Target
Constants ¶
This section is empty.
Variables ¶
var ErrCustody = errors.New("server: the runner is holding this lease's capacity")
ErrCustody means the runner has taken responsibility for a lease's capacity, so the caller must NOT release it.
Returned from Launch when compute may exist that could not be confirmed gone. Releasing then would hand the capacity back while a container is possibly still running on it.
ErrHolderUnavailable means result-dependent teardown has not reached the process that holds the compute, so its durable completion must be retried.
var ErrNoMessage = errors.New("server: no message")
ErrNoMessage means a long poll timed out with nothing to report — the ordinary outcome, not a failure.
A sentinel rather than (nil, nil), which the upstream client returns: a nil message with a nil error is indistinguishable from "something went wrong and nobody said so".
var ErrSessionHeld = errors.New("server: this scale set already has an active message " +
"session, held by a control plane that did not close it")
ErrSessionHeld means a message session for this scale set is already outstanding, held by a control plane that did not close it.
A SENTINEL BECAUSE THE ANSWER IS TO WAIT, NOT TO FAIL. Every other reason a session cannot open is a reason to report and stop; this one resolves by itself when GitHub expires the abandoned session, and it is the ordinary state after any restart that was not graceful.
var ErrUntrustworthySession = errors.New("server: the scale set returned something billet cannot act on")
ErrUntrustworthySession marks a scale-set response billet cannot act on.
FATAL WHENEVER IT ARRIVES, including in the middle of a shutdown. Once GitHub returns an id nobody offered for, billet cannot tell which of its commitments are real — so it must stop rather than keep operating that session, and a cancellation happening at the same moment must not turn that into a drain.
Functions ¶
This section is empty.
Types ¶
type BoundCompletionAwareRunner ¶
type BoundCompletionAwareRunner interface {
DestroyCompletedBound(
ctx context.Context,
requestID int64,
result, leaseID, nodeName string,
leaseEpoch int64,
outcome alloc.Phase,
) error
}
BoundCompletionAwareRunner reconciles teardown with the node and lease that actually held compute before a control-plane restart erased live ownership.
type CompletionAwareRunner ¶
type CompletionAwareRunner interface {
DestroyCompleted(ctx context.Context, requestID int64, result string) error
}
CompletionAwareRunner receives GitHub's authoritative completed-job result. It is optional so runners that have no result-dependent teardown keep the smaller Runner contract.
type ControlPlaneOption ¶
type ControlPlaneOption func(*Server)
ControlPlaneOption configures a Server.
func AdvertiseNothing ¶
func AdvertiseNothing() ControlPlaneOption
AdvertiseNothing makes every listener advertise zero capacity.
It exists so the whole path — App auth, scale-set reconciliation, session, long poll — can be exercised against a REAL organization without accepting a job that nothing in this repository can yet launch. Accepting one would strand somebody's CI, which is a worse first contact than not connecting at all.
func OptionsFromConfig ¶
func OptionsFromConfig(cfg *config.Config) ([]ControlPlaneOption, error)
OptionsFromConfig is the control-plane configuration implied by billet.yaml.
It lives here rather than in cmd/billet so the whole chain — the YAML key, the parse, the control-plane option, and the listener it configures — is one package's worth of code and can be tested end to end. Assembling it at the call site left the only link that matters, a value reaching a listener, spanning two packages with a test on neither side of the join.
func WithCleanupRetry ¶
func WithCleanupRetry(first, ceiling time.Duration) ControlPlaneOption
WithCleanupRetry sets how soon, and at most how far apart, every listener retries a cleanup obligation whose destroy or release failed.
Exposed for the same reason WithReapInterval is: the defaults are sized for a node that will not answer sooner for being asked more often, and a test of a completion that settles on a LATER attempt proves nothing if the later attempt never comes.
func WithCompletionLedger ¶
func WithCompletionLedger(db *state.DB) ControlPlaneOption
WithCompletionLedger durably preserves authoritative results until nodes accept them.
func WithDrainTimeout ¶
func WithDrainTimeout(d time.Duration) ControlPlaneOption
WithDrainTimeout sets when every listener starts REPORTING that its drain is running long.
IT IS NOT A DEADLINE. A drain waits for the jobs already running for as long as they run, and nothing destroys one on the way out. This is the operator's key — server.drain_timeout — reaching the code that honours it; see defaultDrainGrace for why the default is the length of a job rather than the length of a shutdown.
func WithHurry ¶
func WithHurry(c <-chan struct{}) ControlPlaneOption
WithHurry gives every listener the channel that ends its drain wait early.
This is the operator's second signal reaching the code that honours it. See WithHurrySignal for what a listener does with it.
func WithLeadershipLost ¶
func WithLeadershipLost(fn func() bool) ControlPlaneOption
WithLeadershipLost gives every listener the question that decides whether its teardown may act on anything at all.
SUPPLIED BY cmd/billet RATHER THAN DERIVED HERE, because the fact belongs to the ledger handle: `state.DB.LeadershipLost` latches inside the write transaction that was refused, which is before the refusal reaches any listener and therefore before any of them starts unwinding. A server that inferred it from an error would tell the one listener that saw it and leave its siblings tearing down as if nothing had happened.
func WithNodeRunner ¶
func WithNodeRunner(r Runner) ControlPlaneOption
WithNodeRunner attaches the compute every listener launches onto.
Without it the listeners fail closed: they account for capacity and decline the work, rather than accepting jobs nothing can run.
func WithReapInterval ¶
func WithReapInterval(d time.Duration) ControlPlaneOption
WithReapInterval sets how often abandoned capacity is reclaimed.
Exposed so a test can make the reaper actually fire: the default is slow relative to any test, so a reaper/heartbeat test left on it never reaches the reaper and proves nothing while passing.
func WithRolloutCoordinator ¶
func WithRolloutCoordinator(c *rollout.Coordinator, every time.Duration) ControlPlaneOption
WithRolloutCoordinator gives the control plane the driver for a fleet rollout.
OPTIONAL, because a deployment whose runner is not the node plane has nobody to dispatch an upgrade to — and a coordinator that could never dispatch would hold a rollout open forever reporting nothing.
func WithRolloutStarter ¶ added in v0.6.0
func WithRolloutStarter(s *rollout.Starter, every time.Duration) ControlPlaneOption
WithRolloutStarter gives the control plane the driver that begins a rollout when the deployment's channel advances.
OPTIONAL FOR THE REASON THE COORDINATOR IS: a deployment whose runner is not the node plane has nobody to converge, so it has nothing to start either.
func WithStagedCredentialSweeper ¶
func WithStagedCredentialSweeper(s StagedCredentialSweeper) ControlPlaneOption
WithStagedCredentialSweeper attaches the sweep. Without it nothing is swept, which is right for a deployment with no backend that stages a credential.
func WithTargets ¶ added in v0.10.0
func WithTargets(targets ...Target) ControlPlaneOption
WithTargets names the targets this control plane serves.
A TIER IS RESOLVED TO ITS TARGET, AND EVERYTHING ABOUT ITS SCALE SET GOES THROUGH THAT TARGET'S PROVISIONER: reconciliation, the session, the runner registry and the recorded provenance, keyed by the target's GitHub path so a record of one can be told apart from another target's under the same state directory. A tier naming no target resolves to the only one; with several, a tier naming none or naming one not listed here stops Run before any listener starts, because a scale set created with the wrong credential is a scale set on the wrong owner.
type Job ¶
type Job struct {
RequestID int64
RunID int64
// RunnerID and RunnerName identify the pool member GitHub actually bound.
// They are authoritative only on JobStarted and JobCompleted messages.
RunnerID int64
// JobID is GitHub's stable workflow-job identity. It is required when the
// direct-assignment path sends RequestID zero.
JobID string
// CompletionID is the scale-set message that delivered the result. A
// redelivery keeps it; a later reuse of RequestID receives a different one.
CompletionID int64
// RunnerName is GitHub's name for the ephemeral runner. Completed messages
// can omit RequestID, so the billet-issued name is the durable route back to
// the lease and its assigned request.
RunnerName string
// Result is GitHub's conclusion on a completed-job message. It is empty on
// available and assigned messages.
Result string
// The GitHub event that queued this job — retained for diagnostics only. A JIT
// runner joins a pool before GitHub chooses its job, so event is not launch
// authority; the tier's static trust policy is.
Event string
// Owner, Repository and WorkflowRef are GitHub's authenticated cache scope.
// They come from the scale-set assignment, never from a workflow-controlled
// environment variable or by decoding the Actions runtime token.
Owner string
Repository string
WorkflowRef string
}
Job identifies one workflow job.
RequestID is billet's numeric scheduler identity. GitHub's positive runnerRequestId is used unchanged; a direct assignment carrying zero receives a durable negative id keyed by JobID, so concurrent jobs never alias at zero.
type Listener ¶
type Listener struct {
// contains filtered or unexported fields
}
Listener runs one tier's scale set.
func NewListener ¶
NewListener builds a listener for one tier.
func (*Listener) Acquiring ¶
Acquiring reports how many offers this listener has escrow promised to and has not yet been assigned. Exported for tests, which cannot read the guarded field safely.
func (*Listener) Backlog ¶
Backlog is what GitHub last said was assigned to this scale set and not yet finished.
TotalAssignedJobs is the documented scaling signal; counting messages is not, because a response carries at most 50 and a large backlog is truncated.
func (*Listener) Held ¶
Held returns the leases this listener has escrowed and not yet handed to a job.
Exported for tests, which need lease IDENTITY rather than a count: an escrow that was lost and rebuilt has the same size and different ids.
func (*Listener) Run ¶
Run polls until the context is done.
The order of operations is the design: capacity is escrowed BEFORE it is advertised, and only what the escrow actually returned is advertised.
The other way round — advertise what this tier could theoretically take, reserve when GitHub assigns — over-admits by construction on any host with more than one tier: each listener computes a maximum from the same free pool, GitHub fills all of them at once, and reserving on assignment is too late.
The vendor's own listener package computes a desired runner count itself, which is why billet does not use it.
type Message ¶
type Message struct {
MessageID int64
Statistics *Statistics
// Available is work GitHub is OFFERING. Acquiring one of these is how a
// scale set claims it.
Available []Job
// Assigned is work this scale set has been given, which is the confirmation
// that an acquisition succeeded.
Assigned []Job
// Started binds a registered pool member to the job it actually consumed.
// GitHub may choose a different member than the assignment that caused Billet
// to scale up.
Started []Job
Completed []Job
}
Message is one batch of scale-set news.
type Option ¶
type Option func(*Listener)
Option configures a Listener.
func WithCleanupRetryPacing ¶
WithCleanupRetryPacing sets how long a failed cleanup retry waits, and the ceiling it doubles towards.
A first value of zero or less turns pacing off, which only a test wants: in production it lets one unreachable node occupy each pass ahead of a node that has just come back.
func WithCompletionStore ¶
WithCompletionStore makes result delivery and its capacity settlement durable across listener restarts.
func WithDrainGrace ¶
WithDrainGrace sets when a stopping listener starts REPORTING that its drain is running long. It bounds nothing.
Validated against maxDrainGrace rather than maxGrace, because this is the one budget here that waits on somebody else's job rather than on billet's own teardown. See defaultDrainGrace.
func WithFinishGraces ¶
WithFinishGraces bounds the two local phases of the teardown: closing the session, and releasing leases.
Separate from the shutdown grace because they wait on nothing remote, and separate from each other because a slow close must not leave the releases to fail on what is left of a shared budget.
func WithHurrySignal ¶
func WithHurrySignal(c <-chan struct{}) Option
WithHurrySignal gives the listener a channel whose closing ends the drain's wait.
THE ONLY THING THAT ENDS A DRAIN WITH WORK STILL RUNNING, now that nothing bounds it. An operator who cannot wait needs a lever that stops the WAITING without stopping the teardown billet owes; what follows is the session close, the idle escrow, and the destroys a completion already asked for. The jobs still executing are left alone. Without this the only escape is killing the process, which loses billet's bookkeeping rather than the work — but leaves the operator no orderly way out.
func WithLeadershipLostCheck ¶
WithLeadershipLostCheck supplies the question "has this process stopped being this deployment's controller", which the teardown asks before it acts on anything. Named like WithHurrySignal beside it: the listener option carries the specific name and the control-plane one that forwards it carries the short one.
A PREDICATE RATHER THAN A CHANNEL, unlike the hurry signal beside it, and the difference is what each one means. A hurry is an EVENT an operator sends once, and a listener that was not watching when it arrived must still see it. This is a durable FACT about the process — `state.DB.LeadershipLost` latches and never clears — so the only thing a caller ever needs is to ask.
NIL EVERYWHERE BUT THE CONTROL PLANE. Nothing else has a claim to lose.
func WithLogger ¶
WithLogger sets the logger. The default is slog.Default().
func WithMaxCapacity ¶
WithMaxCapacity caps what this listener will ever advertise.
Zero means advertise nothing: connect, reconcile, poll, and tell GitHub there is no room — which is what makes a first run against a real organization safe.
A negative value is rejected rather than clamped: "advertise -1" means the caller computed something wrong, and turning it into 0 hides that.
func WithRunner ¶
WithRunner sets what turns assigned leases into running compute.
THE DEFAULT DECLINES THE JOB, which is the opposite of what this comment used to claim. noRunner fails closed: it returns an error, the ordinary failed-launch path hands the capacity back, and GitHub reassigns. It does not hold capacity and it does not quietly succeed — see noRunner's own documentation, which said so correctly while this said the reverse.
func WithRunnerRegistry ¶
func WithRunnerRegistry(registry RunnerRegistry) Option
WithRunnerRegistry installs the GitHub side of safe runner retirement.
func WithShutdownGrace ¶
WithShutdownGrace bounds the teardown: how long the listener will spend destroying compute and closing its session before giving up and letting the reaper deal with what is left.
Worth setting on a deployment whose provider is genuinely slow to destroy — the alternative to a grace that is too short is not a cleaner shutdown, it is leases nobody releases and containers nobody removes.
func WithStalePromiseAfter ¶
WithStalePromiseAfter sets how long an acquired job may go unassigned before billet reports it. It does not reclaim anything — see defaultStalePromise.
type Provisioner ¶
type Provisioner interface {
// EnsureScaleSet makes a tier's scale set exist. It must be idempotent: it is
// called on every start, and a scale set an operator created by hand is
// adopted rather than treated as a conflict.
EnsureScaleSet(ctx context.Context, name, group string, labels []string) (*ScaleSet, error)
// Session opens a long-poll session on one scale set.
Session(ctx context.Context, scaleSetID int, owner string) (Session, error)
RunnerRegistry
}
Provisioner creates scale sets and opens message sessions on them.
Billet's own interface, implemented by internal/scaleset. Two methods is the whole of what the control plane needs from GitHub's API, and keeping it that small is what lets the scheduler be tested against a fake.
type Runner ¶
type Runner interface {
// The lease is already durable and counted against the budget, so a failure
// here means capacity is held for something that is not running; the caller
// releases it.
Launch(ctx context.Context, lease *alloc.Lease, job Job) error
// MUST be idempotent: it runs on redelivered completions, on shutdown, and on
// paths that have already failed once.
Destroy(ctx context.Context, requestID int64) error
}
Runner turns an assigned lease into running compute, and tears it down again. It is the seam between the control plane and a host.
Both methods are called OUTSIDE the escrow mutex: launching pulls images and talks to a hypervisor, and holding the mutex across that would stall every heartbeat behind it.
type RunnerRegistry ¶
type RunnerRegistry interface {
RemoveRunner(ctx context.Context, runnerID int64, runnerName string) error
}
RunnerRegistry removes a GitHub registration before its guest is destroyed. A failed removal leaves compute and capacity held so GitHub cannot race a new assignment onto a guest Billet is tearing down.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is billet's control plane: one listener per tier, one shared capacity budget between them.
func New ¶
func New( a *alloc.Allocator, prov Provisioner, tiers []config.Tier, owner string, log *slog.Logger, opts ...ControlPlaneOption, ) *Server
New builds a control plane over a configured tier catalog.
func (*Server) Run ¶
Run reconciles a scale set per tier and listens on all of them until the context is done.
Reconciliation happens FIRST and for every tier, before any listener starts. Starting listeners as their scale sets appear would mean a tier whose reconciliation fails leaves the others running against a budget they are quietly splitting with a tier that will never take work — and the operator sees a half-configured control plane reported as healthy.
type Session ¶
type Session interface {
// Returns ErrNoMessage when the poll times out with nothing to report, which
// is the ordinary case.
GetMessage(ctx context.Context, lastMessageID int64, maxCapacity int) (*Message, error)
// An unacknowledged message is redelivered, so everything derived from one
// must be idempotent.
DeleteMessage(ctx context.Context, messageID int64) error
// Returns the ids actually acquired, which may be fewer than asked for.
AcquireJobs(ctx context.Context, requestIDs []int64) ([]int64, error)
// What GitHub said when the session opened, or nil. The only view of a backlog
// that predates the session.
Statistics() *Statistics
Close(ctx context.Context) error
}
Session is the part of a GitHub scale-set message session billet uses.
billet's own interface rather than the vendor's: the scale-set client is a public preview whose interfaces may change, and a four-method fake is what makes the capacity arithmetic testable without a GitHub organization.
type StagedCredentialSweeper ¶
type StagedCredentialSweeper interface {
// SweepStagedCredentials runs one pass. An error is reported and the next
// tick tries again; it never stops the control plane.
SweepStagedCredentials(ctx context.Context) error
}
StagedCredentialSweeper removes runner registrations that were staged OUTSIDE the compute they were minted for and that nothing else will ever remove.
A CodeBuild build cannot be handed a secret, so its registration lives in Parameter Store and outlives the build; a node that dies between staging one and settling its lease leaks it. Only the ledger can authorise the delete — the lease terminal, and closed longer ago than any build could still be running — and only this process holds the ledger, which is why the sweep runs here on the reaper's clock rather than on a node.
type Statistics ¶
type Statistics struct {
TotalAvailableJobs int
TotalAcquiredJobs int
TotalAssignedJobs int
TotalRunningJobs int
TotalRegisteredRunners int
TotalBusyRunners int
TotalIdleRunners int
}
Statistics is GitHub's own view of the scale set.
TotalAssignedJobs is the ONLY field to scale on. A message carries at most 50 job entries and a large backlog is truncated, so counting what arrived undercounts exactly when the undercount is most expensive.
type Sweeper ¶
type Sweeper interface {
// Sweep destroys compute whose lease is no longer open. Called after each
// reap, because reaping is what MAKES a container an orphan.
Sweep(ctx context.Context) error
// Tend advances compute the runner holds capacity for: heartbeating those
// leases, letting adopted work finish, and destroying what is confirmed
// finished. The mirror of Sweep — that finds compute no lease is holding, this
// holds leases whose compute is unaccounted for.
Tend(ctx context.Context) error
// KeepAlive renews held leases until the context ends, on its OWN clock.
// Separate from Tend because renewal must not share a schedule with anything
// that talks to a compute backend: a slow `docker ps` would delay the next
// renewal past the lease TTL and let the reaper reclaim capacity held on
// purpose. Blocks until ctx is done.
KeepAlive(ctx context.Context)
}
Sweeper is a Runner that can also find compute nothing is asking about.
Optional, and asserted for rather than required: Launch and Destroy are per-job, but enumerating everything a backend runs is a whole-host operation a node may not be able to answer during a partition.
type Target ¶ added in v0.10.0
type Target struct {
Config config.GitHubTarget
Provisioner Provisioner
}
Target is one GitHub organization or repository this control plane serves, and the provisioner holding that target's credential.
Source Files