Documentation
¶
Overview ¶
Package scheduler decides which node runs which task, and keeps the promises that decision makes: a lease per assignment, a deadline for accepting one, the spec's own timeout, and the node-health policy that turns a machine going away into either a new attempt or an honest `lost`.
Index ¶
Constants ¶
const ( ReasonNoNodes = "no node is connected to the control plane" ReasonNoLabels = "no online node carries every label this task requires" ReasonAllDraining = "every node that could run this task is draining" ReasonNoRoom = "no online node has enough free CPU or memory for this task" ReasonFull = "every node that could run this task is full" ReasonNotAccepted = "node did not accept assignment" ReasonTimeout = "timeout" ReasonNoLease = "the task is on no node and holds no lease" )
The reasons a task is told it is still queued, and the reasons it is failed. They are constants because the CLI, the UI and the tests all read them, and because an operator staring at "queued" deserves the same sentence every time.
const FastTimerDivisor = 10
FastTimerDivisor is how much PODIUM_TEST_FAST_TIMERS=1 shrinks the intervals by.
const FastTimersEnv = "PODIUM_TEST_FAST_TIMERS"
FastTimersEnv shrinks every interval in Timing tenfold when it is set to a true value. Tests set it so a scenario that waits out a 120s offline threshold takes 12 seconds instead. It is read once, by TimingFromEnv, and never consulted again.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Dispatcher ¶
type Dispatcher interface {
// Candidates is one snapshot per connected node.
Candidates() []nodes.Snapshot
// Assign pushes an assignment onto a node's stream and charges its cost.
Assign(ctx context.Context, nodeID string, a *podiumv1.Assign, cost nodes.TaskCost) error
// Cancel asks a node to stop a task. It does not wait.
Cancel(ctx context.Context, nodeID, taskID, reason string) error
// LoseTask applies the node-loss policy: requeue if the spec allows it, else lost.
LoseTask(ctx context.Context, task store.Task, node store.Node, reason string)
// Release gives back the slot a node was holding for a task that never reached a
// terminal status, which is the only path the event side does not already cover.
Release(taskID string)
}
Dispatcher is the node registry, as much of it as a scheduler needs.
type Resolver ¶
type Resolver interface {
Resolve(ctx context.Context, taskID string, refs []spec.SecretRef) ([]secrets.Resolved, error)
// ResolveRegistries returns the login for each registry the images are pulled from
// that the store holds one for, and nothing for the rest.
ResolveRegistries(ctx context.Context, taskID string, images []string) ([]secrets.RegistryCredential, error)
}
Resolver turns a task's secret references into the plaintext an Assign carries. It is called immediately before the assignment and never earlier: a value should exist in server memory for as short a time as possible.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service is the scheduler: it places queued work on nodes and keeps the promises that placement makes. Two loops, deliberately separate:
- the dispatch tick claims queued tasks and assigns them, woken either by its own ticker or by the database saying a task has just become queued;
- the watchdog sweep looks at every task a node already owes an answer for and applies the deadlines — the 15s to accept an assignment, the spec's timeout, the operator's cancel, the lease.
Neither ever transitions a task from what a session says in memory. Every state change goes through TransitionTask with an explicit `from`, so a heartbeat expiring at the same instant a late event arrives is a rejected transition — a no-op — and not corruption.
func New ¶
func New(st *store.Store, disp Dispatcher, resolver Resolver, timing Timing, logger *slog.Logger) *Service
New returns the scheduler. It does not start anything; call Run.
func (*Service) Retry ¶
Retry is what the log ingest calls when a node reports an error that ended the run but could succeed elsewhere — an image the registry was too busy to serve, an engine that stuttered. The node is finished with the task either way, so the only question is whether anybody else should try, and that is the attempt budget's to answer.
type Timing ¶
type Timing struct {
// Tick is how often queued work is claimed. A pg_notify wake-up beats it to the punch
// for anything freshly submitted; this is the backstop.
Tick time.Duration
// Watchdog is how often leases, timeouts, cancels and node health are swept.
Watchdog time.Duration
// LeaseTTL is how long an assignment's lease lasts before the task has proved it is
// being worked on.
LeaseTTL time.Duration
// ProvisioningDeadline is how long a node has, after Assign, to say provisioning. Past
// it the assignment is revoked and the task requeued.
ProvisioningDeadline time.Duration
// LeaseGrace is added to the spec's timeout to give a live task's lease its expiry, so
// a lease never expires under a task that is still legitimately running.
LeaseGrace time.Duration
// UnreachableAfter is how long without a heartbeat makes a node unreachable. Its tasks
// keep running: a node that cannot talk is not a node that has stopped working.
UnreachableAfter time.Duration
// OfflineAfter is how long without a heartbeat makes a node offline and expires its
// leases.
OfflineAfter time.Duration
// CancelGrace is how long a cancelled task is given to end by itself before the server
// writes the terminal status without the node.
CancelGrace time.Duration
// ClaimLimit is how many queued tasks one tick considers. It is a batch size, not a
// duration, and PODIUM_TEST_FAST_TIMERS does not touch it.
ClaimLimit int
}
Timing is every clock the scheduler runs on, in one struct so a test can shrink them all together and so an operator reading the code finds the whole policy in one place.
The defaults are the canonical numbers from the design: heartbeat every 10s, unreachable at 30s, offline at 120s, a 15s provisioning deadline, a 30s SIGTERM→SIGKILL grace on the node and 60s before the server gives up waiting for a cancelled task's node.
func TimingFromEnv ¶
func TimingFromEnv() Timing
TimingFromEnv is DefaultTiming, shrunk when PODIUM_TEST_FAST_TIMERS says so.
Source Files