Documentation
¶
Overview ¶
Package scheduler is the registered-pool scheduler (design/workstreams/dispatcher-runtime/ LLD/scheduler-and-pools.md): agent registration and tokens, the long-poll lease matcher (queues, budgets, lending, harness resolution, anti-affinity, fairness), heartbeats, lease expiry with the queue's retry policy, and abandon.
It has no HTTP: the API server (RT-S2-T03) calls Register, Authenticate, UpdateAgent, Deregister, Lease, Heartbeat, Complete and Abandon. Run drives the lease controller (expiry, revocation) and the 1 s matching tick.
Index ¶
- Constants
- Variables
- func AgentID(login, host string) string
- func CounterName(scope, name string, t time.Time) string
- type AbandonRequest
- type AgentRefresh
- type AgentToken
- type AgentUpdate
- type Deps
- type EnrollRequest
- type Enrollment
- type EnrollmentSpec
- type EnrollmentStatus
- type Grant
- type HeartbeatRequest
- type HeartbeatResponse
- type Hours
- type Lease
- type LeaseInfo
- type LeaseSpec
- type LeaseStatus
- type LeaseUsage
- type Options
- type Owner
- type RegisterRequest
- type Registration
- type RunAuth
- type Scheduler
- func (s *Scheduler) Abandon(ctx context.Context, agentID, leaseID string, req AbandonRequest) error
- func (s *Scheduler) Authenticate(ctx context.Context, token string) (string, error)
- func (s *Scheduler) AuthenticateRun(ctx context.Context, token string) (RunAuth, error)
- func (s *Scheduler) Complete(ctx context.Context, agentID, leaseID string, res v1alpha1.Result) (v1alpha1.TaskPhase, error)
- func (s *Scheduler) Controller() *controllers.Controller
- func (s *Scheduler) CreateEnrollment(ctx context.Context, req EnrollRequest) (*Enrollment, string, error)
- func (s *Scheduler) DeleteEnrollment(ctx context.Context, id string) error
- func (s *Scheduler) Deregister(ctx context.Context, agentID string) error
- func (s *Scheduler) Explain(ctx context.Context, taskID, agentID string) (string, error)
- func (s *Scheduler) Heartbeat(ctx context.Context, agentID, leaseID string, req HeartbeatRequest) (*HeartbeatResponse, error)
- func (s *Scheduler) Lease(ctx context.Context, agentID string, wait time.Duration) (*Grant, error)
- func (s *Scheduler) ListEnrollments(ctx context.Context) ([]*Enrollment, error)
- func (s *Scheduler) Reconcile(ctx context.Context, taskID string) (controllers.Result, error)
- func (s *Scheduler) Refresh(ctx context.Context) error
- func (s *Scheduler) Register(ctx context.Context, req RegisterRequest) (*Registration, error)
- func (s *Scheduler) RevokeRefresh(ctx context.Context, agentID string) error
- func (s *Scheduler) Run(ctx context.Context) error
- func (s *Scheduler) UpdateAgent(ctx context.Context, agentID string, u AgentUpdate) (*v1alpha1.Agent, error)
- type UsageDelta
- type ValidationError
Constants ¶
const ( ScopeQueue = "queue" ScopeRuleSet = "ruleset" ScopeGlobal = "global" ScopeAgent = "agent" )
Budget counter scopes: budget.<scope>.<name>.<YYYY-MM-DD>. The agent scope holds an agent's daily usage, checked against its lending (maxRunsPerDay, maxTokensPerDay) and mirrored on Agent.status.today.
const ( // KindEnrollment is an enrollment, its token hashed, named by its enl_ id. KindEnrollment = "Enrollment" // KindAgentRefresh is an enrolled agent's refresh token, hashed, named by the agent id. KindAgentRefresh = "AgentRefreshToken" // LabelEnrollmentToken and LabelRefreshToken are the hex SHA-256 of the record's token: // what a registration looks the record up by. LabelEnrollmentToken = "factory.ccf.dev/enrollment-token" LabelRefreshToken = "factory.ccf.dev/refresh-token" // EnrollmentTokenPrefix and RefreshTokenPrefix start the two tokens (the enrollmentToken // and agentRefreshToken schemes). EnrollmentTokenPrefix = "enr_" RefreshTokenPrefix = "arf_" // DefaultEnrollmentTTL and MaxEnrollmentTTL bound an enrollment token's life. DefaultEnrollmentTTL = 7 * 24 * time.Hour MaxEnrollmentTTL = 30 * 24 * time.Hour )
Enrollment (P1 §6): an admin mints a one-time enrollment token for a pool, an owner and a name; the agent host's supervisor exchanges it once for a refresh token, which re-registers it from then on and rotates on each use.
const ( // IdentityReviewer and IdentityAuthor are the TaskSpec identities anti-affinity compares. IdentityReviewer = "reviewer" IdentityAuthor = "author" )
Names the matcher knows.
const ( // MintBackoffInitial is the delay after a task's first failed mint. MintBackoffInitial = 30 * time.Second // MintBackoffMax caps the delay, which doubles on each failure after the first. MintBackoffMax = 10 * time.Minute )
The git-credential backoff: a pending task whose credentials failed to mint isn't offered to any agent again before its delay has passed, so a missing App installation costs one provider call per delay rather than one per poll.
const ( LabelOwner = "factory.ccf.dev/owner" LabelPool = "factory.ccf.dev/pool" )
Agent labels.
const ( DefaultTick = time.Second DefaultTokenTTL = 24 * time.Hour DefaultTokenRenewEvery = 10 * time.Minute DefaultAntiAffinityWait = 10 * time.Minute DefaultMaxWait = 60 * time.Second DefaultLeaseRetention = time.Hour DefaultThrottle = 5 * time.Minute DefaultLeaseTTL = 2 * time.Minute DefaultHeartbeat = 30 * time.Second // DefaultSchedWait bounds the wait for the sched lock, and how long a timeout keeps the // next passes from waiting (P3 §Locks and leases). DefaultSchedWait = 2 * time.Second // DefaultSchedTTL is the sched lock's lease; a commit section runs under half of it. DefaultSchedTTL = 30 * time.Second // CredentialRefreshBefore is how long before its expiry a lease's git credential is // re-minted on a heartbeat. CredentialRefreshBefore = 10 * time.Minute // MaxCredentialTTL caps a git credential (design/02 §Safety rails: ≤ 1h). MaxCredentialTTL = time.Hour )
Defaults for Options and for pools without a lease policy.
const ( FairnessPreferLimited = "prefer-limited" FairnessPreferUnlimited = "prefer-unlimited" )
Pool fairness values (Pool.spec.fairness).
const ( ReasonQueueFull = "queue-full" ReasonBudget = "budget" ReasonAgentBusy = "agent-busy" ReasonAgentThrottled = "agent-throttled" ReasonLending = "lending" ReasonNoCapableAgent = "no-capable-agent" ReasonAntiAffinity = "anti-affinity" // ReasonAgentLost is a lease that expired without a heartbeat (retry reason agent-lost). ReasonAgentLost = string(v1alpha1.RetryAgentLost) // ReasonGitCredential is a task whose git credential could not be minted at lease time. ReasonGitCredential = "git-credential" )
Reasons the scheduler writes on a Pending task's status (the LLD's matching table), plus the expiry and credential reasons.
const ( // KindLease is one lease: which agent holds which task until when, and the usage deltas // its heartbeats reported. It never holds a credential, only its run token's hash. KindLease = "Lease" // KindAgentToken is an agent's bearer token, hashed (SHA-256), named by the agent id. KindAgentToken = "AgentToken" )
Internal kinds the scheduler keeps in the store (store.MetaObject; no API, no schema).
const ( LabelTask = "factory.ccf.dev/task" LabelAgent = "factory.ccf.dev/agent" // LabelRunToken is the hex SHA-256 of the lease's run token: what AuthenticateRun looks // the lease up by. LabelRunToken = "factory.ccf.dev/run-token" )
Labels on Lease records.
const ( LeaseCompleted = "completed" LeaseAbandoned = "abandoned" LeaseExpired = "expired" // LeaseRevoked is a lease whose task left it: a run timeout, a cancel without an agent, // a deregistration, or a task finished by someone else. LeaseRevoked = "revoked" )
Lease ends (LeaseStatus.End).
const AntiAffinityOff time.Duration = -1
AntiAffinityOff as Options.AntiAffinityWait turns anti-affinity off.
const ReasonNotInPool = "not-in-pool"
ReasonNotInPool is Explain's answer for an agent that isn't a candidate at all: the task's queue feeds another pool (or the queue or pool is gone from the catalog).
const RunTokenPrefix = "rtk_"
RunTokenPrefix starts every run token (the runToken scheme's format). It differs from the run IDs' run_ so a token is always recognisable as a credential.
const SchedLock = "sched"
SchedLock is the store lock every lease commit, on every dispatcher, re-checks and writes under (P3 §Locks and leases): one lock, not one per pool, since lending lets a queue's tasks run in other pools.
Variables ¶
var ( ErrUnauthorized = errors.New("unauthorized") // ErrForbidden: authenticated but not allowed: the owner is not in the pool's owners // Group, the capabilities miss Pool.require, or the lease is someone else's (403). ErrForbidden = errors.New("forbidden") // ErrInvalid: a malformed request (400). Validation failures are *ValidationError. ErrInvalid = errors.New("invalid request") // ErrNotFound: no such agent or lease (404). ErrNotFound = errors.New("not found") // ErrLeaseGone: the lease ended (expired, revoked after a timeout or cancel, completed or // abandoned): the agent must stop the run (409). ErrLeaseGone = errors.New("lease gone") // ErrNotLeaseHolder: the lease belongs to another agent (403 not-lease-holder). ErrNotLeaseHolder = fmt.Errorf("not the lease holder: %w", ErrForbidden) )
Errors of the scheduler API. Every error it returns for a bad request or a missing object wraps one of these, so the API server maps them to HTTP statuses with errors.Is.
Functions ¶
Types ¶
type AbandonRequest ¶
type AbandonRequest struct {
// Reason is throttled, agent-error or shutdown.
Reason string
// Until is when a throttled harness's limit resets.
Until time.Time
Message string
}
AbandonRequest is abandonLease's body.
type AgentRefresh ¶
type AgentRefresh struct {
v1alpha1.TypeMeta
Metadata v1alpha1.ObjectMeta `json:"metadata"`
// Hash is "sha256:<hex>" of the current refresh token; each use replaces it.
Hash string `json:"hash"`
// Enrollment is the enrollment that registered the agent; Pool, Owner, OwnerProvider and
// Name are its, which a registration must match.
Enrollment string `json:"enrollment"`
Pool string `json:"pool"`
Owner string `json:"owner"`
OwnerProvider string `json:"ownerProvider"`
Name string `json:"name"`
RotatedAt time.Time `json:"rotatedAt"`
}
AgentRefresh is an enrolled agent's refresh token, stored hashed. An admin's deregistration deletes it.
func (*AgentRefresh) GetName ¶
func (a *AgentRefresh) GetName() string
GetName returns metadata.name (the agent id).
func (*AgentRefresh) GetObjectMeta ¶
func (a *AgentRefresh) GetObjectMeta() *v1alpha1.ObjectMeta
GetObjectMeta implements store.MetaObject.
type AgentToken ¶
type AgentToken struct {
v1alpha1.TypeMeta
Metadata v1alpha1.ObjectMeta `json:"metadata"`
// Hash is "sha256:<hex>" of the token's secret part.
Hash string `json:"hash"`
ExpiresAt time.Time `json:"expiresAt"`
RenewedAt time.Time `json:"renewedAt"`
}
AgentToken is an agent's token, stored hashed.
func (*AgentToken) GetName ¶
func (a *AgentToken) GetName() string
GetName returns metadata.name (the agent id).
func (*AgentToken) GetObjectMeta ¶
func (a *AgentToken) GetObjectMeta() *v1alpha1.ObjectMeta
GetObjectMeta implements store.MetaObject.
type AgentUpdate ¶
type AgentUpdate struct {
// Phase is Ready, Throttled (with Until), Draining or Offline; empty keeps it.
Phase v1alpha1.AgentPhase
Until time.Time
// Lending, when set, replaces the agent's lending.
Lending *v1alpha1.Lending
}
AgentUpdate is updateAgent (PATCH /agents/{id}): a phase report and/or a lending update.
type Deps ¶
type Deps struct {
Store store.Store
Catalog controllers.CatalogProvider
// Conns mints git credentials.
Conns controllers.ConnSource
// Runs is the run controller: lease issued, complete, abandon, cancel.
Runs controllers.RunAPI
}
Deps are what the scheduler works with.
type EnrollRequest ¶
type EnrollRequest struct {
Pool, Owner, Name string
// TTL is the token's life; 0 is DefaultEnrollmentTTL, at most MaxEnrollmentTTL.
TTL time.Duration
// OwnerProvider is the Provider the owner's login is on; CreatedBy the admin's actor id.
OwnerProvider, CreatedBy string
}
EnrollRequest is createEnrollment, plus who asks and the owners' provider.
type Enrollment ¶
type Enrollment struct {
v1alpha1.TypeMeta
Metadata v1alpha1.ObjectMeta `json:"metadata"`
Spec EnrollmentSpec `json:"spec"`
Status EnrollmentStatus `json:"status"`
}
Enrollment is the record of one enrollment. It never holds its token, only the hash.
func (*Enrollment) GetName ¶
func (e *Enrollment) GetName() string
GetName returns metadata.name (the enl_ id).
func (*Enrollment) GetObjectMeta ¶
func (e *Enrollment) GetObjectMeta() *v1alpha1.ObjectMeta
GetObjectMeta implements store.MetaObject.
type EnrollmentSpec ¶
type EnrollmentSpec struct {
Pool string `json:"pool"`
// Owner is the owner's login on OwnerProvider (the dispatcher's sign-in provider).
Owner string `json:"owner"`
OwnerProvider string `json:"ownerProvider"`
// Name is the agent host's name: the agent is agt_<owner>-<name>.
Name string `json:"name"`
CreatedBy string `json:"createdBy"`
ExpiresAt time.Time `json:"expiresAt"`
// TokenHash is "sha256:<hex>" of the enrollment token.
TokenHash string `json:"tokenHash"`
}
EnrollmentSpec is what the admin fixed.
type EnrollmentStatus ¶
type EnrollmentStatus struct {
UsedAt time.Time `json:"usedAt,omitzero"`
Agent string `json:"agent,omitzero"`
}
EnrollmentStatus says whether the token was spent, and on which agent.
type Grant ¶
type Grant struct {
Lease LeaseInfo
Task *v1alpha1.Task
// GitCredential is minted for this lease and never stored; nil when the task has no git
// credential request.
GitCredential *provider.Credential
// ReadCredentials are the read-only tokens (contents: read), one per private read repo;
// none for public reads. Minted for this lease and never stored.
ReadCredentials []provider.Credential
// RunToken is the lease's run token (P1 §5): minted for this lease, returned only here,
// stored only as its hash. Never log it.
RunToken string
}
Grant is a lease handed to an agent: leaseTask's 200 response.
type HeartbeatRequest ¶
type HeartbeatRequest struct {
Progress []v1alpha1.ProgressEvent
Usage *UsageDelta
}
HeartbeatRequest is the progress and usage since the previous heartbeat.
type HeartbeatResponse ¶
type HeartbeatResponse struct {
ExpiresAt time.Time
// Cancel tells the agent to stop the run: the task is being cancelled.
Cancel bool
// RefreshedGitCredential is a new credential when the first of the lease's credentials to
// expire had under 10 minutes left.
RefreshedGitCredential *provider.Credential
// RefreshedReadCredentials replace the lease's read-only credentials, refreshed with
// RefreshedGitCredential; empty when the task has none.
RefreshedReadCredentials []provider.Credential
}
HeartbeatResponse is heartbeatLease's response.
type Hours ¶
type Hours struct {
// contains filtered or unexported fields
}
Hours is a parsed lending.hours window: "[<days>] <HH:MM>-<HH:MM> [<zone>]", e.g. "Mon-Fri 08:00-20:00 Europe/Lisbon".
- days: comma-separated days or ranges (Mon, Mon-Fri, Fri-Mon wraps); default every day;
- the time range is local to the zone (default UTC); an end before the start crosses midnight and belongs to the day it starts on ("Fri 22:00-06:00" includes Saturday 03:00, not Friday 03:00); equal start and end is the whole day.
func ParseHours ¶
ParseHours parses a lending.hours value.
type Lease ¶
type Lease struct {
v1alpha1.TypeMeta
Metadata v1alpha1.ObjectMeta `json:"metadata"`
Spec LeaseSpec `json:"spec"`
Status LeaseStatus `json:"status"`
}
Lease is the record of one lease. Ended leases are kept for LeaseRetention so a retried complete is answered, then deleted.
func (*Lease) GetObjectMeta ¶
func (l *Lease) GetObjectMeta() *v1alpha1.ObjectMeta
GetObjectMeta implements store.MetaObject.
type LeaseInfo ¶
type LeaseInfo struct {
ID string
Agent string
Task string
Run string
ExpiresAt time.Time
HeartbeatEvery time.Duration
}
LeaseInfo is the API's LeaseInfo.
type LeaseSpec ¶
type LeaseSpec struct {
Task string `json:"task"`
Agent string `json:"agent"`
Pool string `json:"pool"`
Queue string `json:"queue"`
RuleSet string `json:"ruleSet,omitzero"`
// Auth is the resolved harness's auth (USD budgets count apiKey runs only).
Auth string `json:"auth,omitzero"`
IssuedAt time.Time `json:"issuedAt"`
TTL time.Duration `json:"ttl"`
// Heartbeat is the pool's heartbeat interval, returned to the agent.
Heartbeat time.Duration `json:"heartbeat"`
// RunTokenHash is "sha256:<hex>" of the lease's run token (the token itself is returned
// once, in the grant, and never stored).
RunTokenHash string `json:"runTokenHash,omitzero"`
}
LeaseSpec is what was leased to whom.
type LeaseStatus ¶
type LeaseStatus struct {
ExpiresAt time.Time `json:"expiresAt"`
LastHeartbeat time.Time `json:"lastHeartbeat,omitzero"`
// CredentialExpiresAt is when the first of the git credentials last minted for the lease
// expires (the credentials themselves are never stored).
CredentialExpiresAt time.Time `json:"credentialExpiresAt,omitzero"`
// Usage is the sum of the heartbeats' usage deltas.
Usage LeaseUsage `json:"usage"`
End string `json:"end,omitzero"`
EndedAt time.Time `json:"endedAt,omitzero"`
}
LeaseStatus is the lease's state.
type LeaseUsage ¶
type LeaseUsage struct {
InputTokens int64 `json:"inputTokens"`
OutputTokens int64 `json:"outputTokens"`
CostUSD float64 `json:"costUSD"`
}
LeaseUsage is usage summed over heartbeats.
type Options ¶
type Options struct {
// Clock times leases, tokens, hours and budgets, and arms the long-poll timers.
Clock clock.Clock
// Logger. Default slog.Default().
Logger *slog.Logger
// Rand is the source of tokens and lease ids. Default crypto/rand.
Rand io.Reader
// Tick is the matching tick: parked polls retry and pending tasks' reasons are
// refreshed this often. Default 1 s.
Tick time.Duration
// TokenTTL is an agent token's lifetime, renewed by use. Default 24 h.
TokenTTL time.Duration
// TokenRenewEvery bounds how often a token's expiry is written. Default 10 min.
TokenRenewEvery time.Duration
// AntiAffinityWait is how long a reviewer task waits for another agent before its
// author's agent may take it on the author's harness (soft anti-affinity). Default
// 10 min; AntiAffinityOff (any negative value) turns anti-affinity off.
AntiAffinityWait time.Duration
// MaxWait caps a long poll. Default 60 s.
MaxWait time.Duration
// LeaseRetention keeps ended leases, so a retried complete is answered. Default 1 h.
LeaseRetention time.Duration
// Throttle is how long an agent throttles when it gives no until. Default 5 min.
Throttle time.Duration
// Workers are the lease controller's workers. Default 2.
Workers int
// Leader, when set, gates the leader-only work: the lease controller's reconciles and
// Refresh (P3 §Several dispatchers). Matching, heartbeats, completes and the tick's wake
// run on every instance. Nil: this process always leads.
Leader controllers.Gate
// SchedWait bounds the wait for the sched lock (DefaultSchedWait); SchedTTL is its lease
// (DefaultSchedTTL).
SchedWait, SchedTTL time.Duration
}
Options configure the scheduler. Zero fields take the defaults.
type Owner ¶
Owner is an agent's owner: the Provider instance its login is on, and the login. An enrolled agent's owner is the enrollment's, fixed by the admin who minted it (P1 §6).
type RegisterRequest ¶
type RegisterRequest struct {
// EnrollmentToken is X-Factory-Enrollment-Token and RefreshToken
// X-Factory-Agent-Refresh-Token (P1 §6): exactly one is set. Never stored or logged.
EnrollmentToken string
RefreshToken string
Pool string
Host string
AgentVersion string
Capabilities v1alpha1.AgentCapabilities
Lending *v1alpha1.Lending
}
RegisterRequest is registerAgent: the header credential and the body.
type Registration ¶
type Registration struct {
AgentID string
AgentToken string
ExpiresAt time.Time
// RefreshToken is the agent's next refresh token.
RefreshToken string
}
Registration is registerAgent's response. AgentToken and RefreshToken are returned only here.
type RunAuth ¶
type RunAuth struct {
Lease, Task, Agent string
}
RunAuth is what a run token authenticates: its lease, the lease's task and agent.
type Scheduler ¶
type Scheduler struct {
// contains filtered or unexported fields
}
Scheduler is the registered-pool scheduler. It is safe for concurrent use.
func (*Scheduler) Abandon ¶
Abandon is abandonLease: the run controller returns the task to Pending (throttled: without an attempt, and the agent is Throttled until req.Until) or retries or fails it per the queue's retry policy; the lease ends.
func (*Scheduler) Authenticate ¶
Authenticate checks an agent token ("<agentId>.<secret>") against its stored hash and returns the agent id. It renews the token (now + TokenTTL) at most every TokenRenewEvery: heartbeats and polls keep a live agent's token valid.
func (*Scheduler) AuthenticateRun ¶
AuthenticateRun checks a run token and returns what it authenticates. The token's lease must be live: not ended (completed, abandoned, expired or revoked) and not past its expiry. Every refusal wraps ErrUnauthorized and never carries the token.
func (*Scheduler) Complete ¶
func (s *Scheduler) Complete(ctx context.Context, agentID, leaseID string, res v1alpha1.Result) (v1alpha1.TaskPhase, error)
Complete is completeLease: it hands the Result to the run controller (RunAPI.Complete), then charges the usage the heartbeats didn't report (max(0, total − Σ deltas) per field) and ends the lease. A retried complete of a completed lease returns the task's phase.
func (*Scheduler) Controller ¶
func (s *Scheduler) Controller() *controllers.Controller
Controller returns the lease controller (keyed by task id).
func (*Scheduler) CreateEnrollment ¶
func (s *Scheduler) CreateEnrollment(ctx context.Context, req EnrollRequest) (*Enrollment, string, error)
CreateEnrollment is createEnrollment: it mints an enrollment for a registered pool and returns it and its token, which is returned only here (the record keeps its hash).
func (*Scheduler) DeleteEnrollment ¶
DeleteEnrollment is deleteEnrollment: an unused enrollment's token stops working; a used one's record goes (its agent stays). An unknown id is ErrNotFound.
func (*Scheduler) Deregister ¶
Deregister is deregisterAgent: it revokes the token, marks the agent Offline and abandons its running leases (reason shutdown: retried per the queue's policy).
func (*Scheduler) Explain ¶
Explain runs the matching table for a task and an agent now and returns the first reason it fails ("" when the agent could take it; ReasonNotInPool when it isn't a candidate; git-credential while the task backs off after a failed mint). It changes nothing but the anti-affinity wait.
func (*Scheduler) Heartbeat ¶
func (s *Scheduler) Heartbeat(ctx context.Context, agentID, leaseID string, req HeartbeatRequest) (*HeartbeatResponse, error)
Heartbeat is heartbeatLease: it renews the lease (the task becomes Running on the first one), appends the progress to the run, charges the usage deltas to the budgets and the agent's day, re-mints the git credentials when the first to expire has under 10 minutes left (all of them, the read-only ones included), and says whether to cancel. A lease whose task left it (timeout, cancel, expiry) is ErrLeaseGone.
func (*Scheduler) Lease ¶
Lease is leaseTask: a long poll for a task agentID may take. It returns at once with a grant when one matches, else waits up to wait (capped at Options.MaxWait) and tries again on every task, lease or agent change and on the tick. No task within wait is (nil, nil).
func (*Scheduler) ListEnrollments ¶
func (s *Scheduler) ListEnrollments(ctx context.Context) ([]*Enrollment, error)
ListEnrollments is every enrollment, in id order.
func (*Scheduler) Reconcile ¶
Reconcile is the lease controller, keyed by task id. It
- revokes the leases whose task left them: the run timed out or was cancelled, the task was finished or re-leased (the hook RT-S2-T01 left for timeouts);
- expires the lease of a task whose leaseExpiresAt passed without a heartbeat: back to Pending (attempts + 1, agent-lost) while the queue's retry policy allows, else Failed (agent-lost); a Cancelling task becomes Cancelled;
- deletes ended leases after LeaseRetention.
func (*Scheduler) Refresh ¶
Refresh is the tick's housekeeping: agents whose token expired go Offline, throttles that passed end, the agents' today mirrors follow the day, and each pending task's reason says why no agent can take it now (the reason of the agent that got furthest down the table; a task backing off after a failed mint keeps git-credential). The anti-affinity waits and the mint backoffs of tasks no longer Pending are dropped.
func (*Scheduler) Register ¶
func (s *Scheduler) Register(ctx context.Context, req RegisterRequest) (*Registration, error)
Register is registerAgent (LLD §Registration), with one of two credentials (P1 §6):
- an enrollment token: an unused, unexpired enrollment for the request's pool and host, which it spends; the owner is the enrollment's (the admin vouched for it), and the response adds the agent's first refresh token;
- a refresh token: the agent's current one, which it rotates (the old one stops working); the response carries the next.
Then the capabilities must offer one of Pool.require.harnesses, the Agent agt_<owner>-<host> is created or replaced, and a new agent token (32 random bytes, stored hashed, TTL 24 h renewed by use) is returned once.
func (*Scheduler) RevokeRefresh ¶
RevokeRefresh deletes an agent's refresh token (an admin's deregistration). An agent without one is fine.
func (*Scheduler) Run ¶
Run runs the lease controller and the matching tick until ctx is done. With Options.Leader, the lease controller and Refresh wait for it; the tick wakes the parked polls on every instance.
func (*Scheduler) UpdateAgent ¶
func (s *Scheduler) UpdateAgent(ctx context.Context, agentID string, u AgentUpdate) (*v1alpha1.Agent, error)
UpdateAgent is updateAgent (PATCH /agents/{id}): the agent reports Ready, Throttled{until}, Draining or Offline, and/or re-sends its lending.
type UsageDelta ¶
UsageDelta is usage since the previous heartbeat. CostUSD is nil for subscription runs.
type ValidationError ¶
ValidationError is a request field that fails validation. It matches ErrInvalid.
func (*ValidationError) Error ¶
func (e *ValidationError) Error() string
func (*ValidationError) Is ¶
func (e *ValidationError) Is(target error) bool
Is makes errors.Is(err, ErrInvalid) true.