Documentation
¶
Index ¶
- Constants
- Variables
- func DeriveClaimKey(ctx context.Context, taskType string, taskID string, owner string, repo string) (string, error)
- func DeriveClaimKeysWithBody(ctx context.Context, taskType string, taskID string, owner string, repo string, ...) (string, string, bool, error)
- func RecoverAtStartup(ctx context.Context, manager LeaseManager, viewReady func() bool) error
- func RedriveStreakPhaseComponent(phase string) string
- func RunClaimRecovery(ctx context.Context, manager LeaseManager, viewReady func() bool) error
- type Claim
- type ClaimClient
- type ClaimHeldError
- type ClaimRef
- type ClaimRefClient
- type ClaimTarget
- type HeldClaim
- type JobControl
- type LeaseManager
- type RedriveReasonStore
- type RedriveStreakStore
- type StatusError
- type TakeoverResult
Constants ¶
const ( // LeaseDuration is how long a claim stays valid without a renew. LeaseDuration = 300 * libtime.Second // TickInterval is how often the lease loop ticks. TickInterval = 10 * libtime.Second // RenewInterval is how often a held claim is renewed. RenewInterval = 60 * libtime.Second // FenceMargin is how long before the lease end the holder stops its Job. FenceMargin = 60 * libtime.Second // FenceLead is how much earlier than fenceAt a tick fences, so a 10 s tick // lands at or before fenceAt. FenceLead = 10 * libtime.Second // TakeoverGrace is how long after the lease end a takeover is allowed. TakeoverGrace = 30 * libtime.Second // JoblessGrace is how long a held claim's Job must be continuously absent // before the lease loop releases the claim. JoblessGrace = 300 * libtime.Second // JoblessCheckInterval is how often the lease loop asks JobControl whether a // held claim's Job is still running. JoblessCheckInterval = 60 * libtime.Second )
const ( // RedriveReasonPrefix is the key prefix of every re-drive reason ref. // // It is a separate namespace from the frozen RedriveStreakPrefix on purpose: // parseRedriveStreakRef is exactly five segments and refs written by the // deployed build must keep parsing. A sixth segment on the streak key would // have broken every one of them. RedriveReasonPrefix = "claims/redrive-reason/" // RedriveReasonUnknown is the reason recorded when a failure reason cannot be // parsed into the closed ZombieReason vocabulary. It is never an empty key // segment. RedriveReasonUnknown = "unknown" )
const ( // RedriveStreakCluster is the sentinel cluster field on every streak ref. // git-rest refuses an empty cluster, and this value must never equal a real // executor cluster name: the lease manager's recovery and fence compare on // the cluster field. RedriveStreakCluster = "redrive-streak" // RedriveStreakPrefix is the frozen key prefix of every streak ref. // Changing it strands every existing ref. RedriveStreakPrefix = "claims/redrive/" // RedriveStreakPhaseNone is the phase component used when a task carries // no phase. RedriveStreakPhaseNone = "none" // RedriveStreakLeaseDuration is the informational lease of a streak ref. // Nothing renews or expires it. RedriveStreakLeaseDuration = 24 * libtime.Hour )
const ReadBackTimeout = 10 * libtime.Second
ReadBackTimeout bounds the recovery read-back that follows a claim write which failed without a 409. It is a fresh window, deliberately not a share of the caller's: the caller's request context has already expired in exactly the case this read-back exists to recover, so deriving from it would kill the recovery at the moment it is needed.
It is deliberately shorter than RequestTimeout, which ClaimRefClient.List applies to the same call. The read-back runs uninterruptibly (WithoutCancel) while the caller holds the lease manager's lock, so this window also bounds how long a shutdown or a fence can be held up by a store that has stopped answering.
const RequestTimeout = 30 * libtime.Second
RequestTimeout is the hard per-request cap for every git-rest claim call.
const SuccessGracePeriod = 300 * time.Second
SuccessGracePeriod is how long a claim stays held after its Job succeeded.
A succeeded Job means the agent published its result, but the vault has not necessarily absorbed it yet — the controller writes phase and status back on its own cycle. In that gap a peer cluster sees a task that still reads as eligible, and nothing tells it otherwise: the spawn path's success record is per-pod memory, so a cluster that starts cold has never heard of the result.
Holding the claim across the gap is what stops the peer, because the claim ref is the one artefact both clusters share. A peer's Claim inside the window gets a 409 and does not spawn; once the window has passed the vault carries the result and the normal phase gate decides.
It matches the spawn path's respawn grace window: the two bound one gap from opposite ends — one refuses to spawn, the other refuses to hand the claim over.
Variables ¶
var ( // ErrClaimHeld is returned when another holder owns the claim. The returned // *ClaimHeldError carries who holds it and until when. ErrClaimHeld = stderrors.New("claim held") // ErrClaimFree is returned by Read when nothing holds the claim. ErrClaimFree = stderrors.New("claim free") // ErrLeaseLost is returned by Renew when the token is no longer current. // The caller must re-acquire rather than retry the renew. ErrLeaseLost = stderrors.New("lease lost") // ErrClaimTargetInvalid is returned when a ClaimTarget names neither a task // nor a repository, or names both. It is refused locally: sending an empty // path segment would be a server-side 404 that reads like a missing task, // and an ambiguous target would silently act on one of two objects. ErrClaimTargetInvalid = stderrors.New("claim target invalid") )
Typed-claim errors. Each names a distinct server answer, so a caller can act on them without parsing a body: a held claim is retryable later, a lost lease must be re-acquired, and a free claim is the ordinary idle state.
var ErrLeaseConflict = stderrors.New("lease conflict")
ErrLeaseConflict is returned when git-rest answers 409 on a PUT or DELETE. It is final: the claim client never retries.
var ErrLeaseNotExpired = stderrors.New("lease not expired")
ErrLeaseNotExpired is returned by Takeover when the listed claim is still inside its lease or its lease_until cannot be parsed as RFC3339.
var ErrRefNotFound = stderrors.New("ref not found")
ErrRefNotFound is returned when git-rest answers 404 on a DELETE. It is final: the ref is already gone.
Functions ¶
func DeriveClaimKey ¶
func DeriveClaimKey( ctx context.Context, taskType string, taskID string, owner string, repo string, ) (string, error)
DeriveClaimKey maps a task to the claim key it uses. The three repo-keyed task types share one key per repository; every other type is keyed by task id. Every component is validated before it is used: empty, "/", "..", a leading ".", a trailing ".lock", or a character outside [A-Za-z0-9._-] is rejected.
⚠️ Owner and repo are joined with "/", not "-", and that is load-bearing. A hyphen is legal inside both components, so the old hyphen join was lossy: owner "a-b" with repo "c" and owner "a" with repo "b-c" both derived "claims/repo/a-b-c" — two distinct repositories aliasing one lock, so the mutual exclusion silently did nothing. Because "/" is rejected inside a component, the slash join cannot collide. This also matches the streak keys (claims/redrive/<taskid>/<phase>/<n>), which were already slash-separated.
⚠️ The format is shared with vault-api, which reproduces it rather than deriving its own. The two must change together: a period where one side writes "claims/repo/a/b" and the other reads "claims/repo/a-b" is a period where the lock excludes nothing. In-flight claims carry a 5-minute lease, so a rolling deploy orphans each one for at most one lease and the next acquire takes it over (see AcquireClaimHandler in vault-api, which carries an expired ref's own SHA as the compare-and-swap).
func DeriveClaimKeysWithBody ¶ added in v0.35.0
func DeriveClaimKeysWithBody( ctx context.Context, taskType string, taskID string, owner string, repo string, body string, ) (string, string, bool, error)
DeriveClaimKeysWithBody maps a task to the claim refs it uses. The first return value is the primary key. The second is the secondary key, which is always empty: no task type writes a pair any more, so every claim is a single ref. The third reports that a pr-review task carried no usable pull-request URL and was keyed by its task id instead; it is meaningful only when the error is nil.
A pr-review task whose body carries a standalone pull-request URL is keyed per pull request, never by the shared repository key. Making the repository key the primary serialised every review of one repository behind a single claim: the first holder refused every sibling at takeover with a bare lease conflict for as long as it renewed, so no review of that repository could start.
func RecoverAtStartup ¶ added in v0.30.2
func RecoverAtStartup( ctx context.Context, manager LeaseManager, viewReady func() bool, ) error
RecoverAtStartup re-adopts, releases or fences the claims this cluster held before a restart, retried a bounded number of times.
It exists because the held set is in memory: a replaced pod starts with it empty, so without this step nothing renews and nothing fences the Jobs that pod was running. Their leases lapse, another cluster's takeover is legitimate, and the original Job runs on to its natural end. Every rollout replaces a pod, so that is not a rare path.
It never returns an error. A git-rest outage at boot must not crash-loop an executor that can still process tasks: the failure is logged and the claims stay unheld, which is the behaviour before this step existed.
func RedriveStreakPhaseComponent ¶ added in v0.38.1
RedriveStreakPhaseComponent maps an empty phase to the frozen "none" component; every other value is used verbatim. Exported so a caller reading a Snapshot keys it exactly the way Count does.
func RunClaimRecovery ¶ added in v0.30.3
func RunClaimRecovery( ctx context.Context, manager LeaseManager, viewReady func() bool, ) error
RunClaimRecovery is RecoverAtStartup as a run-group member: it recovers once, then holds until the context is cancelled.
The hold is load-bearing, not tidiness. service.Run wraps every run func in run.CancelOnFirstFinishWait, which cancels all the others as soon as ANY one returns. A recovery step that returned once it had recovered therefore ends the whole service a few seconds after boot — v0.30.2 did exactly that on nuke dev, nuke prod and sun, exiting 0 with `claim lease tick failed: context canceled` while every other component was still healthy. Recovering is a startup action, not a component with a lifetime, so it waits on the same context everything else waits on.
Types ¶
type Claim ¶ added in v0.40.0
type Claim struct {
Scope string
ID string
Holder string
LeaseUntil libtime.DateTime
RenewBefore libtime.DateTime
LeaseToken string
}
Claim is the server's view of a held lease.
type ClaimClient ¶ added in v0.40.0
type ClaimClient interface {
// Acquire takes the claim for holder with the given TTL in seconds. A held
// claim returns a *ClaimHeldError. An expired lease is taken over, so a
// dead holder cannot block the target forever.
Acquire(ctx context.Context, target ClaimTarget, holder string, ttlSeconds int) (*Claim, error)
// Renew extends an existing lease. It never takes over: a stale token
// returns ErrLeaseLost and the caller re-acquires.
Renew(
ctx context.Context,
target ClaimTarget,
leaseToken string,
ttlSeconds int,
) (*Claim, error)
// Release drops the claim. A token that is no longer current is
// ErrLeaseLost, not a silent success.
Release(ctx context.Context, target ClaimTarget, leaseToken string) error
// Read returns the current claim, or ErrClaimFree when there is none.
Read(ctx context.Context, target ClaimTarget) (*Claim, error)
}
ClaimClient is the typed vault-api claim contract. It replaces the key/SHA-shaped ClaimRefClient for lease work: a claim is acquired on a target with a holder and a TTL, and released with the lease token the acquire returned. There is no key, no expected-SHA and no prefix listing — those are git-rest's vocabulary, not the claim's.
⚠️ ClaimRefClient stays: RedriveStreakStore uses the same git-rest refs as a counter, which is not a lease and has no typed home yet.
func NewVaultAPIClaimClient ¶ added in v0.40.0
func NewVaultAPIClaimClient(baseURL, gatewaySecret, gatewayInitiator string) ClaimClient
NewVaultAPIClaimClient creates a ClaimClient for the vault API at baseURL. gatewaySecret empty means no auth headers are sent. The secret is held in memory only and must never appear in a log line or an error string.
type ClaimHeldError ¶ added in v0.40.0
ClaimHeldError names the holder of a claim that could not be acquired. It wraps ErrClaimHeld so callers can test with errors.Is and still report who blocked them.
func (*ClaimHeldError) Error ¶ added in v0.40.0
func (e *ClaimHeldError) Error() string
func (*ClaimHeldError) Unwrap ¶ added in v0.40.0
func (e *ClaimHeldError) Unwrap() error
Unwrap makes errors.Is(err, ErrClaimHeld) true for every held answer.
type ClaimRef ¶
type ClaimRef struct {
Ref string `json:"ref"`
Sha string `json:"sha"`
Cluster string `json:"cluster"`
TaskID string `json:"task_id"`
LeaseUntil string `json:"lease_until"`
}
ClaimRef is one entry of GET /api/v1/refs. Ref is the full ref name ("refs/claims/..."), not the key. LeaseUntil is the raw RFC3339 string the server returned; callers parse it and never trust it without validation.
type ClaimRefClient ¶
type ClaimRefClient interface {
// List returns every claim ref whose path is prefix or lies under it as a
// whole path component. prefix is passed as the ?prefix= query value.
List(ctx context.Context, prefix string) ([]ClaimRef, error)
// Put creates or replaces the claim at key by compare-and-swap. An empty
// expectedSha means "must not exist". It returns the new sha from the
// server response. A 409 is ErrLeaseConflict.
Put(
ctx context.Context,
key string,
expectedSha string,
cluster string,
taskID string,
leaseUntil libtime.DateTime,
) (string, error)
// Delete removes the claim at key by compare-and-swap. A 409 is
// ErrLeaseConflict.
Delete(ctx context.Context, key string, expectedSha string) error
}
ClaimRefClient is the git-rest leased claim-ref API (git-rest spec 018).
func NewClaimRefClient ¶
func NewClaimRefClient(baseURL, gatewaySecret, gatewayInitiator string) ClaimRefClient
NewClaimRefClient creates a ClaimRefClient for the git-rest instance at baseURL. gatewaySecret empty means no auth headers are sent. The secret is held in memory only and must never appear in a log line or an error string.
type ClaimTarget ¶ added in v0.40.0
type ClaimTarget struct {
// TaskID addresses a task claim. Mutually exclusive with Owner/Repo.
TaskID string
// Owner and Repo address a repository claim.
Owner string
Repo string
}
ClaimTarget is what a claim is held ON: a task identifier or a repository. It is the typed replacement for git-rest's free-form ref key — a caller never spells a path. Exactly one addressing form must be set; a target that sets both, or neither, is refused rather than silently resolved.
func NewRepoClaim ¶ added in v0.40.0
func NewRepoClaim(owner, repo string) ClaimTarget
NewRepoClaim returns a target addressed by repository.
func NewTaskClaim ¶ added in v0.40.0
func NewTaskClaim(taskID string) ClaimTarget
NewTaskClaim returns a target addressed by task identifier.
type JobControl ¶
type JobControl interface {
// IsJobRunning reports whether a non-terminal Job exists for taskID.
IsJobRunning(ctx context.Context, taskID string) (bool, error)
// DeleteJob deletes the Job for taskID. Deleting an absent Job is not an
// error (idempotent).
DeleteJob(ctx context.Context, taskID string) error
}
JobControl is the small slice of Kubernetes Job control the lease manager needs. It is injected so the lease logic can be tested on a fake clock.
type LeaseManager ¶
type LeaseManager interface {
Claim(ctx context.Context, key string, taskID string) (*HeldClaim, error)
// ClaimPair creates both refs of a claim. The primary is written first; the
// secondary is written only when it is non-empty. The claim is held only
// when every write succeeded, or when a failed primary write was read back
// as this manager's own. A secondary conflict releases the primary and
// returns ErrLeaseConflict.
ClaimPair(
ctx context.Context,
primary string,
secondary string,
taskID string,
) (*HeldClaim, error)
Release(ctx context.Context, key string) error
// ReleaseAfterSuccess releases the claim like Release, but not before
// SuccessGracePeriod has elapsed since completedAt. Until then the claim
// stays held and keeps being renewed, so a peer cluster's Claim gets a 409
// instead of a task another cluster has already run.
ReleaseAfterSuccess(ctx context.Context, key string, completedAt libtime.DateTime) error
Takeover(ctx context.Context, key string, taskID string) (*TakeoverResult, error)
// TakeoverPair is Takeover over both refs of a claim. An empty secondary
// behaves exactly like Takeover.
TakeoverPair(
ctx context.Context,
primary string,
secondary string,
taskID string,
) (*TakeoverResult, error)
Tick(ctx context.Context) error
Run(ctx context.Context) error
Recover(ctx context.Context) error
// HeldClaims returns a snapshot sorted by Key. Claims in pending Job
// deletion or pending release are included: they are still held until the
// pending action completes or the fence drops them.
HeldClaims() []HeldClaim
// AuditMismatch compares the claims this manager holds with the given set of
// running Job task ids. It adds 1 to executor_claim_mismatch_total for every
// running Job with no held claim and for every held claim with no running
// Job, and logs which case each was. A task present in both is not counted.
AuditMismatch(ctx context.Context, runningTaskIDs map[string]bool)
}
LeaseManager owns this executor's claims for one cluster name.
func NewLeaseManager ¶
func NewLeaseManager( cluster string, refClient ClaimRefClient, jobControl JobControl, currentDateTime libtime.CurrentDateTimeGetter, waiter libtime.WaiterDuration, ) (LeaseManager, error)
NewLeaseManager constructs a LeaseManager. An empty cluster is an error. waiter is used only by Run; tests pass libtime.NewWaiterDuration() or the counterfeiter fake.
type RedriveReasonStore ¶ added in v0.40.0
type RedriveReasonStore interface {
// Record appends one failure reason for taskID at phase. An empty reason is
// recorded as RedriveReasonUnknown. Recording at a new phase deletes the
// task's refs at every other phase, exactly as the streak store does.
Record(ctx context.Context, taskID string, phase string, reason string) error
// Reasons returns one element per ref at (taskID, phase), ordered by the
// numeric suffix ascending. The order is load-bearing: the classifier reads
// only the most recent count entries, so a reason left behind by an earlier
// streak cannot skew the class.
Reasons(ctx context.Context, taskID string, phase string) ([]string, error)
// Snapshot returns every parsed reason ref as taskID → phase component →
// reasons, from ONE list call over the whole reason prefix, each slice
// ordered exactly as Reasons returns it. The reconcile pass feeds the same
// classifier, so the two must agree on order.
Snapshot(ctx context.Context) (map[string]map[string][]string, error)
// Clear removes every reason ref of taskID, at every phase.
Clear(ctx context.Context, taskID string) error
}
RedriveReasonStore persists one failure reason per spawn attempt of a task at a phase, so the cap can tell a repeated broken signature from a transient failure instead of treating every three-times-failed record the same way.
It shares the claim ref store with RedriveStreakStore and reuses the RedriveStreakCluster sentinel. That sentinel, not the prefix, is what makes these refs invisible to an older binary: its lease-manager recovery skips every ref carrying it.
func NewRedriveReasonStore ¶ added in v0.40.0
func NewRedriveReasonStore( refClient ClaimRefClient, currentDateTime libtime.CurrentDateTimeGetter, ) RedriveReasonStore
NewRedriveReasonStore creates a RedriveReasonStore over refClient. The injected clock supplies lease_until; no wall clock is read.
type RedriveStreakStore ¶ added in v0.30.8
type RedriveStreakStore interface {
// Count returns how many streak refs taskID has at phase. Refs at other
// phases and refs whose key does not parse are not counted.
Count(ctx context.Context, taskID string, phase string) (int, error)
// Snapshot returns every parsed streak ref as taskID → phase component →
// count, from ONE list call over the whole streak prefix. The reconcile pass
// reads it once and answers every candidate's cap check from the result;
// Count issues one list per task, which is what put a pass over a large
// candidate set over its budget (see handler/redrive_streak_cache.go).
// Keys are normalized with RedriveStreakPhaseComponent.
Snapshot(ctx context.Context) (map[string]map[string]int, error)
// Record appends one spawn of taskID at phase, then deletes the task's
// refs at every other phase. A 409 create is retried on the next free
// number at most redriveStreakMaxCreateAttempts times.
Record(ctx context.Context, taskID string, phase string) error
// Clear removes every parsed streak ref of taskID, at every phase.
Clear(ctx context.Context, taskID string) error
}
RedriveStreakStore persists the consecutive-spawn streak of a task per phase in the shared git-rest claim store, so every cluster and every restarted pod reads the same count.
func NewRedriveStreakStore ¶ added in v0.30.8
func NewRedriveStreakStore( refClient ClaimRefClient, currentDateTime libtime.CurrentDateTimeGetter, ) RedriveStreakStore
NewRedriveStreakStore creates a RedriveStreakStore over refClient. The injected clock supplies lease_until; no wall clock is read.
type StatusError ¶ added in v0.40.0
StatusError is returned for a status no route-specific branch claims. It exposes the code as a field so a caller can tell a retryable 5xx from a terminal 4xx without parsing the message.
func (*StatusError) Error ¶ added in v0.40.0
func (e *StatusError) Error() string
func (*StatusError) Retryable ¶ added in v0.40.0
func (e *StatusError) Retryable() bool
Retryable reports whether the status suggests a transient failure.
type TakeoverResult ¶
type TakeoverResult struct {
Key string
TaskID string
Sha string
LeaseUntil libtime.DateTime
PreviousTaskID string
}
TakeoverResult is returned by Takeover. PreviousTaskID is the task id of the expired claim that was replaced, or "" when the key was absent and a create was used instead.