Documentation
¶
Index ¶
- Constants
- Variables
- func DeriveClaimKey(ctx context.Context, taskType string, taskID string, owner string, repo string) (string, error)
- func RecoverAtStartup(ctx context.Context, manager LeaseManager) error
- type ClaimRef
- type ClaimRefClient
- type HeldClaim
- type JobControl
- type LeaseManager
- type TakeoverResult
Constants ¶
const ( // LeaseDuration is how long a claim stays valid without a renew. LeaseDuration = 300 * libtime.Second // TickInterval is how often the lease loop ticks. TickInterval = 10 * libtime.Second // RenewInterval is how often a held claim is renewed. RenewInterval = 60 * libtime.Second // FenceMargin is how long before the lease end the holder stops its Job. FenceMargin = 60 * libtime.Second // FenceLead is how much earlier than fenceAt a tick fences, so a 10 s tick // lands at or before fenceAt. FenceLead = 10 * libtime.Second // TakeoverGrace is how long after the lease end a takeover is allowed. TakeoverGrace = 30 * libtime.Second )
const RequestTimeout = 30 * libtime.Second
RequestTimeout is the hard per-request cap for every git-rest claim call.
Variables ¶
var ErrLeaseConflict = stderrors.New("lease conflict")
ErrLeaseConflict is returned when git-rest answers 409 on a PUT or DELETE. It is final: the claim client never retries.
var ErrLeaseNotExpired = stderrors.New("lease not expired")
ErrLeaseNotExpired is returned by Takeover when the listed claim is still inside its lease or its lease_until cannot be parsed as RFC3339.
Functions ¶
func DeriveClaimKey ¶
func DeriveClaimKey( ctx context.Context, taskType string, taskID string, owner string, repo string, ) (string, error)
DeriveClaimKey maps a task to the claim key it uses. The four repo-keyed task types share one key per repository; every other type is keyed by task id. Every component is validated before it is used: empty, "/", "..", a leading ".", a trailing ".lock", or a character outside [A-Za-z0-9._-] is rejected.
func RecoverAtStartup ¶ added in v0.30.2
func RecoverAtStartup(ctx context.Context, manager LeaseManager) error
RecoverAtStartup re-adopts, releases or fences the claims this cluster held before a restart, retried a bounded number of times.
It exists because the held set is in memory: a replaced pod starts with it empty, so without this step nothing renews and nothing fences the Jobs that pod was running. Their leases lapse, another cluster's takeover is legitimate, and the original Job runs on to its natural end. Every rollout replaces a pod, so that is not a rare path.
It never returns an error. A git-rest outage at boot must not crash-loop an executor that can still process tasks: the failure is logged and the claims stay unheld, which is the behaviour before this step existed.
Types ¶
type ClaimRef ¶
type ClaimRef struct {
Ref string `json:"ref"`
Sha string `json:"sha"`
Cluster string `json:"cluster"`
TaskID string `json:"task_id"`
LeaseUntil string `json:"lease_until"`
}
ClaimRef is one entry of GET /api/v1/refs. Ref is the full ref name ("refs/claims/..."), not the key. LeaseUntil is the raw RFC3339 string the server returned; callers parse it and never trust it without validation.
type ClaimRefClient ¶
type ClaimRefClient interface {
// List returns every claim ref whose path is prefix or lies under it as a
// whole path component. prefix is passed as the ?prefix= query value.
List(ctx context.Context, prefix string) ([]ClaimRef, error)
// Put creates or replaces the claim at key by compare-and-swap. An empty
// expectedSha means "must not exist". It returns the new sha from the
// server response. A 409 is ErrLeaseConflict.
Put(
ctx context.Context,
key string,
expectedSha string,
cluster string,
taskID string,
leaseUntil libtime.DateTime,
) (string, error)
// Delete removes the claim at key by compare-and-swap. A 409 is
// ErrLeaseConflict.
Delete(ctx context.Context, key string, expectedSha string) error
}
ClaimRefClient is the git-rest leased claim-ref API (git-rest spec 018).
func NewClaimRefClient ¶
func NewClaimRefClient(baseURL, gatewaySecret, gatewayInitiator string) ClaimRefClient
NewClaimRefClient creates a ClaimRefClient for the git-rest instance at baseURL. gatewaySecret empty means no auth headers are sent. The secret is held in memory only and must never appear in a log line or an error string.
type JobControl ¶
type JobControl interface {
// IsJobRunning reports whether a non-terminal Job exists for taskID.
IsJobRunning(ctx context.Context, taskID string) (bool, error)
// DeleteJob deletes the Job for taskID. Deleting an absent Job is not an
// error (idempotent).
DeleteJob(ctx context.Context, taskID string) error
}
JobControl is the small slice of Kubernetes Job control the lease manager needs. It is injected so the lease logic can be tested on a fake clock.
type LeaseManager ¶
type LeaseManager interface {
Claim(ctx context.Context, key string, taskID string) (*HeldClaim, error)
Release(ctx context.Context, key string) error
Takeover(ctx context.Context, key string, taskID string) (*TakeoverResult, error)
Tick(ctx context.Context) error
Run(ctx context.Context) error
Recover(ctx context.Context) error
// HeldClaims returns a snapshot sorted by Key. Claims in pending Job
// deletion or pending release are included: they are still held until the
// pending action completes or the fence drops them.
HeldClaims() []HeldClaim
// AuditMismatch compares the claims this manager holds with the given set of
// running Job task ids. It adds 1 to executor_claim_mismatch_total for every
// running Job with no held claim and for every held claim with no running
// Job, and logs which case each was. A task present in both is not counted.
AuditMismatch(ctx context.Context, runningTaskIDs map[string]bool)
}
LeaseManager owns this executor's claims for one cluster name.
func NewLeaseManager ¶
func NewLeaseManager( cluster string, refClient ClaimRefClient, jobControl JobControl, currentDateTime libtime.CurrentDateTimeGetter, waiter libtime.WaiterDuration, ) (LeaseManager, error)
NewLeaseManager constructs a LeaseManager. An empty cluster is an error. waiter is used only by Run; tests pass libtime.NewWaiterDuration() or the counterfeiter fake.
type TakeoverResult ¶
type TakeoverResult struct {
Key string
TaskID string
Sha string
LeaseUntil libtime.DateTime
PreviousTaskID string
}
TakeoverResult is returned by Takeover. PreviousTaskID is the task id of the expired claim that was replaced, or "" when the key was absent and a create was used instead.