Documentation
¶
Overview ¶
Package autoscalingstate hosts the read-snapshot / write-accumulator abstraction used by the SandboxPool autoscaler.
The package follows a three-step pipeline:
Loader.Load(pool) — build a Snapshot capturing every input the decision logic needs (the Pool, its owning SandboxEnv, sibling Pools in the same scaling group, the in-process PoolScheduler queue stats, the in-process LastCreateTracker value, idle Pod ages).
Decide(snap, mut) — pure function in this same package consumes the Snapshot and accumulates writes into a Mutator via PatchStatus / SetTargetReplicas / Mark* helpers. No K8s I/O.
Mutator.Commit(ctx, client, recorder) — applies the accumulated writes in a single pass: at most one Env-spec patch, one Pool-status sub-resource patch, and N per-Pod annotation patches, each wrapped in retry.RetryOnConflict.
The separation matters because:
The decision logic stays pure and is easy to unit-test by hand-building a Snapshot and asserting on the resulting Mutator without standing up a fake K8s client.
All status writes coalesce: a single reconcile pass writes the SandboxPool status at most once. This avoids the cache-race class of bugs where multiple intra-reconcile status patches against a slowly-propagating informer cache silently drop bookkeeping fields and let the cooldown gate be bypassed.
Index ¶
- func Decide(snap *Snapshot, mut *Mutator)
- type Clock
- type LastCreateTracker
- type Loader
- type Mutator
- func (m *Mutator) Commit(ctx context.Context, c client.Client, recorder events.EventRecorder) error
- func (m *Mutator) EmitEvent(eventType, action, reason, format string, args ...any)
- func (m *Mutator) HasWrites() bool
- func (m *Mutator) MarkPodScaleDownProtected(pod *corev1.Pod, at time.Time)
- func (m *Mutator) PatchStatus(fn StatusMutateFunc)
- func (m *Mutator) PendingScaleUpAttempt() (from, target int32, present bool)
- func (m *Mutator) ScaleDownTransition() ScaleDownTransition
- func (m *Mutator) ScaleUpAttempt(from, target int32)
- func (m *Mutator) SetScaleDownTransition(tr ScaleDownTransition)
- func (m *Mutator) SetTargetReplicas(n int32)
- func (m *Mutator) Snapshot() *Snapshot
- func (m *Mutator) TargetReplicas() (int32, bool)
- func (m *Mutator) UnmarkPodScaleDownProtected(pod *corev1.Pod)
- type Prober
- type ScaleDownSessionView
- type ScaleDownTracker
- type ScaleDownTransition
- type ScaleDownTransitionKind
- type SchedulerLookup
- type Snapshot
- type StatusMutateFunc
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Decide ¶
Decide is the pure-function entry point of the Pool autoscaler. Given a fully-loaded Snapshot it computes whatever bookkeeping + spec writes the autoscaler wants for this reconcile cycle and stages them on mut. No K8s I/O happens here — that is Commit's job.
Decision flow:
Idle-zero bookkeeping. Maintains Pool.Status.AutoScaling.IdleZeroSince — set when idleReplicas hits 0, cleared when it climbs back above 0. The timestamp drives the proactive scale-up trigger; the same bookkeeping runs even when autoscaling is disabled so the field doesn't stay stale forever after a toggle.
Scale-up evaluation. Runs only when autoscaling is enabled. Returns early (without trying scale-down) when it actually committed a scale-up — same-cycle scale-down on a Pool we just decided to grow is never desirable.
Scale-down evaluation. Only runs when no scale-up happened this cycle. Honours the group MinReplicas aggregate floor, scale-down stabilization, and the per-Pod idleTimeoutSeconds gate.
All cooldown / quiet-window / idle-threshold semantics come from the group's PoolScaleUpPolicy / PoolScaleDownPolicy, which the CRD's kubebuilder defaults guarantee are populated.
Types ¶
type Clock ¶
Clock is the wall-clock abstraction used by the Snapshot and Mutator. Tests inject a fixed-time clock so timestamp comparisons (cooldown, quiet window) are deterministic.
func SystemClock ¶
func SystemClock() Clock
SystemClock returns a Clock backed by time.Now(). Convenience for production wiring.
type LastCreateTracker ¶
type LastCreateTracker interface {
// Get returns the most recent Create timestamp the tracker has seen
// for the given pool, and whether any timestamp was recorded.
Get(namespace, poolName string) (time.Time, bool)
}
LastCreateTracker exposes the in-process "most recent Create" timestamp for a Pool. Production wires this to lifecycle/lastcreate.Tracker; tests inject a fake. nil is treated as "tracker not wired" — the Snapshot loader falls back to the LastSandboxCreateTimeAnnotationKey on the Pool object.
type Loader ¶
type Loader struct {
// Client is the controller-runtime client (typically an informer-cache
// backed reader). Required.
Client client.Reader
// Schedulers exposes the in-process PoolScheduler registry. nil is
// allowed; the loader then leaves Snapshot.PoolSchedSnap nil for
// every Pool, which the decision logic treats as "no reactive
// signal".
Schedulers SchedulerLookup
// LastCreate exposes the in-process Create-time tracker. nil is
// allowed; the loader then falls back to the persisted annotation
// on the Pool object.
LastCreate LastCreateTracker
// Prober is the admission probe runner injected into every
// Snapshot. nil is allowed; the autoscaler then treats every
// scale-up target as admissible (matching the unit-test
// fast path).
Prober Prober
// Clock provides Now(). When nil, SystemClock() is used.
Clock Clock
// ScaleDown is the shared per-Pool scale-down session tracker. nil is
// allowed; the loader then leaves Snapshot.ScaleDownSession zero,
// which the decision logic reads as "no session" and falls back to
// the persisted status timestamp alone. Must be the SAME instance the
// reconciler holds — two instances silently defeat the cache-lag
// gate.
ScaleDown *ScaleDownTracker
}
Loader builds Snapshots from the K8s cache plus the two in-process state sources. All dependencies are interfaces so unit tests can inject fakes; production wires controller-runtime client + k8sSandboxService + lifecycle/lastcreate.Tracker.
func (*Loader) Load ¶
func (l *Loader) Load(ctx context.Context, pool *agentsv1alpha1.SandboxPool) (*Snapshot, error)
Load assembles a Snapshot for the given Pool. The Pool argument is stored by reference; callers must not mutate it after Load returns.
Errors are returned only for unexpected I/O failures (cache list / get returning a non-NotFound error). "Soft" misses — Env missing, scaling group not configured, no idle Pods — produce a successful Snapshot with the corresponding fields left nil/empty and are signalled to the decision logic via the IsAutoscalingEnabled helper.
type Mutator ¶
type Mutator struct {
// contains filtered or unexported fields
}
Mutator accumulates the writes the autoscaler decision logic wants to apply this reconcile cycle. It performs no I/O until Commit is called:
PatchStatus schedules one or more mutations against Pool.Status.AutoScaling. Multiple PatchStatus calls compose (applied in registration order); Commit folds them into a single SandboxPool status sub-resource patch.
SetTargetReplicas requests a desired-replicas update. The write target is Env.Spec.Clusters[i].Members[j].Spec.Replicas on the owning SandboxEnv (NOT Pool.Spec.Replicas directly). The existing Env reconciler's drift loop propagates the change onto the live Pool, which keeps a single writer of Pool.Spec.Replicas and reuses the manual UpdateMember path for free. May be called at most once per Mutator; the last call wins. Requires Snapshot.Env to be non-nil — Decide is responsible for the guard.
MarkPodScaleDownProtected / UnmarkPodScaleDownProtected queue per-Pod annotation patches.
EmitEvent queues a Kubernetes event for the Pool object. Events are recorded after spec/status patches succeed so they reflect what was actually persisted, not what was attempted.
Mutator is single-threaded: it is built, populated, and committed from the same goroutine within one Reconcile call. There is no internal locking.
func NewMutator ¶
NewMutator returns an empty Mutator bound to snap. The Snapshot is retained read-only and used at Commit time to derive object keys.
func (*Mutator) Commit ¶
Commit applies the accumulated writes in fixed order:
- Env spec patch (Env.Spec.Clusters[i].Members[j].Spec.Replicas)
- Pool status patch (Pool.Status.AutoScaling)
- Per-Pod annotation patches
- Events
Each K8s patch is wrapped in retry.RetryOnConflict so a transient version race re-reads and re-applies the mutation against the latest object. Non-conflict errors abort Commit and propagate to the caller; the reconciler is expected to requeue.
recorder may be nil; events are then dropped silently (useful in tests and in code paths that have not yet wired up the event sink).
func (*Mutator) EmitEvent ¶
EmitEvent queues an event for the Pool. The arguments map onto the k8s.io/client-go/tools/events.EventRecorder.Eventf signature:
- eventType matches corev1.EventTypeNormal / EventTypeWarning;
- action is a short verb describing what the controller did (e.g. "ScaleUp", "ScaleDown"). It feeds into the event's `action` field, which the newer events API surfaces alongside reason for richer telemetry;
- reason is the more specific CamelCase code conventionally prefixed with the subsystem (e.g. "AutoscalerScaleUp");
- format/args produce the human-readable message via fmt.Sprintf.
func (*Mutator) HasWrites ¶
HasWrites reports whether Commit would issue any K8s write. Useful for the reconciler's "nothing changed" fast-path logging.
func (*Mutator) MarkPodScaleDownProtected ¶
MarkPodScaleDownProtected stamps the scale-down-protected annotation onto pod with the given timestamp (RFC3339 UTC). Used by the scale-down two-phase protection flow.
Idempotent at Commit time: if the live Pod already carries the same timestamp the patch is skipped.
func (*Mutator) PatchStatus ¶
func (m *Mutator) PatchStatus(fn StatusMutateFunc)
PatchStatus registers fn to run against Pool.Status.AutoScaling during Commit. Multiple calls compose: fns run in the order they were registered. The fn receives a pointer to a non-nil PoolAutoScalingStatus — when the live object has no AutoScaling block yet, Commit allocates one before invoking the chain.
func (*Mutator) PendingScaleUpAttempt ¶
PendingScaleUpAttempt reports the buffered intent, useful for test assertions on the decision logic before Commit runs the probe.
func (*Mutator) ScaleDownTransition ¶
func (m *Mutator) ScaleDownTransition() ScaleDownTransition
ScaleDownTransition reports the buffered session transition. The reconciler applies it to the ScaleDownTracker only after Commit succeeds.
func (*Mutator) ScaleUpAttempt ¶
ScaleUpAttempt registers a scale-up intent that Commit will resolve by calling the Snapshot's Prober. Use this from Decide instead of SetTargetReplicas for autoscaler-initiated scale-ups so the probe is consulted and saturation state is recorded. Calling ScaleUpAttempt with target <= from is a no-op; later calls overwrite earlier ones.
The resulting writes (status + spec + event) are NOT computed until Commit, because the probe is I/O. ScaleUpAttempt only buffers the intent.
func (*Mutator) SetScaleDownTransition ¶
func (m *Mutator) SetScaleDownTransition(tr ScaleDownTransition)
SetScaleDownTransition records the scale-down session state change to apply to the shared ScaleDownTracker after this cycle commits. Calling it multiple times keeps the last value. Buffering only — the tracker is not touched until the reconciler reads ScaleDownTransition post-Commit, so a session never advances ahead of a persisted decrement.
func (*Mutator) SetTargetReplicas ¶
SetTargetReplicas requests a write of n to Pool.Spec.Replicas. Calling SetTargetReplicas multiple times keeps the most recent value. Negative targets are clamped to 0 to match the CRD's Minimum=0 validation.
func (*Mutator) Snapshot ¶
Snapshot returns the Snapshot this Mutator is bound to. Useful when a decision helper composes Mutator updates without needing to thread the Snapshot through every call.
func (*Mutator) TargetReplicas ¶
TargetReplicas reports the pending replicas write, if any. Exposed primarily so tests and downstream decision composition can inspect accumulated state without committing.
func (*Mutator) UnmarkPodScaleDownProtected ¶
UnmarkPodScaleDownProtected removes the scale-down-protected annotation from pod. A JSON merge patch with a null value deletes the key.
type Prober ¶
type Prober interface {
// Probe asks the plugin chain whether scaling pool's replicas
// from `current` to `target` is admissible. Implementations must
// satisfy:
// - current <= Accepted <= target
// - Accepted == target ⇔ Result == PoolScaleUpAttemptEnough
// - errMsg is empty when Result == Enough; otherwise it carries
// a short, single-line diagnostic suitable for surfacing on
// PoolAutoScalingStatus.ScaleUpErrorMessage.
Probe(ctx context.Context, pool *agentsv1alpha1.SandboxPool, current, target int32) (accepted int32, result agentsv1alpha1.PoolScaleUpAttemptResult, errMsg string)
}
Prober runs a PreUpdatePool admission probe against the cluster to discover how many additional replicas the plugin chain (scheduler reservation, quota, ...) will actually accept. The autoscaler uses it before committing a scale-up so it can patch a partial value and record SaturatedUntil-style cooldown state when the cluster has no headroom.
Production wires Prober to plugins.ProbeAcceptedReplicas via the SandboxPool reconciler's PluginManager. Tests inject a fake that returns canned (Accepted, Result) tuples. nil Prober is treated as "every probe trivially succeeds" — useful in unit tests that don't care about plugin admission.
type ScaleDownSessionView ¶
type ScaleDownSessionView struct {
LastDecisionAt time.Time
StartAt time.Time
LastStuckAt time.Time
Active bool
StartReplicas int32
}
ScaleDownSessionView is the immutable projection of a session handed to the decision logic via the Snapshot. The zero value means "no session" (no entry, or expired): Active is false and every timestamp is zero.
type ScaleDownTracker ¶
type ScaleDownTracker struct {
// contains filtered or unexported fields
}
ScaleDownTracker holds the per-Pool scale-down sessions. A single instance is shared between the Loader (which reads a View into each Snapshot) and the reconciler (which Applies transitions after a successful Commit). All methods are safe for concurrent use; the same Pool key is never reconciled concurrently, so the mutex only guards cross-Pool map access.
func NewScaleDownTracker ¶
func NewScaleDownTracker() *ScaleDownTracker
NewScaleDownTracker returns an initialized ScaleDownTracker.
func (*ScaleDownTracker) Apply ¶
func (t *ScaleDownTracker) Apply(key types.NamespacedName, tr ScaleDownTransition, now time.Time)
Apply mutates the session for key according to tr. It is called from the reconciler after a successful Commit, using the same Snapshot.Now that the cycle's gate read so lastDecisionAt lines up exactly with the next reconcile's comparison base.
func (*ScaleDownTracker) DeleteSession ¶
func (t *ScaleDownTracker) DeleteSession(key types.NamespacedName)
DeleteSession drops the entry for key. Called when the Pool is deleted so the map does not grow unboundedly.
func (*ScaleDownTracker) View ¶
func (t *ScaleDownTracker) View(key types.NamespacedName, now time.Time) ScaleDownSessionView
View returns the current session projection for key. An entry older than scaleDownSessionTTL (no decrement observed for that long) is treated as stale, deleted, and reported as the zero value.
type ScaleDownTransition ¶
type ScaleDownTransition struct {
Kind ScaleDownTransitionKind
StartReplicas int32
}
ScaleDownTransition is the session change a Decide cycle wants applied after Commit. StartReplicas is meaningful only for ScaleDownStarted.
type ScaleDownTransitionKind ¶
type ScaleDownTransitionKind int
ScaleDownTransitionKind enumerates the session state changes the decision logic can request. The reconciler applies the recorded transition to the tracker only after the cycle's writes commit, keeping the in-process gate from ever running ahead of a persisted decrement.
const ( // ScaleDownNoTransition is the zero value: the cycle touched no // session state. ScaleDownNoTransition ScaleDownTransitionKind = iota // ScaleDownStarted opens a new session on the first decrement. ScaleDownStarted // ScaleDownStepped records a subsequent decrement within an open // session (no event emitted). ScaleDownStepped // ScaleDownCompleted closes a session that ended naturally (nothing // left to remove). ScaleDownCompleted // ScaleDownAborted closes a session cut short by scale-up or reactive // demand (no Completed event). ScaleDownAborted // ScaleDownStuckMarked records that a Stuck warning fired this cycle; // the session stays open. ScaleDownStuckMarked )
type SchedulerLookup ¶
type SchedulerLookup interface {
// GetScheduler returns the running PoolScheduler for the named pool
// or nil when none has been created yet.
GetScheduler(namespace, poolName string) *schedule.PoolScheduler
}
SchedulerLookup is the read-only view onto the apiserver's per-Pool scheduler registry that the autoscaler needs. The production implementation is the same map maintained by k8sSandboxService; tests inject a fake. nil is allowed and is treated as "no scheduler running" for every Pool (useful in unit tests that don't exercise reactive demand signals).
type Snapshot ¶
type Snapshot struct {
// Pool is the SandboxPool being reconciled. Always non-nil. The
// reconciler hands a deep copy to Load; the Snapshot retains the
// reference for downstream readers.
Pool *agentsv1alpha1.SandboxPool
// Env is the SandboxEnv that owns Pool, resolved via the
// agentbox.navix.sh/env label first, falling back to ownerReferences.
// nil when the Pool has no owning Env (treated as "unmanaged Pool,
// autoscaling disabled"). nil also when the Env was deleted between
// Pool list and Env get — the loader does not treat this as an error
// because the next reconcile will see the Pool's own deletion soon.
Env *agentsv1alpha1.SandboxEnv
// MemberConfig is the Pool's EnvClusterMemberConfig from
// env.spec.clusters[].members[].config. nil when Env is nil OR when
// the Pool's name is not in the Env's member list (which would be a
// stale orphan — autoscaling is disabled in that case).
MemberConfig *agentsv1alpha1.EnvClusterMemberConfig
// Group is env.spec.autoscaling.groups[name == MemberConfig.ScalingGroup].
// nil when:
// - Env is nil
// - MemberConfig.ScalingGroup is empty
// - The named group is not declared on env.spec.autoscaling.Groups
// - env.spec.autoscaling is nil
// nil means autoscaling is disabled for this Pool; the decision logic
// must skip every scale-up / scale-down evaluation.
Group *agentsv1alpha1.EnvAutoscalingGroup
// SiblingPools is the set of Pools in the same Env and same scaling
// group (this Pool included). Sorted by metadata.name for stable
// tie-breaks in cross-Pool priority comparisons. Empty when Group is
// nil. Pools are deep copies returned from the cache via client.List.
SiblingPools []*agentsv1alpha1.SandboxPool
// PoolSchedSnap is the in-process snapshot of this Pool's
// PoolScheduler (queue length, idle ready count, reserved count,
// last dispatch). nil when no scheduler has been registered for this
// Pool yet (first-ever Create has not been observed by the apiserver).
PoolSchedSnap *schedule.Snapshot
// LastCreateAt is the timestamp of the most recent Sandbox.Create
// request for this Pool, resolved with the following priority:
// 1. In-process LastCreateTracker (fresh; updated synchronously by
// the Create handler);
// 2. Pool.Annotations[LastSandboxCreateTimeAnnotationKey] (the
// throttled flush mirror; survives process restart).
// nil when no Create has ever been recorded for this Pool.
LastCreateAt *time.Time
// IdlePodAges contains one entry per idle Pod owned by this Pool, in
// descending order (oldest first). Each entry is `Now - GetPodPhaseSince(p, Idle)`.
// Pods whose phase-since timestamp could not be resolved are skipped.
// Used by the scale-down candidate selector to compare against
// scaleDownPolicy.idleTimeoutSeconds.
IdlePodAges []time.Duration
// Prober runs the cluster-admission probe (PreUpdatePool plugin
// chain) the autoscaler consults before committing a scale-up.
// nil disables probing and behaves as "every target is admissible"
// — convenient in unit tests, equivalent to the pre-S6 behaviour
// where the autoscaler trusted its own computed target.
Prober Prober
// Now is the wall-clock time captured at Load entry. Used by every
// duration comparison downstream so a single Snapshot evaluates
// consistently end-to-end even if the decision logic runs slowly.
Now time.Time
// ScaleDownSession is the in-process view of this Pool's ongoing
// scale-down lifecycle (gate timestamp + whether a session is open).
// The zero value means "no session". Populated from Loader.ScaleDown
// when wired; left zero in unit tests that don't exercise the
// cache-lag gate or the Started/Completed event pairing.
ScaleDownSession ScaleDownSessionView
}
Snapshot is the read-only view of every input the Pool autoscaler's decision logic needs in one reconcile cycle. Built by Loader.Load; once returned, the decision logic treats it as immutable.
Pointers are used for "may be absent" inputs (Env not found, group not configured, no PoolScheduler running yet, etc.) so the decision logic can distinguish "missing" from "zero".
func (*Snapshot) GroupDesiredTotal ¶
GroupDesiredTotal returns the sum of spec.replicas across every Pool in the same scaling group, including self. Used by the group MaxReplicas ceiling check before a scale-up is committed.
func (*Snapshot) GroupIdleTotal ¶
GroupIdleTotal returns the sum of status.idleReplicas across every Pool in the same scaling group, including self.
func (*Snapshot) IsAutoscalingEnabled ¶
IsAutoscalingEnabled reports whether the decision logic should consider this Pool for scale-up / scale-down. Both an owning group and that group's Enabled flag must be true.
func (*Snapshot) IsReactiveDemand ¶
IsReactiveDemand reports whether the in-process scheduler currently has at least one claim request queued AND no idle Pod available to serve it. This is the autoscaler's immediate scale-up signal: an unsatisfied claim means the cluster has real, present-tense demand for an additional replica.
Returns false when no PoolScheduler has been registered for this Pool yet — in that case there can be no queued claims either.
type StatusMutateFunc ¶
type StatusMutateFunc func(*agentsv1alpha1.PoolAutoScalingStatus)
StatusMutateFunc mutates the autoscaling sub-status in place. The argument is guaranteed non-nil — Commit allocates it before invoking the mutators when the field was absent on the live object.
Source Files