Documentation
¶
Overview ¶
Package autoscale is the deterministic replica autoscaler: a level-triggered controller that turns observed load into a desired replica count.
Index ¶
- Constants
- Variables
- func CeilingFromEnv(lookup func(string) string) int
- func FlapThresholdFromEnv(lookup func(string) string) int
- func Flaps(events []Event, now time.Time, window time.Duration) int
- func IntervalFromEnv(lookup func(string) string) time.Duration
- func MetricMaxAgeFromEnv(lookup func(string) string) time.Duration
- func OSEnv(key string) string
- func WindowFromEnv(lookup func(string) string) time.Duration
- type Condition
- type Controller
- type Decision
- type Event
- type Gauge
- type Input
- type Policy
- type Reading
- type Recorder
- type Signals
- type State
- type Status
- type Store
Constants ¶
const ( EventScaledUp = "scaled_up" EventScaledDown = "scaled_down" EventHeld = "held" EventFailed = "failed" EventConfigured = "configured" EventCleared = "cleared" )
Event kinds recorded in an app's history.
const ( ReasonWithinTarget = "WithinTarget" ReasonScaledUp = "ScaledUp" ReasonScaledDown = "ScaledDown" ReasonCooldown = "CooldownActive" ReasonStabilizing = "Stabilizing" ReasonNoCapacity = "NoCapacity" ReasonSpendCapPaused = "SpendCapPaused" ReasonAtMaximum = "AtMaximum" ReasonAtMinimum = "AtMinimum" ReasonBoundsCorrected = "BoundsCorrected" ReasonSuspended = "Suspended" )
Decision reasons, shown in the status condition and the history.
const ( BlockedCooldown = "cooldown" BlockedStabilization = "stabilization" BlockedSpendCap = "spend_cap" BlockedNoCapacity = "no_capacity" BlockedMaxReplicas = "max_replicas" BlockedMinReplicas = "min_replicas" BlockedGlobalCeiling = "global_ceiling" BlockedStepLimit = "step_limit" BlockedNoMetrics = "no_metrics" BlockedSuspended = "suspended" )
Blockers recorded when a wanted change did not happen, or only partly.
const ( MetricCPUPercent = "cpu_percent" MetricMemoryPercent = "memory_percent" MetricRequestsPerReplica = "requests_per_replica" )
Metrics a policy can target.
const ( DefaultScaleUpTolerancePct = 10 DefaultScaleDownTolerancePct = 30 DefaultScaleUpCooldown = 60 * time.Second DefaultScaleDownCooldown = 10 * time.Minute DefaultStabilization = 5 * time.Minute DefaultMaxStepUp = 4 DefaultMaxStepDown = 1 )
Defaults applied by Normalize to unset policy fields.
const ( EnvMaxReplicasCeiling = "APP_AUTOSCALE_MAX_REPLICAS" EnvInterval = "APP_AUTOSCALE_INTERVAL" EnvWindow = "APP_AUTOSCALE_WINDOW" EnvMetricMaxAge = "APP_AUTOSCALE_METRIC_MAX_AGE" EnvFlapThreshold = "APP_AUTOSCALE_FLAP_THRESHOLD" DefaultMaxReplicasCeiling = 20 DefaultInterval = 30 * time.Second DefaultWindow = 2 * time.Minute DefaultMetricMaxAge = 90 * time.Second DefaultFlapThreshold = 3 )
Environment settings read by the control plane.
const ( ReasonPending = "Pending" ReasonFlapping = "Flapping" )
Reasons only Status produces.
const ReasonScaleFailed = "ScaleFailed"
ReasonScaleFailed is recorded when the replica write itself failed.
Variables ¶
var ErrNotFound = errors.New("autoscale: no policy for this app")
ErrNotFound means an app has no autoscale policy.
Functions ¶
func CeilingFromEnv ¶
CeilingFromEnv is the global maximum any policy may ask for.
func FlapThresholdFromEnv ¶
FlapThresholdFromEnv is the number of direction reversals in an hour that counts as flapping.
func Flaps ¶
Flaps counts direction reversals between consecutive scale events within window of now. events may be in any order.
func IntervalFromEnv ¶
IntervalFromEnv is how often the controller re-evaluates every policy.
func MetricMaxAgeFromEnv ¶
MetricMaxAgeFromEnv is how old the newest sample may be before the metric counts as unknown.
Types ¶
type Condition ¶
type Condition struct {
Healthy bool `json:"healthy"`
Reason string `json:"reason"`
Message string `json:"message"`
}
Condition is the controller's status for one app.
type Controller ¶
type Controller struct {
Store Store
Signals Signals
Recorder Recorder
Nudge func()
Ceiling int
FlapThreshold int
Logger *slog.Logger
Now func() time.Time
}
Controller converges each autoscaled app's replica count on its load. It holds no state of its own: every tick re-derives the decision from the store and the signals, so a restart or a missed tick changes nothing.
type Decision ¶
type Decision struct {
Current int
Desired int
Value float64
Reason string
BlockedBy string
Message string
DownSince time.Time
DownFloor int
}
Decision is the outcome: the count to converge on and why.
func Decide ¶
Decide computes the desired replica count. It is pure: no clock, no I/O.
The count follows ceil(current * value / target), the same proportional rule Kubernetes uses, but only outside a dead band (scale up above target plus the up tolerance, scale down below target minus the down tolerance), so a reading near the target never moves anything. Scale up is fast (short cooldown); scale down waits for a stabilization window and a longer cooldown, and lands on the highest count recommended inside that window.
type Event ¶
type Event struct {
ID int64 `json:"id"`
App string `json:"app"`
Kind string `json:"kind"`
Current int `json:"current"`
Desired int `json:"desired"`
Value float64 `json:"value"`
Reason string `json:"reason"`
BlockedBy string `json:"blocked_by"`
Detail string `json:"detail"`
At time.Time `json:"at"`
}
Event is one entry of an app's decision history.
type Gauge ¶
Gauge is one app's snapshot, recorded as metrics so a plain threshold alert rule can watch NoCapacity and flapping.
type Input ¶
type Input struct {
Now time.Time
Current int
// Value is the metric in the units of the policy target; it is only
// meaningful while ValueKnown is true.
Value float64
ValueKnown bool
// MetricNote says why the value is unknown.
MetricNote string
// Ceiling is the operator's global maximum; 0 means none.
Ceiling int
LastScaleUp time.Time
LastScaleDown time.Time
// DownSince and DownFloor carry the scale down stabilization window
// between calls: when a lower count was first recommended and the highest
// count recommended since.
DownSince time.Time
DownFloor int
// SpendBlocked is true while a spend cap is being enforced.
SpendBlocked bool
// Headroom reports how many of extra more replicas the placement can
// hold. A negative answer, or a nil func, means unknown and never blocks.
Headroom func(extra int) int
}
Input is everything one decision depends on. Decide reads nothing else, so the same Input always produces the same Decision.
type Policy ¶
type Policy struct {
App string `json:"app"`
MinReplicas int `json:"min_replicas"`
MaxReplicas int `json:"max_replicas"`
Metric string `json:"metric"`
Target float64 `json:"target"`
// Scale up only when the metric is this far above Target, scale down only
// when it is this far below: the gap between them is the dead band.
ScaleUpTolerancePct int `json:"scale_up_tolerance_pct"`
ScaleDownTolerancePct int `json:"scale_down_tolerance_pct"`
ScaleUpCooldownSecs int `json:"scale_up_cooldown_secs"`
ScaleDownCooldownSecs int `json:"scale_down_cooldown_secs"`
StabilizationSecs int `json:"stabilization_secs"`
MaxStepUp int `json:"max_step_up"`
MaxStepDown int `json:"max_step_down"`
}
Policy is one app's autoscaling contract.
func (Policy) Normalize ¶
Normalize fills unset fields with their defaults. Hysteresis cannot be switched off: zero means the default, never "none".
func (Policy) ScaleDownCooldown ¶
ScaleDownCooldown is the minimum gap between a scale down and the last change.
func (Policy) ScaleUpCooldown ¶
ScaleUpCooldown is the minimum gap between two scale ups.
func (Policy) Stabilization ¶
Stabilization is how long a lower count must keep being recommended before it is applied.
type Signals ¶
type Signals interface {
Metric(ctx context.Context, p Policy, current int, now time.Time) (Reading, error)
// Headroom reports how many of extra more replicas fit on the app's node,
// or a negative number when that cannot be told.
Headroom(ctx context.Context, app string, extra int) (int, error)
// SpendEnforced returns the spend cap action in force for the app, "" when none.
SpendEnforced(ctx context.Context, app string) (string, error)
}
Signals is the observed world the controller reads.
type State ¶
type State struct {
App string
Current int
Desired int
Value float64
ValueKnown bool
Reason string
BlockedBy string
Message string
LastScaleUp time.Time
LastScaleDown time.Time
DownSince time.Time
DownFloor int
UpdatedAt time.Time
}
State is what the controller last decided for an app, plus the memory a later decision needs (cooldown timestamps and the stabilization window).
type Status ¶
type Status struct {
Policy
EffectiveMax int `json:"effective_max"`
Current int `json:"current"`
Desired int `json:"desired"`
Value float64 `json:"value"`
ValueKnown bool `json:"value_known"`
BlockedBy string `json:"blocked_by"`
Flaps1h int `json:"flaps_1h"`
Flapping bool `json:"flapping"`
LastScaleUp *time.Time `json:"last_scale_up,omitempty"`
LastScaleDn *time.Time `json:"last_scale_down,omitempty"`
EvaluatedAt *time.Time `json:"evaluated_at,omitempty"`
Condition Condition `json:"condition"`
}
Status is one app's policy with its live position and condition.
type Store ¶
type Store interface {
ListAutoscalePolicies(ctx context.Context) ([]Policy, error)
GetAutoscalePolicy(ctx context.Context, app string) (Policy, error)
SaveAutoscalePolicy(ctx context.Context, p Policy, ev Event, now time.Time) error
DeleteAutoscalePolicy(ctx context.Context, app string, ev Event) error
GetAutoscaleState(ctx context.Context, app string) (State, bool, error)
SaveAutoscaleState(ctx context.Context, st State, events []Event) error
ListAutoscaleEvents(ctx context.Context, app string, limit int) ([]Event, error)
// AutoscaleReplicas returns the app's desired replica count and whether it
// is suspended, or ErrNotFound.
AutoscaleReplicas(ctx context.Context, app string) (replicas int, suspended bool, err error)
// SetAutoscaleReplicas writes the app's desired replica count. It is the
// only writer of that column besides a manual update.
SetAutoscaleReplicas(ctx context.Context, app string, replicas int) error
}
Store is the persistence the controller needs; *store.DB satisfies it.