scale

package
v0.14.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 20, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package scale is the opt-in, host-aware process auto-scaler (plan §8A). It scales container REPLICAS of one edge-fronted HTTP service — never VMs, never the whole project — and is conservative-by-construction: it REFUSES rather than queues, treats a refusal as an alertable signal, and on a small box collapses to a safe no-op (effective_max = 1).

This file is the candidacy gate. The decision core (controller hysteresis) and the load-bearing host-capacity guard are in controller.go / capacity.go. All three are pure so the safety properties are exhaustively testable; the watcher (watcher.go) supplies the inputs, applies the §0 gate + semaphore, and performs the scale.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Candidacy

func Candidacy(s ServiceSpec) (ok bool, reason string)

Candidacy reports whether a service may be auto-scaled, with the first failing reason. Default is NOT scalable: every condition must hold. A stateful service is rejected with a clear reason — it is a config-file/cert-binding app (§7.4), not a scaling candidate.

func MaxReplicas

func MaxReplicas(in CapacityInput) (ceiling int, nearOOM bool, reason string)

MaxReplicas returns the hard replica ceiling this service may run RIGHT NOW. It is never below 1 (a service always keeps its base replica) and is capped by the policy max and BOTH resource budgets. nearOOM=true means the box is critically low on memory and scaling is a no-op (effective_max = 1) — a wanted scale-up above this is a refusal the caller must surface as scale_refused_no_capacity.

func StatefulImage

func StatefulImage(image string) bool

StatefulImage reports whether an image reference belongs to a known stateful / clustered family (C4). It strips the registry, tag, and digest, then matches the final repository path component against the denylist (so "ghcr.io/acme/postgres:16" and "postgres" both match, but "my-postgres-helper" does not).

Types

type Action

type Action string

Action is what the watcher should do with the decision.

const (
	ActNone    Action = "none"
	ActUp      Action = "up"
	ActDown    Action = "down"
	ActRefused Action = "refused" // wanted to scale up but the capacity ceiling blocked it
)

type Budget

type Budget struct {
	HostTotal  uint64 // total host resource
	HostFree   uint64 // measured-free right now
	Reserved   uint64 // everything NOT this service: control plane + edge + safety floor + OTHER apps' desired replicas
	FreeFloor  uint64 // keep at least this much free (applied to the measured budget)
	PerReplica uint64 // this service's per-replica reservation (required, non-zero)
	Current    int    // this service's current replica count
}

Budget is one resource's accounting (memory in bytes, or CPU in milli-units).

type CapacityInput

type CapacityInput struct {
	Mem Budget
	CPU Budget

	PolicyMax          int    // operator's configured max_replicas
	PerReplicaMemFloor uint64 // an implausibly small per-replica mem reservation is rejected
	NearOOMFreeBytes   uint64 // host mem free below this → effective_max collapses to 1
}

CapacityInput is everything MaxReplicas needs for one service this tick.

type Config

type Config struct {
	Store        *Store
	Alerts       *alertstore.Store // nil → refusals are logged only
	Snap         func() *monitor.Snapshot
	Sem          *dockerexec.Semaphore
	Scaler       Scaler
	Edge         EdgeReconciler // optional
	Reserves     Reserves
	Log          *slog.Logger
	Interval     time.Duration
	WritePlaneOK bool
	HostCPUMilli uint64                                        // total host CPU (milli); 0 disables the CPU budget
	IsCandidate  func(app, service string) (ServiceSpec, bool) // C1–C6 from compose; nil → trust the policy opt-in
	// EdgeStats returns the edge-measured p95 latency (ms) and request rate (req/s) for a service
	// over the rolling window, plus whether ANY request was sampled in it. It backs source:edge
	// metrics. nil = no managed edge on this host → source:edge metrics are OMITTED (inert), so they
	// neither help nor pin scaling.
	EdgeStats func(app, service string) (p95Ms, reqPerSec float64, present bool)
	// EdgeLive reports whether the edge is delivering access logs at all right now (any service).
	// It lets edgeSignal tell a genuinely-idle service (edge live, no samples → may scale down) from
	// a blind window (enable-time lag / broken capture → HOLD, never shed on data we don't have).
	// Wired together with EdgeStats (both set, or both nil).
	EdgeLive func() bool
	// Held returns the set of operator-HELD services (a manual stop / "pause auto-restart"). A held
	// service is skipped entirely each tick — never reconciled, scaled, or reserved-for — so a
	// service the operator deliberately stopped is not relaunched by the scaler. Read once per tick.
	// nil → nothing held.
	Held func() map[Key]bool
	// CircuitOpen returns services whose self-heal circuit is OPEN (self-heal gave up on a crash loop
	// and is paging instead of restarting). The scaler must NOT relaunch them — reconciling a crashed
	// replica back up would re-arm exactly the loop the breaker exists to stop. Read once per tick.
	CircuitOpen func() map[Key]bool
	// BusyApps returns apps holding a self-heal expected_down lease (an operator lifecycle/deploy
	// action is in flight). The scaler skips their services meanwhile — it must not fight the write
	// plane, and this also closes the brief window during an operator STOP before its hold row lands
	// (self-heal holds the lease until after the hold is written). Read once per tick.
	BusyApps func() map[string]bool
	Now      func() int64
}

Config configures the auto-scaling Watcher.

type Decision

type Decision struct {
	Target int
	Action Action
	Reason string
	Next   State
}

Decision is the pure outcome; the watcher persists Next and, on Up/Down, performs the scale (+ edge-pool reconcile). On Refused it raises scale_refused_no_capacity.

func Decide

func Decide(st State, m Metrics, p Policy, ceiling int, now int64) Decision

Decide steps the controller for one service. ceiling is the host-capacity guard's hard cap for this tick (from MaxReplicas).

type EdgeReconciler

type EdgeReconciler interface {
	ReconcilePool(ctx context.Context, app, service string, replicas int) error
}

EdgeReconciler updates the edge replica pool for a service after a count change (discover live replicas → validated pool → reload). May be nil (then the route's single upstream DNS-round-robins across replicas).

type Key

type Key struct{ App, Service string }

Key identifies one scaled service.

type MetricSpec added in v0.10.0

type MetricSpec struct {
	Name   string  `json:"name"`
	Source string  `json:"source"` // "ops" (app queue depth) | "edge" (Mooring-measured latency / req-rate)
	Select string  `json:"select,omitempty"`
	Up     float64 `json:"up"`
	Down   float64 `json:"down"`
}

MetricSpec is a custom scaling signal's full spec: how to SOURCE it (Source/Select, used by the watcher to read the value) plus its thresholds (Up/Down, used by the controller). Persisted with the policy as JSON.

type Metrics

type Metrics struct {
	CPUMeanPct float64
	MemMaxPct  float64
	AllHealthy bool
	Signals    []Signal // custom per-service signals, keyed by name
}

Metrics is the per-service signal: per-replica CPU MEAN and mem MAX aggregated across the running replicas (plan §8A), plus whether every replica is healthy, plus any CUSTOM signals (e.g. queue depth from the ops probe) already aggregated to a per-replica value by the watcher.

type Policy

type Policy struct {
	Min, Max         int
	UpCPUPct         float64
	UpMemPct         float64
	DownCPUPct       float64
	DownMemPct       float64
	BreachForSecs    int64
	CooldownUpSecs   int64
	CooldownDownSecs int64
	Signals          []SignalPolicy // custom signals (additive; CPU/mem unchanged)
}

Policy is the operator's scaling policy for one service.

func (Policy) Valid

func (p Policy) Valid() (bool, string)

Valid checks the policy invariants config validation must enforce: a sane replica range, the ≥20-pt dead band on BOTH signals, and up-eager/down-lazy cooldowns.

type PolicyRow

type PolicyRow struct {
	Policy
	Enabled       bool
	PerReplicaMem uint64
	PerReplicaCPU uint64
	Metrics       []MetricSpec // custom scaling signals (source + thresholds); mirrored into Policy.Signals
}

PolicyRow is a stored policy plus its per-replica reservations + enabled flag.

type Reserves

type Reserves struct {
	MemReserveBytes    uint64 // control plane + edge + safety floor (memory)
	CPUReserveMilli    uint64 // control plane + edge (cpu)
	MemFreeFloor       uint64 // keep at least this much memory free (measured budget)
	CPUFreeFloor       uint64
	NearOOMFreeBytes   uint64
	PerReplicaMemFloor uint64
}

Reserves are the host headroom the capacity guard subtracts before funding this app's replicas (control plane + edge slice + a safety floor), plus the near-OOM and per-replica-floor guards.

type Scaler

type Scaler interface {
	Scale(ctx context.Context, app, service string, replicas int) error
}

Scaler performs the actual replica change for a service (static-argv `docker compose up -d --no-deps --no-recreate --scale <svc>=<n>`). The watcher calls it only after the §0 gate + a non-blocking semaphore acquire, holding the one-docker-child semaphore.

type ServiceSpec

type ServiceSpec struct {
	Name                string
	EdgeUpstream        bool // C1: an edge HTTP upstream with a known internal port
	L4Upstream          bool // C1 (alt): a managed L4 (TCP/UDP) upstream — fronted by an edge l4_route, replicas internal-only
	FixedHostPort       bool // C2 (disqualifies): publishes a fixed host port the LB does NOT own
	RWVolume            bool // C3 (disqualifies): has an exclusive read-write volume
	Stateful            bool // C4 (disqualifies): a DB/broker/coordination store
	IdentityPlaceholder bool // C5 (disqualifies): a deploy-time identity (node cookie/name/seed/host-bound port)
	StatelessContract   bool // C6: honors the stateless restart contract (operator attests)
	OptedIn             bool // C7: the operator explicitly enabled scaling for this service
}

ServiceSpec is the deploy-time view of one service, derived from its compose definition + the managed edge routes. Candidacy is re-evaluated on every deploy / config change; a service that GAINS a host port or RW volume loses candidacy and is scaled back to 1.

type Signal added in v0.10.0

type Signal struct {
	Name    string
	Value   float64
	Present bool
}

Signal is one custom metric input for a tick, already aggregated by the watcher to the per-replica value the thresholds compare against (so a service-TOTAL like queue depth is divided by the running replica count → target-tracking falls out of the same threshold compare). Present=false means the metric was unavailable this tick (probe down / degraded); a missing signal must never drive a scale-up and must BLOCK scale-down (we can't confirm the load has dropped), so the service holds rather than shedding capacity blind.

type SignalPolicy added in v0.10.0

type SignalPolicy struct {
	Name string
	Up   float64
	Down float64
}

SignalPolicy is one custom signal's scale rule: a per-replica up threshold (scale up when the per-replica value is at/above Up) and a down threshold (below which it permits scale-down). Up>Down is the per-signal dead band. For a target-tracking metric like queue depth, Up is the target queue-per-replica.

type State

type State struct {
	Replicas    int
	BreachSince int64 // unix sec; when the current up-breach started (0 = not breaching)
	LastChange  int64 // unix sec; last scale action (for the cooldowns)
}

State is the persisted controller state for one service. Replicas is the DESIRED count the controller is driving toward (the watcher reconciles observed→desired).

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store persists scaling policies (operator opt-in + thresholds) and controller state (desired replicas + hysteresis timers, recovered on restart).

func NewStore

func NewStore(db *store.DB) *Store

NewStore builds a Store.

func (*Store) DeleteApp

func (s *Store) DeleteApp(ctx context.Context, app string) error

SavePolicy validates + upserts a policy. A policy must pass Policy.Valid() and carry non-zero per-replica reservations before it can be enabled. DeleteApp removes ALL scaling policies and controller state for every service of an app. Used by the app-delete teardown.

func (*Store) EnabledPolicies

func (s *Store) EnabledPolicies() (map[Key]PolicyRow, error)

EnabledPolicies returns every enabled policy keyed by (app,service).

func (*Store) HasEdgeMetric added in v0.11.0

func (s *Store) HasEdgeMetric() bool

HasEdgeMetric reports whether ANY enabled scaling policy uses a source:edge metric. The edge reconciler consults this each cycle to decide whether to render Caddy's per-request access log (which feeds the latency aggregator) — so an edge with no such metric never pays for access logging, and enabling one takes effect on the next reconcile with no restart. A query error is treated as false (fail-safe: don't turn on logging we can't justify).

func (*Store) LoadStates

func (s *Store) LoadStates() (map[Key]State, error)

LoadStates returns all persisted controller states.

func (*Store) PolicyFor

func (s *Store) PolicyFor(k Key) (PolicyRow, bool, error)

PolicyFor returns the policy for one service (enabled flag included), or ok=false.

func (*Store) SavePolicy

func (s *Store) SavePolicy(ctx context.Context, k Key, pr PolicyRow) error

func (*Store) SaveState

func (s *Store) SaveState(ctx context.Context, k Key, st State, now int64) error

SaveState upserts one controller state.

type Watcher

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher is the auto-scaling controller loop (plan §8A).

func New

func New(cfg Config) *Watcher

New builds a Watcher.

func (*Watcher) Nudge added in v0.14.0

func (w *Watcher) Nudge(app, service string, delta int)

Nudge requests a manual one-step change (+1 / −1) to a service's desired replica count. It only RECORDS the request under a short lock; the controller applies it at the start of the next tick, so it uses fresh capacity data and never races the state map. A nudged service stays under normal autoscaling afterward: it scales back down under sustained no-load (after the down-cooldown) and up under load — the manual step is a boost the controller still manages, not a pin. Repeated clicks before the next tick accumulate. A held or non-scalable service's nudge is a harmless no-op (the tick simply never applies it).

func (*Watcher) Run

func (w *Watcher) Run(ctx context.Context)

Run recovers state and ticks until ctx is cancelled.

func (*Watcher) Tick

func (w *Watcher) Tick(ctx context.Context)

Tick runs one control pass. Exported for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL