Documentation
¶
Overview ¶
Package capacity implements admission control for sandbox creation.
Admission is purely resource-math — there is no fixed "max sandboxes" number. The admitter combines two signals before letting a sandbox in:
- Sum-of-reservations against host CPU cores and total memory, scaled by configurable ratios. This is predictable: the operator knows that accepted requests will not collectively exceed the ratio of host capacity, regardless of whether the sandboxes are actually using it. A 64-core box can run 200 tiny sandboxes or 8 huge ones — whatever the math allows.
- A live free-memory floor read from /proc/meminfo. This catches the case where reservations look fine but the host is genuinely tight right now (e.g. another process is eating RAM).
Reservations are tracked in-process. They must be replayed at startup from persistent state so a daemon restart doesn't reset accounting to zero.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrCapacityExceeded = errors.New("host capacity exceeded")
ErrCapacityExceeded is returned by Admit when the host cannot accept a new sandbox. The wrapped error carries human-readable reasons; callers can use errors.Is to map this to a 503 response.
Functions ¶
func ParkReservationID ¶
ParkReservationID is the admitter key for a warm-pool parked slot.
func ReleaseParkReservation ¶
ReleaseParkReservation is the service-side transfer after adopt.
Types ¶
type Admitter ¶
type Admitter struct {
// contains filtered or unexported fields
}
Admitter is the thread-safe admission controller. Construct with New and keep a single instance per daemon.
func New ¶
New builds an admitter. host should reflect the machine's total capacity; callers that don't care about per-host detection can use DetectHost.
New constructs without an RSSSource (the Phase 5 memory-overcommit axis is disabled). Callers that want it should use NewWithRSSSource — kept as a separate constructor rather than adding a fifth positional arg here so existing call sites in tests and cluster bootstrapping stay untouched.
func NewWithRSSSource ¶
NewWithRSSSource builds an admitter wired to a Phase 5 RSS source. nil rss is equivalent to New — the watermark check is skipped.
func (*Admitter) Admit ¶
Admit decides whether a new sandbox with the given resource request can be accepted right now. On success it reserves the request under sandboxID and returns nil. On failure it returns an error wrapping ErrCapacityExceeded with reasons attached; no reservation is made.
Callers that fail downstream (e.g. docker create errors) must call Release to free the reservation. Reserve is idempotent per sandboxID — re-admitting the same ID overwrites the prior reservation.
func (*Admitter) CanAdmitRequest ¶
CanAdmitRequest is a non-mutating admission probe for a specific shape.
func (*Admitter) Release ¶
Release frees the reservation for sandboxID. Safe to call for unknown IDs.
type HostInfo ¶
type HostInfo struct {
CPUCores int
MemoryTotalMB int
// DiskTotalGB is the disk budget for sandbox storage. It is auto-detected
// from the host filesystem by default and can be overridden by
// SB_HOST_DISK_GB when the sandbox pool is smaller than the filesystem.
DiskTotalGB int
// DiskFreeGB is the currently available space on the detected sandbox
// storage filesystem. It is an observability signal; reservation
// admission uses DiskTotalGB × DiskReservationRatio.
DiskFreeGB int
// GPUCount and GPUVendor describe the GPU inventory the operator has
// wired up to Docker. Operator-declared (SB_HOST_GPU_COUNT /
// SB_HOST_GPU_VENDOR); 0 means no GPUs available.
GPUCount int
GPUVendor string
// SupportedRuntimes lists the OCI runtimes installed on this host.
// Empty defaults to []{"docker"} at Admitter construction so a node
// that doesn't set the field still admits the common case.
SupportedRuntimes []string
}
HostInfo describes the static host capacity.
func DetectHost ¶
DetectHost returns CPU core count from runtime and memory total from /proc/meminfo via the default probe. Callers may override either field.
type Limits ¶
type Limits struct {
// CPUReservationRatio is the maximum fraction of host CPU cores that may
// be reserved across all sandboxes. 0 = unlimited.
CPUReservationRatio float64
// MemoryReservationRatio is the maximum fraction of host memory that may
// be reserved across all sandboxes. 0 = unlimited.
MemoryReservationRatio float64
// DiskReservationRatio is the maximum fraction of host disk that may be
// reserved across all sandboxes. Disk is sized in GB rather than MB
// because Docker's --storage-opt size is GB-granular and operators
// reason about disk in GB. 0 = unlimited.
DiskReservationRatio float64
// MemoryFloorRatio is the minimum live MemAvailable the host must retain
// after admitting the request, expressed as a fraction of total host
// memory (e.g. 0.05 = keep at least 5% of RAM free). Expressing this as a
// ratio means a 16 GB laptop and a 256 GB box scale their headroom
// proportionally — a fixed MB floor would be either pointless on big
// hosts or starve small ones. 0 = no floor.
MemoryFloorRatio float64
// CPUOverProvisionFactor multiplies the CPU reservation budget. Docker
// --cpus is a CFS cap, not a hard reservation, so idle sandboxes share
// cores happily — a 10× default lets typical workloads pack densely.
// Values below 1.0 are clamped to 1.0 (no overcommit). 0 is treated as
// the default 1.0 so a zero-value Limits behaves as before.
CPUOverProvisionFactor float64
// MemoryOverProvisionFactor multiplies the memory reservation budget.
// Memory pressure is harder to recover from than CPU contention (OOM
// killer fires on hard limits), so the live MemoryFloorRatio check is
// the real backstop when this is set high. Same clamping as the CPU
// factor.
MemoryOverProvisionFactor float64
// RSSWatermarkRatio is the Phase 5 memory-overcommit safety floor,
// expressed as a fraction of host memory. Admission rejects a
// request when EffectiveMemoryFree (HostMem - sum-of-VMM-RSS) would
// drop below this watermark after admit. 0 disables the check (the
// default) — operators opt in by setting the env var, and the check
// is also gated on a non-nil RSSSource whose Ready() returns true,
// so a pre-tick daemon falls back to nominal accounting. The point
// of this axis is to safely raise MemoryOverProvisionFactor:
// nominal accounting caps you at "host RAM × ratio × factor" but
// can't see whether VMMs are actually touching the memory they
// reserved; the watermark says "regardless of nominal, keep this
// much real free RAM in reserve for the next RSS spike."
RSSWatermarkRatio float64
}
Limits configures the admitter. Zero values disable the corresponding check.
type MemProbe ¶
MemProbe reports live free memory in MB. The default implementation reads /proc/meminfo MemAvailable. Tests can substitute a fake.
func NewProcMeminfoProbe ¶
func NewProcMeminfoProbe() MemProbe
NewProcMeminfoProbe returns a MemProbe backed by /proc/meminfo with a 500ms cache. The cache window is small enough that decisions remain responsive to actual host memory pressure but large enough to absorb a burst of concurrent CreateSandbox calls without 50 file reads.
type ParkGate ¶
type ParkGate struct {
Admitter *Admitter
// GuardShape is reserved for one default-shaped sandbox after each park.
GuardShape Request
}
ParkGate implements warm-pool refill gating against the host admitter.
func (*ParkGate) CanPark ¶
func (g *ParkGate) CanPark(shape dockerpool.ParkShape) bool
CanPark reports whether parking another slot leaves the guard band.
func (*ParkGate) ParkReservation ¶
func (g *ParkGate) ParkReservation(slotID string, shape dockerpool.ParkShape) error
ParkReservation reserves capacity under park:<slot-id>.
func (*ParkGate) ReleasePark ¶
ReleasePark frees a park reservation.
type RSSSource ¶
RSSSource is the Phase 5 bridge for the per-VMM RSS sampler that lives in internal/runtime/firecracker. The structural shape keeps pkg/capacity from importing the runtime package (and forming a cycle through main.go); the sampler satisfies the interface without declaring it.
TotalRSSMB returns the most recent aggregate in MB; Ready reports whether the sampler has produced at least one observation. Admission only consults TotalRSSMB when Ready returns true — pre-tick, the "0 MB resident" reading is indistinguishable from "host is empty" and falsely admits under any non-zero RSSWatermarkRatio.
type Request ¶
type Request struct {
CPU float64
MemoryMB int
DiskGB int
GPUs int
GPUVendor string
Runtime string
TemplateID string
// ModuleRef, when set alongside Runtime="wasm", lets cluster placement
// prefer peers that already have the module cached locally.
ModuleRef string
}
Request is the per-sandbox resource ask, in normalized units. CPU is fractional cores (e.g. 0.5 = half a core); memory is whole MB; disk is whole GB. GPUs is the count requested (0 = no GPU); GPUVendor is the required vendor when GPUs > 0 and is matched against HostInfo.GPUVendor at admission/placement time. Runtime is the OCI runtime identifier ("docker", "gvisor", ...) — empty means "any runtime the host supports." TemplateID, when set alongside Runtime="firecracker", lets placement prefer (and gate on) peers that already have the template's artifacts cached locally — empty means "any host," matching pre-Phase-6 behaviour.
type Snapshot ¶
type Snapshot struct {
HostCPUCores int `json:"host_cpu_cores"`
HostMemoryTotalMB int `json:"host_memory_total_mb"`
HostDiskTotalGB int `json:"host_disk_total_gb,omitempty"`
HostDiskFreeGB int `json:"host_disk_free_gb,omitempty"`
ReservedCPU float64 `json:"reserved_cpu"`
ReservedMemoryMB int `json:"reserved_memory_mb"`
ReservedDiskGB int `json:"reserved_disk_gb,omitempty"`
ReservedGPUs int `json:"reserved_gpus,omitempty"`
AvailableCPU float64 `json:"available_cpu"`
AvailableMemoryMB int `json:"available_memory_mb"`
AvailableDiskGB int `json:"available_disk_gb,omitempty"`
AvailableGPUs int `json:"available_gpus,omitempty"`
LiveMemoryFreeMB int `json:"live_memory_free_mb"`
SandboxesActive int `json:"sandboxes_active"`
// SandboxesByRuntime is the live reservation count keyed by OCI runtime
// ("docker"/containerd, "gvisor", "wasm", "isolate", "firecracker";
// "unspecified" for a runtime-agnostic reservation). Lets dashboards show
// live sandboxes broken down by isolation type — the bare SandboxesActive
// gauge has no runtime label.
SandboxesByRuntime map[string]int `json:"sandboxes_by_runtime,omitempty"`
CanAdmit bool `json:"can_admit"`
Reasons []string `json:"reasons,omitempty"`
CPUReservationRatio float64 `json:"cpu_reservation_ratio"`
MemoryReservationRatio float64 `json:"memory_reservation_ratio"`
DiskReservationRatio float64 `json:"disk_reservation_ratio,omitempty"`
MemoryFloorRatio float64 `json:"memory_floor_ratio"`
CPUOverProvisionFactor float64 `json:"cpu_overprovision_factor"`
MemoryOverProvisionFactor float64 `json:"memory_overprovision_factor"`
// MemoryFloorMB is the absolute floor derived from MemoryFloorRatio and
// host memory, exposed for operators reading /capacity.
MemoryFloorMB int `json:"memory_floor_mb"`
// Phase 5 effective-memory axis. ActualRSSMB is the live aggregate
// from the per-VMM sampler; EffectiveMemoryFreeMB is the headroom
// admission actually decides against (HostMem - ActualRSSMB);
// RSSWatermarkMB is the absolute floor derived from
// RSSWatermarkRatio. All four are 0 on hosts without a sampler so
// peers running older binaries report 0 and placement falls back to
// nominal accounting rather than treating "0" as "host is empty".
ActualRSSMB int `json:"actual_rss_mb,omitempty"`
EffectiveMemoryFreeMB int `json:"effective_memory_free_mb,omitempty"`
RSSWatermarkMB int `json:"rss_watermark_mb,omitempty"`
RSSWatermarkRatio float64 `json:"rss_watermark_ratio,omitempty"`
// CPUBudget and MemoryBudgetMB are the post-overcommit budgets actually
// used by Admit, exposed so operators can see the effective ceiling.
CPUBudget float64 `json:"cpu_budget"`
MemoryBudgetMB int `json:"memory_budget_mb"`
DiskBudgetGB int `json:"disk_budget_gb,omitempty"`
// GPUCount and GPUVendor are static node attributes set by the operator
// (no auto-detection in Phase 1). Placement uses them to skip a peer
// that cannot satisfy a GPU sandbox at all — running the admitter
// remotely after a doomed forward would just 503.
GPUCount int `json:"gpu_count,omitempty"`
GPUVendor string `json:"gpu_vendor,omitempty"`
// SupportedRuntimes lists the OCI runtimes this host can run
// ("docker", "gvisor", ...). Used by placement to skip a peer that
// doesn't have, say, runsc installed. Empty = legacy node, treated as
// supporting any runtime so rolling upgrades don't strand pre-D peers.
SupportedRuntimes []string `json:"supported_runtimes,omitempty"`
// ContainerEngine is an observability tag (docker|containerd) so benches
// and canary nodes compare engines like-for-like. Deliberately NOT a
// placement attribute — see plans/containerd-engine.md §2 D18.
ContainerEngine string `json:"container_engine,omitempty"`
// LocalTemplateInventoryKnown distinguishes an authoritative empty
// Firecracker template inventory from a legacy/unknown peer. When
// false, placement treats LocalTemplateIDs as "unknown, allow" for
// rolling upgrades and just-joined nodes. When true, an empty
// LocalTemplateIDs slice means this host definitely has no local
// templates cached.
LocalTemplateInventoryKnown bool `json:"local_template_inventory_known,omitempty"`
// LocalTemplateIDs lists the Firecracker templates whose artifact
// files (rootfs.ext4 + snapshot.{memory,state}) are present on this
// host. Placement gates a Firecracker create against it so a clone
// against a template the target has never seen lands on a node that
// already has it — preserving the <100ms boot promise. Consult
// LocalTemplateInventoryKnown before treating an empty list as
// authoritative: pre-upgrade peers omit both fields and degrade to
// pre-feature behaviour, while known=true plus an empty list means
// "definitely no local templates here."
LocalTemplateIDs []string `json:"local_template_ids,omitempty"`
// LocalWasmModuleInventoryKnown mirrors LocalTemplateInventoryKnown for
// runtime=wasm module_ref placement (plans/wasm-runtime.md Phase 7).
LocalWasmModuleInventoryKnown bool `json:"local_wasm_module_inventory_known,omitempty"`
// LocalWasmModuleIDs lists module_ref values cached on this host.
LocalWasmModuleIDs []string `json:"local_wasm_module_ids,omitempty"`
}
Snapshot is a read-only view of admitter state, suitable for an HTTP /capacity response. It is also the payload in cluster capacity heartbeats: every field needed to decide "can this peer host this request?" must be present here, otherwise placement scores against stale assumptions and the caller fans out to a node that can't actually admit.