Documentation
¶
Overview ¶
Package resource provides system resource monitoring and control.
Index ¶
- Constants
- func ComputeSlots(capSlots, hardMax int, cpuCores float64, memBudget int64, jobCores float64, ...) int
- type AIMD
- type Capacity
- type Controller
- func (c *Controller) AcquireSlot(ctx context.Context) bool
- func (c *Controller) AcquireSlotBlocking(ctx context.Context) error
- func (c *Controller) AdaptiveMaxJobs() int
- func (c *Controller) ForceGC()
- func (c *Controller) GetMetrics() *SystemMetrics
- func (c *Controller) GetStatus() *ControllerStatus
- func (c *Controller) IsThrottled() bool
- func (c *Controller) ReleaseSlot()
- func (c *Controller) SetMaxConcurrentJobs(max int)
- func (c *Controller) Start(ctx context.Context) error
- func (c *Controller) Stop()
- type ControllerConfig
- type ControllerStatus
- type CostBook
- type HostResources
- type JobSample
- type Manager
- type ManagerConfig
- type Outcome
- type ProbeResult
- type Prober
- type SystemMetrics
- type ToolEstimate
Constants ¶
const DefaultHardMaxSlots = 64
DefaultHardMaxSlots is the most jobs a sensor runs at once when nothing lower is configured.
Variables ¶
This section is empty.
Functions ¶
func ComputeSlots ¶ added in v0.10.0
func ComputeSlots(capSlots, hardMax int, cpuCores float64, memBudget int64, jobCores float64, jobMem int64) int
ComputeSlots is how many jobs fit:
clamp(min(cap (when > 0), ⌊cpuCores / jobCores⌋, ⌊memBudget / jobMem⌋), 1, hardMax)
A resource that is unknown (0) does not limit. hardMax <= 0 is DefaultHardMaxSlots.
Types ¶
type AIMD ¶ added in v0.10.0
type AIMD struct {
// contains filtered or unexported fields
}
AIMD is the slot controller's congestion window: halved on a congestion signal (OOM kill, timeout, CPU throttling), grown by one slot per window of successful jobs. Safe for concurrent use.
func (*AIMD) Decrease ¶ added in v0.10.0
func (a *AIMD) Decrease()
Decrease halves the window (multiplicative decrease, never below 1).
func (*AIMD) DecreaseFrom ¶ added in v0.10.0
DecreaseFrom halves the window from the slots actually in use (current), when they are below the window: congestion at 4 slots under a window of 10 leaves 2, not 5.
type Capacity ¶ added in v0.10.0
type Capacity struct {
// SlotsTotal is how many jobs the sensor runs at once now (dynamic, at
// most the configured cap).
SlotsTotal int `json:"slots_total"`
// SlotsFree is SlotsTotal minus ActiveJobs (never negative).
SlotsFree int `json:"slots_free"`
// ActiveJobs is the jobs it holds now.
ActiveJobs int `json:"active_jobs"`
// PerTool is the per-job cost of each tool the sensor runs.
PerTool map[string]ToolEstimate `json:"per_tool,omitempty"`
}
Capacity is the slot picture a heartbeat carries as "capacity".
type Controller ¶
type Controller struct {
// contains filtered or unexported fields
}
Controller monitors system resources and controls work admission.
func NewController ¶
func NewController(config *ControllerConfig) *Controller
NewController creates a new resource controller.
func (*Controller) AcquireSlot ¶
func (c *Controller) AcquireSlot(ctx context.Context) bool
AcquireSlot attempts to acquire a job slot. Returns true if a slot was acquired, false if throttled or at capacity. The caller MUST call ReleaseSlot when done.
func (*Controller) AcquireSlotBlocking ¶
func (c *Controller) AcquireSlotBlocking(ctx context.Context) error
AcquireSlotBlocking blocks until a slot is available or context is canceled.
func (*Controller) AdaptiveMaxJobs ¶
func (c *Controller) AdaptiveMaxJobs() int
AdaptiveMaxJobs calculates recommended max jobs based on current load.
func (*Controller) GetMetrics ¶
func (c *Controller) GetMetrics() *SystemMetrics
GetMetrics returns the current system metrics.
func (*Controller) GetStatus ¶
func (c *Controller) GetStatus() *ControllerStatus
GetStatus returns the current controller status.
func (*Controller) IsThrottled ¶
func (c *Controller) IsThrottled() bool
IsThrottled returns true if resource thresholds are exceeded.
func (*Controller) ReleaseSlot ¶
func (c *Controller) ReleaseSlot()
ReleaseSlot releases a previously acquired job slot.
func (*Controller) SetMaxConcurrentJobs ¶
func (c *Controller) SetMaxConcurrentJobs(max int)
SetMaxConcurrentJobs dynamically adjusts the max concurrent jobs.
type ControllerConfig ¶
type ControllerConfig struct {
// CPUThreshold is the CPU percentage above which new work is paused.
// Default: 85%
CPUThreshold float64
// MemoryThreshold is the memory percentage above which new work is paused.
// Default: 85%
MemoryThreshold float64
// MaxConcurrentJobs limits the number of concurrent jobs.
// Default: runtime.NumCPU()
MaxConcurrentJobs int
// MinConcurrentJobs is the minimum jobs to allow even under load.
// Default: 1
MinConcurrentJobs int
// SampleInterval is how often to sample system metrics.
// Default: 5 seconds
SampleInterval time.Duration
// CooldownDuration is how long to wait after threshold exceeded.
// Default: 30 seconds
CooldownDuration time.Duration
// OnThresholdExceeded is called when resource threshold is exceeded.
OnThresholdExceeded func(metrics *SystemMetrics, reason string)
// OnThresholdCleared is called when resources return to normal.
OnThresholdCleared func(metrics *SystemMetrics)
// Verbose enables debug logging.
Verbose bool
}
ControllerConfig configures the resource controller.
func DefaultControllerConfig ¶
func DefaultControllerConfig() *ControllerConfig
DefaultControllerConfig returns sensible defaults.
type ControllerStatus ¶
type ControllerStatus struct {
Throttled bool `json:"throttled"`
ThrottledAt time.Time `json:"throttled_at,omitempty"`
ThrottleReason string `json:"throttle_reason,omitempty"`
ActiveJobs int `json:"active_jobs"`
PendingJobs int `json:"pending_jobs"`
RejectedJobs int64 `json:"rejected_jobs"`
MaxJobs int `json:"max_jobs"`
Metrics *SystemMetrics `json:"metrics,omitempty"`
}
ControllerStatus represents the current state of the controller.
type CostBook ¶ added in v0.10.0
type CostBook struct {
// contains filtered or unexported fields
}
CostBook is the per-tool cost history: priors for tools it has not seen, learned from finished jobs, persisted to a small JSON file so it survives restarts. Safe for concurrent use.
func NewCostBook ¶ added in v0.10.0
NewCostBook opens the history at path ("" keeps it in memory). A missing file starts empty; an unreadable or corrupt one also starts empty and is returned as the error (the book is usable either way).
func (*CostBook) Estimate ¶ added in v0.10.0
func (b *CostBook) Estimate(tool string) ToolEstimate
Estimate is a tool's per-job cost: learned, or the prior.
func (*CostBook) Observe ¶ added in v0.10.0
Observe learns from a finished job. Only successes teach CPU and time; a kill raises the memory estimate (half again), since it was most likely the OOM killer.
type HostResources ¶ added in v0.10.0
type HostResources struct {
// CPUCores is the CPU this sensor may use: the online CPUs, narrowed by
// the cgroup's cpuset and CPU quota (cpu.max / cfs_quota_us).
CPUCores float64 `json:"cpu_cores"`
// CPUUsedPct is how busy that CPU was since the previous probe, 0-100:
// the cgroup's usage against its quota when it has one, else the host's.
CPUUsedPct float64 `json:"cpu_used_pct"`
// MemTotalBytes is the memory this sensor may use: the host's, or the
// cgroup's limit when lower.
MemTotalBytes int64 `json:"mem_total_bytes"`
// MemAvailableBytes is what is left of it now (page cache that can be
// reclaimed counts as available).
MemAvailableBytes int64 `json:"mem_available_bytes"`
// Load1 is the host's one-minute load average.
Load1 float64 `json:"load1"`
// DiskFreeBytes is the space available to the sensor on its work
// directory's file system.
DiskFreeBytes int64 `json:"disk_free_bytes"`
}
HostResources is the resource snapshot a heartbeat carries as "resources". 0 means unknown, except CPUUsedPct (0-100, 0 on the first probe: usage needs two samples).
type JobSample ¶ added in v0.10.0
type JobSample struct {
Tool string
Targets int
Wall time.Duration
// CPUSeconds and PeakRSSBytes come from the job's processes; 0 when
// unknown (the job ran no measured process).
CPUSeconds float64
PeakRSSBytes int64
Outcome Outcome
}
JobSample is what a finished job cost.
type Manager ¶ added in v0.10.0
type Manager struct {
// contains filtered or unexported fields
}
Manager turns the sensor's resources and its tools' learned costs into a live slot count: the jobs that fit the CPU and memory it may use, at most the configured cap, narrowed by an AIMD window that halves on OOM kills, timeouts and CPU throttling and slowly grows back on success. Safe for concurrent use.
func NewManager ¶ added in v0.10.0
func NewManager(cfg ManagerConfig) *Manager
NewManager creates a Manager. A history that cannot be loaded is reported to OnError and started over.
func (*Manager) Observe ¶ added in v0.10.0
Observe learns from a finished job and moves the AIMD window: down on a timeout or kill, up on success. The history is saved.
type ManagerConfig ¶ added in v0.10.0
type ManagerConfig struct {
// Cap is the operator's limit on concurrent jobs; 0 sets none (the
// slots then follow the resources, up to HardMax).
Cap int
// HardMax bounds the slots whatever the resources (default
// DefaultHardMaxSlots).
HardMax int
// Tools are the tools the sensor runs; the slots are sized for the most
// demanding of them. Empty: a generic job (1 core, 512 MiB).
Tools []string
// StateFile persists the per-tool cost history ("" keeps it in memory).
StateFile string
// Prober reads the resources (nil: the running system, no work dir).
Prober *Prober
// ProbeInterval is the longest a probe is reused (default 10s).
ProbeInterval time.Duration
// OnError receives non-fatal errors (loading or saving the history).
OnError func(error)
}
ManagerConfig configures a Manager.
func (ManagerConfig) MaxSlots ¶ added in v0.10.0
func (c ManagerConfig) MaxSlots() int
MaxSlots is the most slots there can be: the cap, else HardMax.
type Outcome ¶ added in v0.10.0
type Outcome int
Outcome is how a job ended, for the cost history and the slot controller.
const ( // OutcomeSuccess: the job finished. OutcomeSuccess Outcome = iota // OutcomeFailed: the job failed for its own reasons (bad target, tool // error); says nothing about load. OutcomeFailed // OutcomeTimeout: the job ran out of time. OutcomeTimeout // OutcomeKilled: a process of the job was killed (SIGKILL), typically // by the OOM killer. OutcomeKilled )
type ProbeResult ¶ added in v0.10.0
type ProbeResult struct {
Resources HostResources
// ThrottledRatio is the share of CPU periods (0-1) in which the cgroup
// was throttled by its quota since the previous probe.
ThrottledRatio float64
// OOMKills is how many processes the cgroup's OOM killer ended since the
// previous probe.
OOMKills uint64
}
ProbeResult is one probe: the snapshot plus the pressure signals seen since the previous probe.
type Prober ¶ added in v0.10.0
type Prober struct {
// Root is prepended to /proc and /sys paths ("" or "/": the real ones).
Root string
// WorkDir is the directory whose file system DiskFreeBytes reports
// ("" skips it).
WorkDir string
// NumCPU returns the CPUs the process may run on (default
// runtime.NumCPU).
NumCPU func() int
// DiskFree returns the bytes available on dir's file system (default:
// statfs; 0 where unsupported).
DiskFree func(dir string) int64
// Now is the clock (default time.Now).
Now func() time.Time
// contains filtered or unexported fields
}
Prober reads the host's resources. The zero value probes the running system; tests set Root to a directory holding proc/ and sys/fs/cgroup/ fixtures. Safe for concurrent use.
func (*Prober) Probe ¶ added in v0.10.0
func (p *Prober) Probe() ProbeResult
Probe takes a snapshot.
type SystemMetrics ¶
type SystemMetrics struct {
CPUPercent float64 `json:"cpu_percent"`
MemoryPercent float64 `json:"memory_percent"`
MemoryUsedMB int64 `json:"memory_used_mb"`
MemoryTotalMB int64 `json:"memory_total_mb"`
NumGoroutines int `json:"num_goroutines"`
NumCPU int `json:"num_cpu"`
Timestamp time.Time `json:"timestamp"`
}
SystemMetrics contains current system resource metrics.
type ToolEstimate ¶ added in v0.10.0
type ToolEstimate struct {
// CPUSeconds is the CPU time one job uses (all its processes).
CPUSeconds float64 `json:"est_cpu_s"`
// MemBytes is the peak resident memory of one job.
MemBytes int64 `json:"est_mem_bytes"`
// ThroughputTargetsPerMin is how many targets one job slot scans per
// minute.
ThroughputTargetsPerMin float64 `json:"throughput_targets_per_min"`
}
ToolEstimate is what one job of a tool costs on this sensor, as a heartbeat carries it under capacity.per_tool.<tool>.