resource

package
v0.26.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 13, 2026 License: GPL-3.0 Imports: 18 Imported by: 0

Documentation

Overview

Package resource provides system resource monitoring and control.

Stability: Internal-bound (docs/STABILITY.md): public today because other public packages use it; it moves under internal/ before v1.0.0. Do not import it from a sensor.

Index

Constants

View Source
const (
	MinReservedMem = int64(256) << 20
	MaxReservedMem = int64(1) << 30
)

Default sensor memory reserve (DefaultReservedMem).

View Source
const DefaultHardMaxSlots = 64

DefaultHardMaxSlots is the most jobs a sensor runs at once when nothing lower is configured.

Variables

This section is empty.

Functions

func ComputeSlots added in v0.10.0

func ComputeSlots(capSlots, hardMax int, cpuCores float64, memBudget int64, jobCores float64, jobMem int64) int

ComputeSlots is how many jobs fit:

clamp(min(cap (when > 0), ⌊cpuCores / jobCores⌋, ⌊memBudget / jobMem⌋), 1, hardMax)

A resource that is unknown (0) does not limit. hardMax <= 0 is DefaultHardMaxSlots.

func DefaultReservedMem added in v0.15.0

func DefaultReservedMem(total int64) int64

DefaultReservedMem is the memory reserved for the sensor by default: a tenth of total, at least MinReservedMem and at most MaxReservedMem (MinReservedMem when total is unknown).

Types

type AIMD added in v0.10.0

type AIMD struct {
	// contains filtered or unexported fields
}

AIMD is the slot controller's congestion window: halved on a congestion signal (OOM kill, timeout, CPU throttling), grown by one slot per window of successful jobs. Safe for concurrent use.

func NewAIMD added in v0.10.0

func NewAIMD(maxSlots int) *AIMD

NewAIMD starts at max (no congestion seen yet).

func (*AIMD) Decrease added in v0.10.0

func (a *AIMD) Decrease()

Decrease halves the window (multiplicative decrease, never below 1).

func (*AIMD) DecreaseFrom added in v0.10.0

func (a *AIMD) DecreaseFrom(current int)

DecreaseFrom halves the window from the slots actually in use (current), when they are below the window: congestion at 4 slots under a window of 10 leaves 2, not 5.

func (*AIMD) Increase added in v0.10.0

func (a *AIMD) Increase()

Increase grows the window by 1/window: one slot after a full window of successes (additive increase), up to the maximum.

func (*AIMD) Limit added in v0.10.0

func (a *AIMD) Limit() int

Limit is the current window, >= 1.

func (*AIMD) SetMax added in v0.10.0

func (a *AIMD) SetMax(maxSlots int)

SetMax changes the ceiling (the window is clamped to it).

type Capacity added in v0.10.0

type Capacity struct {
	// SlotsTotal is how many jobs the sensor runs at once now (dynamic, at
	// most the configured cap).
	SlotsTotal int `json:"slots_total"`
	// SlotsFree is SlotsTotal minus ActiveJobs (never negative).
	SlotsFree int `json:"slots_free"`
	// ActiveJobs is the jobs it holds now.
	ActiveJobs int `json:"active_jobs"`
	// PerTool is the per-job cost of each tool the sensor runs.
	PerTool map[string]ToolEstimate `json:"per_tool,omitempty"`
}

Capacity is the slot picture a heartbeat carries as "capacity".

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

Controller monitors system resources and controls work admission.

func NewController

func NewController(config *ControllerConfig) *Controller

NewController creates a new resource controller.

func (*Controller) AcquireSlot

func (c *Controller) AcquireSlot(ctx context.Context) bool

AcquireSlot attempts to acquire a job slot. Returns true if a slot was acquired, false if throttled or at capacity. The caller MUST call ReleaseSlot when done.

func (*Controller) AcquireSlotBlocking

func (c *Controller) AcquireSlotBlocking(ctx context.Context) error

AcquireSlotBlocking blocks until a slot is available or context is canceled.

func (*Controller) AdaptiveMaxJobs

func (c *Controller) AdaptiveMaxJobs() int

AdaptiveMaxJobs calculates recommended max jobs based on current load.

func (*Controller) ForceGC

func (c *Controller) ForceGC()

ForceGC triggers garbage collection.

func (*Controller) GetMetrics

func (c *Controller) GetMetrics() *SystemMetrics

GetMetrics returns the current system metrics.

func (*Controller) GetStatus

func (c *Controller) GetStatus() *ControllerStatus

GetStatus returns the current controller status.

func (*Controller) IsThrottled

func (c *Controller) IsThrottled() bool

IsThrottled returns true if resource thresholds are exceeded.

func (*Controller) ReleaseSlot

func (c *Controller) ReleaseSlot()

ReleaseSlot releases a previously acquired job slot.

func (*Controller) SetMaxConcurrentJobs

func (c *Controller) SetMaxConcurrentJobs(max int)

SetMaxConcurrentJobs dynamically adjusts the max concurrent jobs.

func (*Controller) Start

func (c *Controller) Start(ctx context.Context) error

Start begins resource monitoring.

func (*Controller) Stop

func (c *Controller) Stop()

Stop stops the resource controller.

type ControllerConfig

type ControllerConfig struct {
	// CPUThreshold is the CPU percentage above which new work is paused.
	// Default: 85%
	CPUThreshold float64

	// MemoryThreshold is the memory percentage above which new work is paused.
	// Default: 85%
	MemoryThreshold float64

	// MaxConcurrentJobs limits the number of concurrent jobs.
	// Default: runtime.NumCPU()
	MaxConcurrentJobs int

	// MinConcurrentJobs is the minimum jobs to allow even under load.
	// Default: 1
	MinConcurrentJobs int

	// SampleInterval is how often to sample system metrics.
	// Default: 5 seconds
	SampleInterval time.Duration

	// CooldownDuration is how long to wait after threshold exceeded.
	// Default: 30 seconds
	CooldownDuration time.Duration

	// OnThresholdExceeded is called when resource threshold is exceeded.
	OnThresholdExceeded func(metrics *SystemMetrics, reason string)

	// OnThresholdCleared is called when resources return to normal.
	OnThresholdCleared func(metrics *SystemMetrics)

	// Verbose enables debug logging.
	Verbose bool
}

ControllerConfig configures the resource controller.

func DefaultControllerConfig

func DefaultControllerConfig() *ControllerConfig

DefaultControllerConfig returns sensible defaults.

type ControllerStatus

type ControllerStatus struct {
	Throttled      bool           `json:"throttled"`
	ThrottledAt    time.Time      `json:"throttled_at,omitempty"`
	ThrottleReason string         `json:"throttle_reason,omitempty"`
	ActiveJobs     int            `json:"active_jobs"`
	PendingJobs    int            `json:"pending_jobs"`
	RejectedJobs   int64          `json:"rejected_jobs"`
	MaxJobs        int            `json:"max_jobs"`
	Metrics        *SystemMetrics `json:"metrics,omitempty"`
}

ControllerStatus represents the current state of the controller.

type CostBook added in v0.10.0

type CostBook struct {
	// contains filtered or unexported fields
}

CostBook is the per-tool cost history: priors for tools it has not seen, learned from finished jobs, persisted to a small JSON file so it survives restarts. Safe for concurrent use.

func NewCostBook added in v0.10.0

func NewCostBook(path string) (*CostBook, error)

NewCostBook opens the history at path ("" keeps it in memory). A missing file starts empty; an unreadable or corrupt one also starts empty and is returned as the error (the book is usable either way).

func (*CostBook) Estimate added in v0.10.0

func (b *CostBook) Estimate(tool string) ToolEstimate

Estimate is a tool's per-job cost: learned, or the prior.

func (*CostBook) Observe added in v0.10.0

func (b *CostBook) Observe(s JobSample)

Observe learns from a finished job. Only successes teach CPU and time; a kill raises the memory estimate (half again), since it was most likely the OOM killer.

func (*CostBook) Save added in v0.10.0

func (b *CostBook) Save() error

Save writes the history (atomically: temp file + rename, mode 0600).

func (*CostBook) SetPrior added in v0.11.0

func (b *CostBook) SetPrior(tool string, h ToolCostHint)

SetPrior makes h the prior of tool: the cost assumed until the tool has a learned history. An invalid hint (a member not positive) is ignored.

func (*CostBook) Tools added in v0.10.0

func (b *CostBook) Tools() []string

Tools returns the tools with a learned history, sorted.

type HostResources added in v0.10.0

type HostResources struct {
	// CPUCores is the CPU this sensor may use: the online CPUs, narrowed by
	// the cgroup's cpuset and CPU quota (cpu.max / cfs_quota_us).
	CPUCores float64 `json:"cpu_cores"`
	// CPUUsedPct is how busy that CPU was since the previous probe, 0-100:
	// the cgroup's usage against its quota when it has one, else the host's.
	CPUUsedPct float64 `json:"cpu_used_pct"`
	// MemTotalBytes is the memory this sensor may use: the host's, or the
	// cgroup's limit when lower.
	MemTotalBytes int64 `json:"mem_total_bytes"`
	// MemAvailableBytes is what is left of it now (page cache that can be
	// reclaimed counts as available).
	MemAvailableBytes int64 `json:"mem_available_bytes"`
	// Load1 is the host's one-minute load average.
	Load1 float64 `json:"load1"`
	// DiskFreeBytes is the space available to the sensor on its work
	// directory's file system.
	DiskFreeBytes int64 `json:"disk_free_bytes"`
}

HostResources is the resource snapshot a heartbeat carries as "resources". 0 means unknown, except CPUUsedPct (0-100, 0 on the first probe: usage needs two samples).

type JobSample added in v0.10.0

type JobSample struct {
	Tool    string
	Targets int
	Wall    time.Duration
	// CPUSeconds and PeakRSSBytes come from the job's processes; 0 when
	// unknown (the job ran no measured process).
	CPUSeconds   float64
	PeakRSSBytes int64
	Outcome      Outcome
}

JobSample is what a finished job cost.

type Manager added in v0.10.0

type Manager struct {
	// contains filtered or unexported fields
}

Manager turns the sensor's resources and its tools' learned costs into a live slot count: the jobs that fit the CPU and memory it may use, at most the configured cap, narrowed by an AIMD window that halves on OOM kills, timeouts and CPU throttling and slowly grows back on success. Safe for concurrent use.

func NewManager added in v0.10.0

func NewManager(cfg ManagerConfig) *Manager

NewManager creates a Manager. A history that cannot be loaded is reported to OnError and started over.

func (*Manager) Book added in v0.10.0

func (m *Manager) Book() *CostBook

Book is the cost history.

func (*Manager) Cap added in v0.13.0

func (m *Manager) Cap() int

Cap is the operator's limit on concurrent jobs (ManagerConfig.Cap); 0 when the operator set none and the slots follow the resources alone (up to HardMax, which is a safety bound, not a capacity).

func (*Manager) MaxSlots added in v0.10.0

func (m *Manager) MaxSlots() int

MaxSlots is the most slots this manager hands out.

func (*Manager) Observe added in v0.10.0

func (m *Manager) Observe(s JobSample)

Observe learns from a finished job and moves the AIMD window: down on a timeout or kill, up on success. The history is saved.

func (*Manager) SetTools added in v0.10.0

func (m *Manager) SetTools(tools []string)

SetTools sets the tools the slots are sized for.

func (*Manager) Slots added in v0.10.0

func (m *Manager) Slots(active int) int

Slots is how many jobs the sensor may hold now, with active running.

func (*Manager) Snapshot added in v0.10.0

func (m *Manager) Snapshot(active int) (HostResources, Capacity)

Snapshot is a fresh probe and the capacity with active jobs running, for a heartbeat.

type ManagerConfig added in v0.10.0

type ManagerConfig struct {
	// Cap is the operator's limit on concurrent jobs; 0 sets none (the
	// slots then follow the resources, up to HardMax).
	Cap int
	// HardMax bounds the slots whatever the resources (default
	// DefaultHardMaxSlots).
	HardMax int
	// Tools are the tools the sensor runs; the slots are sized for the most
	// demanding of them. Empty: a generic job (1 core, 512 MiB).
	Tools []string
	// CostHints are the sensor's own per-job cost estimates, the prior of a
	// tool without history (instead of the built-in default); see
	// core.ToolRegistry.CostHints.
	CostHints map[string]ToolCostHint
	// StateFile persists the per-tool cost history ("" keeps it in memory).
	StateFile string
	// Prober reads the resources (nil: the running system, no work dir).
	Prober *Prober
	// ProbeInterval is the longest a probe is reused (default 10s).
	ProbeInterval time.Duration
	// ReservedMemBytes is memory the slots leave free for the sensor itself
	// and the kernel, like a kubelet's system-reserved: what the sensor
	// needs to parse and upload results, heartbeat and poll while its
	// scanners run (api RFC-035 §5.2). 0 is DefaultReservedMem of the
	// memory the sensor may use; negative reserves nothing. CPU is not
	// reserved: it is compressible, and scanners yield it to the sensor
	// (core.ScannerPriority).
	ReservedMemBytes int64
	// OnError receives non-fatal errors (loading or saving the history).
	OnError func(error)
}

ManagerConfig configures a Manager.

func (ManagerConfig) MaxSlots added in v0.10.0

func (c ManagerConfig) MaxSlots() int

MaxSlots is the most slots there can be: the cap, else HardMax.

type Outcome added in v0.10.0

type Outcome int

Outcome is how a job ended, for the cost history and the slot controller.

const (
	// OutcomeSuccess: the job finished.
	OutcomeSuccess Outcome = iota
	// OutcomeFailed: the job failed for its own reasons (bad target, tool
	// error); says nothing about load.
	OutcomeFailed
	// OutcomeTimeout: the job ran out of time.
	OutcomeTimeout
	// OutcomeKilled: a process of the job was killed (SIGKILL), typically
	// by the OOM killer.
	OutcomeKilled
)

type ProbeResult added in v0.10.0

type ProbeResult struct {
	Resources HostResources
	// ThrottledRatio is the share of CPU periods (0-1) in which the cgroup
	// was throttled by its quota since the previous probe.
	ThrottledRatio float64
	// OOMKills is how many processes the cgroup's OOM killer ended since the
	// previous probe.
	OOMKills uint64
}

ProbeResult is one probe: the snapshot plus the pressure signals seen since the previous probe.

type Prober added in v0.10.0

type Prober struct {
	// Root is prepended to /proc and /sys paths ("" or "/": the real ones).
	Root string
	// WorkDir is the directory whose file system DiskFreeBytes reports
	// ("" skips it).
	WorkDir string
	// NumCPU returns the CPUs the process may run on (default
	// runtime.NumCPU).
	NumCPU func() int
	// DiskFree returns the bytes available on dir's file system (default:
	// statfs; 0 where unsupported).
	DiskFree func(dir string) int64
	// Now is the clock (default time.Now).
	Now func() time.Time
	// contains filtered or unexported fields
}

Prober reads the host's resources. The zero value probes the running system; tests set Root to a directory holding proc/ and sys/fs/cgroup/ fixtures. Safe for concurrent use.

func (*Prober) Probe added in v0.10.0

func (p *Prober) Probe() ProbeResult

Probe takes a snapshot.

type SystemMetrics

type SystemMetrics struct {
	CPUPercent    float64   `json:"cpu_percent"`
	MemoryPercent float64   `json:"memory_percent"`
	MemoryUsedMB  int64     `json:"memory_used_mb"`
	MemoryTotalMB int64     `json:"memory_total_mb"`
	NumGoroutines int       `json:"num_goroutines"`
	NumCPU        int       `json:"num_cpu"`
	Timestamp     time.Time `json:"timestamp"`
}

SystemMetrics contains current system resource metrics.

type ToolCostHint added in v0.11.0

type ToolCostHint struct {
	// Cores is the CPU cores one running job keeps busy.
	Cores float64
	// MemBytes is the peak memory of one job.
	MemBytes int64
	// SecondsPerTarget is the wall time one target takes.
	SecondsPerTarget float64
}

ToolCostHint is a sensor author's estimate of what one job of a tool costs, used as the prior for a tool without history (instead of the built-in default). The learned history replaces it after a few jobs.

type ToolEstimate added in v0.10.0

type ToolEstimate struct {
	// CPUSeconds is the CPU time one job uses (all its processes).
	CPUSeconds float64 `json:"est_cpu_s"`
	// MemBytes is the peak resident memory of one job.
	MemBytes int64 `json:"est_mem_bytes"`
	// ThroughputTargetsPerMin is how many targets one job slot scans per
	// minute.
	ThroughputTargetsPerMin float64 `json:"throughput_targets_per_min"`
}

ToolEstimate is what one job of a tool costs on this sensor, as a heartbeat carries it under capacity.per_tool.<tool>.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL