Documentation
¶
Overview ¶
Package hoststats measures host resources over one measurement run and records the identity of the host the agent is actually looking at.
Existing collectors report per-process CPU and RSS, which answers "what did my process do" but never "did the machine have memory, disk or IO left". This package fills that gap from Linux procfs, sysfs and cgroup v2, and it deliberately reports where it looked: inside a container the very same files describe a namespace rather than the machine, so a number without its identity and cgroup scope is unreadable.
It implements runctl.BaselineCollector. Both boundaries return an immutable handle carrying a deep-copied Sample, and Collect derives the interval from those two frozen samples alone — it performs no I/O whatsoever, so a snapshot can never drift while it is being built.
Everything here is fail-open: hoststats is an optional collector, so a missing file, an unsupported kernel or an exhausted budget degrades one source at a time and never fails the run or the measured application.
Index ¶
- Constants
- Variables
- func Enabled(getenv func(key string) string) bool
- func HealthKeys() []string
- type CGroup
- type CGroupRaw
- type Collector
- func (c *Collector) CaptureBaseline(ctx context.Context, runID string, ep runctl.Epoch) (runctl.SampleResult, error)
- func (c *Collector) CaptureFinal(ctx context.Context, runID string, ep runctl.Epoch) (runctl.SampleResult, error)
- func (c *Collector) Collect(base, final runctl.BaselineHandle) (any, error)
- func (c *Collector) Name() string
- func (c *Collector) Release(h runctl.BaselineHandle)
- type Disk
- type DiskRaw
- type FSRaw
- type FSUsage
- type HealthNote
- type Identity
- type Interval
- type MemRaw
- type Memory
- type Options
- type PSI
- type PSIRaw
- type PSIResource
- type Sample
- type Section
Constants ¶
const ( // CollectorName is the runctl registration name and snapshot section key. CollectorName = "hoststats" // EnvEnable turns host collection off when set to "off". Default on. EnvEnable = "ISUTOOLS_HOSTSTATS" // EnvRole labels this agent ("app", "db", "dns", "proxy", ...). Free text: // multi-host aggregation displays it, nothing branches on it. EnvRole = "ISUTOOLS_ROLE" // EnvCGroupScope set to "host" declares that this agent lives in the // initial cgroup namespace. It is never inferred: inside a cgroup // namespace both /proc/self/cgroup and mountinfo are virtualised, so no // in-process check can tell the two cases apart. EnvCGroupScope = "ISUTOOLS_CGROUP_SCOPE" // EnvCGroupPath names the cgroup to read, relative to the cgroup2 mount // root. It exists because the agent and the measured service (mysqld, for // example) often live in different cgroups. EnvCGroupPath = "ISUTOOLS_CGROUP_PATH" )
Collector name and the environment variables this package reads. The flag switches the feature on and off; the two cgroup variables are configuration for a feature that is already on.
const ( // ScopeConfigured is the cgroup named by EnvCGroupPath. ScopeConfigured = "configured-cgroup" // ScopeHost is an operator's explicit declaration that the visible cgroup // tree is the host's. Only EnvCGroupScope produces it. ScopeHost = "host" // ScopeVisibleRoot is the root of whatever cgroup tree is visible. It is // the default because it is the one thing that is always true. ScopeVisibleRoot = "visible-root" // ScopeAgentCGroup is the agent's own cgroup, which may well be a // different cgroup from the service under measurement. ScopeAgentCGroup = "agent-cgroup" )
cgroup.scope values. Which cgroup was read is as load-bearing as the limits themselves, so the scope travels with them everywhere.
const ( // CodeNotCapturedPrefix marks a source that was skipped, with the source // name appended ("not-captured:psi"). CodeNotCapturedPrefix = "not-captured:" // CodeCounterRewindPrefix marks a counter that went backwards, with the // device name or "vmstat" appended. CodeCounterRewindPrefix = "counter-rewind:" // CodeCounterRewind is the per-metric form used in Disk.Code. CodeCounterRewind = "counter-rewind" // CodeLimitChanged reports a cgroup limit that changed mid-run. CodeLimitChanged = "limit-changed" // CodeBootIDChanged reports a reboot inside the interval. CodeBootIDChanged = "boot-id-changed" // CodeMachineIDChanged reports that the samples came from two machines. CodeMachineIDChanged = "machine-id-changed" )
Stable Section and Disk codes. The set is closed: transports and templates switch on these strings.
const ( SourceVMStat = "vmstat" SourceDiskstats = "diskstats" SourcePSI = "psi" SourceStatfs = "statfs" SourceCGroup = "cgroup" )
Source names used in "not-captured:<source>" codes.
const ( // DiskUtilNote warns that util% is not saturation on multi-queue devices: // an NVMe drive at 100% "io time" can still be far from its limit. DiskUtilNote = "" /* 157-byte string literal not displayed */ // CGroupScopeNote warns that the agent's limits may not be the measured // service's limits. CGroupScopeNote = "" /* 180-byte string literal not displayed */ )
Fixed display notes. They are constants rather than template text because both of them exist to prevent a specific misreading, and a misreading prevented only in one of two renderers is not prevented at all.
const ( // HealthSourceSkipped reports sources that could not be read. HealthSourceSkipped = "hoststats-source-skipped" // HealthCGroupPathRejected reports a rejected ISUTOOLS_CGROUP_PATH. It is // separate from a plain skip because it means an operator's explicit // configuration did not take effect. HealthCGroupPathRejected = "hoststats-cgroup-path-rejected" // HealthCGroupV1 reports a host without cgroup v2. HealthCGroupV1 = "hoststats-cgroup-v1" // HealthCounterRewind reports counters that moved backwards. HealthCounterRewind = "hoststats-counter-rewind" // HealthHostChanged reports that the two boundaries saw different hosts or // different boots, which voids every interval delta. HealthHostChanged = "hoststats-host-changed" )
Health keys this package reports. They live in their own namespace, separate from runctl's, and the set is fixed at five: a health view that grows a key per condition stops being readable, so a new condition reuses a key with a different message.
const CodeStatfsWedged = "statfs-wedged"
CodeStatfsWedged marks the filesystem source as permanently given up on: the breaker below tripped, and no further statfs will be issued in this process.
It is deliberately distinct from the ordinary "not-captured:statfs" skip. The two call for different readings — one boundary that ran out of budget is bad luck, a mount that never answers again is a broken host — and a reader who cannot tell them apart will keep waiting for numbers that are never coming back.
Variables ¶
var ( // ErrUnsupportedOS reports that there is no procfs to read: a non-Linux // host, or a Linux host where /proc is not mounted. New returns it so the // caller can skip registration entirely instead of registering a collector // that would fail every boundary. ErrUnsupportedOS = errors.New("hoststats: unsupported OS or missing procfs") // ErrNoSource reports that even the required source (/proc/meminfo) could // not be read, so no sample exists. Every other source is optional and // degrades to a "not-captured:<source>" code instead. ErrNoSource = errors.New("hoststats: no source readable") )
Sentinel errors. Callers match them with errors.Is; wrap them with %w rather than replacing them, so that "this host cannot be measured at all" stays distinguishable from "this particular read failed".
var ErrStatfsWedged = errors.New("statfs: the mount never answered within budget; " + CodeStatfsWedged)
ErrStatfsWedged is the error every statfs attempt reports once the breaker has tripped. It is a sentinel so callers classify with errors.Is rather than by matching message text.
Functions ¶
func Enabled ¶
Enabled reports whether the ISUTOOLS_HOSTSTATS flag leaves host collection on. It defaults to on because the reads are a handful of small files at two boundaries; the flag exists so an operator can prove that by turning it off. A nil getenv reads the process environment.
func HealthKeys ¶
func HealthKeys() []string
HealthKeys returns every health key this package can report, in the order notes are emitted. Tests pin the list so the set cannot grow unnoticed.
Types ¶
type CGroup ¶
type CGroup struct {
Scope string `json:"scope"`
Path string `json:"path"`
CPUMaxCores *float64 `json:"cpu_max_cores,omitempty"`
MemoryMaxBytes *uint64 `json:"memory_max_bytes,omitempty"`
MemoryCurrentBaseline uint64 `json:"memory_current_baseline_bytes"`
MemoryCurrentFinal uint64 `json:"memory_current_final_bytes"`
// Code is CodeLimitChanged when a limit moved during the interval.
Code string `json:"code,omitempty"`
}
CGroup is the cgroup v2 limits and usage, always accompanied by the scope that says which cgroup they belong to.
type CGroupRaw ¶
type CGroupRaw struct {
Scope string
Path string
CPUMaxCores *float64 // nil means "max", i.e. no quota
MemoryMaxBytes *uint64 // nil means "max", i.e. no limit
MemoryCurrentBytes uint64
}
CGroupRaw is one boundary's cgroup v2 reading, together with the scope that says which cgroup it describes.
type Collector ¶
type Collector struct {
// contains filtered or unexported fields
}
Collector implements runctl.BaselineCollector for host resources.
It keeps its samples keyed by (runID, epoch, phase) so that a retried boundary returns the first answer verbatim, timestamp included: a retry that re-samples would silently move the boundary the whole run is measured against.
func New ¶
New builds a Collector. It returns ErrUnsupportedOS when there is no procfs to read, so the caller can decline to register the collector at all rather than registering one that fails every boundary and drags the run to partial.
func (*Collector) CaptureBaseline ¶
func (c *Collector) CaptureBaseline(ctx context.Context, runID string, ep runctl.Epoch) (runctl.SampleResult, error)
CaptureBaseline samples the opening boundary.
func (*Collector) CaptureFinal ¶
func (c *Collector) CaptureFinal(ctx context.Context, runID string, ep runctl.Epoch) (runctl.SampleResult, error)
CaptureFinal samples the closing boundary.
func (*Collector) Collect ¶
func (c *Collector) Collect(base, final runctl.BaselineHandle) (any, error)
Collect derives the interval from two frozen samples.
It reads nothing but the two handles: no procfs, no sysfs, no syscalls, and none of the collector's own fields. That is what makes a snapshot immutable in practice and not just in intent — a Collect that peeked at live state would report the moment the report was built, not the run.
It also never returns an error for a data problem. hoststats is optional, so an error here drops the entire host section from the snapshot, taking the memory, cgroup and identity point observations down with the one counter that misbehaved. Degradation is reported per metric instead.
func (*Collector) Release ¶
func (c *Collector) Release(h runctl.BaselineHandle)
Release drops the sample a handle pins. It is idempotent, and it is safe to call before Collect: the handle carries its own deep copy, so releasing the collector's bookkeeping cannot take data away from an interval.
type Disk ¶
type Disk struct {
Device string `json:"device"`
ReadBytes uint64 `json:"read_bytes"`
WriteBytes uint64 `json:"write_bytes"`
ReadMBPerSec *float64 `json:"read_mb_per_s,omitempty"`
WriteMBPerSec *float64 `json:"write_mb_per_s,omitempty"`
IOTimeMillis uint64 `json:"io_time_ms"`
// UtilPercent is the share of the interval the device spent doing IO. See
// DiskUtilNote: on multi-queue devices it is not saturation.
UtilPercent *float64 `json:"util_percent,omitempty"`
// QueueAvg is the average queue depth, from the kernel's weighted IO time.
QueueAvg *float64 `json:"queue_avg,omitempty"`
// Appeared marks a device absent at the baseline, so it has no interval.
Appeared bool `json:"appeared,omitempty"`
// Code is CodeCounterRewind when the device's counters went backwards.
Code string `json:"code,omitempty"`
}
Disk is one block device's interval. Rate fields are pointers because "the interval was too short to divide by" and "the rate was zero" are different facts, and only nil says the first one.
type DiskRaw ¶
type DiskRaw struct {
ReadSectors uint64 // field 3
WriteSectors uint64 // field 7
IOTicksMS uint64 // field 10: milliseconds spent doing IO
WeightedMS uint64 // field 11: weighted IO milliseconds, i.e. queue integral
}
DiskRaw is one device's cumulative counters. Field numbers refer to the kernel's own numbering in Documentation/admin-guide/iostats.rst, counted after the device name.
type FSUsage ¶
type FSUsage struct {
Path string `json:"path"`
TotalBytes uint64 `json:"total_bytes"`
AvailBaseline uint64 `json:"avail_baseline_bytes"`
AvailFinal uint64 `json:"avail_final_bytes"`
}
FSUsage is one filesystem at both boundaries, so growth during the run is visible rather than inferred.
type HealthNote ¶
HealthNote is one health observation derived from a Section. Callers copy it into whatever health registry they own; this package keeps no global state.
type Identity ¶
type Identity struct {
Hostname string `json:"hostname"`
// MachineIDHash is sha256(machine-id) truncated to 16 hex characters. The
// raw id identifies the host to anyone who reads a snapshot, and a
// truncated hash is enough to tell two hosts apart.
MachineIDHash string `json:"machine_id_hash"`
// BootIDHash changes on reboot, which is how a mid-run reboot is caught.
BootIDHash string `json:"boot_id_hash"`
PIDNS string `json:"pid_ns"`
NetNS string `json:"net_ns"`
MntNS string `json:"mnt_ns"`
// CgroupNS says which namespace the cgroup values were read from. It is
// always displayed next to cgroup.scope: scope says which cgroup, this
// says from where, and neither alone fixes what was measured.
CgroupNS string `json:"cgroup_ns"`
Role string `json:"role,omitempty"`
AgentVersion string `json:"agent_version"`
}
Identity is who and where this agent is. The multi-host hub deduplicates peers on these fields, and the namespace ids are the only evidence of what an agent inside a container can actually see.
type Interval ¶
type Interval struct {
BaselineAt time.Time `json:"baseline_at"`
FinalAt time.Time `json:"final_at"`
Seconds float64 `json:"seconds"`
}
Interval is the measured span between the two boundaries. Rates use these two timestamps rather than the run's boundary window, so another collector being slow cannot distort this host's throughput numbers.
type MemRaw ¶
type MemRaw struct {
TotalBytes uint64
AvailableBytes uint64
CachedBytes uint64
DirtyBytes uint64
SwapTotalBytes uint64
SwapFreeBytes uint64
}
MemRaw is one boundary's /proc/meminfo, already converted from kB to bytes.
type Memory ¶
type Memory struct {
TotalBytes uint64 `json:"total_bytes"`
AvailableBaseline uint64 `json:"available_baseline_bytes"`
AvailableFinal uint64 `json:"available_final_bytes"`
CachedBaseline uint64 `json:"cached_baseline_bytes"`
CachedFinal uint64 `json:"cached_final_bytes"`
DirtyBaseline uint64 `json:"dirty_baseline_bytes"`
DirtyFinal uint64 `json:"dirty_final_bytes"`
SwapTotalBytes uint64 `json:"swap_total_bytes"`
SwapFreeBaseline uint64 `json:"swap_free_baseline_bytes"`
SwapFreeFinal uint64 `json:"swap_free_final_bytes"`
// PageMajorFaults is the interval delta of pgmajfault: page faults that
// had to reach the disk, which is what memory pressure feels like.
PageMajorFaults uint64 `json:"page_major_faults"`
}
Memory is the interval's memory picture, in bytes.
type Options ¶
type Options struct {
// ProcFS is /proc. Also the required source: New fails without it.
ProcFS fs.FS
// SysFS is /sys, used to tell whole devices from partitions.
SysFS fs.FS
// EtcFS is /etc, the source of machine-id.
EtcFS fs.FS
// CGroupFS is the cgroup2 mount. Defaults to the mount resolved from
// /proc/self/mountinfo.
CGroupFS fs.FS
// CGroupRoot is that mount's absolute path, needed to check that a
// configured cgroup path does not escape the mount through a symlink.
CGroupRoot string
// Statfs, Readlink and EvalSymlinks are syscalls that fs.FS cannot express,
// so they are injected as functions instead.
Statfs func(path string) (FSRaw, error)
Readlink func(name string) (string, error)
EvalSymlinks func(name string) (string, error)
// Hostname resolves this host's name.
Hostname func() (string, error)
// Getenv reads configuration. Injected so tests need no process-global
// environment.
Getenv func(key string) string
// DataDir is the second statfs target, typically the database data
// directory: the root filesystem filling up and the data volume filling up
// are different incidents.
DataDir string
// Now supplies boundary timestamps.
Now func() time.Time
}
Options configures a Collector. Every source is a seam, for two reasons: this package parses kernel text formats that must be testable on a developer's macOS laptop, and procfs and sysfs are separate trees — a collector rooted at /proc structurally cannot read /sys/block, which is exactly the bug that made the old process collector unable to tell a disk from a partition.
A nil or empty field falls back to the OS implementation, so the zero value is the production configuration.
type PSI ¶
type PSI struct {
CPU PSIResource `json:"cpu"`
Memory PSIResource `json:"memory"`
IO PSIResource `json:"io"`
}
PSI is pressure stall information for the three resources the kernel exposes.
type PSIRaw ¶
type PSIRaw struct {
SomeAvg10 float64
SomeAvg60 float64
SomeTotalUS uint64
FullAvg10 float64
FullAvg60 float64
FullTotalUS uint64
// HasFull records whether a full line existed. Kernels before 5.13 have
// none for cpu, and "absent" must not be rendered as "zero pressure".
HasFull bool
}
PSIRaw is one pressure file: the kernel's own decaying averages plus the cumulative stall total we turn into an interval ratio.
type PSIResource ¶
type PSIResource struct {
SomeAvg10 float64 `json:"some_avg10"`
SomeAvg60 float64 `json:"some_avg60"`
SomeStallRatio *float64 `json:"some_stall_ratio,omitempty"`
FullAvg10 float64 `json:"full_avg10,omitempty"`
FullAvg60 float64 `json:"full_avg60,omitempty"`
FullStallRatio *float64 `json:"full_stall_ratio,omitempty"`
}
PSIResource is one resource's pressure. The averages are the kernel's own decaying windows read at the closing boundary; the stall ratio is ours, over exactly this run's interval.
type Sample ¶
type Sample struct {
// Phase is the boundary this sample belongs to.
Phase runctl.Phase
// At is the boundary timestamp, taken immediately before the required
// source so the boundary is one instant rather than the span of all reads.
At time.Time
// Identity is captured at both boundaries because a reboot or a swapped
// host mid-run invalidates every cumulative counter below.
Identity Identity
// Mem holds /proc/meminfo converted to bytes.
Mem MemRaw
// MajFault is the cumulative pgmajfault counter from /proc/vmstat.
MajFault uint64
// HasMajFault distinguishes "zero major faults" from "vmstat unreadable".
HasMajFault bool
// Disks holds cumulative /proc/diskstats counters keyed by device.
Disks map[string]DiskRaw
// PSI holds pressure counters keyed by "cpu", "memory" and "io".
PSI map[string]PSIRaw
// FS holds statfs results keyed by mount path.
FS map[string]FSRaw
// CGroup is nil when cgroup reading was skipped.
CGroup *CGroupRaw
// CGroupSkip explains a nil CGroup for health reporting: "v1", "no-mount"
// or "path-rejected:<code>".
CGroupSkip string
// Codes lists per-source skips as "not-captured:<source>".
Codes []string
}
Sample is one boundary's raw observation. It is built once, deep-copied into the handle that leaves the collector, and never written again: Collect must be able to derive an interval from fixed values alone, which is only true if nothing can edit a sample after its timestamp was taken.
type Section ¶
type Section struct {
Identity Identity `json:"identity"`
Interval Interval `json:"interval"`
Memory Memory `json:"memory"`
Disks []Disk `json:"disks,omitempty"` // sorted by Device
PSI *PSI `json:"psi,omitempty"` // nil on kernels without PSI
Filesystems []FSUsage `json:"filesystems,omitempty"` // sorted by Path
CGroup *CGroup `json:"cgroup,omitempty"` // nil when skipped
// Partial marks a section that is usable but incomplete. It never fails
// the run: hoststats is optional, so its worst outcome is a partial run.
Partial bool `json:"partial,omitempty"`
// Codes lists this package's stable codes, sorted. Empty means clean.
Codes []string `json:"codes,omitempty"`
// contains filtered or unexported fields
}
Section is the interval value Collect derives from two frozen samples. It is what lands in the run snapshot under the "hoststats" key.
Point observations (memory, filesystem usage, cgroup limits) are reported at both boundaries rather than as a delta: "available memory fell by 2 GB" and "available memory is 200 MB" are different findings, and only the pair carries both.
func (*Section) HealthNotes ¶
func (s *Section) HealthNotes() []HealthNote
HealthNotes returns the health observations for this section, in the fixed order of HealthKeys. The caller receives a copy.