hoststats

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 14, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package hoststats measures host resources over one measurement run and records the identity of the host the agent is actually looking at.

Existing collectors report per-process CPU and RSS, which answers "what did my process do" but never "did the machine have memory, disk or IO left". This package fills that gap from Linux procfs, sysfs and cgroup v2, and it deliberately reports where it looked: inside a container the very same files describe a namespace rather than the machine, so a number without its identity and cgroup scope is unreadable.

It implements runctl.BaselineCollector. Both boundaries return an immutable handle carrying a deep-copied Sample, and Collect derives the interval from those two frozen samples alone — it performs no I/O whatsoever, so a snapshot can never drift while it is being built.

Everything here is fail-open: hoststats is an optional collector, so a missing file, an unsupported kernel or an exhausted budget degrades one source at a time and never fails the run or the measured application.

Index

Constants

View Source
const (
	// CollectorName is the runctl registration name and snapshot section key.
	CollectorName = "hoststats"

	// EnvEnable turns host collection off when set to "off". Default on.
	EnvEnable = "ISUTOOLS_HOSTSTATS"
	// EnvRole labels this agent ("app", "db", "dns", "proxy", ...). Free text:
	// multi-host aggregation displays it, nothing branches on it.
	EnvRole = "ISUTOOLS_ROLE"
	// EnvCGroupScope set to "host" declares that this agent lives in the
	// initial cgroup namespace. It is never inferred: inside a cgroup
	// namespace both /proc/self/cgroup and mountinfo are virtualised, so no
	// in-process check can tell the two cases apart.
	EnvCGroupScope = "ISUTOOLS_CGROUP_SCOPE"
	// EnvCGroupPath names the cgroup to read, relative to the cgroup2 mount
	// root. It exists because the agent and the measured service (mysqld, for
	// example) often live in different cgroups.
	EnvCGroupPath = "ISUTOOLS_CGROUP_PATH"
)

Collector name and the environment variables this package reads. The flag switches the feature on and off; the two cgroup variables are configuration for a feature that is already on.

View Source
const (
	// ScopeConfigured is the cgroup named by EnvCGroupPath.
	ScopeConfigured = "configured-cgroup"
	// ScopeHost is an operator's explicit declaration that the visible cgroup
	// tree is the host's. Only EnvCGroupScope produces it.
	ScopeHost = "host"
	// ScopeVisibleRoot is the root of whatever cgroup tree is visible. It is
	// the default because it is the one thing that is always true.
	ScopeVisibleRoot = "visible-root"
	// ScopeAgentCGroup is the agent's own cgroup, which may well be a
	// different cgroup from the service under measurement.
	ScopeAgentCGroup = "agent-cgroup"
)

cgroup.scope values. Which cgroup was read is as load-bearing as the limits themselves, so the scope travels with them everywhere.

View Source
const (
	// CodeNotCapturedPrefix marks a source that was skipped, with the source
	// name appended ("not-captured:psi").
	CodeNotCapturedPrefix = "not-captured:"
	// CodeCounterRewindPrefix marks a counter that went backwards, with the
	// device name or "vmstat" appended.
	CodeCounterRewindPrefix = "counter-rewind:"
	// CodeCounterRewind is the per-metric form used in Disk.Code.
	CodeCounterRewind = "counter-rewind"
	// CodeLimitChanged reports a cgroup limit that changed mid-run.
	CodeLimitChanged = "limit-changed"
	// CodeBootIDChanged reports a reboot inside the interval.
	CodeBootIDChanged = "boot-id-changed"
	// CodeMachineIDChanged reports that the samples came from two machines.
	CodeMachineIDChanged = "machine-id-changed"
)

Stable Section and Disk codes. The set is closed: transports and templates switch on these strings.

View Source
const (
	SourceVMStat    = "vmstat"
	SourceDiskstats = "diskstats"
	SourcePSI       = "psi"
	SourceStatfs    = "statfs"
	SourceCGroup    = "cgroup"
)

Source names used in "not-captured:<source>" codes.

View Source
const (
	// DiskUtilNote warns that util% is not saturation on multi-queue devices:
	// an NVMe drive at 100% "io time" can still be far from its limit.
	DiskUtilNote = "" /* 157-byte string literal not displayed */
	// CGroupScopeNote warns that the agent's limits may not be the measured
	// service's limits.
	CGroupScopeNote = "" /* 180-byte string literal not displayed */
)

Fixed display notes. They are constants rather than template text because both of them exist to prevent a specific misreading, and a misreading prevented only in one of two renderers is not prevented at all.

View Source
const (
	// HealthSourceSkipped reports sources that could not be read.
	HealthSourceSkipped = "hoststats-source-skipped"
	// HealthCGroupPathRejected reports a rejected ISUTOOLS_CGROUP_PATH. It is
	// separate from a plain skip because it means an operator's explicit
	// configuration did not take effect.
	HealthCGroupPathRejected = "hoststats-cgroup-path-rejected"
	// HealthCGroupV1 reports a host without cgroup v2.
	HealthCGroupV1 = "hoststats-cgroup-v1"
	// HealthCounterRewind reports counters that moved backwards.
	HealthCounterRewind = "hoststats-counter-rewind"
	// HealthHostChanged reports that the two boundaries saw different hosts or
	// different boots, which voids every interval delta.
	HealthHostChanged = "hoststats-host-changed"
)

Health keys this package reports. They live in their own namespace, separate from runctl's, and the set is fixed at five: a health view that grows a key per condition stops being readable, so a new condition reuses a key with a different message.

View Source
const CodeStatfsWedged = "statfs-wedged"

CodeStatfsWedged marks the filesystem source as permanently given up on: the breaker below tripped, and no further statfs will be issued in this process.

It is deliberately distinct from the ordinary "not-captured:statfs" skip. The two call for different readings — one boundary that ran out of budget is bad luck, a mount that never answers again is a broken host — and a reader who cannot tell them apart will keep waiting for numbers that are never coming back.

Variables

View Source
var (
	// ErrUnsupportedOS reports that there is no procfs to read: a non-Linux
	// host, or a Linux host where /proc is not mounted. New returns it so the
	// caller can skip registration entirely instead of registering a collector
	// that would fail every boundary.
	ErrUnsupportedOS = errors.New("hoststats: unsupported OS or missing procfs")

	// ErrNoSource reports that even the required source (/proc/meminfo) could
	// not be read, so no sample exists. Every other source is optional and
	// degrades to a "not-captured:<source>" code instead.
	ErrNoSource = errors.New("hoststats: no source readable")
)

Sentinel errors. Callers match them with errors.Is; wrap them with %w rather than replacing them, so that "this host cannot be measured at all" stays distinguishable from "this particular read failed".

View Source
var ErrStatfsWedged = errors.New("statfs: the mount never answered within budget; " + CodeStatfsWedged)

ErrStatfsWedged is the error every statfs attempt reports once the breaker has tripped. It is a sentinel so callers classify with errors.Is rather than by matching message text.

Functions

func Enabled

func Enabled(getenv func(key string) string) bool

Enabled reports whether the ISUTOOLS_HOSTSTATS flag leaves host collection on. It defaults to on because the reads are a handful of small files at two boundaries; the flag exists so an operator can prove that by turning it off. A nil getenv reads the process environment.

func HealthKeys

func HealthKeys() []string

HealthKeys returns every health key this package can report, in the order notes are emitted. Tests pin the list so the set cannot grow unnoticed.

Types

type CGroup

type CGroup struct {
	Scope                 string   `json:"scope"`
	Path                  string   `json:"path"`
	CPUMaxCores           *float64 `json:"cpu_max_cores,omitempty"`
	MemoryMaxBytes        *uint64  `json:"memory_max_bytes,omitempty"`
	MemoryCurrentBaseline uint64   `json:"memory_current_baseline_bytes"`
	MemoryCurrentFinal    uint64   `json:"memory_current_final_bytes"`
	// Code is CodeLimitChanged when a limit moved during the interval.
	Code string `json:"code,omitempty"`
}

CGroup is the cgroup v2 limits and usage, always accompanied by the scope that says which cgroup they belong to.

type CGroupRaw

type CGroupRaw struct {
	Scope              string
	Path               string
	CPUMaxCores        *float64 // nil means "max", i.e. no quota
	MemoryMaxBytes     *uint64  // nil means "max", i.e. no limit
	MemoryCurrentBytes uint64
}

CGroupRaw is one boundary's cgroup v2 reading, together with the scope that says which cgroup it describes.

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector implements runctl.BaselineCollector for host resources.

It keeps its samples keyed by (runID, epoch, phase) so that a retried boundary returns the first answer verbatim, timestamp included: a retry that re-samples would silently move the boundary the whole run is measured against.

func New

func New(o Options) (*Collector, error)

New builds a Collector. It returns ErrUnsupportedOS when there is no procfs to read, so the caller can decline to register the collector at all rather than registering one that fails every boundary and drags the run to partial.

func (*Collector) CaptureBaseline

func (c *Collector) CaptureBaseline(ctx context.Context, runID string, ep runctl.Epoch) (runctl.SampleResult, error)

CaptureBaseline samples the opening boundary.

func (*Collector) CaptureFinal

func (c *Collector) CaptureFinal(ctx context.Context, runID string, ep runctl.Epoch) (runctl.SampleResult, error)

CaptureFinal samples the closing boundary.

func (*Collector) Collect

func (c *Collector) Collect(base, final runctl.BaselineHandle) (any, error)

Collect derives the interval from two frozen samples.

It reads nothing but the two handles: no procfs, no sysfs, no syscalls, and none of the collector's own fields. That is what makes a snapshot immutable in practice and not just in intent — a Collect that peeked at live state would report the moment the report was built, not the run.

It also never returns an error for a data problem. hoststats is optional, so an error here drops the entire host section from the snapshot, taking the memory, cgroup and identity point observations down with the one counter that misbehaved. Degradation is reported per metric instead.

func (*Collector) Identity added in v1.5.0

func (c *Collector) Identity() Identity

Identity returns the current credential-free host identity used by the multi-host handshake. Each source degrades independently to an empty field.

func (*Collector) Name

func (c *Collector) Name() string

Name is the registration name and the snapshot section key.

func (*Collector) Point added in v1.4.0

func (c *Collector) Point(ctx context.Context) (TimelinePoint, error)

Point reads only diskstats for the optional high-frequency timeline. It does not invoke statfs, PSI, identity or cgroup reads from the boundary collector.

func (*Collector) Release

func (c *Collector) Release(h runctl.BaselineHandle)

Release drops the sample a handle pins. It is idempotent, and it is safe to call before Collect: the handle carries its own deep copy, so releasing the collector's bookkeeping cannot take data away from an interval.

type Disk

type Disk struct {
	Device        string   `json:"device"`
	ReadBytes     uint64   `json:"read_bytes"`
	WriteBytes    uint64   `json:"write_bytes"`
	ReadMBPerSec  *float64 `json:"read_mb_per_s,omitempty"`
	WriteMBPerSec *float64 `json:"write_mb_per_s,omitempty"`
	IOTimeMillis  uint64   `json:"io_time_ms"`
	// UtilPercent is the share of the interval the device spent doing IO. See
	// DiskUtilNote: on multi-queue devices it is not saturation.
	UtilPercent *float64 `json:"util_percent,omitempty"`
	// QueueAvg is the average queue depth, from the kernel's weighted IO time.
	QueueAvg *float64 `json:"queue_avg,omitempty"`
	// Appeared marks a device absent at the baseline, so it has no interval.
	Appeared bool `json:"appeared,omitempty"`
	// Code is CodeCounterRewind when the device's counters went backwards.
	Code string `json:"code,omitempty"`
}

Disk is one block device's interval. Rate fields are pointers because "the interval was too short to divide by" and "the rate was zero" are different facts, and only nil says the first one.

type DiskRaw

type DiskRaw struct {
	ReadSectors  uint64 // field 3
	WriteSectors uint64 // field 7
	IOTicksMS    uint64 // field 10: milliseconds spent doing IO
	WeightedMS   uint64 // field 11: weighted IO milliseconds, i.e. queue integral
}

DiskRaw is one device's cumulative counters. Field numbers refer to the kernel's own numbering in Documentation/admin-guide/iostats.rst, counted after the device name.

type FSRaw

type FSRaw struct {
	TotalBytes uint64
	AvailBytes uint64
}

FSRaw is one filesystem's size at one boundary.

type FSUsage

type FSUsage struct {
	Path          string `json:"path"`
	TotalBytes    uint64 `json:"total_bytes"`
	AvailBaseline uint64 `json:"avail_baseline_bytes"`
	AvailFinal    uint64 `json:"avail_final_bytes"`
}

FSUsage is one filesystem at both boundaries, so growth during the run is visible rather than inferred.

type HealthNote

type HealthNote struct {
	Key     string
	Message string
}

HealthNote is one health observation derived from a Section. Callers copy it into whatever health registry they own; this package keeps no global state.

type Identity

type Identity struct {
	Hostname string `json:"hostname"`
	// MachineIDHash is sha256(machine-id) truncated to 16 hex characters. The
	// raw id identifies the host to anyone who reads a snapshot, and a
	// truncated hash is enough to tell two hosts apart.
	MachineIDHash string `json:"machine_id_hash"`
	// BootIDHash changes on reboot, which is how a mid-run reboot is caught.
	BootIDHash string `json:"boot_id_hash"`
	PIDNS      string `json:"pid_ns"`
	NetNS      string `json:"net_ns"`
	MntNS      string `json:"mnt_ns"`
	// CgroupNS says which namespace the cgroup values were read from. It is
	// always displayed next to cgroup.scope: scope says which cgroup, this
	// says from where, and neither alone fixes what was measured.
	CgroupNS     string `json:"cgroup_ns"`
	Role         string `json:"role,omitempty"`
	AgentVersion string `json:"agent_version"`
}

Identity is who and where this agent is. The multi-host hub deduplicates peers on these fields, and the namespace ids are the only evidence of what an agent inside a container can actually see.

type Interval

type Interval struct {
	BaselineAt time.Time `json:"baseline_at"`
	FinalAt    time.Time `json:"final_at"`
	Seconds    float64   `json:"seconds"`
}

Interval is the measured span between the two boundaries. Rates use these two timestamps rather than the run's boundary window, so another collector being slow cannot distort this host's throughput numbers.

type MemRaw

type MemRaw struct {
	TotalBytes     uint64
	AvailableBytes uint64
	CachedBytes    uint64
	DirtyBytes     uint64
	SwapTotalBytes uint64
	SwapFreeBytes  uint64
}

MemRaw is one boundary's /proc/meminfo, already converted from kB to bytes.

type Memory

type Memory struct {
	TotalBytes        uint64 `json:"total_bytes"`
	AvailableBaseline uint64 `json:"available_baseline_bytes"`
	AvailableFinal    uint64 `json:"available_final_bytes"`
	CachedBaseline    uint64 `json:"cached_baseline_bytes"`
	CachedFinal       uint64 `json:"cached_final_bytes"`
	DirtyBaseline     uint64 `json:"dirty_baseline_bytes"`
	DirtyFinal        uint64 `json:"dirty_final_bytes"`
	SwapTotalBytes    uint64 `json:"swap_total_bytes"`
	SwapFreeBaseline  uint64 `json:"swap_free_baseline_bytes"`
	SwapFreeFinal     uint64 `json:"swap_free_final_bytes"`
	// PageMajorFaults is the interval delta of pgmajfault: page faults that
	// had to reach the disk, which is what memory pressure feels like.
	PageMajorFaults uint64 `json:"page_major_faults"`
}

Memory is the interval's memory picture, in bytes.

type Options

type Options struct {
	// ProcFS is /proc. Also the required source: New fails without it.
	ProcFS fs.FS
	// SysFS is /sys, used to tell whole devices from partitions.
	SysFS fs.FS
	// EtcFS is /etc, the source of machine-id.
	EtcFS fs.FS
	// CGroupFS is the cgroup2 mount. Defaults to the mount resolved from
	// /proc/self/mountinfo.
	CGroupFS fs.FS
	// CGroupRoot is that mount's absolute path, needed to check that a
	// configured cgroup path does not escape the mount through a symlink.
	CGroupRoot string

	// Statfs, Readlink and EvalSymlinks are syscalls that fs.FS cannot express,
	// so they are injected as functions instead.
	Statfs       func(path string) (FSRaw, error)
	Readlink     func(name string) (string, error)
	EvalSymlinks func(name string) (string, error)
	// Hostname resolves this host's name.
	Hostname func() (string, error)
	// Getenv reads configuration. Injected so tests need no process-global
	// environment.
	Getenv func(key string) string

	// DataDir is the second statfs target, typically the database data
	// directory: the root filesystem filling up and the data volume filling up
	// are different incidents.
	DataDir string
	// Now supplies boundary timestamps.
	Now func() time.Time
}

Options configures a Collector. Every source is a seam, for two reasons: this package parses kernel text formats that must be testable on a developer's macOS laptop, and procfs and sysfs are separate trees — a collector rooted at /proc structurally cannot read /sys/block, which is exactly the bug that made the old process collector unable to tell a disk from a partition.

A nil or empty field falls back to the OS implementation, so the zero value is the production configuration.

type PSI

type PSI struct {
	CPU    PSIResource `json:"cpu"`
	Memory PSIResource `json:"memory"`
	IO     PSIResource `json:"io"`
}

PSI is pressure stall information for the three resources the kernel exposes.

type PSIRaw

type PSIRaw struct {
	SomeAvg10   float64
	SomeAvg60   float64
	SomeTotalUS uint64
	FullAvg10   float64
	FullAvg60   float64
	FullTotalUS uint64
	// HasFull records whether a full line existed. Kernels before 5.13 have
	// none for cpu, and "absent" must not be rendered as "zero pressure".
	HasFull bool
}

PSIRaw is one pressure file: the kernel's own decaying averages plus the cumulative stall total we turn into an interval ratio.

type PSIResource

type PSIResource struct {
	SomeAvg10      float64  `json:"some_avg10"`
	SomeAvg60      float64  `json:"some_avg60"`
	SomeStallRatio *float64 `json:"some_stall_ratio,omitempty"`
	FullAvg10      float64  `json:"full_avg10,omitempty"`
	FullAvg60      float64  `json:"full_avg60,omitempty"`
	FullStallRatio *float64 `json:"full_stall_ratio,omitempty"`
}

PSIResource is one resource's pressure. The averages are the kernel's own decaying windows read at the closing boundary; the stall ratio is ours, over exactly this run's interval.

type Sample

type Sample struct {
	// Phase is the boundary this sample belongs to.
	Phase runctl.Phase
	// At is the boundary timestamp, taken immediately before the required
	// source so the boundary is one instant rather than the span of all reads.
	At time.Time
	// Identity is captured at both boundaries because a reboot or a swapped
	// host mid-run invalidates every cumulative counter below.
	Identity Identity
	// Mem holds /proc/meminfo converted to bytes.
	Mem MemRaw
	// MajFault is the cumulative pgmajfault counter from /proc/vmstat.
	MajFault uint64
	// HasMajFault distinguishes "zero major faults" from "vmstat unreadable".
	HasMajFault bool
	// Disks holds cumulative /proc/diskstats counters keyed by device.
	Disks map[string]DiskRaw
	// PSI holds pressure counters keyed by "cpu", "memory" and "io".
	PSI map[string]PSIRaw
	// FS holds statfs results keyed by mount path.
	FS map[string]FSRaw
	// CGroup is nil when cgroup reading was skipped.
	CGroup *CGroupRaw
	// CGroupSkip explains a nil CGroup for health reporting: "v1", "no-mount"
	// or "path-rejected:<code>".
	CGroupSkip string
	// Codes lists per-source skips as "not-captured:<source>".
	Codes []string
}

Sample is one boundary's raw observation. It is built once, deep-copied into the handle that leaves the collector, and never written again: Collect must be able to derive an interval from fixed values alone, which is only true if nothing can edit a sample after its timestamp was taken.

type Section

type Section struct {
	Identity    Identity  `json:"identity"`
	Interval    Interval  `json:"interval"`
	Memory      Memory    `json:"memory"`
	Disks       []Disk    `json:"disks,omitempty"`       // sorted by Device
	PSI         *PSI      `json:"psi,omitempty"`         // nil on kernels without PSI
	Filesystems []FSUsage `json:"filesystems,omitempty"` // sorted by Path
	CGroup      *CGroup   `json:"cgroup,omitempty"`      // nil when skipped
	// Partial marks a section that is usable but incomplete. It never fails
	// the run: hoststats is optional, so its worst outcome is a partial run.
	Partial bool `json:"partial,omitempty"`
	// Codes lists this package's stable codes, sorted. Empty means clean.
	Codes []string `json:"codes,omitempty"`
	// contains filtered or unexported fields
}

Section is the interval value Collect derives from two frozen samples. It is what lands in the run snapshot under the "hoststats" key.

Point observations (memory, filesystem usage, cgroup limits) are reported at both boundaries rather than as a delta: "available memory fell by 2 GB" and "available memory is 200 MB" are different findings, and only the pair carries both.

func (*Section) HealthNotes

func (s *Section) HealthNotes() []HealthNote

HealthNotes returns the health observations for this section, in the fixed order of HealthKeys. The caller receives a copy.

type TimelinePoint added in v1.4.0

type TimelinePoint struct {
	ReadBytes  uint64
	WriteBytes uint64
	IOTicks    time.Duration
	WeightedIO time.Duration
}

TimelinePoint is a cumulative, whole-device disk reading for bucket deltas.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL