os

package
v0.13.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Overview

Package os queries the host OS information (e.g., kernel version).

Index

Constants

View Source
const (
	// DefaultMaxAllocatedFileHandles is some high number, in case the system is under high file descriptor usage.
	DefaultMaxAllocatedFileHandles = 10000000

	// DefaultMaxRunningPIDs is some high number, in case fd-max is unlimited
	DefaultMaxRunningPIDs = 900000
)
View Source
const (
	// EventNameBlockedProcessesPersistent is emitted on the rising edge of a
	// persistent blocked-process (Linux D-state) episode (LEP-6029).
	EventNameBlockedProcessesPersistent = "blocked_processes_persistent"
	// EventNameBlockedProcessesRecovered is emitted when a persistent
	// blocked-process episode clears.
	EventNameBlockedProcessesRecovered = "blocked_processes_recovered"
)
View Source
const (
	// DefaultBlockedProcessPersistenceThreshold is the number of consecutive
	// checks a process must remain blocked (Linux D-state) before it is
	// considered persistent. The OS component checks once per minute: five
	// consecutive checks means the process has been blocked for at least four
	// minutes of wall time (persistenceThreshold check intervals).
	//
	// Why five: transient D-state is normal -- most blocking disk I/O
	// (reads, writes, page faults hitting storage) passes through D for
	// milliseconds to seconds -- so a single observation must never flag a
	// node. The 2026-08-12 validation found zero D-state processes on healthy
	// production GPU nodes (instant scans plus a 30x10s sampling window),
	// while an induced real D-state process (dd on a suspended device-mapper
	// device) persisted >=6 minutes. Five one-minute checks sit between the
	// two: long enough to ignore ordinary I/O waits, short enough to surface
	// a wedged kernel path within minutes after it stalls.
	DefaultBlockedProcessPersistenceThreshold = 5
)
View Source
const (

	// DefaultBlockedProcessRebootThreshold is the number of reboot events (with
	// no recovery in between) after which a persistent blocked-process
	// condition escalates from reboot suggestion to hardware inspection.
	// Kept at parity with the XID component's DefaultRebootThreshold
	// (components/accelerator/nvidia/xid).
	DefaultBlockedProcessRebootThreshold = 2
)
View Source
const Name = "os"

Name is the ID of the OS component.

Variables

This section is empty.

Functions

func DefaultBlockedProcessNameRegexes added in v0.13.0

func DefaultBlockedProcessNameRegexes() []string

DefaultBlockedProcessNameRegexes returns the default process-name regexes used to escalate persistent D-state processes on machines with NVIDIA GPUs.

func HasVFSFileMaxLimitReached added in v0.5.0

func HasVFSFileMaxLimitReached(line string) bool

HasVFSFileMaxLimitReached returns true if the line indicates that the file-max limit has been reached. ref. https://docs.kernel.org/admin-guide/sysctl/fs.html#file-max-file-nr

func Match added in v0.5.0

func Match(line string) (eventName string, message string)

Match matches OS-related kernel log lines to event names and messages.

func New

func New(gpudInstance *components.GPUdInstance) (components.Component, error)

New creates the OS component.

func SetDefaultBlockedProcessThresholds added in v0.13.0

func SetDefaultBlockedProcessThresholds(th BlockedProcessThresholds) error

SetDefaultBlockedProcessThresholds validates, compiles, and updates the default thresholds.

func SetStartupBlockedProcessThresholds added in v0.13.0

func SetStartupBlockedProcessThresholds(th BlockedProcessThresholds)

SetStartupBlockedProcessThresholds records the startup-resolved thresholds as the baseline that session updateConfig falls back to when the control-plane config omits the os component. Called once from the gpud run/scan command wiring after flag resolution.

Types

type BlockedProcess added in v0.13.0

type BlockedProcess struct {
	// PID is the process ID.
	PID int32 `json:"pid"`
	// Name is the process name (Linux comm, truncated to 15 chars by the kernel).
	Name string `json:"name"`
	// FirstSeenUnixSeconds is when the process was first observed blocked.
	FirstSeenUnixSeconds int64 `json:"first_seen_unix_seconds"`
	// LastSeenUnixSeconds is when the process was most recently observed blocked.
	LastSeenUnixSeconds int64 `json:"last_seen_unix_seconds"`
	// BlockedSeconds is the observed blocked duration (last seen - first seen).
	BlockedSeconds int64 `json:"blocked_seconds"`
	// ConsecutiveChecks is the number of consecutive checks the process was blocked.
	ConsecutiveChecks int `json:"consecutive_checks"`
}

BlockedProcess describes one process observed in uninterruptible sleep.

type BlockedProcessThresholds added in v0.13.0

type BlockedProcessThresholds struct {
	// PersistenceThreshold is the number of consecutive checks a process must
	// remain blocked before it is considered persistent. Values <= 0 reset to
	// DefaultBlockedProcessPersistenceThreshold.
	PersistenceThreshold int `json:"persistence_threshold"`

	// NameRegexes gates which persistent D-state processes escalate to
	// unhealthy with a repair suggestion. An empty set disables D-state
	// process checking entirely.
	//
	// The gate exists because D-state alone does not identify the cause: a
	// dd stuck on a suspended device and an nvidia-smi stuck in a wedged
	// driver show the same D state letter in /proc/<pid>/stat (both become
	// process.Blocked at the level gpud reads). A persistent D-state process from
	// any name degrades the component (something in the kernel is stuck and
	// SIGKILL cannot clear it), but a reboot suggestion is only attached when
	// the name matches -- i.e., when the stuck process implicates the GPU
	// driver path, as in the LEP-6029 incident (nvidia-smi in D during a
	// kubelet package upgrade).
	NameRegexes []string `json:"name_regexes"`
	// contains filtered or unexported fields
}

BlockedProcessThresholds configures the D-state (blocked) process tracking of the os component (LEP-6029), following the same default-override pattern as components/accelerator/nvidia/gpu-counts/threshold.go.

func GetDefaultBlockedProcessThresholds added in v0.13.0

func GetDefaultBlockedProcessThresholds() BlockedProcessThresholds

GetDefaultBlockedProcessThresholds returns the configured default thresholds.

func GetStartupBlockedProcessThresholds added in v0.13.0

func GetStartupBlockedProcessThresholds() BlockedProcessThresholds

GetStartupBlockedProcessThresholds returns the thresholds resolved at process startup. If none were recorded (e.g., tests), it returns the current defaults so the updateConfig fallback leaves them unchanged.

func (BlockedProcessThresholds) IsZero added in v0.13.0

func (t BlockedProcessThresholds) IsZero() bool

IsZero returns true when no process-name regexes are configured, meaning D-state process checking is disabled.

func (BlockedProcessThresholds) MatchesName added in v0.13.0

func (t BlockedProcessThresholds) MatchesName(name string) bool

MatchesName returns true if the process name matches any configured regex.

type BlockedProcesses added in v0.13.0

type BlockedProcesses struct {
	// CurrentCount is the number of processes blocked at this check.
	CurrentCount int `json:"current_count"`
	// PersistentCount is the number of processes that met the persistence
	// threshold (may exceed len(Persistent) when the output is capped).
	PersistentCount int `json:"persistent_count"`
	// Persistent is the bounded list of persistent blocked processes.
	Persistent []BlockedProcess `json:"persistent,omitempty"`
}

BlockedProcesses summarizes the blocked-process state of one check.

type FileDescriptors added in v0.5.0

type FileDescriptors struct {
	// The number of file descriptors currently allocated on the host (not per process).
	AllocatedFileHandles uint64 `json:"allocated_file_handles"`
	// The number of running PIDs returned by https://pkg.go.dev/github.com/shirou/gopsutil/v4/process#Pids.
	RunningPIDs uint64 `json:"running_pids"`
	Usage       uint64 `json:"usage"`
	Limit       uint64 `json:"limit"`

	// AllocatedFileHandlesPercent is the percentage of file descriptors that are currently allocated,
	// based on the current file descriptor limit and the current number of file descriptors allocated on the host (not per process).
	AllocatedFileHandlesPercent string `json:"allocated_file_handles_percent"`
	// UsedPercent is the percentage of file descriptors that are currently in use,
	// based on the current file descriptor limit on the host (not per process).
	UsedPercent string `json:"used_percent"`

	ThresholdAllocatedFileHandles        uint64 `json:"threshold_allocated_file_handles"`
	ThresholdAllocatedFileHandlesPercent string `json:"threshold_allocated_file_handles_percent"`

	ThresholdRunningPIDs        uint64 `json:"threshold_running_pids"`
	ThresholdRunningPIDsPercent string `json:"threshold_running_pids_percent"`

	// Set to true if the file handles are supported.
	FileHandlesSupported bool `json:"file_handles_supported"`
	// Set to true if the file descriptor limit is supported.
	FDLimitSupported bool `json:"fd_limit_supported"`
}

FileDescriptors represents the file descriptors of the host.

type Host

type Host struct {
	ID string `json:"id"`
}

Host contains host-level identity fields.

type Kernel

type Kernel struct {
	Arch    string `json:"arch"`
	Version string `json:"version"`
}

Kernel contains kernel architecture and version information.

type KernelPanicInstance added in v0.6.0

type KernelPanicInstance struct {
	// Process ID from the CPU line
	PID int
	// CPU number where the panic occurred
	CPU int
	// Process name (e.g., "khungtaskd")
	ProcessName string
}

KernelPanicInstance contains information related to a kernel panic event

func (*KernelPanicInstance) Summary added in v0.6.0

func (k *KernelPanicInstance) Summary() string

Summary returns a human-readable summary of the kernel panic.

type MachineMetadata added in v0.3.5

type MachineMetadata struct {
	BootID        string `json:"boot_id"`
	DmidecodeUUID string `json:"dmidecode_uuid"`
	OSMachineID   string `json:"os_machine_id"`
}

MachineMetadata contains stable host identity values gathered from the OS.

type Platform

type Platform struct {
	Name    string `json:"name"`
	Family  string `json:"family"`
	Version string `json:"version"`
}

Platform contains OS platform metadata.

type Uptimes

type Uptimes struct {
	Seconds             uint64 `json:"seconds"`
	BootTimeUnixSeconds uint64 `json:"boot_time_unix_seconds"`
}

Uptimes contains system uptime values in seconds.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL