gpu

package
v0.2.0-beta.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 5, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package gpu detects NVIDIA GPUs on a node by querying nvidia-smi and the Docker daemon's registered runtimes. It never shells out to the docker CLI.

Index

Constants

View Source 
const (
	StatusOK   = "ok"
	StatusWarn = "warn"
	StatusFail = "fail"
)

Finding statuses, matching the doctor report's vocabulary.

View Source 
const (
	AttachCDI    = "cdi"
	AttachLegacy = "legacy"
	AttachNone   = "none"
)

Attach modes a host can use for GPU containers.

View Source 
const (
	// CDIGenerateCommand writes the NVIDIA CDI spec that CDI attach needs.
	CDIGenerateCommand = "sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml"
	// CDIEnableHint is the remedy when a spec exists but Docker lists no CDI devices.
	CDIEnableHint = `` /* 147-byte string literal not displayed */

)
View Source 
const DriverHint = "No NVIDIA driver detected. On Ubuntu run ubuntu-drivers install (or apt-get install nvidia-driver-XXX), then reboot."

DriverHint is the operator-facing remedy when no NVIDIA driver answers.

View Source 
const InstallCommand = "" /* 135-byte string literal not displayed */

InstallCommand is the copy-pasteable fix for a missing nvidia runtime, assuming NVIDIA's apt repository is already configured.

View Source 
const InstallHint = "" /* 209-byte string literal not displayed */

InstallHint is the operator-facing remedy when GPUs exist but Docker has no nvidia runtime.

Variables

View Source 
var (
	ErrNoGPU          = errors.New("node reports no NVIDIA GPU")
	ErrRuntimeMissing = errors.New("the nvidia container runtime is not registered with Docker on the node")
	ErrUnknownDevice  = errors.New("GPU device not present on the node")
	ErrDeviceTaken    = errors.New("GPU device already reserved")
	ErrInsufficient   = errors.New("not enough free GPUs")
)

Sentinel reasons a claim does not fit a node.

Functions

func RunCollector 

func RunCollector(ctx context.Context, sink Sink, r Runner, rl RuntimeLister, nodeID string, interval time.Duration, logger *slog.Logger)

RunCollector detects this host's GPUs immediately and then every interval, writing each snapshot for nodeID to sink until ctx ends.

Types

type CDILister 

type CDILister interface {
	CDIDevices(ctx context.Context) ([]string, error)
}

CDILister reports the NVIDIA CDI devices the Docker daemon discovered.

type Claim 

type Claim struct {
	Name      string
	Count     int
	DeviceIDs []string
}

Claim is one workload's GPU reservation. With no DeviceIDs, Count <= 0 means every GPU on the node, matching how Docker's request is built.

func (Claim) All 

func (c Claim) All() bool

All reports whether the claim takes every GPU on the node.

type Device 

type Device struct {
	Index              int
	UUID               string
	Name               string
	VRAMTotalMiB       int64
	VRAMUsedMiB        int64
	UtilizationPercent int
}

Device is one GPU as reported by nvidia-smi.

func ParseSMI 

func ParseSMI(out string) (driver string, devices []Device, err error)

ParseSMI parses the CSV emitted by nvidia-smi with smiArgs.

type DockerFacts 

type DockerFacts interface {
	RuntimeNames(ctx context.Context) ([]string, error)
	CDIDevices(ctx context.Context) ([]string, error)
}

DockerFacts is what the diagnosis asks the Docker daemon.

type ExecRunner 

type ExecRunner struct{}

ExecRunner runs commands with os/exec.

func (ExecRunner) Run 

func (ExecRunner) Run(ctx context.Context, name string, args ...string) ([]byte, error)

Run implements Runner.

type Finding 

type Finding struct {
	Code    string `json:"code"`
	Name    string `json:"name"`
	Status  string `json:"status"`
	Message string `json:"message"`
	Fix     string `json:"fix,omitempty"`
}

Finding is one diagnosed aspect of a host's GPU setup.

type HostDiagnosis 

type HostDiagnosis struct {
	Info           Info
	ToolkitVersion string
	CDISpecFiles   []string
	CDIDevices     []string
	Attach         string
	Findings       []Finding
}

HostDiagnosis is the detailed GPU picture of one host.

func DiagnoseHost 

func DiagnoseHost(ctx context.Context, r Runner, d DockerFacts, glob func(pattern string) []string) HostDiagnosis

DiagnoseHost inspects this host's GPU stack: driver, per-GPU memory, the NVIDIA container toolkit, CDI specs and how Docker can attach GPUs. glob may be nil to use the real filesystem.

type Info 

type Info struct {
	Present          bool
	DriverVersion    string
	Devices          []Device
	RuntimeInstalled bool
}

Info is a node's GPU capability snapshot.

func Detect 

func Detect(ctx context.Context, r Runner, rl RuntimeLister) Info

Detect returns the node's GPU snapshot. A missing nvidia-smi or a failing query means no GPU, not an error: most nodes have none.

func (Info) Count 

func (i Info) Count() int

Count returns the number of GPUs.

func (Info) Hint 

func (i Info) Hint() string

Hint returns remediation text when the node has a GPU problem, or empty when there is nothing to fix (including nodes with no GPU).

func (Info) TotalVRAMMiB 

func (i Info) TotalVRAMMiB() int64

TotalVRAMMiB sums VRAM across devices.

func (Info) Usable 

func (i Info) Usable() bool

Usable reports whether a container can be given a GPU on this node.

type Ledger 

type Ledger struct {
	// contains filtered or unexported fields
}

Ledger tracks which GPUs of one node are reserved. Docker does not enforce exclusivity, so this is scheduling accounting, not isolation.

func NewLedger 

func NewLedger(info Info, claims []Claim) Ledger

NewLedger builds the ledger for a node from the claims already on it.

func (Ledger) Fit 

func (l Ledger) Fit(c Claim) error

Fit reports why c cannot be placed on this node, or nil when it can.

func (Ledger) Free 

func (l Ledger) Free() int

Free is Total minus Reserved.

func (Ledger) Holders 

func (l Ledger) Holders() []string

Holders lists the workloads holding a reservation, sorted.

func (Ledger) Reserved 

func (l Ledger) Reserved() int

Reserved is how many GPUs are spoken for, never above Total.

func (Ledger) Total 

func (l Ledger) Total() int

Total is the node's GPU count.

type Runner 

type Runner interface {
	Run(ctx context.Context, name string, args ...string) ([]byte, error)
}

Runner runs a host command, returning its stdout.

type RuntimeLister 

type RuntimeLister interface {
	RuntimeNames(ctx context.Context) ([]string, error)
}

RuntimeLister reports the OCI runtimes registered with Docker.

type Sink 

type Sink interface {
	SetNodeGPU(ctx context.Context, nodeID string, info Info) error
}

Sink stores a node's GPU snapshot.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.