allocmode

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var APIVersionPreference = []string{"v1", "v1beta2", versionV1beta1}

APIVersionPreference is the preference-ordered list of resource.k8s.io API versions the DRA checks accept, newest first. DRA graduated to GA (v1) in K8s 1.34, but the pinned NVIDIA DRA driver chart also supports K8s 1.32/1.33 clusters that serve only v1beta1/v1beta2 (the driver itself accepts v1|v1beta2|v1beta1), so the checks must not require v1.

Functions

func ClassifyK8sReadError

func ClassifyK8sReadError(err error, subject string) error

ClassifyK8sReadError wraps a Kubernetes object-read failure with the error code matching its cause — the SHARED classifier for probe/object reads so call sites cannot diverge: a true NotFound keeps ErrCodeNotFound, timeout forms (see IsK8sTimeoutErr) map to ErrCodeTimeout, and everything else (RBAC denials, transient apiserver errors) stays ErrCodeInternal. subject names the object being read, e.g. "ResourceClaim gpu-claim-abc".

func DiscoverServedVersion

func DiscoverServedVersion(ctx context.Context, clientset kubernetes.Interface) (string, *metav1.APIResourceList, error)

DiscoverServedVersion returns the newest SERVED version of the resource.k8s.io API group (per APIVersionPreference) along with its discovered resource list. The probe honors ctx (see draGroupVersionResources) so a validator timeout can abort it. Only NotFound is treated as "this version is not served"; any other discovery error is ambiguous and propagates (fail closed). When no version is served it returns ("", nil, nil) — the caller decides whether that is a failure (driver in recipe scope) or not.

func EligibleReadySchedulableNodes

func EligibleReadySchedulableNodes(nodeList *corev1.NodeList) map[string]*corev1.Node

EligibleReadySchedulableNodes indexes the Ready, schedulable (uncordoned) nodes from the list by name.

func GVRAt

func GVRAt(version, resource string) schema.GroupVersionResource

GVRAt returns the GroupVersionResource for a resource.k8s.io resource at the given served API version (v1, v1beta2, or v1beta1).

func IsK8sTimeoutErr

func IsK8sTimeoutErr(err error) bool

IsK8sTimeoutErr reports whether err represents a timeout rather than a generic failure: client-side context cancellation/deadline expiry, a Kubernetes apiserver Timeout/ServerTimeout status, a transport-level network timeout (*url.Error / net.Error with Timeout() true), or client-go's rate-limiter refusing because the wait would exceed the context deadline. Callers use it to map such errors to ErrCodeTimeout instead of ErrCodeInternal; RBAC and other transient API errors stay ErrCodeInternal.

func SortedNodeNames

func SortedNodeNames(set map[string]struct{}) []string

func UsableDriverDeviceCounts

func UsableDriverDeviceCounts(ctx context.Context, items []unstructured.Unstructured, driver string, eligible map[string]*corev1.Node) (map[string]int, int, error)

UsableDriverDeviceCounts is UsableDriverSliceNodes with per-node usable DEVICE counts: for each Ready, schedulable node it returns how many untainted devices the driver advertises in complete, current-generation pools. Sizing consumers (DRA-mode worker counts) need the device count, not just node membership.

Counting invariant (#1620 review): under allNodes, multi-node nodeSelector, or per-device allNodes topologies one physical device would increment the count for EVERY reachable node, over-reporting aggregate capacity N×. That state is deliberately unreachable for gpu.nvidia.com consumers: non-node-local topologies are surfaced via Mode.NonNodeLocalGPUSlices and rejected fail-fast (#1652) BEFORE any sizing reads these counts. If that fail-fast is ever relaxed, this counting must be revisited first.

func UsableDriverSliceNodes

func UsableDriverSliceNodes(ctx context.Context, items []unstructured.Unstructured, driver string, eligible map[string]*corev1.Node) (map[string]struct{}, int, error)

UsableDriverSliceNodes validates the ResourceSlices published by the given DRA driver and returns the set of Ready, schedulable node names reachable from at least one untainted advertised device. A slice only counts when it belongs to a complete, current-generation pool (see currentPoolSlices) — incomplete, stale, or inconsistent pools are ignored. The second return value is the number of slices the driver published (before validation), so callers can distinguish "driver publishes nothing" from "slices exist but none pass validation".

func Verify

func Verify(policy string, mode *Mode) error

Verify is the VERIFY step of the #1327 contract ("configuration selects the allocation policy, cluster inspection verifies it, mismatch fails closed"): it compares the recipe-configured whole-GPU allocation policy against the facts Detect inspected, and returns ErrCodeInvalidRequest when the cluster cannot serve the configured mechanism — no silent fallback to the other mechanism. A cluster drifted from its recipe is a validation failure by design.

GPUAllocationPolicyUnspecified (and "") verifies nothing: standalone runs keep capability-driven automatic selection. Any policy this build does not know how to verify — including the reserved dra-extended-resource — fails closed rather than slipping through unverified.

Types

type Mode

type Mode struct {
	// APIVersion is the served resource.k8s.io API version the DRA probe ran
	// against (per APIVersionPreference: v1, v1beta2, or v1beta1), or ""
	// when the cluster serves no version of the group. Consumers that create
	// DRA resources (e.g. the secure-access ResourceClaim) must use it so the
	// probe and the behavioral test operate at the same group-version.
	APIVersion string

	// DRAUsable is true when the gpu.nvidia.com DeviceClass exists AND at
	// least one complete, current-generation ResourceSlice pool from driver
	// gpu.nvidia.com advertises an untainted device reachable from a Ready,
	// schedulable node.
	DRAUsable bool
	// DRADetail explains why DRA is or is not usable.
	DRADetail string
	// DRANodes is the sorted set of Ready, schedulable node names with
	// usable full-GPU DRA devices.
	DRANodes []string
	// DRANodeDevices maps each DRANodes entry to its USABLE full-GPU DRA
	// device count (untainted devices in complete, current-generation pools).
	// Consumers that size DRA-mode workloads (e.g. inference-perf worker
	// count) must use this — NOT scalar allocatable nvidia.com/gpu, which on
	// dual-advertised nodes can differ from the DRA-usable device count.
	DRANodeDevices map[string]int
	// NodeLocalGPUSliceDevices maps node name → RAW node-local
	// gpu.nvidia.com device count: slices attributed by raw spec.nodeName
	// only, devices counted without pool/taint validation. This mirrors how
	// kai-scheduler v0.14.1 attributes slices to nodes
	// (kubernetes_lister.go:198, cluster_info.go:270 — kai then rejects
	// scalar device-plugin GPU pods on every counted node, node_info.go (the scalar-GPU-on-DRA-node rejection, upstream-marked "temporary fix"))
	// and doubles as the SUPPORTED-configuration detector: in AICR's
	// supported full-GPU DRA state the NVIDIA driver publishes one
	// node-local pool per node, so a node with usable validated devices
	// (DRANodes) always appears here too. A node present here but absent
	// from DRANodes carries slices kai counts but AICR cannot validate.
	// Populated whenever the DRA API is served — independent of DeviceClass
	// existence, exactly like kai.
	NodeLocalGPUSliceDevices map[string]int
	// ForeignGPUSliceDrivers lists "gpu"-named ResourceSlice drivers OTHER
	// than gpu.nvidia.com (kai's driver predicate is lowercase-contains
	// "gpu" — dra.go:149 — so e.g. gpu.amd.com slices consume kai's shared
	// GPU capacity vector). AICR's inference validation does not model mixed
	// GPU DRA drivers: consumers fail fast when this is non-empty (#1652).
	// ComputeDomain drivers contain no "gpu" substring and never appear.
	ForeignGPUSliceDrivers []string
	// NonNodeLocalGPUSlices lists gpu.nvidia.com ResourceSlice names using
	// nodeSelector/allNodes/perDeviceNodeSelection topologies. kai cannot
	// node-attribute them (invisible to its capacity vector while the DRA
	// driver can still bind them), which AICR's inference validation does
	// not model: consumers fail fast when this is non-empty (#1652). The
	// conformance checks' generic topology validation is unaffected.
	NonNodeLocalGPUSlices []string
	// GPUPoolNodes maps each node-local gpu.nvidia.com ResourceSlice POOL
	// name to the node (spec.nodeName) whose slices publish it. The K8s API
	// documents that a pool name "is often the node name, but this is not
	// required" (resource/v1 ResourcePool.Name) — so occupancy attribution
	// must resolve pool → node through this slice-derived map, never through
	// pool==node name equality, which mis-attributes whenever a pool is
	// named after a DIFFERENT existing node. Pools listed in
	// AmbiguousGPUPools are excluded.
	GPUPoolNodes map[string]string
	// AmbiguousGPUPools lists gpu.nvidia.com pool names that node-local
	// slices of MULTIPLE nodes publish — pool → node attribution would be a
	// guess: consumers fail fast when this is non-empty (#1652). The NVIDIA
	// driver publishes one pool per node, so this is empty in AICR's
	// supported configurations.
	AmbiguousGPUPools []string

	// DevicePluginUsable is true when a Ready, schedulable node has
	// allocatable nvidia.com/gpu > 0. Kubernetes semantics: a Ready node
	// advertising scalar nvidia.com/gpu allocatable is device-plugin-backed
	// even when a DeviceClass maps that extended resource to DRA (KEP-5004
	// DRAExtendedResource) — the scheduler routes the extended-resource
	// request to DRA only on nodes where scalar allocatable is absent or
	// zero. See ExtendedResourceDRABacked for the recorded attribution caveat.
	DevicePluginUsable bool
	// DevicePluginDetail explains why the device plugin is or is not usable.
	DevicePluginDetail string
	// DevicePluginNodes is the sorted set of Ready, schedulable node names
	// advertising allocatable nvidia.com/gpu.
	DevicePluginNodes []string

	// DualAdvertisedNodes lists Ready, schedulable nodes where BOTH
	// mechanisms advertise GPUs. Dual advertisement risks GPU over-admission
	// (the same physical GPU allocatable through both paths). Currently a
	// warning only; a later PR promotes this to an error.
	DualAdvertisedNodes []string

	// ExtendedResourceDRABacked is true when a DeviceClass maps the
	// nvidia.com/gpu extended resource to DRA via spec.extendedResourceName
	// (KEP-5004 DRAExtendedResource). Recorded as attribution evidence only —
	// it does NOT clear DevicePluginUsable: on nodes with scalar allocatable
	// nvidia.com/gpu the request is device-plugin-served by definition; the
	// DRA mapping applies only where scalar allocatable is absent/zero.
	// Consumers that need deterministic per-node attribution should pin their
	// workload to a node with scalar allocatable (see the secure-access check).
	ExtendedResourceDRABacked bool
	// ExtendedResourceDetail explains the extended-resource attribution state.
	ExtendedResourceDetail string
}

Mode reports which GPU allocation mechanisms are currently usable in the cluster. The CNCF secure_accelerator_access requirement permits either DRA or the device plugin; the NVIDIA DRA driver's supported configuration is ComputeDomain-only (no gpu.nvidia.com DeviceClass), with whole GPUs allocated by the device plugin (nvidia.com/gpu). Mirrors the detection semantics of kubernetes-sigs/ai-conformance#75. (#1327)

func Detect

func Detect(parent context.Context, clientset kubernetes.Interface, dynClient dynamic.Interface) (*Mode, error)

Detect probes the cluster for usable GPU allocation mechanisms: full-GPU DRA (gpu.nvidia.com) and the device plugin (nvidia.com/gpu). The DRA probe runs at the newest SERVED resource.k8s.io API version (v1, v1beta2, or v1beta1 — see DiscoverServedVersion), so beta-only clusters (K8s 1.32/1.33) are detected as DRA-usable too. When both mechanisms are live on the same node(s) it emits a warning about GPU over-admission risk (it does NOT fail — a later PR promotes dual advertisement to an error).

func (*Mode) Summary

func (m *Mode) Summary() string

Summary renders the detection result for evidence artifacts.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL