validations

func CheckAcceleratedSelectorMissing ¶

func CheckAcceleratedSelectorMissing(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckAcceleratedSelectorMissing checks if accelerated-node-selector is missing when conditions are met. This is a generic check that can be used by any component.

func CheckDriverOwnershipCoherence ¶ added in v0.18.0

func CheckDriverOwnershipCoherence(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckDriverOwnershipCoherence fails a bundle whose FINAL effective values (recipe merge plus all --set/--set-json/--set-file overrides) render an incoherent GPU driver-ownership profile. Two rules:

Rule 1 (driverless cluster, gated on recorded snapshot state): when the snapshot that produced the recipe observed no NVIDIA kernel driver on the sampled GPU node (metadata.gpuDriverState=absent — recorded by pkg/client/v1's snapshot-driven resolution, the `--gpu-driver none` signature), the effective config must have the operator install the full stack: driver.enabled=true and, when declared, toolkit.enabled not false. Deploying the preinstalled-driver assumption onto that cluster leaves GPU nodes driverless — nothing on the node provides a driver and the recipe does not install one. Recipes without a recorded state (criteria-only resolves, older recipes, snapshots without a usable driver-loaded reading) are not gated by this rule. A recorded state outside the two documented constants is rejected outright — the empty-string disarm is deliberate, an unrecognized nonempty spelling in a loaded or hand-edited recipe is not.

Rule 2 (DRA driver-root lockstep, metadata-independent): when nvidia-dra-driver-gpu is bundled alongside gpu-operator, its nvidiaDriverRoot must track the driver owner — see pkg/recipe/driver_root_lockstep_test.go for the full invariant rationale (issue #1087). With driver.enabled=true the DRA kubelet plugin must read the operator install dir (hostPaths.driverInstallDir), or CDI spec generation fails and DRA-allocated pods stall in ContainerCreating; with driver.enabled=false the root must not be the operator container root /run/nvidia/driver, which nothing populates in that mode — the signature of a legacy pre-flip recipe whose valuesFile now resolves the preinstalled-driver defaults while its baked DRA override still points at the operator path. Because this rule evaluates effective values only, it catches those legacy recipes with no recorded gpuDriverState.

Independent of both rules, an explicitly declared gpu-operator hostPaths.driverInstallDir that cleans to "/" is always rejected: it is the host path the operator-validator bind-mounts as the driver-validation container's rootfs target, and runc rejects a mount whose destination is "/" — the issue #1106 regression (see pkg/recipe/driver_root_lockstep_test.go, invariant 1). The DRA nvidiaDriverRoot of "/" is NOT flagged — it is the legitimate preinstalled-driver value.

The check runs at bundle generation — not at snapshot-driven recipe resolution, which only warns — because this is the first point where the user's --set ownership overrides are known: `aicr recipe` has no --set, so a resolution-time hard failure would leave supported LEGACY GPU-Operator-managed clusters unable to reach the documented override. On ADR-015-profiled recipes (AKS gpuStack) the --set escape does not apply — ownership paths are profile-owned and per-path flips are rejected — so the profiled remedy is out-of-band (fix/recreate pools, recapture, regenerate with --profile); see driverAbsentRemedy. Registered with severity error on gpu-operator (recipes/registry.yaml), which converts the returned messages into a blocking ErrCodeInvalidRequest in RunValidations. The check returns hard errors only when a component's effective values cannot be resolved or the user's overrides cannot be reapplied to them (see effectiveComponentValues): either way the values this gate must verify cannot be reconstructed, so coherence fails closed.

func CheckHostMofedWithoutNetworkOperator ¶ added in v0.13.0

func CheckHostMofedWithoutNetworkOperator(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckHostMofedWithoutNetworkOperator warns when network-operator is disabled via --set but gpu-operator still has driver.rdma.useHostMofed=true (the AKS default). Without network-operator, no host MOFED is present and useHostMofed should be set to false.

func CheckMariaDBOperatorOwnershipCoherence ¶ added in v0.19.0

func CheckMariaDBOperatorOwnershipCoherence(_ context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckMariaDBOperatorOwnershipCoherence enforces the snapshot-driven installation-safety policy for AICR-provided Slurm accounting. Existing MariaDB CRs and inconclusive discovery block bundling; an API with no detected CRs produces a warning; conclusive absence is silent. An empty state means no snapshot evidence was recorded and produces a non-blocking warning so criteria-only and older-snapshot workflows remain compatible.

func CheckNVSentinelDriverLabelDetectable ¶ added in v0.20.0

func CheckNVSentinelDriverLabelDetectable(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

NVSENTINEL GATE POLICY — what an nvsentinel gate means when parts of nvsentinel are disabled or filtered away (three review rounds probed this from different angles; keep the rules in one place):

1. Evidence is the DECLARED union; enforcement follows the OUTPUT. The gates read other components (gpu-operator) from the pre-filter declaration via resolveGPUOperatorRef — a component excluded from a subset bundle still describes the platform — but a gate only runs at all when nvsentinel itself is rendered (RunComponentValidations iterates the filtered refs). This is the ADR-018 union rule.
2. An explicitly-disabled consumer subchart renders nothing, so it neither triggers a gate nor may deployment validation require it: the RuntimeClass gate skips when metadata-collector is disabled, the driver-label gate skips when BOTH label consumers (metadata-collector, syslog monitors) are disabled, and the health check's DaemonSet assertions use the negative form that tolerates a true 404. A PARTIALLY disabled consumer set does not skip — the remaining consumer still needs the remedy.
3. When rendered, a consumer must be fully rolled out — 0 desired (the #2175 signature) and partial rollout both fail the health check; the gates exist so that state is rejected at bundle time instead.

CheckNVSentinelDriverLabelDetectable blocks a bundle whose NVSentinel deployment would silently come up half-rolled-out.

The NVSentinel labeler decides nvsentinel.dgxc.nvidia.com/driver.installed by watching for a GPU driver pod. Where the driver ships in the node image and no driver pod exists — AKS gpuStack=azure-managed, GKE COS gpuStack=gke-default, OKE — the label is never applied, so metadata-collector and both syslog-health-monitor DaemonSets report 0 desired pods. Nothing reports an error: a DaemonSet whose node selector matches no node is not unhealthy, it emits no event, and gpu-health-monitor keeps running because it selects on the DCGM label instead. The stack looks healthy while half of it was never scheduled (issue #2175).

The chart automates the remedy: labeler.assumeDriverInstalled renders --assume-driver-installed, which is the Manual Labeling Procedure of NVSentinel design 018 expressed as configuration. Manually labeling nodes is NOT an equivalent workaround — the labeler computes an empty desired value when no driver pod exists and removes the label on its next reconcile.

The gate fires only when every one of the following holds, so the flag is never demanded where the GPU Operator owns the driver (setting it there would skip detection and mask an unloaded or unhealthy driver):

• nvsentinel is present and enabled,
• the GPU Operator is present, enabled, and has driver.enabled=false in its FINAL effective values (recipe merge plus the user's --set/--set-json overrides), so the documented GPU-Operator-managed override set clears the gate,
• no other driver pod source the labeler recognizes exists (labelerObservesDriverPod — GKE gpuStack=driver-installer), and
• labeler.assumeDriverInstalled is not truthy in nvsentinel's effective values.

It stays silent when the GPU Operator is absent or disabled: no AICR recipe ships nvsentinel without it, and with no driver-ownership signal to key on the gate would be guessing rather than detecting.

When the driver toggle cannot be read as a boolean the gate defers to CheckDriverOwnershipCoherence, which rejects that on the same values, rather than adding a second message derived from a guessed default. That deferral holds only while the sibling actually runs: it is registered on the GPU Operator (recipes/registry.yaml) and RunComponentValidations iterates the FILTERED ComponentRefs, so a subset bundle (bundlers=nvsentinel) renders nvsentinel without it and nothing would report the malformed toggle. The declared union supplies this gate's evidence but cannot make the sibling execute, so the gate fails closed when the operator it read the toggle from is not itself rendered.

Registered with severity error on nvsentinel (recipes/registry.yaml), which converts the returned message into a blocking ErrCodeInvalidRequest in RunValidations. Hard errors are returned only when a component's effective values cannot be resolved or the user's overrides cannot be reapplied to them (see effectiveComponentValues): the state this gate must verify is then unknown, so it fails closed.

func CheckNVSentinelRuntimeClassCoherence ¶ added in v0.20.0

func CheckNVSentinelRuntimeClassCoherence(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNVSentinelRuntimeClassCoherence blocks a bundle whose NVSentinel metadata-collector pods would be rejected at admission.

The metadata-collector DaemonSet sets runtimeClassName (chart default "nvidia"), and the GPU Operator's ClusterPolicy controller creates the primary RuntimeClass named after operator.runtimeClass (also default "nvidia"). When a recipe retargets operator.runtimeClass — the AKS azure-managed profile sets nvidia-container-runtime because that is the handler preconfigured on the AKS node image — no RuntimeClass named "nvidia" exists on the cluster, and the API server rejects every metadata-collector pod at admission: `pod rejected: RuntimeClass "nvidia" not found` (issue #2176).

The failure mode is easy to misread: the pods are rejected before a pod object is created, so there is nothing to kubectl describe — the DaemonSet shows N desired / 0 created and the only signal is a FailedCreate event on it. Distinct from and additive to the driver-label gap (CheckNVSentinelDriverLabelDetectable, #2175): the label gets metadata-collector scheduled, the runtime class gets its pods admitted.

This is a value comparison, not a platform matrix: the gate fires exactly when the two resolved names differ, treating either side unset as the shared chart default "nvidia" (both defaults verified against the pinned charts — see defaultRuntimeClassName). It therefore passes wherever the recipes leave operator.runtimeClass at its default (EKS, GKE, OKE, AKS operator-managed) and fails AKS azure-managed until the override is passed. An explicitly EMPTY metadata-collector.runtimeClassName also passes: the subchart omits the field entirely then, and a pod without runtimeClassName is always admitted.

The gate stays silent when the GPU Operator is absent or disabled (nothing manages RuntimeClasses, so there is no authoritative name to compare against), when the metadata-collector subchart is disabled (global.metadataCollector.enabled=false — no DaemonSet renders), when either value is present but not a string (the install fails on its own terms; guessing a default here could invert the verdict).

It also runs on a nil bundler config — the values-only Client.BundleComponents path. That path used to be exempt because the remedy was a --set it cannot express; since #2181 the AKS profile owns both operator.runtimeClass and metadata-collector.runtimeClassName, so the coherent state is reachable from resolved values alone.

Registered with severity error on nvsentinel (recipes/registry.yaml). Hard errors are returned only when effective values cannot be resolved (effectiveComponentValues) — the state this gate must verify is then unknown, so it fails closed.

func CheckWildcardAcceleratedToleration ¶ added in v0.17.0

func CheckWildcardAcceleratedToleration(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckWildcardAcceleratedToleration reports when the effective accelerated-node tolerations for a component include a wildcard (keyless operator: Exists) toleration. Scope it via registry conditions to services where the wildcard is harmful — on AKS, admission collapses a pod's toleration list to just the wildcard when one is present, which defeats the nodewright operator's drain exemption for its own package pods and deadlocks packages that declare interrupts (NVIDIA/nodewright#296). That deadlock requires manual node cordon/reboot to recover, so the registry wires this at severity: error to block the bundle until a keyed toleration is supplied.

The default bundle path always hits this: with no --accelerated-node-toleration flag the CLI falls back to snapshotter.DefaultTolerations() (a single bare operator: Exists). An empty toleration list is flagged too, because the tuning manifest template renders its own wildcard fallback when none are injected.

A component disabled via --set (e.g. the documented RDMA opt-out --set nodewrightcustomizations:enabled=false) renders no package pods and cannot deadlock, so it is skipped regardless of the toleration shape.

func CheckWorkloadSelectorMissing ¶

func CheckWorkloadSelectorMissing(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckWorkloadSelectorMissing checks if workload-selector is missing when conditions are met. This is a generic check that can be used by any component.

func GetAll ¶

func GetAll() []string

GetAll returns all registered validation function names.

func Register ¶

func Register(name string, fn ValidationFunc)

Register adds a validation function to the registry. This allows components to register custom validation functions. It's also called from init() functions in check files for auto-registration.

func RunComponentValidations ¶ added in v0.18.0

func RunComponentValidations(ctx context.Context, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config) ([]string, error)

RunComponentValidations executes every registry-declared validation for each component present in recipeResult, in ComponentRefs order. It is the shared component preflight behind DefaultBundler.Make (which passes its bundle config so --set/--set-json overrides participate) and Client.BundleComponents (which passes a nil config — that path has no bundle-time overrides, and every validation that acts solely on bundle-time flags no-ops on a nil config). Returns the accumulated non-blocking warnings and the first blocking error; warnings gathered before a blocking error are still returned so the caller can surface them alongside it.

func RunValidations ¶

func RunValidations(ctx context.Context, componentName string, validations []recipe.ComponentValidationConfig, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config) (warnings []string, errors []error)

RunValidations executes all validations for a component and returns warnings and errors. The optional message from the validation config is appended to each warning/error. Severity determines whether check results become warnings or errors.