validations

func BundleSuppliesGKEDriver ¶ added in v0.21.0

func BundleSuppliesGKEDriver(ctx context.Context, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config) (bool, error)

BundleSuppliesGKEDriver reports whether the composed bundle carries an effectively enabled gcp-driver-installer — i.e. the bundle itself provisions the NVIDIA kernel driver, so metadata.gpuDriverState=absent is the expected pre-deployment state of a correctly provisioned pool (gpu-driver-version=disabled) rather than a misconfiguration. It keys off the EFFECTIVE installer gate (recipe values plus any --set overrides in bundlerConfig), not the selected profile name: a --set that flips the gate must flip this answer with it. The gate mirrors the manifest template exactly (toString(installer.enabled) == "true"), so only a value that actually renders the DaemonSet counts as a producer; anything else — absent, false, or an unrecognized type — leaves the driverless Rule 1 gate armed (fail closed). The lookup runs against the declared-union view so a subset bundle (--bundlers gpu-operator) still observes the installer its sibling bundle carries. bundlerConfig may be nil (the resolution-time caller in pkg/client/v1 has no override channel).

func CheckAcceleratedSelectorMissing ¶

func CheckAcceleratedSelectorMissing(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckAcceleratedSelectorMissing checks if accelerated-node-selector is missing when conditions are met. This is a generic check that can be used by any component.

func CheckDriverOwnershipCoherence ¶ added in v0.18.0

func CheckDriverOwnershipCoherence(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckDriverOwnershipCoherence fails a bundle whose FINAL effective values (recipe merge plus all --set/--set-json/--set-file overrides) render an incoherent GPU driver-ownership profile. Two rules:

Rule 1 (driverless cluster, gated on recorded snapshot state): when the snapshot that produced the recipe observed no NVIDIA kernel driver on the sampled GPU node (metadata.gpuDriverState=absent — recorded by pkg/client/v1's snapshot-driven resolution, the `--gpu-driver none` signature), the effective config must have the operator install the full stack: driver.enabled=true and, when declared, toolkit.enabled not false. Deploying the preinstalled-driver assumption onto that cluster leaves GPU nodes driverless — nothing on the node provides a driver and the recipe does not install one. Recipes without a recorded state (criteria-only resolves, older recipes, snapshots without a usable driver-loaded reading) are not gated by this rule. A recorded state outside the two documented constants is rejected outright — the empty-string disarm is deliberate, an unrecognized nonempty spelling in a loaded or hand-edited recipe is not.

Rule 2 (DRA driver-root lockstep, metadata-independent): when nvidia-dra-driver-gpu is bundled alongside gpu-operator, its nvidiaDriverRoot must track the driver owner — see pkg/recipe/driver_root_lockstep_test.go for the full invariant rationale (issue #1087). With driver.enabled=true the DRA kubelet plugin must read the operator install dir (hostPaths.driverInstallDir), or CDI spec generation fails and DRA-allocated pods stall in ContainerCreating; with driver.enabled=false the root must not be the operator container root /run/nvidia/driver, which nothing populates in that mode — the signature of a legacy pre-flip recipe whose valuesFile now resolves the preinstalled-driver defaults while its baked DRA override still points at the operator path. Because this rule evaluates effective values only, it catches those legacy recipes with no recorded gpuDriverState.

Independent of both rules, an explicitly declared gpu-operator hostPaths.driverInstallDir that cleans to "/" is always rejected: it is the host path the operator-validator bind-mounts as the driver-validation container's rootfs target, and runc rejects a mount whose destination is "/" — the issue #1106 regression (see pkg/recipe/driver_root_lockstep_test.go, invariant 1). The DRA nvidiaDriverRoot of "/" is NOT flagged — it is the legitimate preinstalled-driver value.

The check runs at bundle generation — not at snapshot-driven recipe resolution, which only warns — because this is the first point where the user's --set ownership overrides are known: `aicr recipe` has no --set, so a resolution-time hard failure would leave supported LEGACY GPU-Operator-managed clusters unable to reach the documented override. On ADR-015-profiled recipes (AKS gpuStack) the --set escape does not apply — ownership paths are profile-owned and per-path flips are rejected — so the profiled remedy is out-of-band (fix/recreate pools, recapture, regenerate with --profile); see driverAbsentRemedy. Registered with severity error on gpu-operator (recipes/registry.yaml), which converts the returned messages into a blocking ErrCodeInvalidRequest in RunValidations. The check returns hard errors only when a component's effective values cannot be resolved or the user's overrides cannot be reapplied to them (see effectiveComponentValues): either way the values this gate must verify cannot be reconstructed, so coherence fails closed.

func CheckGB300HostKernelGranule ¶ added in v0.22.0

func CheckGB300HostKernelGranule(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckGB300HostKernelGranule warns that the GB300 tuned profile assumes a 64k-granule ARM64 host kernel. Scope it via registry conditions to the leaf that has no nvidia-setup to pin one (service: generic, accelerator: gb300); every other GB300 route runs nvidia-setup-kernel, which installs the pinned 64k kernel itself.

nvidia-gb300-performance sizes its hugepage pools for that granule (hugepagesz=512M, and no 1G, which a 64k granule cannot register). A 4k-granule host still boots and runs: Linux rejects the invalid hugepagesz clause and silently drops the hugepages= count paired with it, so the node comes up without the 512M pool the profile intended. That is a performance regression rather than a failure, which is why the registry wires this at severity: info — bundle time has no cluster-side signal to tell the two apart, and blocking would refuse a configuration that works.

A component disabled via --set, or one whose tuning is gated off with tuningEnabled=false, renders no nvidia-tuned package and applies no profile, so it is skipped. Both gates are read from the FINAL effective values (recipe merge plus scalar --set and typed --set-json/--set-file, under the canonical name and its registry aliases) rather than from the raw scalar override map: a typed --set-json that suppresses the package must suppress the advisory with it. The tuningEnabled comparison mirrors the manifest's own `ne (toString ...) "false"` gate exactly, so the two cannot disagree.

func CheckGKETCPXOInterfacesCoherence ¶ added in v0.22.0

func CheckGKETCPXOInterfacesCoherence(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckGKETCPXOInterfacesCoherence compares the FINAL resolved kubeflow-trainer tcpxoInterfaces value against the mapping the recipe records in configuration.gke.tcpxoInterfaces. Validity alone is not enough: a value that is well-formed but different from what the recipe records is exactly the failure case — the recipe would attest to one wiring while the bundle renders another (the realistic shape: networks get reprovisioned and someone --sets the current names to make the bundle work). The bundler's ownership enforcement already rejects all four override channels for this path; this check is the defense-in-depth for hand-edited recipes and any future channel.

func CheckHostMofedWithoutNetworkOperator ¶ added in v0.13.0

func CheckHostMofedWithoutNetworkOperator(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckHostMofedWithoutNetworkOperator warns when network-operator is disabled via --set but gpu-operator still has driver.rdma.useHostMofed=true (the AKS default). Without network-operator, no host MOFED is present and useHostMofed should be set to false.

func CheckMariaDBOperatorOwnershipCoherence ¶ added in v0.19.0

func CheckMariaDBOperatorOwnershipCoherence(_ context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckMariaDBOperatorOwnershipCoherence enforces the snapshot-driven installation-safety policy for AICR-provided Slurm accounting. Existing MariaDB CRs and inconclusive discovery block bundling; an API with no detected CRs produces a warning; conclusive absence is silent. An empty state means no snapshot evidence was recorded and produces a non-blocking warning so criteria-only and older-snapshot workflows remain compatible.

func CheckNPDNotDuplicatingProviderNPD ¶ added in v0.22.0

func CheckNPDNotDuplicatingProviderNPD(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNPDNotDuplicatingProviderNPD blocks a bundle that would install node-problem-detector where doing so is unsafe or unverified.

An ALLOWLIST, not a denylist. NPD is a privileged DaemonSet that patches Node status, and getting its ownership wrong fails silently -- two writers produce flapping conditions with no error anywhere. So only platforms where this has actually been checked are permitted; everything else is rejected with a message saying what would settle it. A denylist would have let every unexamined platform through by omission.

Rejected, and why:

• gke, aks: the provider already runs its own (npdProviderRunsItsOwn).
• ocp: NPD needs a privileged SecurityContextConstraints binding that AICR does not ship, so the DaemonSet bundles cleanly and then fails admission.
• os talos: recipes/mixins/os-talos.yaml relocates privileged components into privileged-* namespaces for Pod Security Admission. NPD is not in that list, so it would land in a restricted namespace and be denied.
• anything else (lke, bcm, metal3, generic, k0s, ...): unverified.
• no criteria at all: the platform is unknown, and checkConditions' "nil Criteria means condition not met" would skip the gate entirely. A hand-authored or already-hydrated RecipeResult legitimately carries no criteria (pkg/client/v1's loadedResultFromInternal), and that platform could be any of the above.

func CheckNVSentinelDriverLabelDetectable ¶ added in v0.20.0

func CheckNVSentinelDriverLabelDetectable(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

NVSENTINEL GATE POLICY — what an nvsentinel gate means when parts of nvsentinel are disabled or filtered away (three review rounds probed this from different angles; keep the rules in one place):

1. Evidence is the DECLARED union; enforcement follows the OUTPUT. The gates read other components (gpu-operator) from the pre-filter declaration via resolveGPUOperatorRef — a component excluded from a subset bundle still describes the platform — but a gate only runs at all when nvsentinel itself is rendered (RunComponentValidations iterates the filtered refs). This is the ADR-018 union rule.
2. An explicitly-disabled consumer subchart renders nothing, so it neither triggers a gate nor may deployment validation require it: the RuntimeClass gate skips when metadata-collector is disabled, the driver-label gate skips when BOTH label consumers (metadata-collector, syslog monitors) are disabled, and the health check's DaemonSet assertions use the negative form that tolerates a true 404. A PARTIALLY disabled consumer set does not skip — the remaining consumer still needs the remedy.
3. When rendered, a consumer must be fully rolled out — 0 desired (the #2175 signature) and partial rollout both fail the health check; the gates exist so that state is rejected at bundle time instead.

CheckNVSentinelDriverLabelDetectable blocks a bundle whose NVSentinel deployment would silently come up half-rolled-out.

The NVSentinel labeler decides nvsentinel.dgxc.nvidia.com/driver.installed by watching for a GPU driver pod. Where the driver ships in the node image and no driver pod exists — AKS gpuStack=azure-managed, GKE COS gpuStack=gke-default, OKE — the label is never applied, so metadata-collector and both syslog-health-monitor DaemonSets report 0 desired pods. Nothing reports an error: a DaemonSet whose node selector matches no node is not unhealthy, it emits no event, and gpu-health-monitor keeps running because it selects on the DCGM label instead. The stack looks healthy while half of it was never scheduled (issue #2175).

The chart automates the remedy: labeler.assumeDriverInstalled renders --assume-driver-installed, which is the Manual Labeling Procedure of NVSentinel design 018 expressed as configuration. Manually labeling nodes is NOT an equivalent workaround — the labeler computes an empty desired value when no driver pod exists and removes the label on its next reconcile.

The gate fires only when every one of the following holds, so the flag is never demanded where the GPU Operator owns the driver (setting it there would skip detection and mask an unloaded or unhealthy driver):

• nvsentinel is present and enabled,
• the GPU Operator is present, enabled, and has driver.enabled=false in its FINAL effective values (recipe merge plus the user's --set/--set-json overrides), so the documented GPU-Operator-managed override set clears the gate,
• no other driver pod source the labeler recognizes exists (labelerObservesDriverPod — GKE gpuStack=bundle-installer), and
• labeler.assumeDriverInstalled is not truthy in nvsentinel's effective values.

It stays silent when the GPU Operator is absent or disabled: no AICR recipe ships nvsentinel without it, and with no driver-ownership signal to key on the gate would be guessing rather than detecting.

When the driver toggle cannot be read as a boolean the gate defers to CheckDriverOwnershipCoherence, which rejects that on the same values, rather than adding a second message derived from a guessed default. That deferral holds only while the sibling actually runs: it is registered on the GPU Operator (recipes/registry.yaml) and RunComponentValidations iterates the FILTERED ComponentRefs, so a subset bundle (bundlers=nvsentinel) renders nvsentinel without it and nothing would report the malformed toggle. The declared union supplies this gate's evidence but cannot make the sibling execute, so the gate fails closed when the operator it read the toggle from is not itself rendered.

Registered with severity error on nvsentinel (recipes/registry.yaml), which converts the returned message into a blocking ErrCodeInvalidRequest in RunValidations. Hard errors are returned only when a component's effective values cannot be resolved or the user's overrides cannot be reapplied to them (see effectiveComponentValues): the state this gate must verify is then unknown, so it fails closed.

func CheckNVSentinelNicHealthMonitorRequiresMetadataCollector ¶ added in v0.22.0

func CheckNVSentinelNicHealthMonitorRequiresMetadataCollector(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNVSentinelNicHealthMonitorRequiresMetadataCollector blocks a bundle that enables the nic-health-monitor subchart without the NIC inventory it depends on.

nic-health-monitor's link-state and link-counter checks run against the GPU-to-NIC topology metadata-collector writes to /var/lib/nvsentinel/gpu_metadata.json; the one documented bypass is an operator-supplied nicInclusionRegexOverride, which substitutes a manual device list and forfeits the automatic management-NIC exclusion with it. With metadata-collector disabled and no override the DaemonSet renders, deploys, and discovers zero devices, which nothing downstream reports. Registration details (severity, no-op conditions) are in recipes/registry.yaml.

func CheckNVSentinelPreflightDCGMReachable ¶ added in v0.22.0

func CheckNVSentinelPreflightDCGMReachable(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNVSentinelPreflightDCGMReachable rejects a bundle whose preflight DCGM check cannot reach a hostengine.

The address is read from the preflight-dcgm-diag init container rather than assumed: the mixin restates preflight.initContainers, so a leaf override or --set-json can legitimately retarget it, and a bogus address must fail even when gpu-operator sits where the default expects. Once the address names a cluster-local Service in gpu-operator's own namespace, every part of that has to hold. Three ways it does not, none of them visible until a GPU pod starts in an opted-in namespace:

• gpu-operator is absent or disabled, so nothing serves the Service;
• gpu-operator is relocated (os-talos moves it to privileged-gpu-operator), so the DNS name does not resolve;
• gpu-operator runs with dcgm.enabled: false, which is what the shipped Kind overlay does — only the embedded exporter runs and the standalone hostengine Service is never created.

Registration details are in recipes/registry.yaml.

func CheckNVSentinelPreflightGangSchedulerRequired ¶ added in v0.22.0

func CheckNVSentinelPreflightGangSchedulerRequired(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNVSentinelPreflightGangSchedulerRequired blocks a bundle that enables preflight gang coordination against KAI PodGroups while kai-scheduler is disabled or absent.

The dependencyRefs edge the mixin adds only orders the install; it does not require the scheduler to stay enabled, and the bundler prunes an edge to a declared-but-disabled component as "satisfied externally". Without the PodGroup CRD the preflight controller fails its startup GVR resolution and crash-loops — and because the webhook is failurePolicy: Ignore, every GPU pod in a labeled namespace is then admitted unchecked, silently. Registration details are in recipes/registry.yaml.

func CheckNVSentinelRuntimeClassCoherence ¶ added in v0.20.0

func CheckNVSentinelRuntimeClassCoherence(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNVSentinelRuntimeClassCoherence blocks a bundle whose NVSentinel metadata-collector pods would be rejected at admission.

The metadata-collector DaemonSet sets runtimeClassName (chart default "nvidia"), and the GPU Operator's ClusterPolicy controller creates the primary RuntimeClass named after operator.runtimeClass (also default "nvidia"). When a recipe retargets operator.runtimeClass — the AKS azure-managed profile sets nvidia-container-runtime because that is the handler preconfigured on the AKS node image — no RuntimeClass named "nvidia" exists on the cluster, and the API server rejects every metadata-collector pod at admission: `pod rejected: RuntimeClass "nvidia" not found` (issue #2176).

The failure mode is easy to misread: the pods are rejected before a pod object is created, so there is nothing to kubectl describe — the DaemonSet shows N desired / 0 created and the only signal is a FailedCreate event on it. Distinct from and additive to the driver-label gap (CheckNVSentinelDriverLabelDetectable, #2175): the label gets metadata-collector scheduled, the runtime class gets its pods admitted.

This is a value comparison, not a platform matrix: the gate fires exactly when the two resolved names differ, treating either side unset as the shared chart default "nvidia" (both defaults verified against the pinned charts — see defaultRuntimeClassName). It therefore passes wherever the recipes leave operator.runtimeClass at its default (EKS, GKE, OKE, AKS operator-managed) and fails AKS azure-managed until the override is passed. An explicitly EMPTY metadata-collector.runtimeClassName also passes: the subchart omits the field entirely then, and a pod without runtimeClassName is always admitted.

The gate stays silent when the GPU Operator is absent or disabled (nothing manages RuntimeClasses, so there is no authoritative name to compare against), when the metadata-collector subchart is disabled (global.metadataCollector.enabled=false — no DaemonSet renders), when either value is present but not a string (the install fails on its own terms; guessing a default here could invert the verdict).

It also runs on a nil bundler config — the values-only Client.BundleComponents path. That path used to be exempt because the remedy was a --set it cannot express; since #2181 the AKS profile owns both operator.runtimeClass and metadata-collector.runtimeClassName, so the coherent state is reachable from resolved values alone.

Registered with severity error on nvsentinel (recipes/registry.yaml). Hard errors are returned only when effective values cannot be resolved (effectiveComponentValues) — the state this gate must verify is then unknown, so it fails closed.

func CheckNVSentinelTracingEndpointRequired ¶ added in v0.22.0

func CheckNVSentinelTracingEndpointRequired(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckNVSentinelTracingEndpointRequired blocks a bundle that enables NVSentinel distributed tracing without supplying an OTLP collector endpoint. The chart has no fail-closed guard of its own: templates/daemonset.yaml sets OTEL_EXPORTER_OTLP_ENDPOINT to .Values.global.tracing.endpoint with no `required` guard, so global.tracing.enabled=true with an empty endpoint renders and deploys without error — the pod starts, and the OTel exporter fails at runtime with no signal visible to `aicr bundle`/`aicr validate`. This gate is the only point in the pipeline that inspects resolved Helm values (including --set/--set-json/--dynamic) before a bundle is produced, so it is the only mechanism that can catch this before deploy. Registration details (severity, no-op conditions) are in recipes/registry.yaml.

func CheckWildcardAcceleratedToleration ¶ added in v0.17.0

func CheckWildcardAcceleratedToleration(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckWildcardAcceleratedToleration reports when the effective accelerated-node tolerations for a component include a wildcard (keyless operator: Exists) toleration. Scope it via registry conditions to services where the wildcard is harmful — on AKS, admission collapses a pod's toleration list to just the wildcard when one is present, which defeats the nodewright operator's drain exemption for its own package pods and deadlocks packages that declare interrupts (NVIDIA/nodewright#296). That deadlock requires manual node cordon/reboot to recover, so the registry wires this at severity: error to block the bundle until a keyed toleration is supplied.

The default bundle path always hits this: with no --accelerated-node-toleration flag the CLI falls back to snapshotter.DefaultTolerations() (a single bare operator: Exists). An empty toleration list is flagged too, because the tuning manifest template renders its own wildcard fallback when none are injected.

A component disabled via --set (e.g. the documented RDMA opt-out --set nodewrightcustomizations:enabled=false) renders no package pods and cannot deadlock, so it is skipped regardless of the toleration shape.

func CheckWorkloadSelectorMissing ¶

func CheckWorkloadSelectorMissing(ctx context.Context, componentName string, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config, conditions map[string][]string) ([]string, []error)

CheckWorkloadSelectorMissing checks if workload-selector is missing when conditions are met. This is a generic check that can be used by any component.

func GetAll ¶

func GetAll() []string

GetAll returns all registered validation function names.

func Register ¶

func Register(name string, fn ValidationFunc)

Register adds a validation function to the registry. This allows components to register custom validation functions. It's also called from init() functions in check files for auto-registration.

func RunComponentValidations ¶ added in v0.18.0

func RunComponentValidations(ctx context.Context, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config) ([]string, error)

RunComponentValidations executes every registry-declared validation for each component present in recipeResult, in ComponentRefs order. It is the shared component preflight behind DefaultBundler.Make (which passes its bundle config so --set/--set-json overrides participate) and Client.BundleComponents (which passes a nil config — that path has no bundle-time overrides, and every validation that acts solely on bundle-time flags no-ops on a nil config). Returns the accumulated non-blocking warnings and the first blocking error; warnings gathered before a blocking error are still returned so the caller can surface them alongside it.

func RunValidations ¶

func RunValidations(ctx context.Context, componentName string, validations []recipe.ComponentValidationConfig, recipeResult *recipe.RecipeResult, bundlerConfig *config.Config) (warnings []string, errors []error)

RunValidations executes all validations for a component and returns warnings and errors. The optional message from the validation config is appended to each warning/error. Severity determines whether check results become warnings or errors.