Documentation
¶
Overview ¶
Package validator evaluates a recipe's constraints and validation checks against a cluster snapshot and the live cluster.
The validator runs in two phases:
Readiness pre-flight: top-level constraint expressions, plus any declared under validation.readiness.constraints, are evaluated against the snapshot inline (no cluster access required). A malformed expression fails closed so misconfigured rules cannot masquerade as passing.
In-cluster checks: each declared check is materialized as a short-lived Kubernetes Job. RBAC (ServiceAccount + a per-run ClusterRoleBinding to the built-in cluster-admin ClusterRole) is provisioned via server-side apply under the "aicr" field manager so concurrent validators converge on a single owner, unless WithServiceAccountName names an administrator-provisioned ServiceAccount, in which case the run creates no RBAC and verifies that ServiceAccount's cluster-admin grant instead. Job logs and exit codes are aggregated into a CTRF-formatted report.
Test isolation is mandatory: callers operating without cluster credentials must pass WithNoCluster(true) (or --no-cluster on the CLI). In that mode constraint evaluation still runs, but RBAC creation and Job deployment are skipped and each check is reported as "skipped - no-cluster mode".
Subpackages:
- catalog: built-in validation check catalog
- ctrf: Common Test Report Format emitter
- job: Job lifecycle, RBAC, and result extraction
- labels: standard label/annotation set applied to validator resources
- v1: Validation YAML schema and decoders
Shared constraint extraction helpers live in the top-level pkg/constraints package, not under validator.
Package validator provides a container-per-validator execution engine for AICR cluster validation. Each validator is an OCI container image run as a Kubernetes Job, communicating results via exit codes and termination messages.
Index ¶
- Constants
- Variables
- func EnsureDataConfigMaps(ctx context.Context, clientset kubernetes.Interface, namespace string, ...) error
- func WarnPhasesAgainstRecipe(phases []Phase, rec *recipe.RecipeResult)
- type Option
- func WithCleanup(cleanup bool) Option
- func WithCommit(commit string) Option
- func WithDataProvider(dp recipe.DataProvider) Option
- func WithFailFast(failFast bool) Option
- func WithImagePullSecrets(secrets []string) Option
- func WithImageRegistryOverride(override string) Option
- func WithImageTagOverride(override string) Option
- func WithKubeconfig(kubeconfig string) Option
- func WithNamespace(namespace string) Option
- func WithNoCluster(noCluster bool) Option
- func WithNodeSelector(nodeSelector map[string]string) Option
- func WithRunID(runID string) Option
- func WithServiceAccountName(name string) Option
- func WithSkipChecks(names ...string) Option
- func WithTolerations(tolerations []corev1.Toleration) Option
- func WithVersion(version string) Option
- type Phase
- type PhaseResult
- type Validator
- func (v *Validator) PreflightServiceAccount(ctx context.Context) error
- func (v *Validator) PreflightSkipChecks(ctx context.Context, phases []Phase, validationInput *v1.ValidationInput) error
- func (v *Validator) ValidatePhase(ctx context.Context, phase Phase, validationInput *v1.ValidationInput, ...) (result *PhaseResult, err error)
- func (v *Validator) ValidatePhases(ctx context.Context, phases []Phase, validationInput *v1.ValidationInput, ...) (results []*PhaseResult, err error)
Constants ¶
const ( PhaseDeployment = v1.PhaseDeployment PhasePerformance = v1.PhasePerformance PhaseConformance = v1.PhaseConformance )
Re-exported phase constants from pkg/validator/v1.
const PhaseAll = "all"
PhaseAll is the wildcard string accepted by both the `aicr validate --phase` CLI flag and the spec.validate.execution.phases config field to mean "run every phase." It is not a Phase value — the CLI parser collapses it into a nil selection that ValidatePhases interprets as "run all phases."
const SkipCheckReasonCode = "named-in-skip-checks"
SkipCheckReasonCode is the same fact as skipCheckReason in the one channel a signed evidence bundle preserves. The default (minimal) redaction policy blanks TestResult.Message for every test, so a reason carried only in the message reaches the attestation as the empty string: a bundle that records WHICH check was withheld but not WHY. The code rides TestResult.Extra under the allowlisted `skipReason` key instead, and is listed in pkg/evidence/redact's ctrfSkipReasons closed set; adding a code there and minting it here is one change, per that package's contract.
It is distinct from the codes validators mint for themselves (validators/deployment/nvidia_smi.go) precisely so an auditor can tell a caller-declared skip from a check that found its own capability inapplicable.
Variables ¶
var PhaseNames = []string{ string(PhaseDeployment), string(PhasePerformance), string(PhaseConformance), PhaseAll, }
PhaseNames is the canonical user-facing vocabulary accepted by the --phase flag and spec.validate.execution.phases. The typed Phase constants in PhaseOrder plus the PhaseAll wildcard. Single source of truth so the CLI parser and the config-load validator stay in sync when a phase is added or removed.
var PhaseOrder = []Phase{PhaseDeployment, PhaseConformance, PhasePerformance}
PhaseOrder defines the execution order for ValidatePhases. All phases run by default; set Validator.FailFast to stop after the first phase that reports StatusFailed.
Order rationale: deployment (cheap install/health checks) → conformance → performance. Performance runs LAST because its inference-perf benchmark saturates every GPU on the node and the DynamoGraphDeployment teardown releases those DRA ResourceClaims asynchronously; running it before conformance starved conformance's GPU-needing checks (e.g. dra-support, whose 1-GPU test pod failed "cannot allocate all claims" on single-node clusters). Running performance last also keeps a flaky perf phase from blocking conformance under FailFast.
Note: Readiness phase is NOT included. It remains in pkg/validator and uses inline constraint evaluation (no containers).
Functions ¶
func EnsureDataConfigMaps ¶
func EnsureDataConfigMaps( ctx context.Context, clientset kubernetes.Interface, namespace string, runID string, snap *snapshotter.Snapshot, validationInput *v1.ValidationInput, ) error
EnsureDataConfigMaps creates or updates snapshot and validation ConfigMaps. Creates ConfigMaps named aicr-snapshot-{runID} and aicr-validation-{runID} with create-or-update semantics. External controllers should call this after generating a runID and before rendering validator Jobs. The Jobs mount these ConfigMaps at /data/snapshot and /data/validation.
func WarnPhasesAgainstRecipe ¶
func WarnPhasesAgainstRecipe(phases []Phase, rec *recipe.RecipeResult)
WarnPhasesAgainstRecipe warns when a requested phase has no checks defined in the recipe. The phase will still run but produce 0 tests in the CTRF report. This is purely advisory — it emits slog warnings and never fails the run.
Types ¶
type Option ¶
type Option func(*Validator)
Option is a functional option for configuring Validator instances.
func WithCleanup ¶
WithCleanup controls whether to delete Jobs, ConfigMaps, and RBAC after validation. Default: true.
func WithCommit ¶
WithCommit sets the git commit SHA (typically the CLI build commit). Used for resolving dev-build validator images to SHA-tagged images.
func WithDataProvider ¶
func WithDataProvider(dp recipe.DataProvider) Option
WithDataProvider binds the recipe DataProvider used to load the validator catalog. When unset, the catalog loads from the package-global provider.
func WithFailFast ¶
WithFailFast controls whether ValidatePhases stops after the first phase whose status is blocking per ctrf.IsFailingStatus (StatusFailed or StatusOther — an inconclusive outcome fails closed). Default: false (all phases run regardless of earlier failures). Set true to restore the historical fail-fast behavior.
func WithImagePullSecrets ¶
WithImagePullSecrets sets image pull secrets for validator Jobs.
func WithImageRegistryOverride ¶
WithImageRegistryOverride sets the image registry prefix override for validator container images. When non-empty, replaces the default registry (e.g., ghcr.io/nvidia) with the specified prefix (e.g., localhost:5001). Forwarded to validator Jobs via the AICR_VALIDATOR_IMAGE_REGISTRY env var.
func WithImageTagOverride ¶
WithImageTagOverride sets the image tag override for validator container images. When non-empty, overrides the resolved tag on every validator image. Intended for feature-branch dev builds whose commit SHA has no published image. Forwarded to validator Jobs via the AICR_VALIDATOR_IMAGE_TAG env var.
func WithKubeconfig ¶
WithKubeconfig sets an explicit, run-scoped kubeconfig path for all Kubernetes API operations performed by the validation run. The file is reloaded for each run. Empty uses the shared default Kubernetes client and its standard KUBECONFIG, ~/.kube/config, then in-cluster discovery chain.
func WithNamespace ¶
WithNamespace sets the Kubernetes namespace for validation Jobs. Default: "aicr-validation".
func WithNoCluster ¶
WithNoCluster controls cluster access. When true, all validators are reported as skipped and no K8s API calls are made. Default: false.
func WithNodeSelector ¶
WithNodeSelector sets node selector labels to override inner workload scheduling. When set, validators pass these selectors to the workloads they create (e.g., NCCL benchmark pods), replacing platform-specific defaults. Does not affect the orchestrator Job.
func WithRunID ¶
WithRunID sets the RunID for this validation run. Used when resuming a previous run.
func WithServiceAccountName ¶
WithServiceAccountName runs validator Jobs as an existing, administrator-provisioned ServiceAccount instead of a per-run one. See Validator.ServiceAccountName. Default: empty (per-run RBAC).
func WithSkipChecks ¶
WithSkipChecks names checks to withhold from every requested phase. Each is reported as skipped rather than dropped, and the run is rejected up front if a name matches no validator in the catalog or if the list would empty a requested phase. Default: none. See skip_checks.go.
func WithTolerations ¶
func WithTolerations(tolerations []corev1.Toleration) Option
WithTolerations sets tolerations to override inner workload scheduling. When set, validators pass these tolerations to the workloads they create (e.g., NCCL benchmark pods), replacing default tolerate-all policy. Does not affect the orchestrator Job.
func WithVersion ¶
WithVersion sets the validator version string (typically the CLI version).
type Phase ¶
Phase re-exports pkg/validator/v1.Phase so callers that work in the pkg/validator orchestration layer do not have to import the wire package directly.
func ParsePhase ¶
ParsePhase converts a user-facing phase name to its typed Phase value. Returns false for PhaseAll (the wildcard, which has no Phase value) and for unrecognized inputs. Callers that want to accept the wildcard handle it separately, typically by collapsing the whole selection to nil (= run every phase).
func ParsePhaseSelection ¶
ParsePhaseSelection parses a list of user-facing phase names (from the `--phase` CLI flag or the spec.validate.execution.phases config field) into typed Phase values. The PhaseAll wildcard collapses the whole selection to nil (= run every phase), matching the documented "Default: all phases" behavior. PhaseAll is exclusive: combining it with any specific phase is a hard error rather than silently treating the selection as wildcard, so a typo like `--phase deployment --phase all` does not mask the user's mistake.
Every entry is parsed before the wildcard collapse, so an invalid phase name surfaces an error even when "all" is also present.
type PhaseResult ¶
type PhaseResult struct {
// Phase is the phase that was executed.
Phase Phase
// Status is the overall phase status derived from the CTRF summary.
Status string
// Report is the CTRF report for this phase.
Report *ctrf.Report
// Duration is the wall-clock time for the entire phase.
Duration time.Duration
}
PhaseResult is the outcome of running all validators in a single phase.
type Validator ¶
type Validator struct {
// Version is the validator version (typically the CLI version).
Version string
// Commit is the git commit SHA from the CLI build. Used to resolve
// dev-build validator images to SHA-tagged images pushed by on-push CI.
Commit string
// Kubeconfig is an optional, run-scoped path overriding the Kubernetes
// client configuration used for validation namespace, RBAC, ConfigMap, Job,
// and result operations. Empty uses the shared default client.
Kubeconfig string
// Namespace is the Kubernetes namespace for validation Jobs.
Namespace string
// RunID is a unique identifier for this validation run.
RunID string
// Cleanup controls whether to delete Jobs, ConfigMaps, and RBAC after validation.
Cleanup bool
// ServiceAccountName, when non-empty, names an existing ServiceAccount in
// Namespace that validator Jobs run as verbatim. The run then creates and
// deletes no ServiceAccount or ClusterRoleBinding and does not apply the
// namespace: an administrator provisions all three, binding the
// ServiceAccount to cluster-admin, and the run verifies that binding
// before deploying anything. Empty (the default) creates a per-run
// ServiceAccount bound to cluster-admin and deletes both at cleanup.
ServiceAccountName string
// ImagePullSecrets are secret names for pulling validator images.
ImagePullSecrets []string
// NoCluster controls whether to skip cluster operations (dry-run mode).
NoCluster bool
// Tolerations are passed to validation workloads (e.g., NCCL benchmark pods)
// to override their default scheduling constraints. Does not affect the
// orchestrator Job itself.
Tolerations []corev1.Toleration
// NodeSelector is passed to validation workloads (e.g., NCCL benchmark pods)
// to override platform-specific node selectors. Use when GPU nodes have
// non-standard labels. Does not affect the orchestrator Job itself.
NodeSelector map[string]string
// ImageRegistryOverride, when non-empty, replaces the registry prefix
// of all validator container images. Forwarded to the validator Job's
// container env as AICR_VALIDATOR_IMAGE_REGISTRY so inner workloads
// (e.g., AIPerf benchmark images) resolve from the same registry.
ImageRegistryOverride string
// ImageTagOverride, when non-empty, overrides the resolved image tag
// of all validator container images. Forwarded to the validator Job's
// container env as AICR_VALIDATOR_IMAGE_TAG. Intended for feature-branch
// dev builds whose commit SHA has no published image; typical value: "latest".
ImageTagOverride string
// FailFast, when true, stops validation after the first phase that reports
// StatusFailed. By default (false) all phases run and produce results.
FailFast bool
// SkipChecks names checks the caller has declared out of scope for this
// run: they are withheld from every requested phase and recorded in that
// phase's report as skipped. Empty (the default) runs every check the
// validation declares. See skip_checks.go for what this is for, why it is
// a skip list rather than an allow list, and the two guards that keep it
// from turning a red run green.
SkipChecks []string
// contains filtered or unexported fields
}
Validator orchestrates validation runs using containerized validators.
func (*Validator) PreflightServiceAccount ¶
PreflightServiceAccount verifies the administrator-provisioned ServiceAccount named by ServiceAccountName exactly as a run does before deploying anything, so a caller that does its own cluster work first can fail before it. It issues only reads, and returns nil without contacting the cluster when ServiceAccountName is empty or NoCluster is set.
func (*Validator) PreflightSkipChecks ¶
func (v *Validator) PreflightSkipChecks( ctx context.Context, phases []Phase, validationInput *v1.ValidationInput, ) error
PreflightSkipChecks runs the skip-list guard on its own, loading the catalog the same way ValidatePhases does, for a caller that performs cluster work of its own BEFORE it reaches ValidatePhases.
The CLI is that caller: with neither --snapshot nor --no-cluster, `aicr validate` deploys a snapshot-capture agent before it ever constructs the validator, so the guard inside ValidatePhases fires only after a ServiceAccount, a Role and a Job exist. The --skip-check help text promises the opposite ("Rejected before any validation resource is created"), and this is what lets the CLI keep that promise.
It does not replace the ValidatePhases and ValidatePhase calls: an SDK or server caller reaches those directly and must stay guarded there. Calling both is idempotent, since the guard only reads.
Returns nil immediately when the skip list is empty, so a caller on the default path pays nothing, not even a catalog load. Empty phases means the full PhaseOrder, matching ValidatePhases, so the emptied-phase arm sees the same phase set in both places.
func (*Validator) ValidatePhase ¶
func (v *Validator) ValidatePhase( ctx context.Context, phase Phase, validationInput *v1.ValidationInput, snap *snapshotter.Snapshot, ) (result *PhaseResult, err error)
ValidatePhase runs a single validation phase. The readiness pre-flight runs first, exactly as in ValidatePhases: per-phase SDK callers must not be able to execute a phase against a cluster that fails the recipe's readiness gates (e.g. the GKE device-plugin ownership constraint, issue #1755).
func (*Validator) ValidatePhases ¶
func (v *Validator) ValidatePhases( ctx context.Context, phases []Phase, validationInput *v1.ValidationInput, snap *snapshotter.Snapshot, ) (results []*PhaseResult, err error)
ValidatePhases runs the specified phases sequentially and returns one PhaseResult per phase. Pass nil or empty phases to run all phases. By default all phases run and produce results regardless of failures. Set FailFast to stop after the first phase that reports StatusFailed.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package catalog provides the declarative validator catalog.
|
Package catalog provides the declarative validator catalog. |
|
Package ctrf provides Go types and utilities for the Common Test Report Format (CTRF).
|
Package ctrf provides Go types and utilities for the Common Test Report Format (CTRF). |
|
Package labels provides shared label constants for validation resources.
|
Package labels provides shared label constants for validation resources. |
|
Package v1 defines AICR's validator input format (v1alpha1).
|
Package v1 defines AICR's validator input format (v1alpha1). |