Documentation
¶
Overview ¶
Package validators provides shared utilities for v2 validator containers.
Index ¶
- Constants
- Variables
- func EmitExtra(extra map[string]string) error
- func EmitExtraOrWarn(extra map[string]string)
- func EmitRuntimeProvenance(p *ctrf.RuntimeProvenance) error
- func FormatBool(b bool) string
- func FormatDecimal(v float64) string
- func FormatSignedDecimal(v float64) string
- func GPUAllocationPolicyExtra(policy string) map[string]string
- func GPUAllocationResolvedExtra(dra, nodeLocal bool) map[string]string
- func ImagePullSecretNames() []string
- func IsSkip(err error) bool
- func ObservationOnlyExtra() map[string]string
- func RecipeDeclares(ctx *Context, component string) bool
- func ResultSchemaExtra() map[string]string
- func Run(checks map[string]CheckFunc)
- func Skip(reason string) error
- func SkipWithReason(code, reason string) error
- func ValidatorNamespace() string
- type Capability
- type CheckFunc
- type Context
- type GatewayDataPlaneProbe
- type InferenceGatewayProfile
Constants ¶
const ( ExtraKeyNodesValidated = "nodesValidated" ExtraKeyNodesTotal = "nodesTotal" ExtraKeyNodesFailed = "nodesFailed" ExtraKeyNodesSkipped = "nodesSkipped" ExtraKeySkipReason = ctrf.ExtraKeySkipReason ExtraKeyRuntimeSource = "runtimeSource" ExtraKeyAchievedGbps = "achievedGbps" ExtraKeyThresholdGbps = "thresholdGbps" ExtraKeyTolerancePct = "tolerancePct" ExtraKeyPassMargin = "passMargin" ExtraKeyWorkerNodes = "workerNodes" ExtraKeyGPUsInvolved = "gpusInvolved" ExtraKeyTransport = "transport" ExtraKeyFabric = "fabric" ExtraKeyDRAReady = "draReady" ExtraKeyDRADesired = "draDesired" ExtraKeyNodewrightComplete = "nodewrightComplete" ExtraKeyNodewrightExpected = "nodewrightExpected" ExtraKeyNamespacesActive = "namespacesActive" ExtraKeyNamespacesExpected = "namespacesExpected" ExtraKeyThroughputTokPerSec = "throughputTokPerSec" ExtraKeyTTFTP99Ms = "ttftP99Ms" ExtraKeyThroughputThreshold = "throughputThreshold" ExtraKeyTTFTThresholdMs = "ttftThresholdMs" ExtraKeyScaledThreshold = "scaledThreshold" ExtraKeyGPUsBenchmarked = "gpusBenchmarked" ExtraKeyGPUsPerNode = "gpusPerNode" ExtraKeyConfiguredPolicy = "configuredPolicy" ExtraKeyResolvedMode = "resolvedMode" ExtraKeyNodeLocal = "nodeLocal" ExtraKeyDetectedVersion = "detectedVersion" ExtraKeyVersionSatisfied = "versionSatisfied" ExtraKeyObservationOnly = ctrf.ExtraKeyObservationOnly ExtraKeyResultSchema = ctrf.ExtraKeyResultSchema )
Extra keys a check may publish with EmitExtra. Each must have a value validator in pkg/evidence/redact's ctrfExtraAllowlist, or minimal redaction drops it. The keys the evidence readers consume by name are declared in pkg/validator/ctrf and re-exported here, so the publisher and the readers share one definition.
const ( SkipReasonNoGPUNodes = "no-gpu-nodes" SkipReasonNoSchedulableGPUNodes = "no-schedulable-gpu-nodes" SkipReasonNodesBusy = "nodes-busy" SkipReasonNoConstraint = "no-constraint" SkipReasonComponentNotInRecipe = "component-not-in-recipe" SkipReasonNoValidationInput = "no-validation-input" SkipReasonUnsupportedTarget = "unsupported-target" SkipReasonProfileLacksVariant = "profile-lacks-variant" SkipReasonUnsupportedService = "unsupported-service" SkipReasonUnrecognizedPlatform = "unrecognized-platform" SkipReasonNoGPUBackedNodeSet = "no-gpu-backed-nodeset" SkipReasonNoVerifiableNodes = "no-verifiable-nodes" SkipReasonDRADriverNotInstalled = "dra-driver-not-installed" SkipReasonSlurmManagedGPUs = "slurm-managed-gpus" SkipReasonSimulatedNodes = "simulated-nodes" SkipReasonNoMetricsBackend = "no-metrics-backend" )
Skip codes published under ExtraKeySkipReason. Each must also be listed in pkg/evidence/redact's ctrfSkipReasons.
const ( GatewayProbeServiceEndpoints = v1.GatewayProbeServiceEndpoints GatewayProbeGatewayAddress = v1.GatewayProbeGatewayAddress DefaultInferenceGatewayName = v1.DefaultInferenceGatewayName )
const ImagePullSecretsEnv = v1.ImagePullSecretsEnv
ImagePullSecretsEnv carries the names of the --image-pull-secret Secrets the orchestrator attached to the validator Job, comma-separated, so a check that creates a per-run namespace can copy them there and let its probe pods pull with the same credentials. It is pkg/validator/v1's constant, which forwards it, so the two ends cannot drift.
Variables ¶
var ( GatewayGVR = schema.GroupVersionResource{Group: "gateway.networking.k8s.io", Version: "v1", Resource: "gateways"} HTTPRouteGVR = schema.GroupVersionResource{Group: "gateway.networking.k8s.io", Version: "v1", Resource: "httproutes"} )
The Gateway API resources the validators read, declared once so a version bump is one edit for every check that reads them.
Functions ¶
func EmitExtra ¶
EmitExtra writes extra to stdout as one ctrf.ExtraLinePrefix line, which the orchestrator parses into ctrf.TestResult.Extra. Keys are the ExtraKey* constants, and values must take the shapes documented on ctrf.TestResult.Extra. Calls merge, a later value winning per key. An empty extra writes nothing.
func EmitExtraOrWarn ¶
EmitExtraOrWarn calls EmitExtra and logs, rather than returns, any error. Use it for informational extras only (counts, measured values, the allocation mode a check exercised); an extra that decides how a result is read goes through EmitExtra with the error returned.
func EmitRuntimeProvenance ¶
func EmitRuntimeProvenance(p *ctrf.RuntimeProvenance) error
EmitRuntimeProvenance writes a derived runtime's RuntimeProvenance record as a single JSON line prefixed with ctrf.ProvenanceLinePrefix. The orchestrator parses it into ctrf.TestResult.RuntimeProvenance, which survives minimal redaction under the bounded rules in pkg/evidence/redact (sha256 digests, template KEYS only — never values). A nil record is a no-op.
func FormatDecimal ¶
FormatDecimal formats v rounded to one decimal place, without trailing zeros ("305.4", "300").
Special cases, all rejected by the redaction allowlist, are:
- a v that rounds below zero keeps its minus sign
- NaN and ±Inf format as "NaN", "+Inf", and "-Inf"
func FormatSignedDecimal ¶
FormatSignedDecimal is FormatDecimal with an explicit sign ("+1.8", "-12.3", "+0").
func GPUAllocationPolicyExtra ¶
GPUAllocationPolicyExtra returns the Extra recording policy as the configured allocation policy. Emit it before verifying policy, so a mismatch still records it.
func GPUAllocationResolvedExtra ¶
GPUAllocationResolvedExtra returns the Extra recording the allocation mode a check exercised, DRA resource claims if dra is set and device-plugin extended resources otherwise. nodeLocal records whether the mode was chosen for one node rather than from cluster-wide capability.
func ImagePullSecretNames ¶
func ImagePullSecretNames() []string
ImagePullSecretNames returns the Secret names ImagePullSecretsEnv carries, trimmed, with empty entries dropped; nil when the variable is unset or names nothing.
func IsSkip ¶
IsSkip reports whether err (or anything in its Unwrap chain) came from Skip. errSkip is unexported, so this is the supported way for callers and tests outside this package to classify a skip without matching on rendered error text.
func ObservationOnlyExtra ¶
ObservationOnlyExtra returns the Extra marking a passed result as observation-only: the check recorded the cluster's arrangement and drove no behavioral test, so its pass is not a demonstration. The CNCF requirement aggregation (pkg/evidence/cncf) renders such a result OBSERVED rather than PASSED and does not let it establish a MUST (#3681). Emit it with EmitExtra and fail the check when the write fails: a lost marker would turn the observation into a demonstration.
func RecipeDeclares ¶
RecipeDeclares reports whether component is present AND enabled in the resolved recipe's ComponentRefs (ctx.ValidationInput). A component that is absent or explicitly disabled is not declared: it will not be deployed, so its capability is genuinely inapplicable and a capability-gated check may Skip. Nil-safe: a nil Context or nil ValidationInput reports false, which is the standalone/no-recipe path that preserves capability-driven selection (#1327).
func ResultSchemaExtra ¶
ResultSchemaExtra returns the Extra stamping the result-contract version this image speaks (ctrf.ResultSchemaVersion). The runner publishes it before the check runs, so every result the orchestrator reads carries it. The CNCF aggregation reads an unmarked pass of an observation-only check as observed with or without it; the stamp only tells it whether to report an image that predates the marker or a marker that went missing (#3703). Emit it with EmitExtra and fail the check when the write fails.
func Run ¶
Run is the main entry point for v2 validator containers. It loads the context, dispatches to the named check, and handles exit codes and termination log writing.
Usage in main.go:
func main() {
validators.Run(map[string]validators.CheckFunc{
"operator-health": checkOperatorHealth,
"expected-resources": checkExpectedResources,
})
}
func Skip ¶
Skip returns a skip sentinel error with the given reason. When returned from a CheckFunc, the runner exits with code 2 (skip).
func SkipWithReason ¶
SkipWithReason publishes code as the skip reason and returns Skip(reason). Only code survives minimal redaction, and the CNCF aggregation reads it as the NOT TESTED reason, so a code that could not be written fails the check instead of skipping it: a skip with no reason would read as an unexplained gap in the evidence.
Errors:
- ErrCodeInternal when the skip reason could not be written to stdout.
func ValidatorNamespace ¶
func ValidatorNamespace() string
ValidatorNamespace returns the namespace the validator pod runs in: the mounted ServiceAccount namespace, else AICR_NAMESPACE, else "default". It is the namespace --image-pull-secret Secrets live in.
Types ¶
type Capability ¶
type Capability struct {
// Component is the recipe componentRef name that supplies this capability
// (e.g. "kai-scheduler"). RecipeDeclares(ctx, Component) decides whether the
// recipe makes the capability applicable.
Component string
// Subject names the probed prerequisite for diagnostics — the concrete
// object/API the probe read (e.g. "kai-scheduler Deployment
// kai-scheduler/kai-scheduler-default"). Used in the classified infra-error
// messages so operators see exactly what could not be read.
Subject string
// AbsentMsg is the actionable operator message emitted when the capability
// is DECLARED but its prerequisite is cleanly missing (NotFound / empty
// result). It should tell the operator how to remediate, e.g. "recipe
// declares kai-scheduler but its Deployment is absent — apply the bundle or
// check RBAC".
AbsentMsg string
// InapplicableMsg is the Skip reason emitted when the capability is NOT
// declared and is therefore genuinely inapplicable (e.g. "KAI scheduler not
// found — cluster may use a different scheduler"). Optional: when empty a
// generic reason is derived from Component and Subject.
InapplicableMsg string
}
Capability describes a recipe-declared capability whose live prerequisite a check probes before proceeding. Require() turns the probe outcome into the correct verdict (proceed / Skip / fail-closed) per the #2122 contract above.
func (Capability) Require ¶
func (c Capability) Require(ctx *Context, probeErr error, present bool) error
Require resolves a capability-gated check's fate from the outcome of probing its live prerequisite:
- probeErr is the error the probe returned (nil on a clean read).
- present reports whether the probe found the prerequisite. It is consulted ONLY when probeErr is nil — e.g. a List that returned zero items, or a discovery call that returned without the expected resource.
Decision table (declared == RecipeDeclares(ctx, c.Component)):
│ recipe DECLARES │ recipe does NOT declare ────────────────────────────┼────────────────────────┼──────────────────────── clean read, present │ nil (proceed) │ nil (proceed) clean read, absent/empty │ FAIL (NotFound) │ Skip probe err: NotFound │ FAIL (NotFound) │ Skip probe err: Forbidden/401 │ FAIL (Unauthorized) — always probe err: timeout/deadline │ FAIL (Timeout) — always probe err: transport/503 │ FAIL (Unavailable) — always probe err: other/discovery │ FAIL (Internal) — always
Infra errors (Forbidden, timeout, transport, API discovery) NEVER Skip, even when the recipe does not declare the component: a missing RBAC grant or an apiserver hiccup is not evidence that the capability is inapplicable. Only a clean NotFound / empty result on a NON-declared capability may Skip (#2122).
func (Capability) RequireList ¶
func (c Capability) RequireList(probeErr error) error
RequireList resolves a capability-gated check that probes a LIST/collection endpoint whose EMPTY (non-error) result — not an error — is what signals inapplicability. Unlike Require, a List *error* is never Skip-eligible: even a NotFound on a collection endpoint is an apiserver/aggregation-layer anomaly, not the clean absence of a single object, so every List error blocks with a classified code (Forbidden→Unauthorized, deadline→Timeout, transport→ Unavailable, else Internal) and never masquerades as an inapplicable Skip. Callers handle the empty-result inapplicability case themselves (e.g. len(items) == 0 → "", nil). This closes the #2122 fail-open where routing a List error through Require would Skip on the (rare but unenforced) NotFound shape for an undeclared capability.
type CheckFunc ¶
CheckFunc is the signature for a v2 validator check function. Return nil for pass, non-nil error for fail, Skip() for skip. Evidence goes to stdout, debug logs to stderr.
type Context ¶
type Context struct {
// Ctx is the parent context with timeout. It is also canceled when the
// container receives SIGTERM (Job deadline, eviction, or node drain), so a
// check that honors it runs its deferred cleanup inside the termination
// grace period. Cleanup must therefore use a fresh context, never Ctx.
Ctx context.Context
// Cancel releases resources, including the SIGTERM subscription. Must be
// called when done.
Cancel context.CancelFunc
// Clientset is the Kubernetes typed client.
Clientset kubernetes.Interface
// RESTConfig is the Kubernetes REST config (for exec, dynamic client, etc.).
RESTConfig *rest.Config
// DynamicClient is the Kubernetes dynamic client for CRD access.
DynamicClient dynamic.Interface
// Snapshot is the captured cluster state.
Snapshot *snapshotter.Snapshot
// ValidationInput is the validation specification (config + context).
ValidationInput *v1.ValidationInput
// Namespace is the validation namespace.
Namespace string
// NodeSelector overrides platform-specific node selectors on inner workloads
// (e.g., NCCL benchmark worker pods). Nil means use the validator's default selectors.
// Set from the AICR_NODE_SELECTOR env var (comma-separated key=value pairs).
NodeSelector map[string]string
// Tolerations overrides the default tolerate-all policy on inner workloads.
// Nil means use the validator's default tolerations.
// Set from the AICR_TOLERATIONS env var (comma-separated key=value:effect entries).
Tolerations []corev1.Toleration
}
Context holds all dependencies for a validator check function.
func LoadContext ¶
LoadContext creates a Context from the v2 container environment. Reads snapshot and recipe from mounted ConfigMap paths. Builds a K8s client from in-cluster config or KUBECONFIG.
The caller MUST call ctx.Cancel() when done.
type GatewayDataPlaneProbe ¶
type GatewayDataPlaneProbe = v1.GatewayDataPlaneProbe
The inference gateway profile and its vocabulary live in pkg/validator/v1 beside the allocation-policy resolver, so the SDK facade (CNCF evidence collection) and the validator binaries read one declaration. The aliases keep check code on the package it already imports.
type InferenceGatewayProfile ¶
type InferenceGatewayProfile = v1.InferenceGatewayProfile
The inference gateway profile and its vocabulary live in pkg/validator/v1 beside the allocation-policy resolver, so the SDK facade (CNCF evidence collection) and the validator binaries read one declaration. The aliases keep check code on the package it already imports.
func DefaultInferenceGatewayProfile ¶
func DefaultInferenceGatewayProfile() InferenceGatewayProfile
DefaultInferenceGatewayProfile is the built-in agentgateway profile, the target of a validator that runs without a recipe.
func ResolveInferenceGatewayProfile ¶
func ResolveInferenceGatewayProfile(ctx *Context) (InferenceGatewayProfile, bool, error)
ResolveInferenceGatewayProfile resolves the recipe's inference gateway from the validation input. Nil-safe: a nil Context or no recipe reports not found, which is the standalone path.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
conformance is a validator container for all conformance phase checks.
|
conformance is a validator container for all conformance phase checks. |
|
deployment is a validator container for all deployment phase checks.
|
deployment is a validator container for all deployment phase checks. |
|
Package helper provides shared utilities for v2 validator containers.
|
Package helper provides shared utilities for v2 validator containers. |
|
internal
|
|
|
gkenet
Package gkenet discovers the GKE multi-NIC networking objects that GPUDirect TCPXO depends on.
|
Package gkenet discovers the GKE multi-NIC networking objects that GPUDirect TCPXO depends on. |
|
performance is a validator container for all performance phase checks.
|
performance is a validator container for all performance phase checks. |