platform

func ApplyEnvToJobTemplate ¶ added in v0.8.0

func ApplyEnvToJobTemplate(
	jt *nvcrev1alpha1.JobTemplateSpec, deps []nvcrev1alpha1.DependencySpec, env []corev1.EnvVar,
) error

ApplyEnvToJobTemplate writes the owner's resolved environment variables onto a resolved job template, the sibling of ApplyImageToJobTemplate for the Certification tier's spec.env and spec.categories[].options.env (ADR-097). It is a no-op when env is empty, and it never creates structure: a template without a trainJob workload or without a trainer has nowhere to put a variable and is left alone.

Where a variable lands depends on how the entry launches its ranks, and the two surfaces are not interchangeable:

• MPI entries get "-x NAME=VALUE" pairs in trainer.args. Their ranks start under sshd with a fresh session environment and never read container env, and their trainjob-ancestor-step label sits on the launcher replicated job, so a trainer env var would reach mpirun's own container and no rank.
• Every other entry gets a trainer env var. Kubeflow upserts it by name onto the container named "node" in the ancestor replicated job, which is where the workload actually runs, and that upsert already wins over the dependency runtime's own value. No dependency rewrite is needed.

The shape is read from the referenced TrainingRuntime's mlPolicy rather than guessed from the argument list: "has a -x pair" conflates being MPI with having an anchor, and an MPI arm with no -x pairs would then be silently written to an environment its ranks never read — the invisible no-op this whole field exists to remove.

func ApplyGangSchedulerToDependencies ¶ added in v0.2.0

func ApplyGangSchedulerToDependencies(deps []nvcrev1alpha1.DependencySpec, gs *nvcrev1alpha1.GangSchedulerSpec) error

ApplyGangSchedulerToDependencies rewrites every TrainingRuntime dependency in place so each of its replicatedJobs runs under the configured gang scheduler. For each replicatedJob it sets schedulerName on the pod spec and the queue label (gangScheduler.queueLabelKey, "kai.scheduler/queue" when unset) on both the job template metadata and the pod template metadata, matching what BuildTorchRuntime and BuildMPIRuntime already emit for a WorkloadRun. The pod-level copy keeps queue assignment from depending on the Trainer/JobSet layer propagating template metadata onto the pods.

For the KAI + MPI shape it also rewrites the JobSet ordering: it deletes the launcher's dependsOn gate together with the JobSet startupPolicy (the two are mutually exclusive) and appends the wait-for-workers init container, because KAI requires every sub-group of the PodGroup to have pods before the group is schedulable — an ordered or gated launcher sub-group is empty until the workers are ready and can never be admitted. The wait therefore lives inside the launcher pod (launcherWaitScript); a dependency between replicated jobs that is not the launcher's node-ready gate keeps its meaning (measured live on KAI v0.16.4; see schedulerNameKAI).

It is a no-op when gs is nil, so a Certification that does not ask for gang scheduling renders byte-identically to before. It is also a no-op when the scheduler name is empty, matching applyGangScheduler: the CRD requires a non-empty schedulerName, but nvcrectl renders straight from a file without consulting the API server, so without this guard a typo would render pod templates pinned to an empty scheduler and carrying a stray queue label.

Callers must invoke this after overrides are resolved. The scheduler name is overwritten unconditionally rather than filled in only when absent, because some catalog entries hardcode schedulerName: default-scheduler and the whole point of the field is to replace it.

func ApplyImageToDependencies ¶ added in v0.3.0

func ApplyImageToDependencies(deps []nvcrev1alpha1.DependencySpec, image string) error

ApplyImageToDependencies rewrites every TrainingRuntime dependency in place so the image of the primary workload container, containers[0], of each replicatedJob's pod template is the given image. Init containers whose image exactly equals the primary's pre-override image follow it: the catalog derives those inits from the workload ref (the MPI launcher's fix-ssh-permissions, the training entries' megatron-clone), so they must run whatever image the workload runs, including any source the operator provisioned inside it. Init containers with a distinct image are infrastructure and stay pinned by construction (the GCP tcpxo-daemon init container keeps GCP's RxDM image). Non-init containers beyond index 0 are never touched, whatever their image.

It is a strict no-op when image is empty, so a Certification that never set options.image renders byte-identically to before, matching ApplyGangSchedulerToDependencies's guard. Unlike that helper's ensureMap labels write, it never creates structure: a replicatedJob with no containers list is left alone, byte for byte.

Callers must invoke this after overrides are resolved, at the same point gang scheduling is applied: platform overrides choose images too (the AWS EFA overrides swap the workers to an nccl-tests build), and the whole point of options.image is to replace whatever the catalog landed on. It is a deliberate structural twin of ApplyGangSchedulerToDependencies rather than a shared walker; a third post-resolve rewriter (issue #212 is the candidate) is the refactor trigger.

func ApplyImageToJobTemplate ¶ added in v0.3.0

func ApplyImageToJobTemplate(jt *nvcrev1alpha1.JobTemplateSpec, image string)

ApplyImageToJobTemplate points the job template's trainer at the given image (jobTemplate.spec.workload.trainJob.trainer.image), the sibling of ApplyImageToDependencies for the typed half of the workload. It is a no-op when image is empty, and it never creates structure: a template without a trainJob workload or without a trainer has no trainer image to replace and is left alone.

func ApplyResolvedWorkflowTransforms ¶ added in v0.5.0

func ApplyResolvedWorkflowTransforms(
	spec *nvcrev1alpha1.WorkflowSpec, t ResolvedWorkflowTransforms,
) error

ApplyResolvedWorkflowTransforms applies every post-resolve transform to a resolved WorkflowSpec in one named stage, then persists the gang-scheduling intent and checks the result.

It exists because a third post-resolve rewrite landed: ADR-077 deliberately left gang scheduling and image replacement as two focused helpers called in sequence at three call sites, and named this the refactor trigger. Those helpers and their tests are unchanged; this composes them.

The order is pinned:

1. ApplyGangSchedulerToDependencies
2. ApplyImageToJobTemplate
3. ApplyImageToDependencies
4. ApplyEnvToJobTemplate
5. resolve and write JobTemplate.spec.workloadMetadata

Gang scheduling, image replacement and the environment write touch disjoint fields, so only step 5's position matters: writing workload metadata last makes the resolved catalog and override JobSpec the base map, which makes its precedence unambiguous.

It then records the resolved intent on spec.gangScheduler and runs ValidateResolvedJobTemplate, so a caller cannot forget either.

func ApplyWorkloadRunScheduling ¶ added in v0.5.0

func ApplyWorkloadRunScheduling(
	spec *nvcrev1alpha1.WorkflowSpec,
	gs *nvcrev1alpha1.GangSchedulerSpec,
	md *nvcrev1alpha1.WorkloadMetadata,
) error

ApplyWorkloadRunScheduling composes the workload-object metadata and records the resolved gang-scheduling intent on a WorkflowSpec that WorkloadRun just built. It is the construction-time half of the same contract ApplyResolvedWorkflowTransforms provides for Certification.

WorkloadRun needs no ApplyGangSchedulerToDependencies call: its runtime dependency is built with the scheduler and queue already in place by BuildTorchRuntime, BuildMPIRuntime and BuildExecRuntime, unlike a catalog runtime that ADR-076 has to rewrite after overrides resolve.

This merge is provisional. WorkloadRun's structural overrides run later and may change either metadata level, so the persisted intent is what makes the Workflow controller's post-override check authoritative.

func BaseNCCLEnvVars ¶

func BaseNCCLEnvVars(enableMNNVL bool) []corev1.EnvVar

BaseNCCLEnvVars returns model-independent NCCL environment variables that are safe to auto-inject for any workload. These are common across all training models and NCCL tests.

Platform-specific NCCL vars (FastRak, IB settings) are injected via overrides, not here.

NCCL_NVLS_ENABLE is deliberately absent. NCCL's own default of 2 is the only value that probes CU_DEVICE_ATTRIBUTE_MULTICAST_SUPPORTED before using NVLink SHARP; forcing 1 asserts support on hardware that may not have it, and on NCCL older than 2.29 it is the one value that turns a failed multicast bind into an aborted job rather than a logged fallback. A site that needs NVLS off sets it through spec.env or mpiArgs, which take precedence over everything here.

Leaving it out here does not mean no render sets it. Several platform fragments set 1 for their own hardware (_lib/nccl/onprem-ib-env.yaml, nscale-ib-env.yaml, togetherai-ib-env.yaml, aws-gb300-roce-transport-env.yaml among them), and those values are field-tuned for fabrics this function knows nothing about. What this changes is the default a workload gets when no fragment has an opinion: NCCL's, rather than ours.

func BuildExecRuntime ¶

func BuildExecRuntime(cfg RuntimeConfig) nvcrev1alpha1.DependencySpec

BuildExecRuntime creates a TrainingRuntime dependency for arbitrary command execution. Uses a simple single-replicatedJob layout with torch mlPolicy.

func BuildMPIRuntime ¶

func BuildMPIRuntime(cfg RuntimeConfig) nvcrev1alpha1.DependencySpec

BuildMPIRuntime creates TrainingRuntime dependencies for MPI-based workloads. Generates: - A runtime with MPI mlPolicy and launcher+node replicatedJobs - Worker nodes with sshd, capabilities, an SSH readiness probe, and cfg.Env - Launcher with mpirun, SSH key setup, and cfg.Env

The worker's openssh-server install (sshdInstallCommand) is guarded by `test -x /usr/sbin/sshd`, so images that already ship sshd start without any package-manager egress (issue #319: air-gapped and restricted-egress clusters). The guard tests the exact path the chain execs (/usr/sbin/sshd); a PATH lookup could disagree with it in either direction.

cfg.Env goes on both containers as container-level env, the same way BuildTorchRuntime emits it (issue #68: it used to be dropped here, so spec.env behaved differently between the two frameworks). On the launcher the variables reach mpirun directly; on the worker they reach the sshd process, but sshd gives each SSH session a fresh environment, so ranks launched through it may not inherit them. A variable that must reach the ranks should be passed as "-x NAME=value" in mpiArgs, which forwards it through mpirun itself.

cfg.Volumes, cfg.VolumeMounts and cfg.InitContainers go on the worker pod and its "node" container, after the built-in mpi-ssh-auth and dshm entries, the same way BuildTorchRuntime emits them (issue #455: they used to be dropped here, so spec.volumes, spec.volumeMounts, spec.config and spec.initContainers were a silent no-op on the mpi framework). The launcher gets none of them, with one exception below. The ranks run on the worker (mpirun on the launcher starts them there over ssh; runLauncherAsNode is off, so the launcher hosts no rank), and the launcher is a coordinator with fixed resources (cpu 2, memory 1Gi, no GPU; spec.resources never applies to it), so a hugepages emptyDir there would fail at volume setup on the kubelet, which refuses the medium without a hugepages request on the pod, and a ReadWriteOnce PVC could not attach on a second node. This matches the catalog precedent in pkg/catalog/entries/_lib/overrides/hostpath-rdma.yaml and ADR-088, which scope the RDMA hostPath mount to the node container.

The exception is the spec.config volume (ConfigVolumeName), which the launcher container also mounts, read-only. mpirun runs on the launcher, so a rankfile, an appfile or an MCA parameter file under /config that mpiArgs names is only usable if the launcher sees it, and a binary under /config is then present where mpirun resolves it instead of relying on mpirun not checking for it locally. None of the reasons above apply to a ConfigMap, which mounts on any node and requests nothing. Env is different again: the launcher's own MPI initialisation reads it, so it stays on both containers.

func BuildTorchRuntime ¶

func BuildTorchRuntime(cfg RuntimeConfig) nvcrev1alpha1.DependencySpec

BuildTorchRuntime creates a TrainingRuntime dependency for PyTorch distributed training. Generates a runtime with torch mlPolicy and a single "node" replicatedJob.

func MPIEnvArgs ¶ added in v0.7.0

func MPIEnvArgs(env []corev1.EnvVar, mpiArgs []string) []string

MPIEnvArgs returns mpirun -x args forwarding env to the MPI ranks, which start under sshd with a fresh environment and never see container env. Names mpiArgs already forwards are skipped, so platform and user mpiArgs keep precedence over env.

func MergeEnvVars ¶

func MergeEnvVars(base, user []corev1.EnvVar) []corev1.EnvVar

MergeEnvVars merges base env vars with user-provided env vars. User-provided values take precedence (override by name).

func Names ¶

func Names() []string

Names returns every valid platform name, in the order used in flag help and error messages.

func NamesList ¶

func NamesList() string

NamesList returns the valid platform names joined with ", " for flag help and error messages.

func OverridesFS ¶ added in v0.7.0

func OverridesFS() embed.FS

OverridesFS returns the embedded platform overrides filesystem, which declares dependencies of its own. Exported so test/helm can derive the set of dependency kinds the manager role has to grant, rather than restating it.

func PreserveSchedulingFields ¶ added in v0.5.0

func PreserveSchedulingFields(
	original, renamed []byte, gs *nvcrev1alpha1.GangSchedulerSpec,
) ([]byte, error)

PreserveSchedulingFields copies the gang-scheduling fields ADR-076 writes from original into renamed, and returns the corrected document.

It exists because per-job dependency renaming rewrites every quoted occurrence of a dependency name, which also catches a queue label value that happens to equal one — a queue named after the runtime it serves. The queue a user configured is not a reference and must survive renaming intact.

Crucially this preserves rather than repairs. Unlike ApplyGangSchedulerToDependencies it asserts nothing and inserts nothing: a field absent from original stays absent, and a value the operator overrode to something else is carried through unchanged. An override that redirects the queue therefore survives renaming and is rejected by validation, which is the reject-without-repair contract. Re-applying the configured intent here instead would silently overwrite that override and let the Job run in a queue the operator did not choose.

It is a no-op when gs configures nothing, since then there is no queue label key to identify.

func ResolvedGangScheduler ¶ added in v0.5.0

func ResolvedGangScheduler(gs *nvcrev1alpha1.GangSchedulerSpec) *nvcrev1alpha1.GangSchedulerSpec

ResolvedGangScheduler returns a copy of gs with queue and queueLabelKey filled in with the defaults the write paths already apply, so the value can be persisted on a Workflow and later compared without re-deriving them. It returns nil when gs configures nothing, matching the guard in ApplyGangSchedulerToDependencies: the CRD requires a non-empty schedulerName, but nvcrectl renders straight from a file without consulting the API server.

func ValidateFlag ¶

func ValidateFlag(name string) error

ValidateFlag validates a --platform flag value. An empty value is allowed (platform not specified). The error text derives the valid names from Names() so the message always matches the actual set.

func ValidateResolvedGangScheduling ¶ added in v0.5.0

func ValidateResolvedGangScheduling(
	js *nvcrev1alpha1.JobSpec,
	deps []nvcrev1alpha1.DependencySpec,
	gs *nvcrev1alpha1.GangSchedulerSpec,
	workloadLabelsPath string,
) error

ValidateResolvedGangScheduling checks that the effective manifests NVCRE is about to submit still place the workload in the queue its owner configured, and restores the one label NVCRE itself owns.

The asymmetry between the two halves is deliberate:

• The workload-object queue label is NVCRE's to write. It follows the insert/same/conflict rule, so an override that removed it gets it back and an override that changed it fails.
• Runtime queue labels and pod scheduler names are assertions over dependency payloads the operator may have redirected. A missing or conflicting runtime value fails and is never repaired, because silently rewriting someone's runtime override would hide the disagreement rather than surface it.

js is the working copy used to create children, not the stored Workflow, so restoring the label here does not touch the Workflow's immutable metadata.

A nil gs is a no-op. Callers wanting the checks that apply whatever the scheduling configuration should use ValidateResolvedJobTemplate instead.

gs is re-resolved rather than trusted. NVCRE persists it already resolved, but queue and queueLabelKey are optional on the CRD, so a hand-authored Workflow can name only a schedulerName. Defaulting here keeps that case checking the same queue the write paths would have used instead of an empty key. Re-resolving an already-resolved value changes nothing.

func ValidateResolvedJobTemplate ¶ added in v0.5.0

func ValidateResolvedJobTemplate(
	js *nvcrev1alpha1.JobSpec,
	deps []nvcrev1alpha1.DependencySpec,
	gs *nvcrev1alpha1.GangSchedulerSpec,
	workloadLabelsPath string,
) error

ValidateResolvedJobTemplate checks a resolved Job spec against everything NVCRE promises about it, and is the entry point every post-override call site should use.

Workload metadata is validated whatever the scheduling configuration. Admission covers a Job or Workflow that reaches an API server, but offline `nvcrectl render` never does, and an override can introduce a reserved key or an oversized map after the owning resource was admitted. Gating that check behind gangScheduler, as ValidateResolvedGangScheduling alone does, would let resolved offline output carry labels the cluster will reject.

The gang-scheduling contract is then checked when an intent is persisted.

func ValidateWorkloadInitContainers ¶ added in v0.8.0

func ValidateWorkloadInitContainers(initContainers []corev1.Container) error

ValidateWorkloadInitContainers rejects a spec.initContainers entry that takes the workload container's name. Every builder names that container "node" on the pod the init containers join, and a pod's container names are unique across its init and regular containers, so the duplicate is otherwise caught at pod creation, after the Workflow and the Job exist, with nothing on the WorkloadRun to say why. The launcher's own init container, fix-ssh-permissions, is not reserved: user init containers never reach the launcher.

func ValidateWorkloadVolumes ¶ added in v0.8.0

func ValidateWorkloadVolumes(volumes []corev1.Volume, mounts []corev1.VolumeMount, framework string, hasConfig bool) error

ValidateWorkloadVolumes rejects a spec.volumes or spec.volumeMounts entry that collides with a volume or mount the builders, or a Kubeflow Trainer plugin, add to the same pod themselves. Both call sites run it before anything is created, so the controller and `nvcrectl workloadrun render` refuse the same specs.

A duplicate volume name or mount path is otherwise caught at pod creation, after the Workflow and the Job exist, and two cases are not caught at all, because Trainer upserts its own entries by name and by path: the MPI plugin's mpi-ssh-auth volume on every worker and launcher pod, and the TrainJobStatus plugin's kubeflow-trainer-token volume on the pod whose job carries the "trainer" ancestor label, which is the node pod on torch and exec. A user volume of either name is replaced by Trainer's, and the user's mount of it ends up holding SSH keys or a service-account token. A user mount at /dev/shm, /tmp/mpi-ssh-raw or /root/.ssh would likewise shadow the built-in memory volume, the Secret the worker copies its keys from, or the directory it copies them into, and break the worker or its sshd before the workload ran. The two SSH directories and the token directory, /var/run/secrets/kubeflow/trainer, are reserved together with every path below them: a mount at /root/.ssh/authorized_keys hides that file as surely as one at /root/.ssh does. /dev/shm and /config are reserved exactly, since nothing built in reads a named file under them, so a mount below either collides with nothing of ours. A mountPath that is not absolute is refused outright, since none of the comparisons here mean anything for one.

framework is one of FrameworkTorch, FrameworkMPI and FrameworkExec. The mpi-only names and paths are checked on FrameworkMPI alone, since the torch and exec runtimes run no sshd and have no launcher; the Trainer token is checked on the other two, since on mpi it lands on the launcher, which user volumes never reach. hasConfig says whether spec.config is set, which is when /config is taken and when a mount may name config-volume at all. The config-volume name itself is reserved either way, so that adding spec.config later never turns a valid spec into an invalid one.