Documentation
¶
Overview ¶
Package catalog maps certification categories (domain/variant) to pre-configured WorkflowSpec builders. Each category is a YAML file under entries/{domain}/{variant}.yaml, embedded at compile time via //go:embed and loaded by loader.go. Adding a new certification category requires only adding a YAML file.
Index ¶
- Constants
- func ConfigArch(arch string) string
- func EntriesFS() embed.FS
- func GPUArchFromNodeSelector(nodeSelector map[string]string) string
- func LoadGPUDefaults() error
- func Register(domain, variant string, entry Entry)
- func TemplateFuncs() template.FuncMap
- func TemplateFuncsWithLib() template.FuncMap
- type BuildConfig
- type CategoryInfo
- type Entry
- type NodeDefaults
- type TemplateData
Constants ¶
const ( DefaultMaxSteps = 50 DefaultExitDurationMins = 30 DefaultSaveInterval = 250 DefaultSaveRetainInterval = 1000 DefaultSaveTopK = 1 DefaultStorageSize = "10Ti" DefaultTestScale = nvcrev1alpha1.TestScaleFullScale DefaultMaxBytes = "16G" DefaultNumIterations = 100 DefaultNumCycles = 10 DefaultMinGroupSize = 2 // Training container resource defaults (DGX-class sizing). Overridable // per value via CategoryOptions.resources (issue #83). DefaultTrainingCPULimit = "128" DefaultTrainingMemoryLimit = "800Gi" DefaultTrainingCPURequest = "64" DefaultTrainingMemoryRequest = "500Gi" // Diagnose mode uses fewer iterations for faster fault detection. DiagnoseNumIterations = 5 DiagnoseNumCycles = 1 DiagnoseTimeoutPerJob = "15m" // DefaultWorkloadRunTimeoutPerJob bounds a WorkloadRun Job when the user // sets no timeoutPerJob. Without it Execution.TimeoutPerJob stays nil, // isJobTimedOut always returns false, and a Job whose pods can never // schedule runs until someone notices: one such run sat InProgress for // 4h10m against a node with no allocatable GPUs. // // This is a backstop against never finishing, not a performance bound. It // is deliberately far longer than the catalog's 1h, because a WorkloadRun // is arbitrary user work and cutting a legitimate long training run short // would be worse than the hang. Set spec.orchestration.timeoutPerJob for // anything tighter. DefaultWorkloadRunTimeoutPerJob = "24h" // DefaultTimeoutPerJob is the default timeout for communication jobs. // Prevents pods from running indefinitely on launcher restarts. DefaultTimeoutPerJob = "1h" )
Default values for template variables when not specified by the user.
Variables ¶
This section is empty.
Functions ¶
func ConfigArch ¶
ConfigArch returns the GPU architecture used for config file lookups. GB300 and GB200 use identical configs, so gb300 maps to gb200.
func EntriesFS ¶
EntriesFS returns the embedded catalog entries filesystem. Used by internal/platform to load shared _lib/ templates.
func GPUArchFromNodeSelector ¶
GPUArchFromNodeSelector extracts the GPU architecture string from a nodeSelector. It parses the nvidia.com/gpu.product label (e.g., "NVIDIA-H100-80GB-HBM3" → "h100"). Returns an empty string if the label is absent.
func LoadGPUDefaults ¶
func LoadGPUDefaults() error
LoadGPUDefaults forces the embedded gpu-defaults.yaml to be parsed and returns any error from that parse. Safe to call multiple times — the file is loaded exactly once and the result is cached.
Call this at process startup (e.g. from cmd/main.go) to fail fast if the embedded catalog data is malformed. Callers that don't invoke this will still trigger a lazy load on the first GPUDefaults call; in that path, parse errors are swallowed and architectures fall back to the safe default.
func TemplateFuncs ¶
TemplateFuncs returns the function map available to catalog entry templates and to the shared fragments under entries/_lib/.
Anything that renders those fragments must use this map. pkg/platform renders _lib fragments too (for WorkloadRun overrides) and previously defined its own reduced map containing only lib and indent. Because a template that calls an unregistered function fails to parse, and both render paths panic on parse failure, a fragment using repeat or trimSuffix rendered fine through the catalog and would have crashed the process through pkg/platform. One map removes that class of divergence.
indent, trimSuffix and repeat are the only sprig functions the templates use (confirmed by grep across pkg/catalog/entries/), which is why this replaces the github.com/Masterminds/sprig/v3 dependency rather than pulling it in.
func TemplateFuncsWithLib ¶
TemplateFuncsWithLib returns TemplateFuncs with the "lib" function wired in, for renderers outside this package that pull in entries/_lib/ fragments but have no catalog entry directory of their own (see pkg/platform).
lib is registered into the same map it renders with, so a fragment may call lib recursively.
Types ¶
type BuildConfig ¶
type BuildConfig struct {
// ImagePullSecrets are references to secrets for pulling container images.
// If empty, the cluster's default image pull configuration is used.
ImagePullSecrets []corev1.LocalObjectReference
// StorageClassName is the StorageClass for PVC dependencies.
// If nil, catalogs that create PVCs omit the field (cluster default).
StorageClassName *string
// NodesPerJob is the number of nodes per job for multi-node workloads.
// Training and communication entries templatize this; single-node entries ignore it.
NodesPerJob int32
// GpusPerNode is the resolved number of GPUs per node (always non-zero).
// The certification controller derives this from the GPU architecture label
// or the user's explicit override.
GpusPerNode int32
// MlnxPerNode is the Mellanox NIC count per node, used by IB/RoCE templates
// (Azure, OCI, TogetherAI). 0 means the templates omit nvidia.com/mlnxnics.
// Resolved from gpu-defaults.yaml + platform overrides + user override.
MlnxPerNode int32
// NicResourceName is the extended resource name of the RDMA NIC devices
// requested by the on-prem GB200/GB300 templates (e.g., "rdma/ib"). Empty
// means the templates omit the NIC resource block. The per-container count
// comes from MlnxPerNode.
NicResourceName string
// Resources overrides the CPU and memory of training containers.
// Nil (or nil sub-fields) means the training entries keep their
// DGX-class defaults (limits: cpu 128 / memory 800Gi; requests:
// cpu 64 / memory 500Gi). Non-training entries ignore it.
Resources *nvcrev1alpha1.CategoryResources
// EnableMNNVL controls the NCCL_MNNVL_ENABLE env var in training entries.
// When true, entries set NCCL_MNNVL_ENABLE=1.
EnableMNNVL bool
// EnableCheckpoint enables checkpointing for training workloads.
EnableCheckpoint bool
// MaxSteps is the maximum training steps for NeMo 4 workloads.
// 0 means use template default (50).
MaxSteps int32
// ExitDurationMins is the training duration in minutes for NeMo 6 workloads.
// 0 means use template default (30).
ExitDurationMins int32
// GPUArchitecture is the GPU architecture string (e.g., "h100", "gb200").
// Derived from the target nodeSelector's nvidia.com/gpu.product label.
// Required — Build returns an error if empty.
GPUArchitecture string
// SaveInterval is the checkpoint save frequency in training steps.
// 0 means use default (250).
SaveInterval int32
// SaveRetainInterval retains checkpoints at multiples of this value (NeMo 6).
// 0 means use default (1000).
SaveRetainInterval int32
// SaveTopK keeps only the top K checkpoints by metric (NeMo 4).
// 0 means use default (1).
SaveTopK int32
// StorageSize is the PVC size for checkpoint storage.
// Empty means use default ("10Ti").
StorageSize string
// TestScale controls orchestration strategy for NCCL tests.
// Empty means "full-scale" (default).
TestScale string
// MaxBytes is the max message size for NCCL tests (e.g., "16G", "32G").
// Empty means use default ("16G").
MaxBytes string
// NumIterations is the NCCL -n flag (timed iterations per message size).
// 0 means use default (100).
NumIterations int32
// NumCycles is the NCCL -N flag (run cycles, each printed separately).
// 0 means use default (10).
NumCycles int32
// MaxConcurrent limits simultaneous jobs. 0 means unlimited.
MaxConcurrent int32
// MinGroupSize is the smallest group at which diagnose's bisection
// stops splitting. 0 means use default (2).
MinGroupSize int32
// TimeoutPerJob is the max duration per job (e.g., "1h", "30m").
// Empty means use default ("1h", or "15m" for diagnose).
TimeoutPerJob string
// MeasurementTimeout is the max duration to wait after Job success for
// measurement data (e.g., "5m", "10m"). Empty means use controller default ("5m").
MeasurementTimeout string
// Thresholds maps metric names to CEL expressions for performance validation.
Thresholds map[string]string
// RepeatCount overrides the orchestration iteration count.
// 0 means use catalog default.
RepeatCount int32
// MaxRestarts overrides the checkpoint maxRestarts for training workloads.
// 0 means use catalog default. Only used when EnableCheckpoint is true.
MaxRestarts int32
// SourceRepo is the Git repository URL for the source checkout cloned by
// entries that fetch source at pod start. Empty means each entry uses its
// own default upstream; entries that clone no source ignore it.
SourceRepo string
}
BuildConfig holds deployment-specific configuration passed from the Certification spec to catalog builders. Fields are optional — catalogs use them when applicable.
type CategoryInfo ¶
CategoryInfo describes a registered catalog category.
func List ¶
func List() []CategoryInfo
List returns all registered catalog categories, sorted by domain then variant.
type Entry ¶
type Entry struct {
// Build returns a WorkflowSpec configured for the given target nodes
// and deployment-specific configuration. Returns an error if the
// configuration is invalid (e.g., nodesPerJob not a power of 2,
// or fewer GPUs than required).
Build func(target nvcrev1alpha1.TargetSpec, config BuildConfig) (nvcrev1alpha1.WorkflowSpec, error)
// MinGPUs is the minimum number of GPUs (nodesPerJob × gpusPerNode)
// required by this entry. 0 means no minimum. Build returns an error
// if the requested GPU count is below this threshold.
MinGPUs int32
// TimeoutPerJob is the entry's default job timeout from its meta.yaml
// (e.g., "2h" for dcgm-level4). Empty means the global default applies
// (DefaultTimeoutPerJob, or DiagnoseTimeoutPerJob for diagnose scale).
// Exposed so callers (e.g., the nvcrectl --wait timeout derivation) can
// compute a category's job budget without a full Build.
TimeoutPerJob string
// Iterations is the entry's default orchestration iteration count,
// parsed from the entry template. Always >= 1.
Iterations int
// MaxValidNodes returns the largest node count <= availableNodes that
// satisfies the entry's constraints (minGPUs, TP×PP divisibility).
// Returns 0 if no valid count exists. When nil, any node count is valid.
MaxValidNodes func(availableNodes, gpusPerNode int32, gpuArch string) int32
}
Entry holds a builder that produces a WorkflowSpec for a given target.
func Lookup ¶
Lookup returns the catalog entry for the given domain and variant. Returns nil if the category is not registered.
func (*Entry) EffectiveTimeoutPerJob ¶
EffectiveTimeoutPerJob returns the timeoutPerJob string that Build would render for this entry: an explicit user override wins, then the entry's meta.yaml default, then the test-scale-dependent global default. Shares resolveTimeoutPerJob with buildTemplateData so the two cannot diverge.
type NodeDefaults ¶
type NodeDefaults struct {
GpusPerNode int32 `json:"gpusPerNode" yaml:"gpusPerNode"`
MlnxPerNode int32 `json:"mlnxPerNode" yaml:"mlnxPerNode"`
}
NodeDefaults holds per-node hardware counts for a GPU architecture. Populated from entries/_lib/gpu-defaults.yaml on first use and consumed by the controller and CLI to derive GpusPerNode/MlnxPerNode without hardcoding.
func GPUDefaults ¶
func GPUDefaults(gpuArch, platform string) NodeDefaults
GPUDefaults returns node hardware defaults for the given GPU architecture and platform. Platform overrides (e.g. OCI L40s) are applied on top of architecture defaults — only fields explicitly set in the platform override take effect; unset fields fall through to the architecture defaults.
Unknown architectures get {GpusPerNode: fallbackGpusPerNode, MlnxPerNode: 0}. An empty platform skips override resolution and returns architecture defaults only — this preserves callers that don't yet have platform context.
type TemplateData ¶
type TemplateData struct {
// ImagePullSecrets are references to secrets for pulling container images.
// Templates use: {{- if .ImagePullSecrets }} ... {{- range .ImagePullSecrets }}
ImagePullSecrets []corev1.LocalObjectReference
// StorageClassName is the StorageClass for PVC dependencies.
// Empty string means omit (use cluster default).
// Templates use: {{- if .StorageClassName }}
StorageClassName string
// NodesPerJob is the number of nodes per job for multi-node workloads.
// Templates use: {{ .NodesPerJob }}
NodesPerJob int32
// GpusPerNode is the resolved number of GPUs per node (always non-zero at Build time).
// Templates use: {{ .GpusPerNode }}
GpusPerNode int32
// MlnxPerNode is the Mellanox NIC count per node for IB/RoCE platforms.
// 0 means omit nvidia.com/mlnxnics. Templates use: {{ .MlnxPerNode }}
MlnxPerNode int32
// NicResourceName is the extended resource name of the RDMA NIC devices
// for the on-prem GB200/GB300 templates. Empty means omit the NIC resource
// block. Templates use: {{- if .NicResourceName }} ... {{ .NicResourceName }}
NicResourceName string
// TrainingCPULimit is the CPU limit for training containers
// (always non-empty after defaults). Templates use: {{ .TrainingCPULimit }}
TrainingCPULimit string
// TrainingMemoryLimit is the memory limit for training containers
// (always non-empty after defaults). Templates use: {{ .TrainingMemoryLimit }}
TrainingMemoryLimit string
// TrainingCPURequest is the CPU request for training containers
// (always non-empty after defaults). Templates use: {{ .TrainingCPURequest }}
TrainingCPURequest string
// TrainingMemoryRequest is the memory request for training containers
// (always non-empty after defaults). Templates use: {{ .TrainingMemoryRequest }}
TrainingMemoryRequest string
// EnableMNNVL controls the NCCL_MNNVL_ENABLE env var.
// Templates use: {{ if .EnableMNNVL }}1{{ else }}0{{ end }}
EnableMNNVL bool
// EnableCheckpoint enables checkpointing for training workloads.
// Templates use: {{ .EnableCheckpoint }}
EnableCheckpoint bool
// MaxSteps is the maximum training steps (always non-zero after defaults).
// Templates use: {{ .MaxSteps }}
MaxSteps int32
// ExitDurationMins is the training duration in minutes (always non-zero after defaults).
// Templates use: {{ .ExitDurationMins }}
ExitDurationMins int32
// GPUArchitecture is the GPU architecture string (e.g., "h100", "gb200").
// Templates use: {{ .GPUArchitecture }}
GPUArchitecture string
// ConfigArch is the GPU architecture used for config file lookups.
// GB300 and GB200 use identical configs, so this maps gb300 → gb200.
// Templates use: {{ .ConfigArch }} in includeTemplate/includeFile paths.
ConfigArch string
// EntryName is the variant name of the catalog entry (e.g., "nccl-all-gather").
// Auto-populated by the loader from the entry's filesystem path.
// Shared library templates use this to construct entry-specific resource names
// like {{ .EntryName }}-runtime or {{ .EntryName }}-compute-domain.
EntryName string
// SaveInterval is the checkpoint save frequency in training steps.
// Templates use: {{ .SaveInterval }}
SaveInterval int32
// SaveRetainInterval retains checkpoints at multiples of this value (NeMo 6).
// Templates use: {{ .SaveRetainInterval }}
SaveRetainInterval int32
// SaveTopK keeps only the top K checkpoints by metric (NeMo 4).
// Templates use: {{ .SaveTopK }}
SaveTopK int32
// StorageSize is the PVC size for checkpoint storage.
// Templates use: {{ .StorageSize }}
StorageSize string
// TestScale controls orchestration strategy for NCCL tests.
// Templates use: {{ .TestScale }}
TestScale string
// MaxBytes is the max message size for NCCL tests.
// Templates use: {{ .MaxBytes }}
MaxBytes string
// NumIterations is the NCCL -n flag.
// Templates use: {{ .NumIterations }}
NumIterations int32
// NumCycles is the NCCL -N flag.
// Templates use: {{ .NumCycles }}
NumCycles int32
// MaxConcurrent limits simultaneous jobs.
// Templates use: {{ .MaxConcurrent }}
MaxConcurrent int32
// MinGroupSize is the smallest group at which bisection stops.
// Templates use: {{ .MinGroupSize }}
MinGroupSize int32
// TimeoutPerJob is the max duration per job for diagnose mode.
// Templates use: {{ .TimeoutPerJob }}
TimeoutPerJob string
// MeasurementTimeout is the max duration to wait for measurement data after Job success.
// Templates use: {{ .MeasurementTimeout }}
MeasurementTimeout string
// Thresholds maps metric names to CEL expressions.
// Templates use: {{- range $k, $v := .Thresholds }}
Thresholds map[string]string
// SourceRepo is the Git repository URL overriding the default upstream of
// the source checkout an entry clones at pod start. Stays empty when the
// user sets nothing; each entry supplies its own canonical fallback.
// Templates use: {{ if .SourceRepo }}{{ .SourceRepo }}{{ else }}<entry default>{{ end }}
SourceRepo string
// TP is tensor-model-parallel size from meta.yaml for the resolved architecture.
// Templates use: {{ .TP }}
TP int32
// PP is pipeline-model-parallel size from meta.yaml for the resolved architecture.
// Templates use: {{ .PP }}
PP int32
// EP is expert-model-parallel size from meta.yaml for the resolved architecture.
// Templates use: {{ .EP }}
EP int32
}
TemplateData holds the data passed to catalog YAML templates at Build time. Fields are populated from BuildConfig. Add new fields here for future template variables — no loader changes required.