catalog

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package catalog maps certification categories (domain/variant) to pre-configured WorkflowSpec builders. Each category is a YAML file under entries/{domain}/{variant}.yaml, embedded at compile time via //go:embed and loaded by loader.go. Adding a new certification category requires only adding a YAML file.

Index

Constants

View Source
const (
	DefaultMaxSteps           = 50
	DefaultExitDurationMins   = 30
	DefaultSaveInterval       = 250
	DefaultSaveRetainInterval = 1000
	DefaultSaveTopK           = 1
	DefaultStorageSize        = "10Ti"
	DefaultTestScale          = nvcrev1alpha1.TestScaleFullScale
	DefaultMaxBytes           = "16G"
	DefaultNumIterations      = 100
	DefaultNumCycles          = 10
	DefaultMinGroupSize       = 2

	// Training container resource defaults (DGX-class sizing). Overridable
	// per value via CategoryOptions.resources (issue #83).
	DefaultTrainingCPULimit      = "128"
	DefaultTrainingMemoryLimit   = "800Gi"
	DefaultTrainingCPURequest    = "64"
	DefaultTrainingMemoryRequest = "500Gi"

	// Diagnose mode uses fewer iterations for faster fault detection.
	DiagnoseNumIterations = 5
	DiagnoseNumCycles     = 1
	DiagnoseTimeoutPerJob = "15m"

	// DefaultWorkloadRunTimeoutPerJob bounds a WorkloadRun Job when the user
	// sets no timeoutPerJob. Without it Execution.TimeoutPerJob stays nil,
	// isJobTimedOut always returns false, and a Job whose pods can never
	// schedule runs until someone notices: one such run sat InProgress for
	// 4h10m against a node with no allocatable GPUs.
	//
	// This is a backstop against never finishing, not a performance bound. It
	// is deliberately far longer than the catalog's 1h, because a WorkloadRun
	// is arbitrary user work and cutting a legitimate long training run short
	// would be worse than the hang. Set spec.orchestration.timeoutPerJob for
	// anything tighter.
	DefaultWorkloadRunTimeoutPerJob = "24h"

	// DefaultTimeoutPerJob is the default timeout for communication jobs.
	// Prevents pods from running indefinitely on launcher restarts.
	DefaultTimeoutPerJob = "1h"
)

Default values for template variables when not specified by the user.

Variables

This section is empty.

Functions

func ConfigArch

func ConfigArch(arch string) string

ConfigArch returns the GPU architecture used for config file lookups. GB300 and GB200 use identical configs, so gb300 maps to gb200.

func EntriesFS

func EntriesFS() embed.FS

EntriesFS returns the embedded catalog entries filesystem. Used by internal/platform to load shared _lib/ templates.

func GPUArchFromNodeSelector

func GPUArchFromNodeSelector(nodeSelector map[string]string) string

GPUArchFromNodeSelector extracts the GPU architecture string from a nodeSelector. It parses the nvidia.com/gpu.product label (e.g., "NVIDIA-H100-80GB-HBM3" → "h100"). Returns an empty string if the label is absent.

func LoadGPUDefaults

func LoadGPUDefaults() error

LoadGPUDefaults forces the embedded gpu-defaults.yaml to be parsed and returns any error from that parse. Safe to call multiple times — the file is loaded exactly once and the result is cached.

Call this at process startup (e.g. from cmd/main.go) to fail fast if the embedded catalog data is malformed. Callers that don't invoke this will still trigger a lazy load on the first GPUDefaults call; in that path, parse errors are swallowed and architectures fall back to the safe default.

func Register

func Register(domain, variant string, entry Entry)

Register adds a catalog entry for the given domain and variant.

func TemplateFuncs

func TemplateFuncs() template.FuncMap

TemplateFuncs returns the function map available to catalog entry templates and to the shared fragments under entries/_lib/.

Anything that renders those fragments must use this map. pkg/platform renders _lib fragments too (for WorkloadRun overrides) and previously defined its own reduced map containing only lib and indent. Because a template that calls an unregistered function fails to parse, and both render paths panic on parse failure, a fragment using repeat or trimSuffix rendered fine through the catalog and would have crashed the process through pkg/platform. One map removes that class of divergence.

indent, trimSuffix and repeat are the only sprig functions the templates use (confirmed by grep across pkg/catalog/entries/), which is why this replaces the github.com/Masterminds/sprig/v3 dependency rather than pulling it in.

func TemplateFuncsWithLib

func TemplateFuncsWithLib() template.FuncMap

TemplateFuncsWithLib returns TemplateFuncs with the "lib" function wired in, for renderers outside this package that pull in entries/_lib/ fragments but have no catalog entry directory of their own (see pkg/platform).

lib is registered into the same map it renders with, so a fragment may call lib recursively.

Types

type BuildConfig

type BuildConfig struct {
	// ImagePullSecrets are references to secrets for pulling container images.
	// If empty, the cluster's default image pull configuration is used.
	ImagePullSecrets []corev1.LocalObjectReference

	// StorageClassName is the StorageClass for PVC dependencies.
	// If nil, catalogs that create PVCs omit the field (cluster default).
	StorageClassName *string

	// NodesPerJob is the number of nodes per job for multi-node workloads.
	// Training and communication entries templatize this; single-node entries ignore it.
	NodesPerJob int32

	// GpusPerNode is the resolved number of GPUs per node (always non-zero).
	// The certification controller derives this from the GPU architecture label
	// or the user's explicit override.
	GpusPerNode int32

	// MlnxPerNode is the Mellanox NIC count per node, used by IB/RoCE templates
	// (Azure, OCI, TogetherAI). 0 means the templates omit nvidia.com/mlnxnics.
	// Resolved from gpu-defaults.yaml + platform overrides + user override.
	MlnxPerNode int32

	// Resources overrides the CPU and memory of training containers.
	// Nil (or nil sub-fields) means the training entries keep their
	// DGX-class defaults (limits: cpu 128 / memory 800Gi; requests:
	// cpu 64 / memory 500Gi). Non-training entries ignore it.
	Resources *nvcrev1alpha1.CategoryResources

	// EnableMNNVL controls the NCCL_MNNVL_ENABLE env var in training entries.
	// When true, entries set NCCL_MNNVL_ENABLE=1.
	EnableMNNVL bool

	// EnableCheckpoint enables checkpointing for training workloads.
	EnableCheckpoint bool

	// MaxSteps is the maximum training steps for NeMo 4 workloads.
	// 0 means use template default (50).
	MaxSteps int32

	// ExitDurationMins is the training duration in minutes for NeMo 6 workloads.
	// 0 means use template default (30).
	ExitDurationMins int32

	// GPUArchitecture is the GPU architecture string (e.g., "h100", "gb200").
	// Derived from the target nodeSelector's nvidia.com/gpu.product label.
	// Required — Build returns an error if empty.
	GPUArchitecture string

	// SaveInterval is the checkpoint save frequency in training steps.
	// 0 means use default (250).
	SaveInterval int32

	// SaveRetainInterval retains checkpoints at multiples of this value (NeMo 6).
	// 0 means use default (1000).
	SaveRetainInterval int32

	// SaveTopK keeps only the top K checkpoints by metric (NeMo 4).
	// 0 means use default (1).
	SaveTopK int32

	// StorageSize is the PVC size for checkpoint storage.
	// Empty means use default ("10Ti").
	StorageSize string

	// TestScale controls orchestration strategy for NCCL tests.
	// Empty means "full-scale" (default).
	TestScale string

	// MaxBytes is the max message size for NCCL tests (e.g., "16G", "32G").
	// Empty means use default ("16G").
	MaxBytes string

	// NumIterations is the NCCL -n flag (timed iterations per message size).
	// 0 means use default (100).
	NumIterations int32

	// NumCycles is the NCCL -N flag (run cycles, each printed separately).
	// 0 means use default (10).
	NumCycles int32

	// MaxConcurrent limits simultaneous jobs. 0 means unlimited.
	MaxConcurrent int32

	// MinGroupSize is the smallest group at which diagnose's bisection
	// stops splitting. 0 means use default (2).
	MinGroupSize int32

	// TimeoutPerJob is the max duration per job (e.g., "1h", "30m").
	// Empty means use default ("1h", or "15m" for diagnose).
	TimeoutPerJob string

	// MeasurementTimeout is the max duration to wait after Job success for
	// measurement data (e.g., "5m", "10m"). Empty means use controller default ("5m").
	MeasurementTimeout string

	// Thresholds maps metric names to CEL expressions for performance validation.
	Thresholds map[string]string

	// RepeatCount overrides the orchestration iteration count.
	// 0 means use catalog default.
	RepeatCount int32

	// MaxRestarts overrides the checkpoint maxRestarts for training workloads.
	// 0 means use catalog default. Only used when EnableCheckpoint is true.
	MaxRestarts int32
}

BuildConfig holds deployment-specific configuration passed from the Certification spec to catalog builders. Fields are optional — catalogs use them when applicable.

type CategoryInfo

type CategoryInfo struct {
	Domain  string `json:"domain"`
	Variant string `json:"variant"`
}

CategoryInfo describes a registered catalog category.

func List

func List() []CategoryInfo

List returns all registered catalog categories, sorted by domain then variant.

type Entry

type Entry struct {
	// Build returns a WorkflowSpec configured for the given target nodes
	// and deployment-specific configuration. Returns an error if the
	// configuration is invalid (e.g., nodesPerJob not a power of 2,
	// or fewer GPUs than required).
	Build func(target nvcrev1alpha1.TargetSpec, config BuildConfig) (nvcrev1alpha1.WorkflowSpec, error)

	// MinGPUs is the minimum number of GPUs (nodesPerJob × gpusPerNode)
	// required by this entry. 0 means no minimum. Build returns an error
	// if the requested GPU count is below this threshold.
	MinGPUs int32

	// TimeoutPerJob is the entry's default job timeout from its meta.yaml
	// (e.g., "2h" for dcgm-level4). Empty means the global default applies
	// (DefaultTimeoutPerJob, or DiagnoseTimeoutPerJob for diagnose scale).
	// Exposed so callers (e.g., the nvcrectl --wait timeout derivation) can
	// compute a category's job budget without a full Build.
	TimeoutPerJob string

	// Iterations is the entry's default orchestration iteration count,
	// parsed from the entry template. Always >= 1.
	Iterations int

	// MaxValidNodes returns the largest node count <= availableNodes that
	// satisfies the entry's constraints (minGPUs, TP×PP divisibility).
	// Returns 0 if no valid count exists. When nil, any node count is valid.
	MaxValidNodes func(availableNodes, gpusPerNode int32, gpuArch string) int32
}

Entry holds a builder that produces a WorkflowSpec for a given target.

func Lookup

func Lookup(domain, variant string) *Entry

Lookup returns the catalog entry for the given domain and variant. Returns nil if the category is not registered.

func (*Entry) EffectiveTimeoutPerJob

func (e *Entry) EffectiveTimeoutPerJob(userTimeout, testScale string) string

EffectiveTimeoutPerJob returns the timeoutPerJob string that Build would render for this entry: an explicit user override wins, then the entry's meta.yaml default, then the test-scale-dependent global default. Shares resolveTimeoutPerJob with buildTemplateData so the two cannot diverge.

type NodeDefaults

type NodeDefaults struct {
	GpusPerNode int32 `json:"gpusPerNode" yaml:"gpusPerNode"`
	MlnxPerNode int32 `json:"mlnxPerNode" yaml:"mlnxPerNode"`
}

NodeDefaults holds per-node hardware counts for a GPU architecture. Populated from entries/_lib/gpu-defaults.yaml on first use and consumed by the controller and CLI to derive GpusPerNode/MlnxPerNode without hardcoding.

func GPUDefaults

func GPUDefaults(gpuArch, platform string) NodeDefaults

GPUDefaults returns node hardware defaults for the given GPU architecture and platform. Platform overrides (e.g. OCI L40s) are applied on top of architecture defaults — only fields explicitly set in the platform override take effect; unset fields fall through to the architecture defaults.

Unknown architectures get {GpusPerNode: fallbackGpusPerNode, MlnxPerNode: 0}. An empty platform skips override resolution and returns architecture defaults only — this preserves callers that don't yet have platform context.

type TemplateData

type TemplateData struct {
	// ImagePullSecrets are references to secrets for pulling container images.
	// Templates use: {{- if .ImagePullSecrets }} ... {{- range .ImagePullSecrets }}
	ImagePullSecrets []corev1.LocalObjectReference

	// StorageClassName is the StorageClass for PVC dependencies.
	// Empty string means omit (use cluster default).
	// Templates use: {{- if .StorageClassName }}
	StorageClassName string

	// NodesPerJob is the number of nodes per job for multi-node workloads.
	// Templates use: {{ .NodesPerJob }}
	NodesPerJob int32

	// GpusPerNode is the resolved number of GPUs per node (always non-zero at Build time).
	// Templates use: {{ .GpusPerNode }}
	GpusPerNode int32

	// MlnxPerNode is the Mellanox NIC count per node for IB/RoCE platforms.
	// 0 means omit nvidia.com/mlnxnics. Templates use: {{ .MlnxPerNode }}
	MlnxPerNode int32

	// TrainingCPULimit is the CPU limit for training containers
	// (always non-empty after defaults). Templates use: {{ .TrainingCPULimit }}
	TrainingCPULimit string

	// TrainingMemoryLimit is the memory limit for training containers
	// (always non-empty after defaults). Templates use: {{ .TrainingMemoryLimit }}
	TrainingMemoryLimit string

	// TrainingCPURequest is the CPU request for training containers
	// (always non-empty after defaults). Templates use: {{ .TrainingCPURequest }}
	TrainingCPURequest string

	// TrainingMemoryRequest is the memory request for training containers
	// (always non-empty after defaults). Templates use: {{ .TrainingMemoryRequest }}
	TrainingMemoryRequest string

	// EnableMNNVL controls the NCCL_MNNVL_ENABLE env var.
	// Templates use: {{ if .EnableMNNVL }}1{{ else }}0{{ end }}
	EnableMNNVL bool

	// EnableCheckpoint enables checkpointing for training workloads.
	// Templates use: {{ .EnableCheckpoint }}
	EnableCheckpoint bool

	// MaxSteps is the maximum training steps (always non-zero after defaults).
	// Templates use: {{ .MaxSteps }}
	MaxSteps int32

	// ExitDurationMins is the training duration in minutes (always non-zero after defaults).
	// Templates use: {{ .ExitDurationMins }}
	ExitDurationMins int32

	// GPUArchitecture is the GPU architecture string (e.g., "h100", "gb200").
	// Templates use: {{ .GPUArchitecture }}
	GPUArchitecture string

	// ConfigArch is the GPU architecture used for config file lookups.
	// GB300 and GB200 use identical configs, so this maps gb300 → gb200.
	// Templates use: {{ .ConfigArch }} in includeTemplate/includeFile paths.
	ConfigArch string

	// EntryName is the variant name of the catalog entry (e.g., "nccl-all-gather").
	// Auto-populated by the loader from the entry's filesystem path.
	// Shared library templates use this to construct entry-specific resource names
	// like {{ .EntryName }}-runtime or {{ .EntryName }}-compute-domain.
	EntryName string

	// SaveInterval is the checkpoint save frequency in training steps.
	// Templates use: {{ .SaveInterval }}
	SaveInterval int32

	// SaveRetainInterval retains checkpoints at multiples of this value (NeMo 6).
	// Templates use: {{ .SaveRetainInterval }}
	SaveRetainInterval int32

	// SaveTopK keeps only the top K checkpoints by metric (NeMo 4).
	// Templates use: {{ .SaveTopK }}
	SaveTopK int32

	// StorageSize is the PVC size for checkpoint storage.
	// Templates use: {{ .StorageSize }}
	StorageSize string

	// TestScale controls orchestration strategy for NCCL tests.
	// Templates use: {{ .TestScale }}
	TestScale string

	// MaxBytes is the max message size for NCCL tests.
	// Templates use: {{ .MaxBytes }}
	MaxBytes string

	// NumIterations is the NCCL -n flag.
	// Templates use: {{ .NumIterations }}
	NumIterations int32

	// NumCycles is the NCCL -N flag.
	// Templates use: {{ .NumCycles }}
	NumCycles int32

	// MaxConcurrent limits simultaneous jobs.
	// Templates use: {{ .MaxConcurrent }}
	MaxConcurrent int32

	// MinGroupSize is the smallest group at which bisection stops.
	// Templates use: {{ .MinGroupSize }}
	MinGroupSize int32

	// TimeoutPerJob is the max duration per job for diagnose mode.
	// Templates use: {{ .TimeoutPerJob }}
	TimeoutPerJob string

	// MeasurementTimeout is the max duration to wait for measurement data after Job success.
	// Templates use: {{ .MeasurementTimeout }}
	MeasurementTimeout string

	// Thresholds maps metric names to CEL expressions.
	// Templates use: {{- range $k, $v := .Thresholds }}
	Thresholds map[string]string

	// TP is tensor-model-parallel size from meta.yaml for the resolved architecture.
	// Templates use: {{ .TP }}
	TP int32

	// PP is pipeline-model-parallel size from meta.yaml for the resolved architecture.
	// Templates use: {{ .PP }}
	PP int32

	// EP is expert-model-parallel size from meta.yaml for the resolved architecture.
	// Templates use: {{ .EP }}
	EP int32
}

TemplateData holds the data passed to catalog YAML templates at Build time. Fields are populated from BuildConfig. Add new fields here for future template variables — no loader changes required.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL