controller

package
v0.1.0-rc.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2026 License: Apache-2.0 Imports: 61 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// ReasonWaitingForNodes marks a Certification that found no schedulable nodes
	// and is retrying. Without it the wait is invisible: the retry only logged, so
	// the CR carried no conditions at all for up to nodeDiscoveryTimeout and an
	// operator had nothing to explain why nothing was happening.
	ReasonWaitingForNodes          = "WaitingForNodes"
	ReasonWorkflowCreated          = "WorkflowCreated"
	ReasonWorkflowRunning          = "WorkflowRunning"
	ReasonAllWorkflowsSucceeded    = "AllWorkflowsSucceeded"
	ReasonWorkflowFailed           = "WorkflowFailed"
	ReasonWorkflowValidationFailed = "WorkflowValidationFailed"
	ReasonCategoryNotFound         = "CategoryNotFound"
	ReasonBuildFailed              = "BuildFailed"
	ReasonWorkflowDeleted          = "WorkflowDeleted"
	ReasonWorkflowSucceeded        = "WorkflowSucceeded"
	ReasonThresholdViolation       = "ThresholdViolation"
)

Certification tier reasons (Certification → Workflow).

View Source 
const (
	ReasonJobCreated          = "JobCreated"
	ReasonJobRunning          = "JobRunning"
	ReasonJobCompleted        = "JobCompleted"
	ReasonJobFailed           = "JobFailed"
	ReasonJobHardwareFailed   = "JobHardwareFailed"
	ReasonJobCreationError    = "JobCreationError"
	ReasonJobValidationFailed = "JobValidationFailed"

	ReasonGroupsPartitioned  = "GroupsPartitioned"
	ReasonIterationCompleted = "IterationCompleted"
	ReasonIterationsFailed   = "IterationsFailed"

	ReasonDependencyCreationError = "DependencyCreationError"
	ReasonNodeDiscoveryError      = "NodeDiscoveryError"
	ReasonPartitionError          = "PartitionError"
)

Workflow tier reasons (Workflow → Job).

View Source 
const (
	ReasonWorkloadCreated       = "WorkloadCreated"
	ReasonWorkloadRunning       = "WorkloadRunning"
	ReasonWorkloadCompleted     = "WorkloadCompleted"
	ReasonWorkloadFailed        = "WorkloadFailed"
	ReasonWorkloadCreationError = "WorkloadCreationError"
	ReasonWorkloadStalled       = "WorkloadStalled"

	// ReasonMeasurementCreationError indicates a GoodputMeasurement or
	// BandwidthMeasurement child resource could not be created. Handling is
	// non-fatal, so this event is the operator-visible signal; a threshold that
	// depends on the missing measurement still fails closed via
	// ReasonMeasurementTimeout.
	ReasonMeasurementCreationError = "MeasurementCreationError"

	ReasonHardwareFailureDetected = "HardwareFailureDetected"
)

Job tier reasons (Job → Workload).

View Source 
const (
	FrameworkTorch = "torch"
	FrameworkMPI   = "mpi"
	FrameworkExec  = "exec"
)

Framework type constants for WorkloadRun.

View Source 
const GPUNodeLabel = "nvidia.com/gpu.present"

GPUNodeLabel marks a node as GPU-equipped. The controllers only ever operate on GPU nodes — discoverTargetNodes filters on this label before a node can be partitioned into a group — so it also defines the Node cache boundary.

View Source 
const (
	// ReasonNotApplicable is the default condition reason for inactive conditions.
	ReasonNotApplicable = "NotApplicable"
)

Variables

This section is empty.

Functions

func ApplyOverridesWithTracking 

func ApplyOverridesWithTracking(spec *nvcrev1alpha1.WorkflowSpec, octx OverrideContext) ([]nvcrev1alpha1.AppliedOverride, error)

ApplyOverridesWithTracking is the exported version of applyOverridesWithTracking for use by CLI tools.

func ApplyWRPreTemplateOverrides 

func ApplyWRPreTemplateOverrides(spec *nvcrev1alpha1.WorkloadRunSpec, overrides []platform.WorkloadRunOverride, octx OverrideContext)

ApplyWRPreTemplateOverrides is the exported version of applyWRPreTemplateOverrides for use by CLI tools, so the "workloadrun render" preview bakes the same platform mpirun args into the spec that the controller bakes at reconcile time.

func CacheOptions 

func CacheOptions() cache.Options

CacheOptions returns the informer cache configuration for the manager.

Without it, controller-runtime caches every object of every watched kind, cluster-wide and in full. This controller watches Nodes, Pods and PersistentVolumes, so on a large GPU cluster the default cache holds the entire Pod population of the cluster in the controller's heap — against a 1Gi limit. Two reductions are applied:

  • Nodes are restricted to GPU nodes. Non-GPU nodes (control plane, CPU workers, infra) are never targets, never run burn-in pods, and never need health evaluation, so caching and watching them is pure overhead.

  • managedFields is stripped from every cached object. It is metadata the controllers never read and is frequently a large fraction of an object's serialized size, particularly for Pods and CRs written by several controllers.

Pods are deliberately not label-scoped. The obvious selector, nvcre.nvidia.com/job, is injected only into the worker replicatedJob's pod template — MPI launcher pods do not carry it (see workerTargetJob in pkg/workload), and the launcher is exactly the pod that NCCL bandwidth measurement and timeout log capture read from. Scoping on that label would make launchers invisible to the cache. Pod reads are instead kept cheap via field indexes; see indexes.go.

func CondIsTrue 

func CondIsTrue(conditions []metav1.Condition, condType string) bool

CondIsTrue returns true if the named condition has Status=True.

func CondMessage 

func CondMessage(conditions []metav1.Condition, condType string) string

CondMessage returns the message for a condition type, or empty string if not found.

func DefaultEnableMNNVL 

func DefaultEnableMNNVL(gpuArch string) bool

DefaultEnableMNNVL returns whether MNNVL should be enabled for the given GPU architecture. GB200 and GB300 use Multi-Node NVLink and benefit from MNNVL.

func DetectGPUArchitecture 

func DetectGPUArchitecture(nodes []corev1.Node) string

DetectGPUArchitecture is the exported version of detectGPUArchitecture for use by CLI tools.

func DetectPlatform 

func DetectPlatform(nodes []corev1.Node) string

DetectPlatform is the exported version of detectPlatform for use by CLI tools.

func DiscoverTargetNodes 

func DiscoverTargetNodes(ctx context.Context, reader client.Reader, target *nvcrev1alpha1.TargetSpec) ([]corev1.Node, error)

DiscoverTargetNodes is the exported version of discoverTargetNodes for use by CLI tools.

It drops the cordoned-node list that discoverTargetNodes also returns. Only the Workflow reconciler records coverage on status; CLI callers want the nodes a run would actually use. Widen this if a CLI ever needs to report what was skipped.

func RegisterFieldIndexes 

func RegisterFieldIndexes(ctx context.Context, indexer client.FieldIndexer) error

RegisterFieldIndexes registers every field index the controllers rely on.

It must be called once per manager, before the controllers start, and is the single source of truth for both production (cmd/manager) and the integration harness — registering these in two places previously allowed prod and test to drift apart.

func ResolveOptions 

func ResolveOptions(global *nvcrev1alpha1.CategoryOptions, override *nvcrev1alpha1.CategoryOptions) nvcrev1alpha1.CategoryOptions

ResolveOptions merges per-category overrides with global defaults. Returns a flat CategoryOptions with all values resolved.

Types

type BandwidthMeasurementReconciler 

type BandwidthMeasurementReconciler struct {
	client.Client
	// APIReader is an uncached client used for pod discovery. Pod watch events
	// from the informer cache can lag behind the API server by several seconds,
	// causing the "pod not found" requeue loop to spin longer than necessary.
	// A direct API read guarantees the freshest view at the cost of one extra
	// API server call per reconcile cycle while waiting for pods.
	APIReader  client.Reader
	Scheme     *runtime.Scheme
	Clientset  *kubernetes.Clientset
	LogFetcher podlogs.PodLogFetcher // if nil, defaults to Clientset-backed fetcher
	// MaxConcurrentReconciles bounds the number of BandwidthMeasurement objects reconciled concurrently.
	MaxConcurrentReconciles int

	// RequeueInterval is the interval between reconcile cycles when polling.
	// Tests set this to 1s; production uses 15s.
	RequeueInterval time.Duration
	// contains filtered or unexported fields
}

BandwidthMeasurementReconciler reconciles a BandwidthMeasurement object.

func (*BandwidthMeasurementReconciler) Reconcile 

func (r *BandwidthMeasurementReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is part of the main kubernetes reconciliation loop.

func (*BandwidthMeasurementReconciler) SetupWithManager 

func (r *BandwidthMeasurementReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager sets up the controller with the Manager.

type CertificationReconciler 

type CertificationReconciler struct {
	client.Client
	Scheme                  *runtime.Scheme
	WorkflowRequeueInterval time.Duration
	// MaxConcurrentReconciles bounds the number of Certification objects reconciled concurrently.
	MaxConcurrentReconciles int
}

CertificationReconciler reconciles a Certification object

func (*CertificationReconciler) Reconcile 

func (r *CertificationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is part of the main kubernetes reconciliation loop which aims to move the current state of the cluster closer to the desired state.

func (*CertificationReconciler) SetupWithManager 

func (r *CertificationReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager sets up the controller with the Manager.

type GoodputMeasurementReconciler 

type GoodputMeasurementReconciler struct {
	client.Client
	Scheme     *runtime.Scheme
	Clientset  *kubernetes.Clientset
	LogFetcher podlogs.PodLogFetcher // if nil, defaults to Clientset-backed fetcher
	// MaxConcurrentReconciles bounds the number of GoodputMeasurement objects reconciled concurrently.
	MaxConcurrentReconciles int
	// contains filtered or unexported fields
}

GoodputMeasurementReconciler reconciles a GoodputMeasurement object.

func (*GoodputMeasurementReconciler) Reconcile 

func (r *GoodputMeasurementReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is part of the main kubernetes reconciliation loop.

func (*GoodputMeasurementReconciler) SetupWithManager 

func (r *GoodputMeasurementReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager sets up the controller with the Manager.

type JobReconciler 

type JobReconciler struct {
	client.Client
	Scheme         *runtime.Scheme
	Clientset      *kubernetes.Clientset
	NodeDiscoverer *nodemonitor.NodeDiscoverer
	Recorder       events.EventRecorder
	// MaxConcurrentReconciles bounds the number of Job objects reconciled concurrently.
	MaxConcurrentReconciles int

	// WorkloadRequeueInterval is how often to poll workload status when job is in progress.
	// If zero, defaults to workloadRequeueInterval (15s).
	WorkloadRequeueInterval time.Duration

	// MeasurementTimeout is how long after the Job succeeds to wait for measurement
	// data before failing threshold validation. If zero, defaults to defaultMeasurementTimeout (5m).
	MeasurementTimeout time.Duration
	// contains filtered or unexported fields
}

JobReconciler reconciles a Job object

func (*JobReconciler) Reconcile 

func (r *JobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is part of the main kubernetes reconciliation loop which aims to move the current state of the cluster closer to the desired state.

func (*JobReconciler) SetupWithManager 

func (r *JobReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager sets up the controller with the Manager.

type OverrideContext 

type OverrideContext struct {
	Platform        string
	GPUArchitecture string
	WorkloadKind    string
	TopologyMode    string
	DomainCount     int
	Config          *apiextensionsv1.JSON
}

OverrideContext holds all detected values used for WhenSpec matching.

func BuildOverrideContext 

func BuildOverrideContext(spec *nvcrev1alpha1.WorkflowSpec, orch *nvcrev1alpha1.OrchestrationStatus, nodes []corev1.Node) OverrideContext

BuildOverrideContext is the exported version of buildOverrideContext for use by CLI tools.

type WorkflowReconciler 

type WorkflowReconciler struct {
	client.Client
	Scheme             *runtime.Scheme
	Clientset          *kubernetes.Clientset
	Recorder           events.EventRecorder
	JobRequeueInterval time.Duration
	// MaxConcurrentReconciles bounds the number of Workflow objects reconciled concurrently.
	MaxConcurrentReconciles int
}

WorkflowReconciler reconciles a Workflow object

func (*WorkflowReconciler) Reconcile 

func (r *WorkflowReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

Reconcile is part of the main kubernetes reconciliation loop which aims to move the current state of the cluster closer to the desired state.

func (*WorkflowReconciler) SetupWithManager 

func (r *WorkflowReconciler) SetupWithManager(mgr ctrl.Manager) error

SetupWithManager sets up the controller with the Manager.

type WorkloadRunReconciler 

type WorkloadRunReconciler struct {
	client.Client
	Scheme *kruntime.Scheme
	// MaxConcurrentReconciles bounds the number of WorkloadRun objects reconciled concurrently.
	MaxConcurrentReconciles int
}

WorkloadRunReconciler reconciles a WorkloadRun object.

func (*WorkloadRunReconciler) Reconcile 

func (r *WorkloadRunReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error)

func (*WorkloadRunReconciler) SetupWithManager 

func (r *WorkloadRunReconciler) SetupWithManager(mgr ctrl.Manager) error

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.