Documentation
¶
Overview ¶
Package gkenet discovers the GKE multi-NIC networking objects that GPUDirect TCPXO depends on. It is shared by the deployment-phase prerequisite check and the performance-phase NCCL benchmark so both gate on the same definition of "the cluster has GPU NIC networks".
Index ¶
- Constants
- Variables
- func DeployedTCPXOMapping(obj *unstructured.Unstructured) ([]recipe.NetworkInterfaceMapping, error)
- func DiscoverGPUNICNetworks(ctx context.Context, dynamicClient dynamic.Interface) ([]string, error)
- func FabricRuntimeDelivered(refs []recipe.ComponentRef) ([]recipe.NetworkInterfaceMapping, bool, error)
- func NodeTemplateOf(obj *unstructured.Unstructured) (map[string]any, error)
- func ParseInterfacesAnnotation(raw string) ([]recipe.NetworkInterfaceMapping, error)
- func ReadDeployedTCPXORuntime(ctx context.Context, dyn dynamic.Interface) (*unstructured.Unstructured, error)
- func ReadErrorCode(err error) errors.ErrorCode
- func VerifyMappingMatchesRecipe(recorded, deployed []recipe.NetworkInterfaceMapping) error
- func VerifyNetworksExist(deployed []recipe.NetworkInterfaceMapping, discovered []string) error
Constants ¶
const ( // TCPXORuntimeName is the ClusterTrainingRuntime the kubeflow-trainer // component ships for GPUDirect-TCPXO (torch-distributed-tcpxo-cluster- // training-runtime.yaml). It is cluster-scoped, so there is exactly one. TCPXORuntimeName = "torch-distributed-tcpxo" // InterfacesAnnotation carries the ordered GKE multi-NIC mapping on the // worker pod template: eth0 -> default first, then eth1..eth8 -> GPU NICs. InterfacesAnnotation = "networking.gke.io/interfaces" // DefaultInterfaceAnnotation names the pod's primary interface (eth0). DefaultInterfaceAnnotation = "networking.gke.io/default-interface" // TCPXONodeJob is the replicatedJob the shipped runtime places its worker // pod template under. It is the same name the benchmark runtime uses, which // is what makes the derivation a template swap rather than a rename. TCPXONodeJob = "node" )
const RequiredGPUNICNetworks = 8
RequiredGPUNICNetworks is the number of GPU NIC networks GPUDirect TCPXO needs on an a3-megagpu-8g node — one per GPU, bound to eth1..eth8. Both the deployment-phase prerequisite check and the performance-phase NCCL benchmark gate on this value so a cluster that clears the deployment gate cannot fail the benchmark's own discovery for the same reason.
The floor is fixed to the a3-megagpu-8g shape rather than derived from the node's GPU count: only the H100 GKE training recipe declares gke-nccl-tcpxo, and that recipe targets a3-megagpu-8g. Making it SKU-aware is tracked separately.
Variables ¶
var ClusterTrainingRuntimeGVR = schema.GroupVersionResource{
Group: "trainer.kubeflow.org", Version: "v1alpha1", Resource: "clustertrainingruntimes",
}
ClusterTrainingRuntimeGVR addresses Kubeflow Trainer's cluster-scoped runtime catalog. The namespaced sibling (trainingruntimes) is what the benchmark creates for itself; this is what the recipe ships.
var NetworkGVR = schema.GroupVersionResource{
Group: "networking.gke.io", Version: "v1", Resource: "networks",
}
NetworkGVR is the cluster-scoped GKE Network CR that multi-networking binds into the cluster. Its absence is what this package detects.
Functions ¶
func DeployedTCPXOMapping ¶ added in v0.22.0
func DeployedTCPXOMapping(obj *unstructured.Unstructured) ([]recipe.NetworkInterfaceMapping, error)
DeployedTCPXOMapping extracts the eth1..eth8 -> network mapping the deployed runtime will actually give its workers, from the interfaces annotation on the node template. It is the "deployed" leg of the three-way comparison.
func DiscoverGPUNICNetworks ¶
DiscoverGPUNICNetworks lists networks.networking.gke.io and returns the GPU NIC network names, sorted alphabetically.
The returned error is the RAW Kubernetes API error, deliberately not wrapped in a pkg/errors code, because callers classify it by shape and would lose that ability behind a wrap. The deployment check relies on exactly this: it routes an apierrors.IsNotFound error (the cluster does not serve this GVR at all, i.e. it was created without --enable-multi-networking) through validators.Capability.Require, which is declaration-gated, and every other error through RequireList, which always blocks. Do not wrap or normalize these errors without updating that caller.
A cluster that serves the API but has no matching networks returns an empty slice and a nil error — "none found" is a verdict for the caller, distinct from both a failure to read and an absent API.
func FabricRuntimeDelivered ¶ added in v0.22.0
func FabricRuntimeDelivered(refs []recipe.ComponentRef) ([]recipe.NetworkInterfaceMapping, bool, error)
FabricRuntimeDelivered reports whether the recipe ships a fabric-wired runtime, and returns the eth1..eth8 -> network mapping it recorded for it.
It is answered from the recipe alone and needs BOTH halves #2297 names: the enabled kubeflow-trainer componentRef must declare the TCPXO runtime manifest (the same test recipe.ShipsGKETCPXORuntime applies — a mapping without the artifact describes nothing) AND carry the typed tcpxoInterfaces override. It is never answered from finding a live runtime, because a stray runtime must not change who owns the evidence. Declaring kubeflow-trainer is NOT sufficient: the platform-kubeflow mixin ships only torch-distributed, so every kubeflow leaf except h100-gke-cos-training-kubeflow returns (nil, false, nil). A recorded override that fails to normalize is an error, not "not delivered": the recipe claims a runtime it cannot describe, and silently downgrading that to the capability-fixture path would hide exactly the divergence this exists to catch.
func NodeTemplateOf ¶ added in v0.22.0
func NodeTemplateOf(obj *unstructured.Unstructured) (map[string]any, error)
NodeTemplateOf returns the shipped runtime's worker PodTemplateSpec — the "node" replicatedJob's template.spec.template — as a map holding both "metadata" and "spec". Both halves matter: the GKE fabric annotations live in template.metadata, before spec begins, so a caller that copied only the PodSpec would drop the wiring entirely (Kubeflow models the two separately in PodTemplatePatch for the same reason). The returned map is a deep copy (unstructured.NestedMap copies), so mutating it does not change obj; callers that need to mutate the runtime must write the map back.
func ParseInterfacesAnnotation ¶ added in v0.22.0
func ParseInterfacesAnnotation(raw string) ([]recipe.NetworkInterfaceMapping, error)
ParseInterfacesAnnotation decodes a networking.gke.io/interfaces value and returns the secondary (eth1..eth8) entries as a mapping, after asserting the leading eth0 -> default entry. It does not re-validate the mapping's shape beyond that; callers compare it against the recipe's already-validated value (VerifyMappingMatchesRecipe) or run recipe.ValidateGKETCPXOInterfaces.
func ReadDeployedTCPXORuntime ¶ added in v0.22.0
func ReadDeployedTCPXORuntime(ctx context.Context, dyn dynamic.Interface) (*unstructured.Unstructured, error)
ReadDeployedTCPXORuntime fetches the shipped ClusterTrainingRuntime from the live API. The manifest in the repo is a Helm template and cannot be parsed at validation time; reading the deployed object is also what turns the derived benchmark into evidence about what was actually installed rather than about the recipe author's intent. A NotFound is returned unwrapped-by-code as ErrCodeNotFound so callers can distinguish "not deployed" from an apiserver fault.
func ReadErrorCode ¶ added in v0.22.0
ReadErrorCode classifies a non-NotFound Kubernetes read failure: an expired deadline is ErrCodeTimeout and an operator abort is ErrCodeCanceled — both are outcomes of the run's context, not product faults — and anything else is ErrCodeInternal. Callers already handle NotFound before reaching this.
func VerifyMappingMatchesRecipe ¶ added in v0.22.0
func VerifyMappingMatchesRecipe(recorded, deployed []recipe.NetworkInterfaceMapping) error
VerifyMappingMatchesRecipe is the recipe-vs-deployed leg: the deployed runtime must carry EXACTLY the ordered mapping the recipe recorded. Order is part of the contract — ethN is bound to the Nth GPU NIC — so this is an element-wise comparison, and any difference is a failure rather than a recorded finding. #2296's bundler-side rejection prevents this in-band; this arm catches the generated artifact being modified outside AICR.
func VerifyNetworksExist ¶ added in v0.22.0
func VerifyNetworksExist(deployed []recipe.NetworkInterfaceMapping, discovered []string) error
VerifyNetworksExist is the deployed-vs-cluster leg: every network the runtime selects must exist on this cluster. It is an explicit SET comparison, never index-wise: DiscoverGPUNICNetworks returns names sorted alphabetically, which happens to match numeric order for eight single-digit suffixes today and would silently break on any naming that does not — so the discovered list is treated as a set, and no ordering is inferred from it.
Types ¶
This section is empty.
Source Files