Documentation
¶
Overview ¶
Package utils provides common cluster utilities: topology validation, CLI management, node filtering, and operator health checks.
Index ¶
- Constants
- func AddConstraint(oc *exutil.CLI, nodeName string, resourceName string, targetNode string) error
- func CountReadyNodes(nodes []corev1.Node) int
- func DecodeObject[T runtime.Object](data string, target T) error
- func EnsureTNFDegradedOrSkip(oc *exutil.CLI)
- func GetMemberState(node *corev1.Node, members []*etcdserverpb.Member) (started, learner bool, err error)
- func GetMembers(etcdClientFactory helpers.EtcdClientCreator) ([]*etcdserverpb.Member, error)
- func GetNodeInternalIP(node *corev1.Node) string
- func GetNodes(oc *exutil.CLI, roleLabel string) (*corev1.NodeList, error)
- func GetReadyMasterNode(ctx context.Context, oc *exutil.CLI) (*corev1.Node, error)
- func HasNodeRebooted(oc *exutil.CLI, node *corev1.Node) (bool, error)
- func IsAPIResponding(oc *exutil.CLI) bool
- func IsClusterHealthyWithTimeout(oc *exutil.CLI, timeout time.Duration) error
- func IsClusterOperatorAvailable(operator *v1.ClusterOperator) bool
- func IsClusterOperatorDegraded(operator *v1.ClusterOperator) bool
- func IsNodeReady(oc *exutil.CLI, nodeName string) bool
- func IsResourceStopped(oc *exutil.CLI, nodeName string, resourceName string) (bool, error)
- func IsServiceRunning(oc *exutil.CLI, execNode string, targetNode string, serviceName string) bool
- func LogEtcdClusterStatus(oc *exutil.CLI, testContext string, ...) error
- func MonitorClusterOperators(oc *exutil.CLI, timeout time.Duration, FiveSecondPollInterval time.Duration) (string, error)
- func RemoveConstraint(oc *exutil.CLI, nodeName string, resourceName string) error
- func SkipIfClusterIsNotHealthy(oc *exutil.CLI, ecf *helpers.EtcdClientFactoryImpl)
- func SkipIfNotTopology(oc *exutil.CLI, wanted v1.TopologyMode)
- func StopKubeletService(oc *exutil.CLI, nodeName string) error
- func TryPacemakerCleanup(oc *exutil.CLI)
- func UnmarshalJSON[T any](jsonData string, target *T) error
- func ValidateClusterOperatorsAvailable(oc *exutil.CLI) error
- func ValidateEssentialOperatorsAvailable(oc *exutil.CLI) error
- type RecentResourceFailure
Constants ¶
const ( AllNodes = "" // No label filter for GetNodes LabelNodeRoleControlPlane = "node-role.kubernetes.io/control-plane" // Control plane node label LabelNodeRoleWorker = "node-role.kubernetes.io/worker" // Worker node label LabelNodeRoleArbiter = "node-role.kubernetes.io/arbiter" // Arbiter node label CLIPrivilegeNonAdmin = false // Standard user CLI CLIPrivilegeAdmin = true // Admin CLI with cluster-admin permissions KubeletPort = "10250" // Kubelet API port // Common poll intervals used across TNF tests FiveSecondPollInterval = 5 * time.Second // Default poll interval for most operations ThirtySecondPollInterval = 30 * time.Second // Poll interval for longer operations (VM state, provisioning) // SecretRecreationTimeout is the time to wait for operator to recreate secrets SecretRecreationTimeout = 5 * time.Minute // SecretRecreationInterval is the poll interval for secret recreation checks SecretRecreationInterval = 5 * time.Second )
Variables ¶
This section is empty.
Functions ¶
func AddConstraint ¶
AddConstraint bans a pacemaker resource from running on a specific node (temporary, doesn't survive reboots).
err := AddConstraint(oc, "master-0", "kubelet-clone", "master-1")
func CountReadyNodes ¶
CountReadyNodes returns the number of nodes in Ready state.
func DecodeObject ¶
DecodeObject decodes YAML or JSON data into a Kubernetes runtime object using generics.
var bmh metal3v1alpha1.BareMetalHost
if err := DecodeObject(yamlData, &bmh); err != nil { return err }
func EnsureTNFDegradedOrSkip ¶
EnsureTNFDegradedOrSkip skips the test if the cluster is not in TNF degraded mode (DualReplica topology with exactly one Ready control-plane node).
func GetMemberState ¶
func GetMemberState(node *corev1.Node, members []*etcdserverpb.Member) (started, learner bool, err error)
GetMemberState returns whether a node's etcd member is started and whether it's a learner
func GetMembers ¶
func GetMembers(etcdClientFactory helpers.EtcdClientCreator) ([]*etcdserverpb.Member, error)
GetMembers returns the etcd member list
func GetNodeInternalIP ¶
GetNodeInternalIP returns the internal IP address of a node, or empty string if not found.
nodeIP := GetNodeInternalIP(&node)
if nodeIP == "" { return fmt.Errorf("no internal IP for node %s", node.Name) }
func GetNodes ¶
GetNodes returns nodes filtered by role label (LabelNodeRoleControlPlane, LabelNodeRoleWorker, etc), or all nodes if roleLabel is AllNodes. This is the preferred method for retrieving nodes in tests instead of calling the Kubernetes API directly. It provides a consistent abstraction and single point of change for node retrieval logic.
allNodes, err := GetNodes(oc, AllNodes) controlPlaneNodes, err := GetNodes(oc, LabelNodeRoleControlPlane) workerNodes, err := GetNodes(oc, LabelNodeRoleWorker)
func GetReadyMasterNode ¶
GetReadyMasterNode returns the first Ready control-plane node.
func HasNodeRebooted ¶
HasNodeRebooted checks if a node has rebooted by comparing its current BootID with a previous snapshot. Returns true if the node's BootID has changed, indicating a reboot occurred.
nodeSnapshot, _ := oc.AdminKubeClient().CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{})
// ... trigger reboot ...
if rebooted, err := HasNodeRebooted(oc, nodeSnapshot); rebooted { /* node rebooted */ }
func IsAPIResponding ¶
IsAPIResponding checks if the Kubernetes API server is responding to requests. Returns true if the API responds successfully, false otherwise.
if !IsAPIResponding(oc) { /* API not ready, continue waiting */ }
func IsClusterHealthyWithTimeout ¶
IsClusterHealthyWithTimeout checks if the cluster is in a healthy state with a configurable timeout. It verifies that all nodes are ready and all cluster operators are available (not degraded or progressing). Returns an error with details if the cluster is not healthy, nil if healthy.
As a first step, this function attempts to run 'pcs resource cleanup' on a Ready node to clear any failed pacemaker resource states, maximizing the odds of etcd recovery.
if err := IsClusterHealthyWithTimeout(oc, 5*time.Minute); err != nil {
return err
}
func IsClusterOperatorAvailable ¶
func IsClusterOperatorAvailable(operator *v1.ClusterOperator) bool
IsClusterOperatorAvailable returns true if operator has Available=True condition.
if !IsClusterOperatorAvailable(etcdOperator) { return fmt.Errorf("etcd not available") }
func IsClusterOperatorDegraded ¶
func IsClusterOperatorDegraded(operator *v1.ClusterOperator) bool
IsClusterOperatorDegraded returns true if operator has Degraded=True condition.
if IsClusterOperatorDegraded(co) { return fmt.Errorf("%s degraded", coName) }
func IsNodeReady ¶
IsNodeReady checks if a node exists and is in Ready state. Returns true if the node exists and has Ready condition, false otherwise.
if !IsNodeReady(oc, "master-0") { /* node not ready, approve CSRs */ }
func IsResourceStopped ¶
IsResourceStopped checks if a pacemaker resource is in stopped state.
stopped, err := IsResourceStopped(oc, "master-0", "kubelet-clone")
func IsServiceRunning ¶
IsServiceRunning checks if a service is running on a specific target node. For kubelet in pacemaker clusters, checks the kubelet-clone resource status. execNode: the node to execute the check command from (important when target node is down) targetNode: the node to check service status for
running := IsServiceRunning(oc, "master-1", "master-0", "kubelet")
func LogEtcdClusterStatus ¶
func LogEtcdClusterStatus(oc *exutil.CLI, testContext string, etcdClientFactory *helpers.EtcdClientFactoryImpl) error
LogEtcdClusterStatus performs comprehensive etcd cluster status logging and validation. This function is designed to be used in AfterEach functions to ensure tests leave the cluster in a known good state. If etcdClientFactory is nil, member promotion status checks will be skipped.
if err := LogEtcdClusterStatus(oc, "BeforeEach validation", nil); err != nil { return err }
if err := LogEtcdClusterStatus(oc, "AfterEach cleanup", etcdClientFactory); err != nil { return err }
func MonitorClusterOperators ¶
func MonitorClusterOperators(oc *exutil.CLI, timeout time.Duration, FiveSecondPollInterval time.Duration) (string, error)
MonitorClusterOperators monitors cluster operators and ensures they are all available. Returns the cluster operators status output and an error if operators are not healthy within timeout.
output, err := MonitorClusterOperators(oc, 5*time.Minute, 15*time.Second)
func RemoveConstraint ¶
RemoveConstraint clears all pacemaker resource bans and failures for a resource (comprehensive cleanup).
err := RemoveConstraint(oc, "master-0", "kubelet-clone")
func SkipIfClusterIsNotHealthy ¶
func SkipIfClusterIsNotHealthy(oc *exutil.CLI, ecf *helpers.EtcdClientFactoryImpl)
SkipIfClusterIsNotHealthy skips the test if cluster health checks fail. Performs comprehensive validation: all nodes ready, all cluster operators healthy, etcd pods running, two voting etcd members, cluster-etcd-operator healthy, and API services available (no stale GroupVersions). Skips are detected by the test runner via preconditions.SkipMarker for JUnit generation.
SkipIfClusterIsNotHealthy(oc, etcdClientFactory)
func SkipIfNotTopology ¶
func SkipIfNotTopology(oc *exutil.CLI, wanted v1.TopologyMode)
SkipIfNotTopology skips the test if cluster topology doesn't match the wanted mode. API errors or empty topology trigger precondition failure; mismatches are valid skips.
SkipIfNotTopology(oc, v1.DualReplicaTopologyMode)
func StopKubeletService ¶
StopKubeletService stops the kubelet service on a specific node.
err := StopKubeletService(oc, "master-0")
func TryPacemakerCleanup ¶
TryPacemakerCleanup clears failed pacemaker resource and STONITH states before health checks. Finds a Ready control-plane node and runs 'pcs resource cleanup' and 'pcs stonith cleanup'. This is a best-effort operation - failures are logged but don't cause errors.
func UnmarshalJSON ¶
UnmarshalJSON parses JSON string into a Go type using generics.
var node corev1.Node
if err := UnmarshalJSON(nodeJSON, &node); err != nil { return err }
func ValidateClusterOperatorsAvailable ¶
ValidateClusterOperatorsAvailable validates that all cluster operators are available and not degraded.
if err := ValidateClusterOperatorsAvailable(oc); err != nil { return err }
func ValidateEssentialOperatorsAvailable ¶
ValidateEssentialOperatorsAvailable validates that essential cluster operators are available for kubelet disruption tests. This is more lenient than ValidateClusterOperatorsAvailable and only checks core operators needed for the test.
if err := ValidateEssentialOperatorsAvailable(oc); err != nil { return err }
Types ¶
type RecentResourceFailure ¶
type RecentResourceFailure struct {
ResourceID string
Task string
NodeName string
RC string
RCText string
LastRCChange time.Time
}
RecentResourceFailure captures details about a resource failure from Pacemaker operation history.
func HasRecentResourceFailure ¶
func HasRecentResourceFailure(oc *exutil.CLI, execNodeName string, resourceID string, timeWindow time.Duration) (bool, []RecentResourceFailure, error)
HasRecentResourceFailure checks if a resource had any failed operations within the given time window. Uses "pcs status xml" to parse the node_history section for operations with non-zero return codes. This is useful for detecting that pacemaker noticed and responded to a resource failure, even if auto-recovery has already restored the resource.
hasFailure, failures, err := HasRecentResourceFailure(oc, "master-0", "kubelet-clone", 5*time.Minute)
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package apis provides BareMetalHost utilities: status checks, provisioning state monitoring, and Metal3 operations.
|
Package apis provides BareMetalHost utilities: status checks, provisioning state monitoring, and Metal3 operations. |
|
Package core provides file utilities: permission constants, temp file creation, and template processing.
|
Package core provides file utilities: permission constants, temp file creation, and template processing. |
|
Package services provides etcd utilities: error classification, retry detection, and learner state handling.
|
Package services provides etcd utilities: error classification, retry detection, and learner state handling. |