cluster

package
v0.0.0-...-658d539 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 24, 2026 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// ClusterLockNamespace is the namespace where the cluster lock ConfigMap is created
	ClusterLockNamespace = "default"
	// ClusterLockConfigMapName is the name of the ConfigMap used to lock the cluster
	ClusterLockConfigMapName = "e2e-cluster-lock"
)

Variables

This section is empty.

Functions

func AcquireClusterLock 

func AcquireClusterLock(ctx context.Context, kubeconfig *rest.Config, testName string) error

AcquireClusterLock creates a ConfigMap in the default namespace to indicate the cluster is busy. If the cluster is already locked, it returns an error with information about who holds the lock. Uses retry logic for transient network errors.

func AddNodesToCluster 

func AddNodesToCluster(ctx context.Context, kubeconfig *rest.Config, clusterDef *config.ClusterDefinition, baseSSHUser, baseSSHHost, sshKeyPath string) error

AddNodesToCluster adds nodes to the cluster It performs the following steps: 1. Gets bootstrap scripts from secrets 2. Runs bootstrap scripts on each node via SSH Note: NodeGroup must be created before calling this function (secrets won't appear until NodeGroup exists) Note: clusterDef must have IPAddress fields filled in for all VM nodes (via GatherVMInfo)

func BootstrapCluster 

func BootstrapCluster(ctx context.Context, sshClient ssh.SSHClient, clusterDef *config.ClusterDefinition, configPath string) error

BootstrapCluster bootstraps a Kubernetes cluster from the setup node to the first master node. It performs the following steps: 1. Logs into the Docker registry using DKP_LICENSE_KEY from config 2. Runs the dhctl bootstrap command in a Docker container Note: clusterDef must have IPAddress fields filled in for all VM nodes (via GatherVMInfo)

func CheckClusterHealth 

func CheckClusterHealth(ctx context.Context, kubeconfig *rest.Config, opts ...CheckClusterHealthOptions) error

CheckClusterHealth checks if the deckhouse deployment has 1 pod running with 2/2 containers ready in the d8-system namespace, optionally verifies that bootstrap secrets are available, and ensures webhook-handler pods are ready. This function is widely used to check cluster health after certain steps. It polls until the deployment is ready or the context times out.

func CleanupCommanderCluster 

func CleanupCommanderCluster(ctx context.Context, resources *CommanderClusterResources) error

CleanupCommanderCluster releases resources and optionally deletes the cluster from Commander. If the cluster was created by us (CreatedByUs is true) and TEST_CLUSTER_CLEANUP is enabled, the cluster will be deleted from Commander. Note: Stress namespace cleanup should be done by the caller before calling this function.

func CleanupExistingCluster 

func CleanupExistingCluster(ctx context.Context, resources *TestClusterResources) error

CleanupExistingCluster releases the cluster lock and closes connections. This should be called after using an existing cluster with UseExistingCluster. Note: Stress namespace cleanup should be done by the caller before calling this function.

func CleanupSetupVM 

func CleanupSetupVM(ctx context.Context, resources *VMResources) error

CleanupSetupVM deletes the setup VM and its associated resources. This should be called after the test cluster bootstrap is complete. Deprecated: Use RemoveVM instead.

func CleanupTestCluster 

func CleanupTestCluster(ctx context.Context, resources *TestClusterResources) error

CleanupTestCluster cleans up all resources created by CreateTestCluster. It performs cleanup in the following order: 1. Stop test cluster tunnel and close test cluster SSH client 2. Close setup SSH client 3. Re-establish base cluster tunnel if needed (for VM cleanup via API) 4. Remove setup VM (always removed) 5. Remove test cluster VMs if TEST_CLUSTER_CLEANUP is enabled 6. Stop base cluster tunnel and close base cluster SSH client

func CleanupTestClusterResources 

func CleanupTestClusterResources(testClusterResources *TestClusterResources, testPassed ...bool)

CleanupTestClusterResources cleans up test cluster resources based on the mode used. If testPassed is true, stress test namespaces will be cleaned up before closing connections.

func ClearCommanderResources 

func ClearCommanderResources()

ClearCommanderResources clears the stored Commander cluster resources

func ForceReleaseClusterLock 

func ForceReleaseClusterLock(ctx context.Context, kubeconfig *rest.Config) error

ForceReleaseClusterLock forcefully removes the cluster lock. Use with caution - this should only be used for cleanup when you're sure no other test is running.

func GatherVMInfo 

func GatherVMInfo(ctx context.Context, virtClient *virtualization.Client, namespace string, clusterDef *config.ClusterDefinition, vmResources *VMResources) error

GatherVMInfo gathers IP addresses for all VMs in the cluster definition and fills them into ClusterDefinition. This should be called once while connected to the base cluster, before switching to test cluster. It modifies clusterDef in-place by setting IPAddress field for each VM node.

func GenerateRandomSuffix 

func GenerateRandomSuffix(length int) string

GenerateRandomSuffix generates a random alphanumeric suffix of specified length

func GetNodeIPAddress 

func GetNodeIPAddress(clusterDef *config.ClusterDefinition, hostname string) (string, error)

GetNodeIPAddress gets the IP address for a node by hostname from ClusterDefinition

func GetSSHPrivateKeyPath 

func GetSSHPrivateKeyPath() (string, error)

GetSSHPrivateKeyPath returns the path to the SSH private key file. If SSHPrivateKey is a file path, it returns the expanded path. If SSHPrivateKey is a base64-encoded string, it decodes it, writes to a temporary file in temp/<test-name>/, and returns that path.

func GetSSHPublicKeyContent 

func GetSSHPublicKeyContent() (string, error)

GetSSHPublicKeyContent returns the SSH public key content as a string. If SSHPublicKey is a file path, it reads and returns the file content. If SSHPublicKey is a plain-text string, it returns it directly.

func GetSetupNode 

func GetSetupNode(clusterDef *config.ClusterDefinition) (*config.ClusterNode, error)

GetSetupNode returns the setup VM node from ClusterDefinition. The setup node is always a separate VM with a unique name (bootstrap-node-<suffix>). Note: clusterDef.Setup.Hostname must be set to the generated VM name (done by GatherVMInfo)

func GetVMIPAddress 

func GetVMIPAddress(ctx context.Context, virtClient *virtualization.Client, namespace, vmName string) (string, error)

GetVMIPAddress gets the IP address of a VM by querying its status It waits for the VM to have an IP address assigned DEPRECATED: Use GatherVMInfo to get all VM info at once, then use VMInfo.GetIPAddress

func IsClusterLocked 

func IsClusterLocked(ctx context.Context, kubeconfig *rest.Config) (bool, error)

IsClusterLocked checks if the cluster is currently locked by checking for the lock ConfigMap.

func OutputEnvironmentVariables 

func OutputEnvironmentVariables()

OutputEnvironmentVariables outputs environment variables to GinkgoWriter for debugging

func PrepareBootstrapConfig 

func PrepareBootstrapConfig(clusterDef *config.ClusterDefinition) (string, error)

PrepareBootstrapConfig prepares the bootstrap configuration file from a template. It takes cluster definition and extracts VM IP addresses to calculate the internal network CIDR. The function generates a config file and saves it to the temp/ directory. Returns the path to the generated config file. Note: clusterDef must have IPAddress fields filled in for all VM nodes (via GatherVMInfo)

func ReleaseClusterLock 

func ReleaseClusterLock(ctx context.Context, kubeconfig *rest.Config) error

ReleaseClusterLock removes the cluster lock ConfigMap. It is safe to call even if the lock doesn't exist (no error will be returned). Uses retry logic for transient network errors.

func RemoveAllVMs 

func RemoveAllVMs(ctx context.Context, resources *VMResources) error

RemoveAllVMs forcefully stops and deletes virtual machines, virtual disks, and virtual images. If a VirtualImage is in use by other resources, it will be skipped but VMs and VDs will still be deleted.

func RemoveVM 

func RemoveVM(ctx context.Context, virtClient *virtualization.Client, namespace, vmName string) error

RemoveVM removes a VM and its associated VirtualDisks, then removes the ClusterVirtualImage if not used by other VMs. It removes resources in order: VM -> VirtualDisks -> ClusterVirtualImage (if unused).

func SetCommanderResources 

func SetCommanderResources(res *CommanderClusterResources)

SetCommanderResources stores Commander cluster resources for later cleanup

func SetExtraCommanderValues 

func SetExtraCommanderValues(values map[string]interface{})

SetExtraCommanderValues sets additional values to be passed when creating a cluster via Commander. These values are merged with COMMANDER_VALUES env var (extra values take precedence over env, but prefix is always set). Call this before UseCommanderCluster() to customize cluster creation parameters.

func UploadBootstrapFiles 

func UploadBootstrapFiles(ctx context.Context, sshClient ssh.SSHClient, privateKeyPath, configPath string) error

UploadBootstrapFiles uploads the private key and config.yml file to the setup node. The private key is uploaded to /home/cloud/.ssh/id_rsa with permissions 0600. The config.yml file is uploaded to /home/cloud/config.yml.

func WaitForAllNodesReady 

func WaitForAllNodesReady(ctx context.Context, kubeconfig *rest.Config, clusterDef *config.ClusterDefinition, timeout time.Duration) error

WaitForAllNodesReady waits for all expected nodes to become Ready in parallel It validates that: 1. All expected nodes are present in the cluster 2. All nodes are in Ready state Expected nodes: all masters (including the first one that was bootstrapped) + all workers

func WaitForDockerReady 

func WaitForDockerReady(ctx context.Context, sshClient ssh.SSHClient) error

WaitForDockerReady waits for Docker to be ready on the setup node. Docker is installed via cloud-init during VM provisioning, so this function waits for cloud-init to complete and verifies Docker is working.

func WaitForSSHReady 

func WaitForSSHReady(ctx context.Context, baseSSHClient ssh.SSHClient, targetIP string) error

WaitForSSHReady waits until port 22 is reachable on the target VM through the base cluster SSH client. This should be called after VMs reach "Running" state but before attempting a full SSH connection, because cloud-init may still be configuring networking and the SSH daemon.

func WaitForTestClusterReady 

func WaitForTestClusterReady(ctx context.Context, resources *TestClusterResources) error

WaitForTestClusterReady waits for all modules in the test cluster to become Ready. It uses the ModuleDeployTimeout from config.

func WaitForWebhookHandler 

func WaitForWebhookHandler(ctx context.Context, kubeconfig *rest.Config, timeout time.Duration) error

WaitForWebhookHandler waits for the webhook-handler deployment to be ready and verifies that the deckhouse service has endpoints registered on port 4223. This should be called before attempting to create/update ModuleConfigs to avoid webhook connection refused errors.

Types

type CheckClusterHealthOptions 

type CheckClusterHealthOptions struct {
	// CheckBootstrapSecrets enables checking for bootstrap secrets.
	// Default is true. Set to false for existing clusters that don't need bootstrap secrets.
	CheckBootstrapSecrets bool
}

CheckClusterHealthOptions defines options for CheckClusterHealth

func DefaultCheckClusterHealthOptions 

func DefaultCheckClusterHealthOptions() CheckClusterHealthOptions

DefaultCheckClusterHealthOptions returns default options with all checks enabled

type ClusterLockInfo 

type ClusterLockInfo struct {
	TestName string
	LockedAt time.Time
	LockedBy string
	Hostname string
	PID      int
}

ClusterLockInfo contains information about who locked the cluster

func GetClusterLockInfo 

func GetClusterLockInfo(ctx context.Context, kubeconfig *rest.Config) (*ClusterLockInfo, error)

GetClusterLockInfo retrieves information about the current cluster lock. Returns an error if the cluster is not locked.

type CommanderClusterResources 

type CommanderClusterResources struct {
	*TestClusterResources
	CommanderClient *commander.Client
	ClusterName     string
	CreatedByUs     bool // True if we created the cluster, false if we used existing
}

CommanderClusterResources holds resources for a Commander-managed cluster

func GetCommanderResources 

func GetCommanderResources() *CommanderClusterResources

GetCommanderResources returns the stored Commander cluster resources

func UseCommanderCluster 

func UseCommanderCluster(ctx context.Context) (*CommanderClusterResources, error)

UseCommanderCluster connects to or creates a cluster via Deckhouse Commander. It performs the following steps: 1. Creates a Commander client 2. Checks if the cluster exists in Commander 3. If COMMANDER_CREATE_IF_NOT_EXISTS is true and cluster doesn't exist, creates it 4. Waits for the cluster to become ready 5. Retrieves kubeconfig and connection info 6. Establishes SSH connection to the cluster 7. Acquires cluster lock

Environment variables used: - COMMANDER_URL: URL of the Commander API - COMMANDER_TOKEN: API token for authentication - COMMANDER_CLUSTER_NAME: Name of the cluster to use/create - COMMANDER_TEMPLATE_NAME: Template for creating new clusters - COMMANDER_TEMPLATE_VERSION: Version of the template (optional) - COMMANDER_CREATE_IF_NOT_EXISTS: Whether to create cluster if it doesn't exist - COMMANDER_WAIT_TIMEOUT: Timeout for waiting for cluster to become ready

type ConnectClusterOptions 

type ConnectClusterOptions struct {
	// Direct connection parameters (used when UseJumpHost is false)
	SSHUser    string
	SSHHost    string
	SSHKeyPath string

	// Jump host parameters (used when UseJumpHost is true)
	UseJumpHost     bool
	JumpHostUser    string // Optional: defaults to SSHUser if empty
	JumpHostHost    string // Optional: defaults to SSHHost if empty
	JumpHostKeyPath string // Optional: defaults to SSHKeyPath if empty
	TargetUser      string // Required when UseJumpHost is true
	TargetHost      string // Required when UseJumpHost is true (IP or hostname)
	TargetKeyPath   string // Optional: defaults to SSHKeyPath if empty
}

ConnectClusterOptions defines options for connecting to a cluster

type OSInfo 

type OSInfo struct {
	ID            string // OS ID (e.g., "debian", "ubuntu", "centos", "redos", "astra", "altlinux")
	IDLike        string // OS ID_LIKE (e.g., "debian", "rhel fedora")
	VersionID     string // OS version ID
	PrettyName    string // OS pretty name
	KernelVersion string // Kernel version (e.g., "5.15.0-91-generic")
}

OSInfo represents detected operating system information

func GetOSInfo 

func GetOSInfo(ctx context.Context, sshClient ssh.SSHClient) (*OSInfo, error)

GetOSInfo detects the operating system and kernel version on a remote host via SSH. This function reads /etc/os-release and runs uname -r to gather OS information.

type TestClusterResources 

type TestClusterResources struct {
	SSHClient          ssh.SSHClient
	Kubeconfig         *rest.Config
	KubeconfigPath     string
	TunnelInfo         *ssh.TunnelInfo
	ClusterDefinition  *config.ClusterDefinition
	VMResources        *VMResources
	BaseClusterClient  ssh.SSHClient   // Base cluster SSH client (for cleanup)
	BaseKubeconfig     *rest.Config    // Base cluster kubeconfig (for cleanup)
	BaseKubeconfigPath string          // Base cluster kubeconfig path (for cleanup)
	BaseTunnelInfo     *ssh.TunnelInfo // Base cluster tunnel (for cleanup, may be nil if stopped)
	SetupSSHClient     ssh.SSHClient   // Setup node SSH client (for cleanup)
}

TestClusterResources holds all resources created for a test cluster connection

func ConnectToCluster 

func ConnectToCluster(ctx context.Context, opts ConnectClusterOptions) (*TestClusterResources, error)

ConnectToCluster establishes SSH connection to a cluster (base or test), retrieves kubeconfig, and sets up port forwarding tunnel.

func CreateOrConnectToTestCluster 

func CreateOrConnectToTestCluster() *TestClusterResources

CreateOrConnectToTestCluster creates a new test cluster or connects to an existing one based on configuration. Returns the TestClusterResources which should be stored for later use in tests and cleanup. Supports three modes: - alwaysUseExisting: Connect to an existing cluster - alwaysCreateNew: Create a new cluster using VMs - commander: Use Deckhouse Commander to create or use a cluster

func CreateTestCluster 

func CreateTestCluster(
	ctx context.Context,
	yamlConfigFilename string,
) (*TestClusterResources, error)

CreateTestCluster creates a complete test cluster by performing all necessary steps: 1. Loading cluster configuration from YAML 2. Connecting to base cluster 3. Verifying virtualization module is Ready 4. Creating test namespace 5. Creating virtual machines 6. Gathering VM information 7. Establishing SSH connection to setup node 8. Installing Docker on setup node 9. Preparing and uploading bootstrap config 10. Bootstrapping cluster 11. Creating NodeGroup for workers 12. Verifying cluster is ready 13. Adding nodes to cluster 14. Enabling and configuring modules

It returns all the resources needed to interact with the test cluster. SSH credentials are obtained from environment variables via config functions.

func UseExistingCluster 

func UseExistingCluster(ctx context.Context) (*TestClusterResources, error)

UseExistingCluster connects to an existing cluster without creating new VMs. It establishes SSH connection, retrieves kubeconfig, and acquires a cluster lock. The lock ensures that only one test uses the cluster at a time. SSH credentials are obtained from environment variables via config functions.

If SSH_JUMP_HOST is set, the connection will go through the jump host first. This is useful for clusters behind a bastion/jump host.

If the cluster is already locked by another test, this function will return an error. The lock is automatically released when CleanupExistingCluster is called.

type VMResources 

type VMResources struct {
	VirtClient  *virtualization.Client
	Namespace   string
	VMNames     []string
	CVMINames   []string // ClusterVirtualImage names (cluster-scoped)
	SetupVMName string   // Name of the setup VM (always created)
}

VMResources tracks VM-related resources created for a test cluster

func CreateVirtualMachines 

func CreateVirtualMachines(ctx context.Context, virtClient *virtualization.Client, clusterDef *config.ClusterDefinition) ([]string, *VMResources, error)

CreateVirtualMachines creates virtual machines from cluster definition. It handles VM name conflicts, creates all VMs in parallel, and returns the list of VM names that were created along with resource tracking info.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.