scenario

package
v0.0.0-...-3a6d9d9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 10, 2026 License: MIT Imports: 84 Imported by: 0

Documentation

Overview

scenario_rcv1p.go contains end-to-end tests for the RCV1P (Root Certificate V1P) cert mode on Linux distros. RCV1P is the next-generation mechanism for distributing Azure root CA certificates to AKS nodes. Instead of relying on hardcoded certificate bundles, RCV1P queries the Azure wireserver at provisioning time to download the latest root certificates and installs them into the OS trust store.

RCV1P requires two conditions on a subscription:

  • The Microsoft.Compute/PlatformSettingsOverride feature flag must be registered.
  • The VMSS must have the tag "platformsettings.host_environment.service.platform_optedin_for_rootcerts=true". On subscriptions with the feature flag, the platform may auto-inject this tag on all VMSSes.

RCV1P tests run against whichever subscription E2E_SUBSCRIPTION_ID points at; the RCV1P pipeline job overrides this to an RCV1P-registered subscription. Positive tests always run and verify cert installation. Negative tests are skipped when RCV1P_TAGS_AUTO_INJECTED=true (platform auto-injects the opt-in tag, making the "no tag" scenario impossible to reproduce).

REVERT ME: this file uses rcv1pWindowsCSEMutator to override CseScriptsPackageURL with a branch-built CSE zip. Remove those overrides once the RCV1P code ships in a published CSE package.

This file contains end-to-end scenarios for the RCV1P cert mode on Windows. Windows uses a different cert installation path than Linux: certificates are downloaded to C:\ca and imported into the Windows root or intermediate LocalMachine certificate store. A scheduled task (aks-ca-certs-refresh-task) is registered to periodically refresh the certificates.

Index

Constants

View Source 
const (
	SharedVNetName        = "abe2e-shared-vnet"
	SharedVNetCIDR        = "10.0.0.0/8"
	SharedVNetIPv6CIDR    = "fd00::/48"
	SharedBastionName     = "abe2e-shared-bastion"
	SharedBastionPIPName  = "abe2e-shared-bastion-pip"
	SharedClusterIdentity = "abe2e-cluster-identity"
	BastionSubnetCIDR     = "10.0.0.0/26"
	FirewallSubnetCIDR    = "10.0.1.0/24"
	PESubnetName          = "abe2e-pe-subnet"
	PESubnetCIDR          = "10.0.2.0/24"
)
View Source 
const (
	SharedFirewallName    = "abe2e-fw"
	SharedFirewallPIPName = "abe2e-fw-pip"
)
View Source 
const (
	CleanupTimeout = 5 * time.Minute
)

Variables

View Source 
var AllowedSSHPrefixes = []string{ssh.KeyAlgoED25519, ssh.KeyAlgoRSA, ssh.KeyAlgoRSASHA256, ssh.KeyAlgoRSASHA512}
View Source 
var CachedBranchCSEPackageURL = cachedFunc(buildBranchCSEPackageURL)

CachedBranchCSEPackageURL builds a CSE zip from staging/cse/windows/ (matching the pipeline packaging in .pipelines/scripts/windows_package_cse.sh), uploads it to the E2E blob storage and returns a SAS-signed URL. The result, including any error, is cached so the zip is built and uploaded at most once per location across all parallel tests.

View Source 
var CachedCompileAndUploadAKSNodeController = cachedFunc(compileAndUploadAKSNodeController)
View Source 
var CachedCreateGallery = cachedFunc(createGallery)
View Source 
var CachedCreateGalleryImage = cachedFunc(createGalleryImage)
View Source 
var CachedCreateVMManagedIdentity = cachedFunc(func(ctx context.Context, location string) (string, error) {
	return config.Azure.CreateVMManagedIdentity(ctx, location)
})
View Source 
var CachedEnsureClusterSubnet = cachedFunc(ensureClusterSubnet)
View Source 
var CachedEnsureResourceGroup = cachedFunc(ensureResourceGroup)
View Source 
var CachedEnsureSharedInfra = cachedFunc(ensureSharedInfra)
View Source 
var CachedGetLatestVMExtensionImageVersion = cachedFunc(
	func(ctx context.Context, req GetLatestExtensionVersionRequest) (string, error) {
		return config.Azure.GetLatestVMExtensionImageVersion(ctx, req.Location, req.ExtType, req.Publisher)
	},
)

CachedGetLatestVMExtensionImageVersion caches the result of querying the Azure API for the latest VM extension image version.

View Source 
var CachedIsVMSizeGen2Only = cachedFunc(func(ctx context.Context, req VMSizeSKURequest) (bool, error) {
	return config.Azure.IsVMSizeGen2Only(ctx, req.Location, req.VMSize)
})

CachedIsVMSizeGen2Only caches the result of querying the Azure Resource SKUs API to determine if a VM size only supports the Gen2 hypervisor.

View Source 
var CachedPlatformSettingsOverrideFeatureFlag = cachedFunc(queryFeatureFlag)

CachedPlatformSettingsOverrideFeatureFlag checks the Microsoft.Compute/PlatformSettingsOverride feature flag, caching the result (including errors) per subscription ID.

View Source 
var CachedPrepareVHD = cachedFunc(prepareVHD)
View Source 
var CachedVMSizeSupportsNVMe = cachedFunc(func(ctx context.Context, req VMSizeSKURequest) (bool, error) {
	return config.Azure.VMSizeSupportsNVMe(ctx, req.Location, req.VMSize)
})

CachedVMSizeSupportsNVMe caches the result of querying the Azure Resource SKUs API to determine if a VM size supports the NVMe disk controller type.

View Source 
var ClusterAzureBootstrapProfileCache = cachedFunc(clusterAzureBootstrapProfileCache)
View Source 
var ClusterAzureNetwork = cachedFunc(clusterAzureNetwork)
View Source 
var ClusterAzureNetworkIsolated = cachedFunc(clusterAzureNetworkIsolated)
View Source 
var ClusterAzureOverlayNetwork = cachedFunc(clusterAzureOverlayNetwork)
View Source 
var ClusterAzureOverlayNetworkDualStack = cachedFunc(clusterAzureOverlayNetworkDualStack)
View Source 
var ClusterCiliumNetwork = cachedFunc(clusterCiliumNetwork)
View Source 
var ClusterKubenet = cachedFunc(clusterKubenet)
View Source 
var ClusterLatestKubernetesVersion = cachedFunc(clusterLatestKubernetesVersion)
View Source 
var ClusterLatestKubernetesVersionAzureBootstrapProfileCache = cachedFunc(clusterLatestKubernetesVersionAzureBootstrapProfileCache)
View Source 
var ClusterLatestKubernetesVersionAzureNetwork = cachedFunc(clusterLatestKubernetesVersionAzureNetwork)
View Source 
var ClusterLatestKubernetesVersionAzureOverlayNetworkDualStack = cachedFunc(clusterLatestKubernetesVersionAzureOverlayNetworkDualStack)
View Source 
var ClusterLatestKubernetesVersionKubenet = cachedFunc(clusterLatestKubernetesVersionKubenet)

Functions

func CreateImage 

func CreateImage(ctx context.Context, s *Scenario) (*config.Image, error)

func CreateSIGImageVersionFromDisk 

func CreateSIGImageVersionFromDisk(ctx context.Context, s *Scenario, version string, diskResourceID string) (*config.Image, error)

CreateSIGImageVersionFromDisk creates a new SIG image version directly from a VM disk

func CustomDataWithNBCCmdHack 

func CustomDataWithNBCCmdHack(customData, binaryURL string) (string, error)

CustomDataWithNBCCmdHack is similar to baker.boothooktemplate, but it uses a hack to run new aks-node-controller binary. Original aks-node-controller isn't run because it fails systemd check validating aks-node-controller-config.json exists (check aks-node-controller.service for details). with a coreos.units block to define and start the service instead.

func DialSSHOverBastion 

func DialSSHOverBastion(
	ctx context.Context,
	bastion *Bastion,
	vmPrivateIP string,
	sshPrivateKey []byte,
) (*ssh.Client, error)

func DualStackConfigMutator 

func DualStackConfigMutator(_ *Cluster, configuration *datamodel.NodeBootstrappingConfiguration)

func DualStackVMConfigMutator 

func DualStackVMConfigMutator(set *armcompute.VirtualMachineScaleSet)

func EmptyBootstrapConfigMutator 

func EmptyBootstrapConfigMutator(_ *Cluster, configuration *datamodel.NodeBootstrappingConfiguration)

func EmptyVMConfigMutator 

func EmptyVMConfigMutator(vmss *armcompute.VirtualMachineScaleSet)

func GetFieldFromJsonObjectOnNode 

func GetFieldFromJsonObjectOnNode(ctx context.Context, s *Scenario, fileName string, jsonPath string) (string, error)

func RebootVMAndWaitForSSH 

func RebootVMAndWaitForSSH(ctx context.Context, s *Scenario) error

func RestartNodeProblemDetector 

func RestartNodeProblemDetector(ctx context.Context, s *Scenario) error

func RunCommand 

func RunCommand(ctx context.Context, s *Scenario, command string) (armcompute.VirtualMachineRunCommandInstanceView, error)

RunCommand executes a script on the VMSS VM with the configured instance ID via the Azure VMSS RunCommand v2 API (VirtualMachineRunCommand resource). This is the API already used by production aks-rp PIS code; using it here keeps test and production on the same surface and avoids the v1 RunCommand extension's failure modes (e.g. the Microsoft.CPlat.Core/RunCommandWindows "Keyset does not exist" error fixed by ADO PR https://msazure.visualstudio.com/CloudNativeCompute/_git/aks-rp/pullrequest/15721814).

Unlike SSH-based exec, this works even when WinRM/SSH are unavailable (e.g. mid-sysprep). It is generally slower than SSH because each call creates a VirtualMachineRunCommand resource on the VM and waits for it to provision.

func ServiceCanRestartValidator 

func ServiceCanRestartValidator(ctx context.Context, s *Scenario, serviceName string, restartTimeoutInSeconds int) error

func ValidateACLFIPSEnabled 

func ValidateACLFIPSEnabled(ctx context.Context, s *Scenario) error

ValidateACLFIPSEnabled asserts ACL-specific FIPS markers are present on the node: the /etc/system-fips marker file written by vhdbuilder/scripts/linux/acl/tool_installs_acl.sh. Kernel FIPS mode (/proc/sys/crypto/fips_enabled == 1) is universal and is asserted by ValidateFIPSProvider; callers should compose the two validators when both are needed.

func ValidateAKSLocalDNSHostsSetupService 

func ValidateAKSLocalDNSHostsSetupService(ctx context.Context, s *Scenario) error

ValidateAKSLocalDNSHostsSetupService checks that aks-localdns-hosts-setup.service ran successfully and the aks-localdns-hosts-setup.timer is active to ensure periodic refresh of /etc/localdns/hosts.

func ValidateAKSLogCollector 

func ValidateAKSLogCollector(ctx context.Context, s *Scenario) error

func ValidateANCLauncherOutput 

func ValidateANCLauncherOutput(ctx context.Context, s *Scenario, expectedContent string) error

ValidateANCLauncherOutput checks that the aks-node-controller-launcher.sh output contains expectedContent, regardless of how the VHD launches it:

  • ACL/Flatcar VHDs still launch the launcher via the aks-node-controller.service systemd unit (ignition doesn't support cloud-boothooks), so its stdout/stderr only lands in the journal.
  • All other VHDs launch the launcher as a direct fork from the cloud-boothook (not a systemd unit, for faster dispatch - see baker.go boothookTemplate), with stdout/stderr redirected to /var/log/azure/aks-node-controller.output.

func ValidateAcceleratedNetworkingTrafficFlowing 

func ValidateAcceleratedNetworkingTrafficFlowing(ctx context.Context, s *Scenario) error

ValidateAcceleratedNetworkingTrafficFlowing checks that network traffic is actually flowing through the accelerated networking VF rather than the slower synthetic (NetVSC) path. It sends HTTP requests from a pod to the node's default gateway and verifies that the VF TX packet counters increase by at least that amount.

func ValidateAcceleratedNetworkingVFBonded 

func ValidateAcceleratedNetworkingVFBonded(ctx context.Context, s *Scenario) error

ValidateAcceleratedNetworkingVFBonded checks that the accelerated networking VF interface exists and is properly bonded to the primary eth0 interface.

func ValidateAcceleratedNetworkingVFHardware 

func ValidateAcceleratedNetworkingVFHardware(ctx context.Context, s *Scenario) error

ValidateAcceleratedNetworkingVFHardware verifies the accelerated networking VF is backed by a PCI function and bound to a kernel network driver.

func ValidateAppArmorBasic 

func ValidateAppArmorBasic(ctx context.Context, s *Scenario) error

ValidateAppArmorBasic validates that AppArmor is running without requiring aa-status

func ValidateArtifactStreamingImagePull 

func ValidateArtifactStreamingImagePull(ctx context.Context, s *Scenario) error

ValidateArtifactStreamingImagePull verifies that artifact streaming actually streams an image on pod launch, rather than merely bootstrapping the overlaybd/acr-mirror services.

Unlike the existing artifact-streaming scenarios (which only assert that overlaybd-snapshotter, overlaybd-tcmu and acr-mirror are running and that /etc/overlaybd exists), this validator:

  1. ensures an overlaybd-converted (artifact-streaming) image exists in the e2e private ACR,
  2. launches a pod from that image and waits for it to run (proving the image was pullable), and
  3. asserts on the node that overlaybd opened a TCMU-backed block device for it — the definitive signal that the image was *streamed* on demand rather than downloaded and unpacked into overlayfs (the fallback path taken for plain OCI images like busybox).

Uses the cluster's ANONYMOUS-pull private ACR (Cluster: ClusterAzureBootstrapProfileCache, which attaches one). Anonymous pull is required because on the standalone e2e VMSS node the acr-mirror service has no managed identity to obtain an AAD token, so it cannot authenticate to a non-anonymous ACR to serve the overlaybd streaming manifest — the pull then silently falls back to overlayfs. Against an anonymous-pull ACR, acr-mirror's anonymous path succeeds and streaming works. (Observed acr-mirror error on a non-anon ACR: "Error with azure sdk, request token error" -> "falling back to anonymous auth" -> 503.)

func ValidateAzureNetworkFiles 

func ValidateAzureNetworkFiles(ctx context.Context, s *Scenario) error

ValidateAzureNetworkFiles checks that udev rules files exist.

func ValidateCiliumIsNotRunningWindows 

func ValidateCiliumIsNotRunningWindows(ctx context.Context, s *Scenario) error

func ValidateCiliumIsRunningWindows 

func ValidateCiliumIsRunningWindows(ctx context.Context, s *Scenario) error

func ValidateCollectWindowsLogsScript 

func ValidateCollectWindowsLogsScript(ctx context.Context, s *Scenario) error

ValidateCollectWindowsLogsScript runs c:\k\debug\collect-windows-logs.ps1 on the node and verifies that a zip archive was produced by the script.

func ValidateCommonLinux 

func ValidateCommonLinux(ctx context.Context, s *Scenario) error

func ValidateCommonWindows 

func ValidateCommonWindows(ctx context.Context, s *Scenario) error

func ValidateContainerRuntimePlugins 

func ValidateContainerRuntimePlugins(ctx context.Context, s *Scenario) error

func ValidateContainerd2Properties 

func ValidateContainerd2Properties(ctx context.Context, s *Scenario, versions []string) error

func ValidateContainerdWindowsPriorityClass 

func ValidateContainerdWindowsPriorityClass(ctx context.Context, s *Scenario) error

ValidateContainerdWindowsPriorityClass verifies that the containerd service is registered with nssm's AppPriority set to ABOVE_NORMAL_PRIORITY_CLASS, and that the running containerd process actually has that OS process priority class applied.

func ValidateCustomLinuxOSConfigPersistsAfterReboot 

func ValidateCustomLinuxOSConfigPersistsAfterReboot(ctx context.Context, s *Scenario, customSysctls map[string]string, customContainerdUlimits map[string]string, swapFileSizeMB int32, thpEnabled, thpDefrag string) error

func ValidateDRAWorkloadSchedulable 

func ValidateDRAWorkloadSchedulable(ctx context.Context, s *Scenario) (err error)

func ValidateDirectoryContent 

func ValidateDirectoryContent(ctx context.Context, s *Scenario, path string, files []string) error

func ValidateDiskQueueService 

func ValidateDiskQueueService(ctx context.Context, s *Scenario) error

func ValidateDllIsNotLoadedWindows 

func ValidateDllIsNotLoadedWindows(ctx context.Context, s *Scenario, dllName string) error

func ValidateDllLoadedWindows 

func ValidateDllLoadedWindows(ctx context.Context, s *Scenario, dllName string) error

func ValidateDotnetNotInstalledWindows 

func ValidateDotnetNotInstalledWindows(ctx context.Context, s *Scenario) error

func ValidateDraDriverNvidiaGpuServiceRunning 

func ValidateDraDriverNvidiaGpuServiceRunning(ctx context.Context, s *Scenario) error

func ValidateEmptyDirectory 

func ValidateEmptyDirectory(ctx context.Context, s *Scenario, dirName string) error

func ValidateEnableNvidiaResource 

func ValidateEnableNvidiaResource(ctx context.Context, s *Scenario) error

func ValidateFIPSProvider 

func ValidateFIPSProvider(ctx context.Context, s *Scenario) error

ValidateFIPSProvider verifies that FIPS is properly configured on the node:

  1. Kernel FIPS mode is enabled (/proc/sys/crypto/fips_enabled == 1).
  2. OpenSSL (3.x) has an active FIPS or SymCrypt provider loaded. The check is skipped on hosts shipping OpenSSL 1.1.x (e.g. Ubuntu 20.04 FIPS), which use the legacy FIPS module rather than the providers interface.
  3. /opt/cni/bin/portmap runs without panicking (regression guard for ICM 51000001009688 where the OpenSSL FIPS provider was not loaded on AzureLinux V3 FIPS nodes).

func ValidateFileDoesNotExist 

func ValidateFileDoesNotExist(ctx context.Context, s *Scenario, fileName string) error

func ValidateFileExcludesContent 

func ValidateFileExcludesContent(ctx context.Context, s *Scenario, fileName string, contents string) error

ValidateFileExcludesContent fails the test if the specified file contains the specified contents. The contents doesn't need to be surrounded by non-word characters. E.g.: searching "bcd" in "abcdef" is a match, thus the validation fails.

func ValidateFileExcludesExactContent 

func ValidateFileExcludesExactContent(ctx context.Context, s *Scenario, fileName string, contents string) error

ValidateFileExcludesExactContent fails the test if the specified file contains the specified contents. The contents needs to be surrounded by non-word characters. E.g.: searching "bcd" in "abcdef" is not a match, thus the validation passes.

func ValidateFileExists 

func ValidateFileExists(ctx context.Context, s *Scenario, fileName string) error

func ValidateFileHasContent 

func ValidateFileHasContent(ctx context.Context, s *Scenario, fileName string, contents string) error

ValidateFileHasContent passes the test if the specified file contains the specified contents. The contents doesn't need to be surrounded by non-word characters. E.g.: searching "bcd" in "abcdef" is a match, thus the validation passes.

func ValidateFileIsRegularFile 

func ValidateFileIsRegularFile(ctx context.Context, s *Scenario, fileName string) error

func ValidateGPUWorkloadSchedulable 

func ValidateGPUWorkloadSchedulable(ctx context.Context, s *Scenario, gpuCount int, resourceName string) error

func ValidateIMDSRestrictionRule 

func ValidateIMDSRestrictionRule(ctx context.Context, s *Scenario, table string) error

func ValidateIPTablesCompatibleWithCiliumEBPF 

func ValidateIPTablesCompatibleWithCiliumEBPF(ctx context.Context, s *Scenario) error

ValidateIPTablesCompatibleWithCiliumEBPF validates that all iptables rules in each table match the provided patterns which are accounted for when eBPF host routing is enabled.

func ValidateInspektorGadget 

func ValidateInspektorGadget(ctx context.Context, s *Scenario) error

func ValidateInstalledPackageVersion 

func ValidateInstalledPackageVersion(ctx context.Context, s *Scenario, component, version string) error

func ValidateJournalctlOutput 

func ValidateJournalctlOutput(ctx context.Context, s *Scenario, serviceName string, expectedContent string) error

ValidateJournalctlOutput checks if specific content exists in the systemd service logs

func ValidateJsonFileDoesNotHaveField 

func ValidateJsonFileDoesNotHaveField(ctx context.Context, s *Scenario, fileName string, jsonPath string, valueNotToBe string) error

func ValidateJsonFileHasField 

func ValidateJsonFileHasField(ctx context.Context, s *Scenario, fileName string, jsonPath string, expectedValue string) error

func ValidateKataContainerdConfig 

func ValidateKataContainerdConfig(ctx context.Context, s *Scenario) error

ValidateKataContainerdConfig asserts that AgentBaker rendered a containerd configuration containing the Kata runtime handlers on a Kata-enabled VHD.

This is the core regression check for the IsKata blocks of the containerd config templates in pkg/agent/baker.go. Note that AgentPoolProfile.IsContainerdV2Distro() returns false for every Kata distro (pkg/agent/datamodel/types.go), so Kata nodes are always rendered from containerdV1ConfigTemplate / containerdV1NoGPUConfigTemplate regardless of the underlying OS. The assertions below therefore target the containerd 1.x plugin paths that those templates emit. If Kata is ever promoted to the V2 templates, this validator should fail loudly rather than silently pass, which is why the plugin paths are asserted explicitly.

func ValidateKataContainerdConfigDump 

func ValidateKataContainerdConfigDump(ctx context.Context, s *Scenario) error

ValidateKataContainerdConfigDump asserts that containerd itself accepted the rendered configuration and actually loaded the Kata runtime handlers.

Checking the file alone is not enough. Kata VHDs ship their own containerd build - CSE skips installing one (see the "azurelinuxkata" entries in parts/common/components.json) - so the containerd major version on the node is decided by the image, not by AgentBaker, while the template AgentBaker renders is decided by the distro (IsContainerdV2Distro short-circuits to the v1 template for every Kata distro). The two can therefore disagree: AzureLinux V3 Kata currently boots containerd 2.x while being handed a containerd 1.x style config.

That combination happens to work today because containerd 2.x migrates the legacy "io.containerd.grpc.v1.cri" runtime handlers onto the current "io.containerd.cri.v1.runtime" paths, but nothing guarantees it keeps doing so. This validator pins the property we actually care about: after containerd has parsed the config, the Kata handlers are present in the effective configuration and containerd raised no warnings while getting there.

func ValidateKataErofsContainerdConfig 

func ValidateKataErofsContainerdConfig(ctx context.Context, s *Scenario) error

ValidateKataErofsContainerdConfig checks that the EROFS snapshotter is configured and that containerd loaded all of its EROFS plugins successfully.

func ValidateKataHostReadiness 

func ValidateKataHostReadiness(ctx context.Context, s *Scenario) error

ValidateKataHostReadiness asserts the host-side prerequisites that the Kata VHD is expected to ship and that the containerd config references. Without these, the containerd config would be syntactically valid but the kata shim would fail at pod sandbox creation time.

func ValidateKataPodIsIsolated 

func ValidateKataPodIsIsolated(ctx context.Context, s *Scenario, handler string) error

ValidateKataPodIsIsolated creates a RuntimeClass bound to the given Kata runtime handler, schedules a pod against it on the node under test, and asserts the pod is genuinely running inside a Kata VM.

This is the end-to-end proof that the containerd config AgentBaker generated is not merely syntactically present but actually usable: if the runtime handler were missing or misconfigured, the kubelet would reject the pod with "RuntimeHandler not supported" and the pod would never reach Running.

Isolation itself is asserted by comparing kernel releases. A Kata pod boots its own guest kernel, so it must report a different `uname -r` than the host; a matching value would mean the pod silently fell back to the shared-kernel runc runtime.

The RuntimeClass is pinned to this scenario's node via Scheduling.NodeSelector so it cannot interfere with other scenarios running in parallel against the same cluster, and is named after the handler so that several handlers can be validated on one node.

func ValidateKernelLogs 

func ValidateKernelLogs(ctx context.Context, s *Scenario) error

ValidateKernelLogs checks kernel logs for critical errors across multiple categories: - Kernel panics/crashes (panic, oops, call trace, BUG, etc.) - CPU lockups/stalls (soft/hard lockup, RCU stall, hung task, watchdog) - Memory issues (OOM killer, page allocation failure, memory corruption) - I/O and filesystem errors (I/O error, filesystem errors, nvme/ata/scsi errors)

func ValidateKubeletActiveFlagsEvent 

func ValidateKubeletActiveFlagsEvent(ctx context.Context, s *Scenario) error

ValidateKubeletActiveFlagsEvent checks that the emit-kubelet-active-flags oneshot service ran successfully and produced a guest agent event file containing kubelet config telemetry. Guarded: skips gracefully on VHDs that don't have the service baked in yet.

func ValidateKubeletArgs 

func ValidateKubeletArgs(ctx context.Context, s *Scenario) error

func ValidateKubeletHasFlags 

func ValidateKubeletHasFlags(ctx context.Context, s *Scenario, filePath string) error

ValidateKubeletHasFlags checks kubelet is started with the right flags and configs.

func ValidateKubeletHasNotStopped 

func ValidateKubeletHasNotStopped(ctx context.Context, s *Scenario) error

func ValidateKubeletNodeIP 

func ValidateKubeletNodeIP(ctx context.Context, s *Scenario) error

func ValidateKubeletServingCertificateRotation 

func ValidateKubeletServingCertificateRotation(ctx context.Context, s *Scenario) error

func ValidateLeakedSecrets 

func ValidateLeakedSecrets(ctx context.Context, s *Scenario) error

func ValidateLocalDNSExporterMetrics 

func ValidateLocalDNSExporterMetrics(ctx context.Context, s *Scenario) error

ValidateLocalDNSExporterMetrics checks if the localdns metrics exporter is working and exports the expected VnetDNS and KubeDNS forward IP metrics.

The validation script is too large (~18KB) to send as a single command over bastion SSH tunnels which have an 8KB WebSocket buffer limit. To work around this, we encode the script in base64, upload it in small chunks via multiple SSH commands, then decode and execute it on the VM.

func ValidateLocalDNSHostsFile 

func ValidateLocalDNSHostsFile(ctx context.Context, s *Scenario, fqdns []string) error

ValidateLocalDNSHostsFile checks that /etc/localdns/hosts contains at least one IPv4 entry for each critical FQDN. This validation approach avoids flakiness with CDN/frontdoor-backed FQDNs (like mcr.microsoft.com) whose A records can rotate between queries. We verify presence, not exact IP matching. The hosts file is populated asynchronously by the aks-localdns-hosts-setup timer/service, so we poll with a timeout.

func ValidateLocalDNSHostsPluginBypass 

func ValidateLocalDNSHostsPluginBypass(ctx context.Context, s *Scenario) error

ValidateLocalDNSHostsPluginBypass verifies that localdns serves FQDNs from /etc/localdns/hosts via the CoreDNS hosts plugin. It checks:

  1. The node has the kubernetes.azure.com/localdns-hosts-plugin=enabled annotation
  2. The Corefile has the hosts plugin configured in both VnetDNS and KubeDNS listeners
  3. The IPs returned by dig match the entries in /etc/localdns/hosts for the same FQDN

We intentionally do NOT assert on DNS flags (AA, RA) because CoreDNS can set these regardless of which plugin served the response.

func ValidateLocalDNSHostsPluginColdStart 

func ValidateLocalDNSHostsPluginColdStart(ctx context.Context, s *Scenario) error

ValidateLocalDNSHostsPluginColdStart verifies that localdns works correctly when started with an empty hosts file — the exact scenario that occurs when localdns starts before aks-localdns-hosts-setup finishes resolving FQDNs.

Test flow:

  1. Truncate hosts file, stop/start localdns — CoreDNS starts fresh with empty hosts file and empty cache
  2. Verify critical and non-critical FQDNs resolve via fallthrough (upstream DNS)
  3. Populate hosts file with a canary entry (simulates aks-localdns-hosts-setup completing)
  4. Wait for CoreDNS reload (5s), verify canary resolves (hosts plugin picks up new file)
  5. Restore original hosts file and stop/start localdns to leave node in clean state

func ValidateLocalDNSHostsPluginIPv6 

func ValidateLocalDNSHostsPluginIPv6(ctx context.Context, s *Scenario) error

ValidateLocalDNSHostsPluginIPv6 checks that IPv6 entries in /etc/localdns/hosts are properly served by CoreDNS's hosts plugin. If the hosts file has no IPv6 entries (some FQDNs don't have AAAA records), the test is skipped gracefully.

Test flow:

  1. Find the first FQDN with an IPv6 entry in the hosts file
  2. Query localdns for AAAA records for that FQDN
  3. Verify the returned IPv6 addresses match the hosts file entries

func ValidateLocalDNSResolution 

func ValidateLocalDNSResolution(ctx context.Context, s *Scenario, server string) error

ValidateLocalDNSResolution checks if the DNS resolution for an external domain is successful from localdns clusterlistenerIP. It uses the 'dig' command to check the DNS resolution and expects a successful response.

func ValidateLocalDNSService 

func ValidateLocalDNSService(ctx context.Context, s *Scenario, state string) error

ValidateLocalDNSService checks if the localdns service is in the expected state (enabled or disabled).

func ValidateMANA 

func ValidateMANA(ctx context.Context, s *Scenario) error

ValidateMANA runs all MANA (Microsoft Azure Network Adapter) checks. It verifies that the MANA PCI device is present, the kernel driver is loaded, the VF interface is bonded to eth0, PCI-backed and driver-bound, and traffic is flowing through the VF.

func ValidateMANADriverLoaded 

func ValidateMANADriverLoaded(ctx context.Context, s *Scenario) error

ValidateMANADriverLoaded checks that the MANA Ethernet driver (mana) is loaded in the running kernel. For built-in drivers they appear in modules.builtin; for loadable modules they must be present in lsmod.

func ValidateMANAPCIDevice 

func ValidateMANAPCIDevice(ctx context.Context, s *Scenario) error

ValidateMANAPCIDevice checks that the MANA PCI device is exposed to the VM. MANA hardware is identified by PCI device ID 0x00ba (Microsoft Corporation).

func ValidateMANATrafficFlowing 

func ValidateMANATrafficFlowing(ctx context.Context, s *Scenario) error

ValidateMANATrafficFlowing checks that network traffic is actually flowing through the MANA Virtual Function rather than the slower synthetic (NetVSC) path.

func ValidateMANAVFBonded 

func ValidateMANAVFBonded(ctx context.Context, s *Scenario) error

ValidateMANAVFBonded checks that the MANA Virtual Function (VF) interface exists and is properly bonded to the primary eth0 interface. When Accelerated Networking is enabled with MANA, a VF interface should appear as a subordinate (SLAVE) of eth0. The VF name varies by VM generation: - V5: enP* (e.g., enP30832p0s0) - V6+: ens1 or enp0s0

func ValidateMIGInstanceProfileCounts 

func ValidateMIGInstanceProfileCounts(ctx context.Context, s *Scenario, expected map[string]int) error

func ValidateMIGInstancesCreated 

func ValidateMIGInstancesCreated(ctx context.Context, s *Scenario, migProfile string, instanceCountExpected int) error

func ValidateMIGModeEnabled 

func ValidateMIGModeEnabled(ctx context.Context, s *Scenario, gpuCountExpected int) error

func ValidateMultipleKubeProxyVersionsExist 

func ValidateMultipleKubeProxyVersionsExist(ctx context.Context, s *Scenario) error

func ValidateNPDFilesystemCorruption 

func ValidateNPDFilesystemCorruption(ctx context.Context, s *Scenario) (err error)

func ValidateNPDGPUCountAfterFailure 

func ValidateNPDGPUCountAfterFailure(ctx context.Context, s *Scenario) error

func ValidateNPDGPUCountCondition 

func ValidateNPDGPUCountCondition(ctx context.Context, s *Scenario) error

func ValidateNPDGPUCountPlugin 

func ValidateNPDGPUCountPlugin(ctx context.Context, s *Scenario) error

func ValidateNPDHealthyNvidiaGridLicenseStatus 

func ValidateNPDHealthyNvidiaGridLicenseStatus(ctx context.Context, s *Scenario) error

func ValidateNPDIBLinkFlappingAfterFailure 

func ValidateNPDIBLinkFlappingAfterFailure(ctx context.Context, s *Scenario) error

func ValidateNPDIBLinkFlappingCondition 

func ValidateNPDIBLinkFlappingCondition(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaDCGMServices 

func ValidateNPDUnhealthyNvidiaDCGMServices(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaDCGMServicesAfterFailure 

func ValidateNPDUnhealthyNvidiaDCGMServicesAfterFailure(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaDCGMServicesCondition 

func ValidateNPDUnhealthyNvidiaDCGMServicesCondition(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaDevicePlugin 

func ValidateNPDUnhealthyNvidiaDevicePlugin(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaDevicePluginAfterFailure 

func ValidateNPDUnhealthyNvidiaDevicePluginAfterFailure(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaDevicePluginCondition 

func ValidateNPDUnhealthyNvidiaDevicePluginCondition(ctx context.Context, s *Scenario) error

func ValidateNPDUnhealthyNvidiaGridLicenseStatusAfterFailure 

func ValidateNPDUnhealthyNvidiaGridLicenseStatusAfterFailure(ctx context.Context, s *Scenario) error

func ValidateNetworkInterfaceConfig 

func ValidateNetworkInterfaceConfig(ctx context.Context, s *Scenario, nicConfig map[string]string) error

ValidateNetworkInterfaceConfig validates network interface configuration settings using ethtool. It identifies network interfaces with slot names matching the enP* pattern (same logic as the udev rule), then verifies that each interface has the expected configuration settings (e.g., rx buffer size). The nicConfig map specifies the ethtool settings to validate (key: setting name, value: expected value).

func ValidateNoFailedSystemdUnits 

func ValidateNoFailedSystemdUnits(ctx context.Context, s *Scenario) error

func ValidateNodeAdvertisesExactGPUResources 

func ValidateNodeAdvertisesExactGPUResources(ctx context.Context, s *Scenario, expected map[string]int64) error

func ValidateNodeAdvertisesGPUResources 

func ValidateNodeAdvertisesGPUResources(ctx context.Context, s *Scenario, gpuCountExpected int64, resourceName string) error

func ValidateNodeCanRunAPod 

func ValidateNodeCanRunAPod(ctx context.Context, s *Scenario) error

func ValidateNodeCanScaleToCapacity 

func ValidateNodeCanScaleToCapacity(ctx context.Context, s *Scenario) (retErr error)

ValidateNodeCanScaleToCapacity fills the scenario node's allocatable pod capacity.

func ValidateNodeExporter 

func ValidateNodeExporter(ctx context.Context, s *Scenario) error

func ValidateNodeHasLabel 

func ValidateNodeHasLabel(ctx context.Context, s *Scenario, labelKey, expectedValue string) error

ValidateNodeHasLabel checks if the node has the expected label with the expected value

func ValidateNodeProblemDetector 

func ValidateNodeProblemDetector(ctx context.Context, s *Scenario) error

func ValidateNonEmptyDirectory 

func ValidateNonEmptyDirectory(ctx context.Context, s *Scenario, dirName string) error

func ValidateNvidiaDCGMExporterIsScrapable 

func ValidateNvidiaDCGMExporterIsScrapable(ctx context.Context, s *Scenario) error

func ValidateNvidiaDCGMExporterScrapeCommonMetric 

func ValidateNvidiaDCGMExporterScrapeCommonMetric(ctx context.Context, s *Scenario, metric string) error

func ValidateNvidiaDCGMExporterSystemDServiceRunning 

func ValidateNvidiaDCGMExporterSystemDServiceRunning(ctx context.Context, s *Scenario) error

func ValidateNvidiaDevicePluginMIGStrategy 

func ValidateNvidiaDevicePluginMIGStrategy(ctx context.Context, s *Scenario, strategy string) error

func ValidateNvidiaDevicePluginServiceRunning 

func ValidateNvidiaDevicePluginServiceRunning(ctx context.Context, s *Scenario) error

func ValidateNvidiaGRIDLicenseValid 

func ValidateNvidiaGRIDLicenseValid(ctx context.Context, s *Scenario) error

func ValidateNvidiaGridV20DriverInstalled 

func ValidateNvidiaGridV20DriverInstalled(ctx context.Context, s *Scenario) error

ValidateNvidiaGridV20DriverInstalled asserts the node installed the grid-v20 (595.x) driver from the aks-gpu-grid-v20 image rather than falling back to a cuda/grid driver. This is the grid-v20-specific check: if SKU->driver-type selection regressed, nvidia-smi would report a different driver major.

func ValidateNvidiaModProbeInstalled 

func ValidateNvidiaModProbeInstalled(ctx context.Context, s *Scenario) error

func ValidateNvidiaPersistencedRunning 

func ValidateNvidiaPersistencedRunning(ctx context.Context, s *Scenario) error

func ValidateNvidiaSMIInstalled 

func ValidateNvidiaSMIInstalled(ctx context.Context, s *Scenario) error

func ValidateNvidiaSMINotInstalled 

func ValidateNvidiaSMINotInstalled(ctx context.Context, s *Scenario) error

func ValidatePodRunning 

func ValidatePodRunning(ctx context.Context, s *Scenario, pod *corev1.Pod) error

func ValidatePodRunningWithRetry 

func ValidatePodRunningWithRetry(ctx context.Context, s *Scenario, pod *corev1.Pod, maxRetries int) error

func ValidatePubkeySSHDisabled 

func ValidatePubkeySSHDisabled(ctx context.Context, s *Scenario) error

ValidatePubkeySSHDisabled validates that SSH with private key authentication is disabled by checking sshd_config

func ValidateRCV1PCertMode 

func ValidateRCV1PCertMode(ctx context.Context, s *Scenario) error

ValidateRCV1PCertMode validates that the rcv1p certificate endpoint mode was used during Linux node provisioning, certificates were downloaded and installed, and a refresh task was scheduled.

func ValidateRCV1PCertModeWindows 

func ValidateRCV1PCertModeWindows(ctx context.Context, s *Scenario) error

ValidateRCV1PCertModeWindows validates that the rcv1p certificate endpoint mode was used during Windows node provisioning, certificates were downloaded and installed, and a refresh task was scheduled.

func ValidateRCV1PNotOptedIn 

func ValidateRCV1PNotOptedIn(ctx context.Context, s *Scenario) error

ValidateRCV1PNotOptedIn validates that when the VM does NOT have the opt-in tag, wireserver returns IsOptedInForRootCerts=false and no certificates are installed, even in the RCV1P subscription with PlatformSettingsOverride registered.

func ValidateRCV1PNotOptedInWindows 

func ValidateRCV1PNotOptedInWindows(ctx context.Context, s *Scenario) error

ValidateRCV1PNotOptedInWindows validates that when the Windows VM does NOT have the opt-in tag, no certificates are installed to C:\ca and no refresh scheduled task is registered, even in the RCV1P subscription with PlatformSettingsOverride registered.

func ValidateRuncVersion 

func ValidateRuncVersion(ctx context.Context, s *Scenario, versions []string) error

func ValidateRxBufferDefault 

func ValidateRxBufferDefault(ctx context.Context, s *Scenario) error

ValidateRxBufferDefault validates rx buffer config using default values based on VM's CPU count

func ValidateSSHKeyLiteralPreservation 

func ValidateSSHKeyLiteralPreservation(ctx context.Context, s *Scenario, expectedKey string) error

ValidateSSHKeyLiteralPreservation reads the Windows authorized_keys file and checks that the expected key string appears literally. This does the comparison in Go rather than in PowerShell to avoid the validator itself interpreting special characters like $() in the key comment.

func ValidateSSHServiceDisabled 

func ValidateSSHServiceDisabled(ctx context.Context, s *Scenario) error

ValidateSSHServiceDisabled validates that the SSH daemon service is disabled and stopped on the node

func ValidateSSHServiceEnabled 

func ValidateSSHServiceEnabled(ctx context.Context, s *Scenario) error

func ValidateScriptlessCSECmd 

func ValidateScriptlessCSECmd(ctx context.Context, s *Scenario) error

ValidateScriptlessCSECmd checks if the node has scriptless cmd correctly enabled

func ValidateScriptlessNBCCSECmd 

func ValidateScriptlessNBCCSECmd(ctx context.Context, s *Scenario) error

ValidateScriptlessNBCCSECmd checks if the node has scriptless NBCCSECmd correctly enabled

func ValidateScriptlessPhase3 

func ValidateScriptlessPhase3(ctx context.Context, s *Scenario) error

ValidateScriptlessPhase3 validates that there are not diffs between ANC generated cse cmd NBC cse cmd vars.

func ValidateSecondaryNICDualStack 

func ValidateSecondaryNICDualStack(ctx context.Context, s *Scenario, ifaceName string) error

ValidateSecondaryNICDualStack checks that the given network interface is UP and has both IPv4 and IPv6 addresses.

func ValidateSecondaryNICUp 

func ValidateSecondaryNICUp(ctx context.Context, s *Scenario, ifaceName string) error

ValidateSecondaryNICUp checks that the given network interface is UP and has an IPv4 address.

func ValidateServiceInSlice 

func ValidateServiceInSlice(ctx context.Context, s *Scenario, service, expectedSlice string) error

ValidateServiceInSlice asserts that the given systemd service is running in the expected slice.

func ValidateServicesDoNotRestartKubelet 

func ValidateServicesDoNotRestartKubelet(ctx context.Context, s *Scenario) error

func ValidateStaleCachedKubeBinariesRemoved 

func ValidateStaleCachedKubeBinariesRemoved(ctx context.Context, s *Scenario) error

ValidateStaleCachedKubeBinariesRemoved validates that stale versioned kube binaries (e.g. kubelet-1.29.0, kubectl-1.29.0) have been removed from /opt/bin/ after the correct version is installed.

func ValidateSwapFileConfig 

func ValidateSwapFileConfig(ctx context.Context, s *Scenario, swapFileSizeMB int32) error

func ValidateSysctlConfig 

func ValidateSysctlConfig(ctx context.Context, s *Scenario, customSysctls map[string]string) error

func ValidateSystemdUnitIsNotFailed 

func ValidateSystemdUnitIsNotFailed(ctx context.Context, s *Scenario, serviceName string) error

func ValidateSystemdUnitIsNotRunning 

func ValidateSystemdUnitIsNotRunning(ctx context.Context, s *Scenario, serviceName string) error

func ValidateSystemdUnitIsRunning 

func ValidateSystemdUnitIsRunning(ctx context.Context, s *Scenario, serviceName string) error

func ValidateSystemdWatchdogForKubernetes132Plus 

func ValidateSystemdWatchdogForKubernetes132Plus(ctx context.Context, s *Scenario) error

func ValidateTLSBootstrapping 

func ValidateTLSBootstrapping(ctx context.Context, s *Scenario) error

func ValidateTaints 

func ValidateTaints(ctx context.Context, s *Scenario, expectedTaints string) error

ValidateTaints checks if the node has the expected taints that are set in the kubelet config with --register-with-taints flag

func ValidateTransparentHugePageConfig 

func ValidateTransparentHugePageConfig(ctx context.Context, s *Scenario, thpEnabled, thpDefrag string) error

func ValidateUlimitSettings 

func ValidateUlimitSettings(ctx context.Context, s *Scenario, ulimits map[string]string) error

func ValidateVulnerableKernelModulesDisabled 

func ValidateVulnerableKernelModulesDisabled(ctx context.Context, s *Scenario) error

ValidateVulnerableKernelModulesDisabled verifies that kernel modules with known LPE vulnerabilities (CVE-2026-31431 / DirtyFrag / Fragnesia: algif_aead, esp4, esp6, rxrpc) are handled correctly per OS:

  • Ubuntu fixed kernels and future Ubuntu releases: assert ABSENCE of the four modprobe blacklist entries. Ubuntu 22.04 linux-azure 5.15.0-1116-azure and Ubuntu 24.04 linux-azure 6.8.0-1058-azure include the fixes, thus new VHDs must stop blocking legitimate module use. Future Ubuntu releases do not inherit this mitigation by default. Ubuntu 20.04 also removes the mitigation on 5.4 Azure FIPS kernels at ABI 1164 or newer.
  • Other Ubuntu 20.04 and vulnerable/unknown 22.04 / 24.04 kernels / Mariner: full check — modprobe config entries are present, modules are NOT loaded, and modprobe refuses to load them.
  • AzureLinux 3.0: assert ABSENCE of the four modprobe blacklist entries. AzL3 is descoped from the mitigation because kernel 6.6.139.1-1.azl3 and later fix all three CVEs upstream, AND customer workloads on AzL3 require those modules (the blacklist actively blocks legitimate use cases). Only those four lines are stripped from modprobe-CIS.conf on newly-built AzL3 VHDs — the rest of the CIS module denylist (dccp/sctp/rds/tipc/cramfs/etc.) is still baked in and asserted present below, so AzL3 keeps the same CIS hardening as every other OS stream. E2E runs against freshly-built VHDs. See https://github.com/Azure/AKS/issues/5753.

To add a new CVE mitigation, append the module name to BOTH lists below — the absence-check list AND the default presence + load-refusal list.

func ValidateWaagentLog 

func ValidateWaagentLog(ctx context.Context, s *Scenario) error

ValidateWaagentLog checks /var/log/waagent.log for expected agent behavior: - AutoUpdate is disabled as expected - The correct version is running as ExtHandler - No errors from ExtHandler Skipped on Flatcar and OSGuard VHDs which manage WALinuxAgent independently.

func ValidateWindowsCiliumIsNotRunning 

func ValidateWindowsCiliumIsNotRunning(ctx context.Context, s *Scenario) error

func ValidateWindowsCiliumIsRunning 

func ValidateWindowsCiliumIsRunning(ctx context.Context, s *Scenario) error

func ValidateWindowsDisplayVersion 

func ValidateWindowsDisplayVersion(ctx context.Context, s *Scenario, displayVersion string) error

func ValidateWindowsProcessContainsArgumentStrings 

func ValidateWindowsProcessContainsArgumentStrings(ctx context.Context, s *Scenario, processName string, substrings []string) error

func ValidateWindowsProcessDoesNotContainArgumentStrings 

func ValidateWindowsProcessDoesNotContainArgumentStrings(ctx context.Context, s *Scenario, processName string, substrings []string) error

func ValidateWindowsProcessHasCliArguments 

func ValidateWindowsProcessHasCliArguments(ctx context.Context, s *Scenario, processName string, arguments []string) error

func ValidateWindowsProductName 

func ValidateWindowsProductName(ctx context.Context, s *Scenario, productName string) error

func ValidateWindowsSecureTLSEnabled 

func ValidateWindowsSecureTLSEnabled(ctx context.Context, s *Scenario) error

ValidateWindowsSecureTLSEnabled asserts that Enable-SecureTls (windowssecuretls.ps1) has hardened the node against protocol downgrade and the Sweet32 birthday attack (CVE-2016-2183 / CVE-2016-6329): TLS 1.2 is enabled, TLS 1.0/1.1 and SSLv2/SSLv3 are disabled, RC4 is disabled, and the configured cipher suite order does not include any 64-bit block ciphers (3DES/DES/RC2).

func ValidateWindowsServiceIsNotRunning 

func ValidateWindowsServiceIsNotRunning(ctx context.Context, s *Scenario, serviceName string) error

func ValidateWindowsServiceIsRunning 

func ValidateWindowsServiceIsRunning(ctx context.Context, s *Scenario, serviceName string) error

func ValidateWindowsSystemServiceRestartConfiguration 

func ValidateWindowsSystemServiceRestartConfiguration(ctx context.Context, s *Scenario, serviceName string) error

func ValidateWindowsSystemServicesRestartConfiguration 

func ValidateWindowsSystemServicesRestartConfiguration(ctx context.Context, s *Scenario) error

func ValidateWindowsVersionFromWindowsSettings 

func ValidateWindowsVersionFromWindowsSettings(ctx context.Context, s *Scenario, windowsVersion string) error

func Windows2025BootstrapConfigMutator 

func Windows2025BootstrapConfigMutator(configuration *datamodel.NodeBootstrappingConfiguration) error

Types

type Bastion 

type Bastion struct {
	// contains filtered or unexported fields
}

func NewBastion 

func NewBastion(credential *azidentity.AzureCLICredential, subscriptionID, resourceGroupName, dnsName string) *Bastion

func (*Bastion) NewTunnelSession 

func (b *Bastion) NewTunnelSession(ctx context.Context, targetHost string, port uint16) (*tunnelSession, error)

type CSEProvisionTiming 

type CSEProvisionTiming struct {
	ExitCode            string          `json:"ExitCode"`
	ExecDuration        string          `json:"ExecDuration"`
	KernelStartTime     string          `json:"KernelStartTime"`
	CloudInitLocalStart string          `json:"CloudInitLocalStartTime"`
	CloudInitStart      string          `json:"CloudInitStartTime"`
	CloudFinalStart     string          `json:"CloudFinalStartTime"`
	CSEStartTime        string          `json:"CSEStartTime"`
	GuestAgentStartTime string          `json:"GuestAgentStartTime"`
	SystemdSummary      string          `json:"SystemdSummary"`
	BootDatapoints      json.RawMessage `json:"BootDatapoints"`
}

CSEProvisionTiming represents the overall provisioning timing from provision.json.

type CSETaskTiming 

type CSETaskTiming struct {
	TaskName  string
	StartTime time.Time
	EndTime   time.Time
	Duration  time.Duration
	Message   string
}

CSETaskTiming represents the timing of a single CSE task.

type CSETimingReport 

type CSETimingReport struct {
	Tasks     []CSETaskTiming
	Provision *CSEProvisionTiming
	// contains filtered or unexported fields
}

CSETimingReport holds all parsed timing data from a VM.

func ExtractCSETimings 

func ExtractCSETimings(ctx context.Context, s *Scenario) (*CSETimingReport, error)

ExtractCSETimings SSHes into the scenario VM and extracts all CSE task timings. Returns an error if no tasks could be parsed, since an empty report would make regression detection ineffective.

func ValidateCSETimings 

func ValidateCSETimings(ctx context.Context, s *Scenario, thresholds CSETimingThresholds) (*CSETimingReport, error)

ValidateCSETimings extracts, logs, and validates CSE task timings.

func (*CSETimingReport) GetTask 

func (r *CSETimingReport) GetTask(name string) *CSETaskTiming

GetTask returns the timing for a specific task, or nil if not found.

func (*CSETimingReport) LogReport 

func (r *CSETimingReport) LogReport(_ context.Context, logger toolkit.Logger)

LogReport logs all task timings to the test logger.

func (*CSETimingReport) TotalCSEDuration 

func (r *CSETimingReport) TotalCSEDuration() time.Duration

TotalCSEDuration returns the duration of the cse_start task if present.

type CSETimingThresholds 

type CSETimingThresholds struct {
	// TaskThresholds maps task name suffixes to maximum duration.
	// Task names are matched by suffix to allow flexible matching
	// (e.g., "installDebPackageFromFile" matches "AKS.CSE.installkubelet.installDebPackageFromFile").
	TaskThresholds map[string]time.Duration

	// TotalCSEThreshold is the maximum acceptable total CSE duration.
	TotalCSEThreshold time.Duration

	// DefaultTaskThreshold is the threshold applied to any task that exceeds it
	// but has no specific entry in TaskThresholds. This ensures that ALL slow tasks
	// appear as sub-tests in ADO Pipeline Analytics, even newly added ones.
	// Tasks below this threshold are silently skipped.
	// Set to 0 to disable dynamic tracking.
	DefaultTaskThreshold time.Duration
}

CSETimingThresholds defines maximum acceptable durations for CSE tasks.

type Cluster 

type Cluster struct {
	Model *armcontainerservice.ManagedCluster

	KubeletIdentity  *armcontainerservice.UserAssignedIdentity
	SubnetID         string
	VNetResourceGUID string
	ClusterParams    *ClusterParams
	Bastion          *Bastion
	ProxyURL         string
	TenantID         string
	// contains filtered or unexported fields
}

func (*Cluster) IsAzureCNI 

func (c *Cluster) IsAzureCNI() (bool, error)

Returns true if the cluster is configured with Azure CNI

func (*Cluster) MaxPodsPerNode 

func (c *Cluster) MaxPodsPerNode() (int, error)

Returns the maximum number of pods per node of the cluster's agentpool

func (*Cluster) NewKubeclientForTest 

func (c *Cluster) NewKubeclientForTest() (*Kubeclient, error)

NewKubeclientForTest creates an independent Kubeclient with its own rate limiter. Use this in individual tests to avoid sharing rate limiter tokens with other parallel tests hitting the same cluster.

type ClusterParams 

type ClusterParams struct {
	CACert         []byte
	BootstrapToken string
	FQDN           string
}

type ClusterRequest 

type ClusterRequest struct {
	Location         string
	K8sSystemPoolSKU string
}

ClusterRequest represents the parameters needed to create a cluster

type ClusterSubnetRequest 

type ClusterSubnetRequest struct {
	Location    string
	ClusterName string
	DualStack   bool
}

type Config 

type Config struct {
	// Cluster creates, updates or re-uses an AKS cluster for the scenario
	Cluster func(ctx context.Context, request ClusterRequest) (*Cluster, error)

	// VHD is the node image used by the scenario.
	VHD *config.Image

	// BootstrapConfigMutator is a function which mutates the base NodeBootstrappingConfig according to the scenario's requirements
	BootstrapConfigMutator func(*Cluster, *datamodel.NodeBootstrappingConfiguration)

	// BootstrapConfigMutatorWithError is used when preparing the bootstrap configuration can fail.
	// It runs after BootstrapConfigMutator.
	BootstrapConfigMutatorWithError func(context.Context, *Cluster, *datamodel.NodeBootstrappingConfiguration) error

	// PreProvisionBootstrapConfigMutator mutates only the NodeBootstrappingConfig used to bake a cached VHD.
	// It runs after BootstrapConfigMutator and after PreProvisionOnly is set. Use it to deliberately make
	// bake-time state differ from provision-time state - e.g. inject a sentinel TLS bootstrap token -
	// so that staleness regressions in the BasePrep->NodePrep split are caught positively.
	PreProvisionBootstrapConfigMutator func(*Cluster, *datamodel.NodeBootstrappingConfiguration)

	// AKSNodeConfigMutator if defined then aks-node-controller will be used to provision nodes
	AKSNodeConfigMutator func(*Cluster, *aksnodeconfigv1.Configuration)

	// VMConfigMutator is a function which mutates the base VMSS model according to the scenario's requirements
	VMConfigMutator func(*armcompute.VirtualMachineScaleSet)

	// VMConfigMutatorWithError is used when preparing the VMSS model can fail.
	// It runs after VMConfigMutator.
	VMConfigMutatorWithError func(context.Context, *armcompute.VirtualMachineScaleSet) error

	// CustomDataWriteFiles injects additional cloud-init write_files entries into rendered customData.
	// This is for e2e-only validation scenarios.
	CustomDataWriteFiles []CustomDataWriteFile

	// Validator is a function where the scenario can perform any extra validation checks
	Validator func(ctx context.Context, s *Scenario) error

	// SkipDefaultValidation is a flag to indicate whether the common validation (like spawning a pod) should be skipped.
	// It shouldn't be used for the majority of scenarios; VHD caching uses it while preparing the image.
	SkipDefaultValidation bool

	// SkipSSHConnectivityValidation is a flag to indicate whether the ssh connectivity validation should be skipped.
	// It shouldn't be used for majority of scenarios, currently only used for scenarios where the node is not expected to be reachable via ssh
	SkipSSHConnectivityValidation bool

	// WaitForSSHAfterReboot if set to non-zero duration, SSH connectivity validation will retry with exponential backoff
	// for up to this duration when encountering reboot-related errors. This is useful for scenarios where the node
	// reboots during provisioning (e.g., MIG-enabled GPU nodes). Default (zero value) means no retry.
	WaitForSSHAfterReboot time.Duration

	// if VHDCaching is set then a VHD will be created first for the test scenario and then a VM will be created from that VHD.
	// The main purpose is to validate VHD Caching logic and ensure a reboot step between basePrep and nodePrep doesn't break anything.
	VHDCaching bool

	// ExpectedError, when set, indicates that VMSS creation is expected to fail with an error containing this substring.
	// The assertion is performed during the scenario run.
	ExpectedError string

	// UseNVMe indicates whether to use NVMe-based disk placement/controller. This is required for certain VM sizes (e.g., v6 and v7 series) which only support NVMe disk controllers.
	UseNVMe bool

	// EagerCSETimingExtraction when true causes CSE timing events to be extracted
	// immediately after SSH is established, before other validators run.
	// This prevents the Guest Agent from sweeping events before they can be read.
	// Only set this on CSE performance test scenarios.
	EagerCSETimingExtraction bool
}

Config represents the configuration of an AgentBaker E2E scenario.

type CreateGalleryImageRequest 

type CreateGalleryImageRequest struct {
	ResourceGroup    string
	GalleryName      string
	Location         string
	Arch             string
	Windows          bool
	HyperVGeneration *armcompute.HyperVGeneration
}

type CreateGalleryRequest 

type CreateGalleryRequest struct {
	Location      string
	ResourceGroup string
}

type CustomDataWriteFile 

type CustomDataWriteFile struct {
	Path        string
	Permissions string
	Owner       string
	Content     string
}

CustomDataWriteFile defines an e2e-only cloud-init write_files entry.

type GetLatestExtensionVersionRequest 

type GetLatestExtensionVersionRequest struct {
	Location  string
	ExtType   string
	Publisher string
}

GetLatestExtensionVersionRequest is the cache key for VM extension version lookups.

type GetVHDRequest 

type GetVHDRequest struct {
	Location string
	Image    config.Image
}

type Kubeclient 

type Kubeclient struct {
	Dynamic    client.Client
	Typed      kubernetes.Interface
	RESTConfig *rest.Config
	KubeConfig []byte
}

func NewKubeclient 

func NewKubeclient(kubeconfigBytes []byte) (*Kubeclient, error)

NewKubeclient creates a Kubeclient from raw kubeconfig bytes. Each call returns an independent client with its own rate limiter, allowing concurrent operations to avoid starving each other.

func (*Kubeclient) CreateDaemonset 

func (k *Kubeclient) CreateDaemonset(ctx context.Context, ds *appsv1.DaemonSet) error

func (*Kubeclient) EnsureDebugDaemonsets 

func (k *Kubeclient) EnsureDebugDaemonsets(ctx context.Context, isNetworkIsolated bool, privateACRName string) error

this is a bit ugly, but we don't want to execute this piece concurrently with other tests

func (*Kubeclient) GetPodNetworkDebugPodForNode 

func (k *Kubeclient) GetPodNetworkDebugPodForNode(ctx context.Context, kubeNodeName string) (*corev1.Pod, error)

GetPodNetworkDebugPodForNode returns a pod that's a member of the 'debugnonhost' daemonset running in the cluster - this will return the name of the pod that is running on the node created for specifically for the test case which is running validation checks.

func (*Kubeclient) GetProxyURL 

func (k *Kubeclient) GetProxyURL(ctx context.Context) (string, error)

GetProxyURL returns the proxy URL after verifying the proxy pod and its backing node are ready on the cluster's permanent managed system pool.

func (*Kubeclient) WaitUntilNodeReady 

func (k *Kubeclient) WaitUntilNodeReady(ctx context.Context, logger toolkit.Logger, vmssName string) (string, error)

func (*Kubeclient) WaitUntilPodRunning 

func (k *Kubeclient) WaitUntilPodRunning(ctx context.Context, namespace string, labelSelector string, fieldSelector string) (*corev1.Pod, error)

type Measurement 

type Measurement struct {
	Name      string
	ClassName string
	Duration  time.Duration
	Message   string
}

Measurement becomes a separate JUnit test case so ADO can track a focused measurement or validation independently from the parent scenario.

type Outcome 

type Outcome struct {
	Error        error
	SkipReason   string
	Measurements []Measurement
}

func Run 

func Run(ctx context.Context, name, artifactName string, logger toolkit.Logger, original *Scenario) Outcome

type Scenario 

type Scenario struct {
	// Name is the stable scenario name used by filters, logs, and test reports.
	Name string

	// Description is a short description of what the scenario does and tests for
	Description string

	// Tags are used for filtering scenarios to run based on the tags provided
	Tags Tags

	// Config contains the configuration of the scenario
	Config

	// Location is the Azure location where the scenario will run. This can be
	// used to override the default location.
	Location string

	// K8sSystemPoolSKU is the VM size to use for the system nodepool. If empty,
	// a default size will be used.
	K8sSystemPoolSKU string

	// Runtime contains the runtime state of the scenario. It's populated in the beginning of the test run
	Runtime *ScenarioRuntime

	// SkipReason disables the scenario with a fixed reason.
	SkipReason string

	// SkipIf returns a reason to skip before the scenario creates Azure resources.
	// An empty reason runs the scenario.
	SkipIf func(context.Context) string

	// Logger writes the scenario log. It is set by the execution flow before the
	// scenario starts and carries no test-control capability.
	Logger toolkit.Logger
	// contains filtered or unexported fields
}

Scenario represents an AgentBaker E2E scenario.

func List 

func List() []*Scenario

func Register 

func Register(s *Scenario) *Scenario

Register adds a declarative scenario to the CLI registry.

func (*Scenario) Cleanup 

func (s *Scenario) Cleanup(fn func(context.Context) error)

Cleanup registers fn to run after the scenario attempt finishes.

func (*Scenario) EffectiveTags 

func (s *Scenario) EffectiveTags() Tags

func (*Scenario) GetClientPrivateKey 

func (s *Scenario) GetClientPrivateKey() string

func (*Scenario) GetContainerRegistryFQDN 

func (s *Scenario) GetContainerRegistryFQDN() string

GetContainerRegistryFQDN returns the container registry FQDN for the cloud environment determined by the cluster's location. Uses Runtime.Cluster.Model.Location so it works for both legacy (NBC) and scriptless (AKSNodeConfig) bootstrap paths.

func (*Scenario) GetDefaultFQDNsForValidation 

func (s *Scenario) GetDefaultFQDNsForValidation() []string

GetDefaultFQDNsForValidation returns the public cloud FQDNs to validate in hosts file checks. AgentBaker e2e only runs in public cloud, so sovereign cloud branches are unnecessary.

func (*Scenario) GetK8sVersion 

func (s *Scenario) GetK8sVersion() string

func (*Scenario) GetServicePrincipalSecret 

func (s *Scenario) GetServicePrincipalSecret() string

func (*Scenario) GetTLSBootstrapToken 

func (s *Scenario) GetTLSBootstrapToken() string

func (*Scenario) HasServicePrincipalData 

func (s *Scenario) HasServicePrincipalData() bool

func (*Scenario) IsHostsPluginEnabled 

func (s *Scenario) IsHostsPluginEnabled() bool

IsHostsPluginEnabled returns true if the hosts plugin is explicitly enabled via either NBC (traditional) or AKSNodeConfig (scriptless) paths.

func (*Scenario) IsLinux 

func (s *Scenario) IsLinux() bool

func (*Scenario) IsWindows 

func (s *Scenario) IsWindows() bool

func (*Scenario) KubeletConfigFileEnabled 

func (s *Scenario) KubeletConfigFileEnabled() bool

func (*Scenario) PrepareVMSSModel 

func (s *Scenario) PrepareVMSSModel(ctx context.Context, vmss *armcompute.VirtualMachineScaleSet) error

PrepareVMSSModel mutates the input VirtualMachineScaleSet based on the scenario's VMConfigMutator, if configured. This method will also use the scenario's configured VHD selector to modify the input VMSS to reference the correct VHD resource.

func (*Scenario) SecureTLSBootstrappingEnabled 

func (s *Scenario) SecureTLSBootstrappingEnabled() bool

type ScenarioRuntime 

type ScenarioRuntime struct {
	NBC                       *datamodel.NodeBootstrappingConfiguration
	AKSNodeConfig             *aksnodeconfigv1.Configuration
	Cluster                   *Cluster
	Kube                      *Kubeclient // per-test client with independent rate limiter
	VM                        *ScenarioVM
	VMSize                    string
	VMSSName                  string
	EnableScriptlessNBCCSECmd bool
	CSETimingReport           *CSETimingReport // eagerly extracted before GA can sweep events
}

type ScenarioVM 

type ScenarioVM struct {
	KubeName  string
	VMSS      *armcompute.VirtualMachineScaleSet
	VM        *armcompute.VirtualMachineScaleSetVM
	PrivateIP string
	SSHClient *ssh.Client
}

func ConfigureAndCreateVMSS 

func ConfigureAndCreateVMSS(ctx context.Context, s *Scenario) (*ScenarioVM, error)

func CreateVMSS 

func CreateVMSS(ctx context.Context, s *Scenario, resourceGroupName string) (*ScenarioVM, error)

func CreateVMSSWithRetry 

func CreateVMSSWithRetry(ctx context.Context, s *Scenario) (*ScenarioVM, error)

type SharedInfra 

type SharedInfra struct {
	VNetName       string
	ResourceGroup  string
	BastionDNSName string
	FirewallIP     string
	IdentityID     string // resource ID of the user-assigned managed identity
	TenantID       string // tenant ID of the user-assigned managed identity
}

type Tags 

type Tags struct {
	Name                   string
	ImageName              string
	OS                     string
	Arch                   string
	NetworkIsolated        bool
	NonAnonymousACR        bool
	GPU                    bool
	WASM                   bool
	Kata                   bool
	BootstrapTokenFallback bool
	KubeletCustomConfig    bool
	Scriptless             bool
	VHDCaching             bool
	MockAzureChinaCloud    bool
	RCV1PCertMode          bool
	VMSeriesCoverageTest   bool
}

type VMSizeSKURequest 

type VMSizeSKURequest struct {
	Location string
	VMSize   string
}

VMSizeSKURequest is the cache key for Resource SKU lookups by VM size and location.

type VNet 

type VNet struct {
	// contains filtered or unexported fields
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.