operator

package
v1.7.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 20, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ParseCapacity added in v1.5.0

func ParseCapacity(cfg *Config) (nodecap.Maximum, error)

ParseCapacity is the static maximum capacity of the lab nodes (nodecap.Maximum): none without a budget.

func ParseLabScheduling

func ParseLabScheduling(cfg *Config) (nodeSelector map[string]string, tolerations []corev1.Toleration, err error)

ParseLabScheduling deserialises the JSON-encoded nodeSelector and tolerations from the operator config into typed values ready for pod specs.

Types

type CacheConfig

type CacheConfig struct {
	Enabled bool `env:"IMAGE_CACHE_ENABLED" envDefault:"false"`
	// Prefix is host:port of the cache as the nodes see it (the node-agent's localhost forwarder); empty = names.RegistryNodePrefix.
	Prefix string `env:"IMAGE_CACHE_PREFIX"`
	// Registries are the upstream registries the cache serves; references to any
	// other registry are pulled directly.
	// PinTTL is how long a resolved image digest is remembered, so labs created in
	// the same wave run the same image even when the upstream tag moves.
	PinTTL     time.Duration `env:"IMAGE_CACHE_PIN_TTL" envDefault:"30m"`
	Registries []string      `env:"IMAGE_CACHE_REGISTRIES" envSeparator:"," envDefault:"docker.io,ghcr.io,quay.io,registry.k8s.io"`
}

CacheConfig configures the image cache rewrite of lab image references.

func (CacheConfig) Rewriter

func (c CacheConfig) Rewriter() imagecache.Rewriter

Rewriter returns the image reference rewriter; the zero one (no rewrite) when the cache is off.

type Config

type Config struct {
	RuntimeObservation bool `env:"RUNTIME_OBSERVATION_ENABLED" envDefault:"false"`
	// Acceptance gate stays false until final native proof; advertising remains false.
	RequiredSnapshotAvailable bool `env:"REQUIRED_SNAPSHOT_AVAILABLE" envDefault:"false"`
	// PublicVPNEndpoint is the host:port advertised to WireGuard clients (demux public address); a host alone gets the default port.
	// Required, no default: the domain comes only from configuration.
	PublicVPNEndpoint string `env:"PUBLIC_VPN_ENDPOINT,notEmpty"`
	// BaseDomain is the cluster ingress base domain used for device exposure URLs.
	// Required, no default.
	BaseDomain string `env:"BASE_DOMAIN,notEmpty"`
	// ProxySourceCIDRs is an optional list of CIDR blocks whose traffic the proxy sends when
	// using hostNetwork (e.g. in Kind, the proxy runs as a hostNetwork pod and its source IP is
	// the node IP, not a pod IP, so namespace/label selectors in NetworkPolicy don't apply).
	// Each CIDR is added as an ipBlock peer in the generated web-exposure NetworkPolicy.
	// Format: comma-separated CIDR strings, e.g. "172.18.0.4/32".
	ProxySourceCIDRs []string `env:"PROXY_SOURCE_CIDRS" envSeparator:","`
	// WGDemuxMode is the mode of the WireGuard demux (userspace, tc or xdp; the chart value proxy.wg.mode). In tc and xdp the demux runs on the
	// host network, so its packets reach the VPN pods with a node address, and the VPN pods' policy admits the nodes on the WireGuard port.
	WGDemuxMode string `env:"WG_DEMUX_MODE" envDefault:"userspace"`
	// VPNBaseNetwork is the base address space for per-lab VPN subnets, VPN half of 10.128.0.0/9 (e.g. "10.128.0.0/10").
	// Each lab is assigned a /24 child subnet by adding its index to the base address.
	VPNBaseNetwork string `env:"VPN_BASE_NETWORK" envDefault:"10.128.0.0/10"`
	// InetBaseNetwork is the base address space for per-lab internet/gateway subnets, internet half (e.g. "10.192.0.0/10").
	InetBaseNetwork string `env:"INET_BASE_NETWORK" envDefault:"10.192.0.0/10"`
	// VPNImage is the container image for per-LabGroup VPN pods. Required, no default: the chart passes
	// repository:tag (the tag is the release tag: image.tag, else the chart appVersion).
	VPNImage string `env:"VPN_IMAGE,notEmpty"`
	// GatewayImage is the container image for per-LabGroup gateway pods. The VPN and the gateway are one image (laboratory-lab, two
	// commands): empty means VPNImage.
	GatewayImage string `env:"GATEWAY_IMAGE"`
	// GroupPods are the resources of the VPN and gateway pods of a new LabGroup (requests = limits).
	GroupPods grouppods.Config
	// GroupPriorityClass and DevicePriorityClass are the PriorityClasses of the VPN and gateway pods and of the device pods
	// (the chart creates them, values priorityClasses). Empty = no class (tests, a development stand).
	GroupPriorityClass  string `env:"PRIORITY_CLASS_GROUP"`
	DevicePriorityClass string `env:"PRIORITY_CLASS_DEVICE"`
	// LabSchedulerName is the kube-scheduler profile that places the pods of labs (devices, VPN, gateway): a bin-packing one, so
	// the nodes fill up one after another and no hole is left that a big device cannot use. Empty = the default scheduler (the
	// chart sets it when labScheduler.enabled). System pods never use it.
	LabSchedulerName string `env:"LAB_SCHEDULER_NAME"`
	// LabPodTerminationGraceSeconds is the terminationGracePeriodSeconds of every pod of a lab (devices, VPN, gateway). They need no
	// graceful drain, and the default 30 s makes a deleted lab's namespace live that long. 0 = the Kubernetes default.
	LabPodTerminationGraceSeconds int64 `env:"LAB_POD_TERMINATION_GRACE_SECONDS" envDefault:"5"`
	// NetConfigImage is the image for the optional device init-container that
	// assigns static IP/routes. Needs iproute2 + sh; node-agent image has both.
	// Required, no default.
	NetConfigImage string `env:"NETCONFIG_IMAGE,notEmpty"`
	// LabNodeSelectorJSON is a JSON-encoded map[string]string of nodeSelector labels
	// applied to all runtime lab pods (VPN, gateway, device).
	LabNodeSelectorJSON string `env:"LAB_NODE_SELECTOR" envDefault:"{}"`
	// LabTolerationsJSON is a JSON-encoded []corev1.Toleration applied to all runtime lab pods.
	LabTolerationsJSON string `env:"LAB_TOLERATIONS" envDefault:"[]"`
	// AgentEnabled gates creation of the management-agent RoleBinding in each LabGroup namespace. On by default, as in the chart: a
	// binding to an agent that does not run grants nothing.
	AgentEnabled bool `env:"AGENT_ENABLED" envDefault:"true"`
	// AgentServiceAccount and AgentServiceNamespace are the management agent's identity bound by that RoleBinding (empty =
	// names.AgentServiceAccount in names.AgentNamespace, the chart's defaults).
	AgentServiceAccount   string `env:"AGENT_SERVICE_ACCOUNT"`
	AgentServiceNamespace string `env:"AGENT_SERVICE_NAMESPACE"`
	// OperatorServiceAccount and OperatorNamespace are the operator's own identity: in every LabGroup namespace it binds that
	// ServiceAccount to the ClusterRole of its working permissions (it has no cluster-wide write access). Empty =
	// names.OperatorServiceAccount in names.SystemNamespace.
	OperatorServiceAccount string `env:"OPERATOR_SERVICE_ACCOUNT"`
	OperatorNamespace      string `env:"OPERATOR_NAMESPACE"`
	// ProxyEnabled gates the RoleBinding that lets the L7 proxy write its traffic reports in each group namespace. On by default,
	// as in the chart.
	ProxyEnabled bool `env:"PROXY_ENABLED" envDefault:"true"`
	// NetworkPolicyEnabled gates creation of the default-deny NetworkPolicy
	// baseline in each LabGroup namespace.
	NetworkPolicyEnabled bool `env:"NETWORK_POLICY_ENABLED" envDefault:"true"`
	// VPNStatsInterval is how often the VPN pod of a NEW group samples traffic statistics (env STATS_INTERVAL of the pod).
	VPNStatsInterval time.Duration `env:"VPN_STATS_INTERVAL" envDefault:"30s"`
	// ImagePullSecrets lists registry Secrets (kubernetes.io/dockerconfigjson) in the
	// operator namespace, created outside the chart. The operator copies them into
	// every group namespace and sets them on the VPN, gateway and device pods, so a
	// private registry works for lab workloads. Empty means public images only.
	ImagePullSecrets []string `env:"IMAGE_PULL_SECRETS" envSeparator:","`
	// SupportEmail is the contact address shown to participants on the VPN
	// probe page. Passed to every per-LabGroup VPN pod as SUPPORT_EMAIL.
	// Required and not empty: the deploy always passes it.
	SupportEmail string `env:"SUPPORT_EMAIL,notEmpty"`

	// TenantEnrollmentTTL is how long an unused tenant enrollment token works.
	TenantEnrollmentTTL time.Duration `env:"TENANT_ENROLLMENT_TTL" envDefault:"24h"`

	// SchedulerEnabled turns the queue on. Off: every pod starts as soon as its
	// object is created.
	SchedulerEnabled bool `env:"SCHEDULER_ENABLED" envDefault:"true"`
	// SchedulerMaxPods is the largest number of pods starting at once. A pod holds
	// its slot from dispatch until it is Ready or declared failed. 0 sets no limit
	// (prepull and the resource check still apply).
	SchedulerMaxPods int `env:"SCHEDULER_MAX_PODS" envDefault:"20"`
	// SchedulerStartupTimeout is how long a dispatched pod may take to become Ready
	// before it is declared failed and its slot is freed.
	SchedulerStartupTimeout time.Duration `env:"SCHEDULER_STARTUP_TIMEOUT" envDefault:"5m"`
	// SchedulerRestartThreshold is the number of restarts after which a pod that is
	// not Ready is declared failed before the timeout.
	SchedulerRestartThreshold int `env:"SCHEDULER_RESTART_THRESHOLD" envDefault:"5"`
	// SchedulerPlatformReservePercent is the platform reserve: the share of the schedulable CPU and memory
	// (allocatable, so after the kubelet's own reserve, and counted after DaemonSet and proxy requests) that user
	// labs can never consume. SchedulerPlatformReserveCPU and SchedulerPlatformReserveMemory are an extra
	// absolute reserve on EVERY schedulable node (quantities, "0" = none).
	SchedulerPlatformReservePercent int    `env:"SCHEDULER_PLATFORM_RESERVE_PERCENT" envDefault:"10"`
	SchedulerPlatformReserveCPU     string `env:"SCHEDULER_PLATFORM_RESERVE_CPU" envDefault:"0"`
	SchedulerPlatformReserveMemory  string `env:"SCHEDULER_PLATFORM_RESERVE_MEMORY" envDefault:"0"`
	// SchedulerPackingReservePercent is the hidden packing reserve taken per node from what is free there after the non-lab
	// pods. SchedulerPackingReserveCPU and SchedulerPackingReserveMemory, when not "0", replace the percent for that
	// resource. The agent reads the same values, so both report the same room.
	SchedulerPackingReservePercent int    `env:"SCHEDULER_PACKING_RESERVE_PERCENT" envDefault:"15"`
	SchedulerPackingReserveCPU     string `env:"SCHEDULER_PACKING_RESERVE_CPU" envDefault:"0"`
	SchedulerPackingReserveMemory  string `env:"SCHEDULER_PACKING_RESERVE_MEMORY" envDefault:"0"`
	// SchedulerResourceCheck holds a pod back while no node has room for its requests.
	SchedulerResourceCheck bool `env:"SCHEDULER_RESOURCE_CHECK" envDefault:"true"`
	// SchedulerPrepull pulls the images of a group onto the nodes before its first pod starts.
	SchedulerPrepull bool `env:"SCHEDULER_PREPULL" envDefault:"true"`
	// SchedulerPrepullTimeout bounds the wait for the image prepull; dispatch goes on after it.
	SchedulerPrepullTimeout time.Duration `env:"SCHEDULER_PREPULL_TIMEOUT" envDefault:"5m"`
	// SchedulerAgingSeconds is how long a gang waits before the room it needs is kept for it (the oldest such gang only), so a
	// stream of small gangs cannot keep a big one out; 0: no aging. Off by default: the kept room is held against every tenant
	// with no time limit, so enable it once the reservation scopes the aged room.
	SchedulerAgingSeconds int `env:"SCHEDULER_AGING_SECONDS" envDefault:"0"`

	// Capacity (DEPLOY.md, "Maximum capacity and node autoscaling"). CapacityBudgetCPU and CapacityBudgetMemory are the room
	// for labs the autoscaled lab nodes (label cybericebox.com/node-group) may give at most (both, or neither: no budget, the
	// maximum is the current capacity). The agent reads the same values. CapacityNodeTimeout is how long every dispatched pod of
	// a group has to get a node before the whole group is withdrawn and queued again (the gang commit, always on).
	CapacityBudgetCPU    string        `env:"CAPACITY_BUDGET_CPU"`
	CapacityBudgetMemory string        `env:"CAPACITY_BUDGET_MEMORY"`
	CapacityNodeTimeout  time.Duration `env:"CAPACITY_NODE_TIMEOUT" envDefault:"15m"`

	// ReservationEndingNoticeSeconds is how long before the end of a Reservation its Ending condition is set (and how long a
	// reservation whose tenant is gone still lives); at the end its groups are deleted. 0: no notice.
	ReservationEndingNoticeSeconds int `env:"RESERVATION_ENDING_NOTICE_SECONDS" envDefault:"300"`

	// RequireAdmissionPolicy makes the operator check at start that the chart's admission policies are enforced on it and
	// exit if they are not (the chart sets it with operator.admissionPolicy.enabled). On by default, as in the chart: an operator that
	// is not confined must be asked for by name (OPERATOR_REQUIRE_ADMISSION_POLICY=false: a developer's machine, where it is not the
	// ServiceAccount the policies are about, or a deploy without the chart).
	RequireAdmissionPolicy bool `env:"OPERATOR_REQUIRE_ADMISSION_POLICY" envDefault:"true"`
	// AdmissionPolicyTimeout is how long the operator waits at start for the policies to be enforced.
	AdmissionPolicyTimeout time.Duration `env:"OPERATOR_ADMISSION_POLICY_TIMEOUT" envDefault:"90s"`

	// DeviceDefaultCPU and DeviceDefaultMemory are the requests and limits of a
	// device container that declares neither (requests always equal limits, so
	// the pod is Guaranteed). Empty leaves such a device without resources.
	DeviceDefaultCPU    string `env:"DEVICE_DEFAULT_CPU" envDefault:"100m"`
	DeviceDefaultMemory string `env:"DEVICE_DEFAULT_MEMORY" envDefault:"256Mi"`
	// DeviceMaxCPU and DeviceMaxMemory are the chart maxima of one device (limits.device.maxCpu / maxMemory): a
	// device object that asks for more (it predates the agent's check, or was made by hand) is clamped to them.
	DeviceMaxCPU    string `env:"DEVICE_MAX_CPU" envDefault:"2000m"`
	DeviceMaxMemory string `env:"DEVICE_MAX_MEMORY" envDefault:"4Gi"`

	// DeviceUserNamespaces runs device pods with hostUsers: false (default on; a hidden setting, not shown in any UI).
	// DeviceEphemeralStorage limits the writable layer, logs and emptyDirs of a device ("" or "0" = no limit).
	DeviceUserNamespaces   bool   `env:"DEVICE_USER_NAMESPACES" envDefault:"true"`
	DeviceEphemeralStorage string `env:"DEVICE_EPHEMERAL_STORAGE" envDefault:"2Gi"`

	// State is the device state persistence configuration (snapshot registry
	// and snapshot policy). Every field is optional; Enabled=false is today's
	// behaviour.
	State StateConfig

	// Cache is the image cache: lab images are pulled through the platform's
	// pull-through cache (zot with on-demand sync) instead of straight from the
	// upstream registries. Independent of State.
	Cache CacheConfig
}

func LoadConfig

func LoadConfig() (*Config, error)

type StateConfig

type StateConfig struct {
	Enabled bool `env:"STATE_PERSISTENCE_ENABLED" envDefault:"false"`
	// RegistryAddr is host:port of the snapshot registry Service, used by the operator to drop snapshots (device reset, retention);
	// empty = names.RegistryServiceAddr (the chart always installs the registry there).
	RegistryAddr     string `env:"STATE_REGISTRY_ADDR"`
	RegistryUser     string `env:"STATE_REGISTRY_USER"`
	RegistryPassword string `env:"STATE_REGISTRY_PASSWORD"`
	// Debounce is how long a device's writable layer must stay quiet before a snapshot.
	Debounce time.Duration `env:"STATE_DEBOUNCE" envDefault:"5s"`
	// ExcludePaths are never snapshotted (comma-separated absolute paths).
	ExcludePaths []string `env:"STATE_EXCLUDE_PATHS" envSeparator:"," envDefault:"/tmp,/var/tmp,/run"`
	// WriteQuota is the quota per device, a Kubernetes quantity ("512Mi").
	WriteQuota string `env:"STATE_WRITE_QUOTA" envDefault:"512Mi"`
	// MaxFileSize: a regular file larger than this is left out of a snapshot (a Kubernetes quantity).
	MaxFileSize string `env:"STATE_MAX_FILE_SIZE" envDefault:"256Mi"`
	// MaxEntries caps the entries of one snapshot layer (the byte quota does not count empty files).
	MaxEntries int `env:"STATE_MAX_ENTRIES" envDefault:"100000"`
	// TenantQuota is what the snapshots of all of one tenant's devices may take in the registry together (a Kubernetes
	// quantity; "0" = no limit); a tenant may be given less (Tenant.spec.persistence.registryQuota).
	TenantQuota string `env:"STATE_TENANT_QUOTA" envDefault:"10Gi"`
	// MaxLayers is the snapshot layer count after which the chain is squashed.
	MaxLayers int32 `env:"STATE_MAX_LAYERS" envDefault:"10"`
	// Retention is how long the snapshots of a deleted lab are kept.
	Retention time.Duration `env:"STATE_RETENTION" envDefault:"168h"`
	// NodeLossForceDeleteAfter: a snapshot-backed device pod that stays Terminating this long on a NotReady or missing node is
	// force-deleted and the device recreated from its last snapshot ("0" = never).
	NodeLossForceDeleteAfter time.Duration `env:"STATE_NODE_LOSS_FORCE_DELETE_AFTER" envDefault:"5m"`
	// RetentionInterval is how often the retention sweep runs.
	RetentionInterval time.Duration `env:"STATE_RETENTION_INTERVAL" envDefault:"10m"`
}

StateConfig configures device state persistence: devices of new labs run as bare Pods whose writable layer the node-agent snapshots into the platform snapshot registry.

func (StateConfig) MaxFileBytes

func (c StateConfig) MaxFileBytes() (int64, error)

MaxFileBytes parses MaxFileSize.

func (StateConfig) TenantQuotaBytes

func (c StateConfig) TenantQuotaBytes() (int64, error)

TenantQuotaBytes parses TenantQuota.

func (StateConfig) WriteQuotaBytes

func (c StateConfig) WriteQuotaBytes() (int64, error)

WriteQuotaBytes parses WriteQuota.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL