k3s

package
v0.0.0-...-6f76ffe Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package k3s is the config tier of a Kubernetes cluster SC stands up itself, on machines described by a `vm` resource (pkg/clouds/vm). It is the cheapest Kubernetes SC can offer in a cloud that has no managed Kubernetes at all — which is most of the Russian market — and it is deliberately a thin thing: everything above the kubeconfig (`kubernetes-cloudrun`, the Caddy edge, the helm operators) already exists and is reused unchanged.

Design: forge/docs/design/2026-10-13/sc-russia-market-readiness/k3s-all-clouds.md, roadmap item sc-k3s-cluster-resource (slice K.0).

Index

Constants

View Source
const (
	NodePortRangeStart = 30000
	NodePortRangeEnd   = 32767
)

The NodePort range a Kubernetes apiserver allocates `type: NodePort` services from, and the range k3s uses because SC never passes `--service-node-port-range`. Both ends inclusive.

They live here rather than in the firewall code that needs them because they are a property of the control plane, not of a cloud: `pkg/clouds/vm` imports this package for the `k3s` entry of a vm resource's `ingressPorts` preset table. That import direction is the reason this package must keep importing nothing from `pkg/clouds/vm` — the name of a vm resource (K3sClusterConfig.VmResource) is all it has ever needed, and widening it to the type would close a cycle.

View Source
const (
	// FlannelVxlanPort carries every packet between pods on different nodes. UDP.
	FlannelVxlanPort = 8472
	// KubeletPort serves metrics and `kubectl logs`/`exec`/`port-forward` — which
	// is also why it must never be public: it reaches into any pod on the node.
	KubeletPort = 10250
	// EtcdClientPort and EtcdPeerPort are the embedded etcd of a multi-server
	// cluster. Contiguous, and used as a range.
	EtcdClientPort = 2379
	EtcdPeerPort   = 2380
)

The ports k3s nodes use to talk to EACH OTHER. Node-to-node only — see vm.PortRange.Internal for why the distinction is per-port and not per-machine.

Taken from k3s's own "Inbound Rules for K3s Nodes" and narrowed to the backends SC actually runs: flannel's default VXLAN backend (so no 51820/51821, which are the wireguard-native ones), and embedded etcd, which is what `--cluster-init` gives an HA control plane.

View Source
const APIPort = 6443

APIPort is the port the k3s API server listens on. Not configurable: it is baked into the kubeconfig k3s itself writes, into `--tls-san` handling, and into every firewall note in the design doc. A flag here would have to be threaded through all three to mean anything.

View Source
const DefaultInstallScriptURL = "https://get.k3s.io"

DefaultInstallScriptURL is Rancher's installer. It resolves the binary from GitHub releases, and both hosts were measured reachable from inside Russia on 2026-10-17 at 6–8 MB/s (roadmap item sc-ru-market-readiness, slice B0) — which is why this is the default rather than a bundled binary. InstallScriptURL exists for the networks where that stops being true.

View Source
const DefaultVersion = "v1.36.5+k3s1"

DefaultVersion is the k3s release SC installs when a resource does not pin one.

A literal version and NOT a channel ("stable", "latest"), deliberately, and the same reasoning refuses a channel in Validate: a channel is resolved by the install script at install time, so the same SC config provisioned twice a month apart stands up two different control planes, and the "is the installed version the wanted one?" guard that keeps a re-provision from restarting the API server cannot be evaluated at all. Bump this by hand, from `curl -s https://update.k3s.io/v1-release/channels` (the `stable` entry), and only together with a smoke — a control-plane upgrade is not a dependency bump.

View Source
const ProviderType = "k3s"

ProviderType is the discriminator K3sClusterConfig reports from ProviderType(), the key pkg/clouds/pulumi/k3s registers its Provider under, and the directory generated schemas land in (docs/schemas/k3s).

It is its OWN provider type rather than the vm resource's "vm", and that is not a stylistic choice: provision.go's initProvider looks the provider up by the config's ProviderType() (pkg/clouds/pulumi/provision.go:311), so a k3s-cluster config that reported "vm" would be handed to the vm provider, which casts its input to *vm.VmConfig and would refuse it with a type-assertion error.

View Source
const ResourceTypeK3sCluster = "k3s-cluster"

ResourceTypeK3sCluster is the `type:` a parent stack's resource declares.

View Source
const TemplateTypeK3sCloudrun = "k3s-cloudrun"

TemplateTypeK3sCloudrun deploys a cloud-compose stack onto a k3s-cluster resource of the parent stack.

It exists for one reason, and it is not a difference in what gets deployed: the existing `kubernetes-cloudrun` template takes its kubeconfig as a LITERAL string in the template config (k8s.KubernetesConfig.Kubeconfig, surfaced through CredentialsValue() so provision.go's initProvider can build the provider from it). A k3s cluster's kubeconfig does not exist until the cluster has been provisioned, so it can only arrive as a parent-stack output — the same way gke-autopilot's does. Hence a template that names a cluster resource instead of carrying credentials.

Everything below the kubeconfig is reused unchanged: the compose→containers mapping is k8s.ToKubernetesRunConfig, and the deploy is kubernetes.KubeRunWithProvider.

Variables

View Source
var DisableableComponents = []string{
	"traefik",
	"servicelb",
	"local-storage",
	"metrics-server",
}

The k3s packaged components SC is willing to turn off. k3s accepts `--disable` for each of these; anything else is refused rather than passed through, because a misspelled component name leaves the component RUNNING and k3s says nothing — the operator believes traefik is gone while it holds port 80.

Functions

func KubeconfigURL

func KubeconfigURL(endpoint string) string

KubeconfigURL is the `server:` URL the exported kubeconfig must carry.

func ReadK3sCloudrunTemplateConfig

func ReadK3sCloudrunTemplateConfig(config *api.Config) (api.Config, error)

func ReadK3sClusterConfig

func ReadK3sClusterConfig(config *api.Config) (api.Config, error)

func ToK3sCloudrunConfig

func ToK3sCloudrunConfig(tpl any, composeCfg compose.Config, stackCfg *api.StackConfigCompose) (any, error)

ToK3sCloudrunConfig converts a cloud-compose client stack for the k3s path.

It delegates the entire mapping to k8s.ToKubernetesRunConfig against a synthesised k8s.CloudrunTemplate. That is not laziness: ConvertComposeToContainers, the cloudExtras handling (probes, VPA, topology spread, security context) and ingress container detection are a few hundred lines that are identical for every Kubernetes flavour, and a second copy would drift silently — the symptom being a cloudExtras field that works on GKE and is ignored on k3s.

Types

type K3sCloudrunInput

type K3sCloudrunInput struct {
	k8s.KubeRunInput `json:",inline" yaml:",inline"`

	// K3sClusterResource is carried through from the template because the
	// provisioning side needs it to spell the parent's export names, and the
	// embedded KubeRunInput has nowhere to put it.
	K3sClusterResource string `json:"k3sClusterResource" yaml:"k3sClusterResource"`
}

K3sCloudrunInput is what a converted client stack carries into provisioning. The embedded k8s.KubeRunInput is the point: it is byte-for-byte what kubernetes.KubeRunWithProvider consumes, so the deploy below the kubeconfig is the existing code path and not a second copy of it.

func (*K3sCloudrunInput) CredentialsValue

func (i *K3sCloudrunInput) CredentialsValue() string

func (*K3sCloudrunInput) ProjectIdValue

func (i *K3sCloudrunInput) ProjectIdValue() string

func (*K3sCloudrunInput) ProviderType

func (i *K3sCloudrunInput) ProviderType() string

ProviderType/ProjectIdValue/CredentialsValue are re-declared on the input rather than inherited: the embedded k8s.KubeRunInput promotes k8s.KubernetesConfig's versions, which report "kubernetes" and hand back an empty kubeconfig as credentials. See K3sCloudrunTemplate.ProviderType for what that would cost.

type K3sCloudrunTemplate

type K3sCloudrunTemplate struct {
	api.Credentials `json:",inline" yaml:",inline"`

	// K3sClusterResource is the name of the k3s-cluster resource in the same parent
	// stack whose kubeconfig this template deploys through. Same role as
	// gke-autopilot's `gkeClusterResource`.
	K3sClusterResource string `json:"k3sClusterResource" yaml:"k3sClusterResource"`

	// The registry images are pushed to and pulled from. Required, and there is no
	// fallback: unlike the vm-compose path, which can ship an image over SSH with
	// `imageDelivery: ssh-tar`, a kubelet can only pull. A k3s stand with no reachable
	// registry is a stand that cannot deploy — which on the RU-market track is a
	// deliberate thing to surface at config time rather than at image-pull time.
	docker.RegistryCredentials `json:",inline" yaml:",inline"`

	CaddyResource *string `json:"caddyResource,omitempty" yaml:"caddyResource,omitempty"` // name of the caddy resource in the parent stack

	// TlsConfig carries `clientTls` / `originTls` and the deprecated `useSSL`,
	// inline — the same vocabulary as the kubernetes-cloudrun template, because
	// this template builds one.
	k8s.TlsConfig `json:",inline" yaml:",inline"`
}

K3sCloudrunTemplate is the parent-stack template for deploying onto a k3s cluster.

Deliberately NOT embedding k8s.CloudrunTemplate: that would inherit a `kubeconfig` field which must always be empty here, and a config field that must never be set is a field someone will eventually set. The provisioning side builds a k8s.KubeRunInput from these values instead.

func (*K3sCloudrunTemplate) CredentialsValue

func (t *K3sCloudrunTemplate) CredentialsValue() string

CredentialsValue returns an empty JSON object rather than "": api.ConvertAuth json.Unmarshals this, and "" is not valid JSON. There are no credentials to carry — reaching the cluster needs the parent's kubeconfig output, and pushing images needs the registry fields above.

func (*K3sCloudrunTemplate) ProjectIdValue

func (t *K3sCloudrunTemplate) ProjectIdValue() string

func (*K3sCloudrunTemplate) ProviderType

func (t *K3sCloudrunTemplate) ProviderType() string

ProviderType reports k3s for the same reason K3sClusterConfig does, and with a sharper consequence here: provision.go's initProvider builds a provider from this value on the DEPLOY path too (deploy.go:193 → getProvisionParams → initProvider, against the converted client-stack config). Reporting "kubernetes" would send it to kubernetes.Provider, which calls CredentialsValue() and builds a real pulumi-kubernetes provider out of an empty kubeconfig — and pulumi-kubernetes with no kubeconfig falls back to the AMBIENT one. On a laptop with a live ~/.kube/config that is a provider pointed at whatever cluster the operator last used.

So: k3s, whose provider is pulumi-command's and is simply unused on this path. K3sCloudrun overrides params.Provider with the real Kubernetes provider it builds from the parent's kubeconfig export.

func (*K3sCloudrunTemplate) Validate

func (t *K3sCloudrunTemplate) Validate() error

type K3sClusterConfig

type K3sClusterConfig struct {
	// VmResource is the name of the `type: vm` resource, in the same parent stack and
	// the same environment, whose hosts become this cluster's nodes.
	VmResource string `json:"vmResource" yaml:"vmResource"`

	// Version pins the k3s release. Defaults to DefaultVersion; a channel name is
	// refused (see DefaultVersion).
	Version string `json:"version,omitempty" yaml:"version,omitempty"`

	// Endpoint is the host or IP that the EXPORTED kubeconfig points at, and the
	// first `--tls-san`.
	//
	// It defaults to the node's own address, which is correct for the single-node
	// stand K.0 builds. Set it when the API server is reached through a name or a
	// load balancer rather than directly — an endpoint that is not a SAN on the
	// serving certificate fails every request with an x509 hostname error, which is
	// why this field and the SAN list are the same field by default.
	Endpoint string `json:"endpoint,omitempty" yaml:"endpoint,omitempty"`

	// Servers is how many of the vm resource's hosts run the control plane. The rest
	// are agents.
	//
	// Default 1 (see EffectiveServers): one server and N-1 agents, which is the
	// topology worth reaching for first — it schedules pods across the whole fleet
	// and survives the loss of any agent, and it is the only shape a 2-machine pool
	// can have. Its control plane is a single point of failure: while host 0 is down
	// the API server is unreachable and nothing can be deployed, though pods already
	// running on the agents keep serving.
	//
	// Set it to 3 (or 5) to run embedded etcd instead, which survives the loss of a
	// control-plane node. It must be ODD: etcd needs a majority of its members to
	// elect a leader, so 2 servers tolerate zero failures — strictly worse than 1,
	// because either machine going away takes the API server down with it. Refused
	// rather than rounded.
	//
	// Ignored on a byo pool that names its own `hosts[].role`, where the operator
	// chose which machine runs what; a value disagreeing with that choice is refused
	// rather than silently losing (pkg/clouds/pulumi/k3s/cluster.go).
	Servers int `json:"servers,omitempty" yaml:"servers,omitempty"`

	// TlsSans are additional names to put on the API server certificate, beyond
	// Endpoint and the node address. Spend them freely: adding a SAN later means
	// re-issuing the certificate and restarting the API server, while an unused SAN
	// costs nothing (k3s-all-clouds.md §4).
	TlsSans []string `json:"tlsSans,omitempty" yaml:"tlsSans,omitempty"`

	// Disable overrides which packaged components are turned off. Leave it unset:
	// the default is derived from the host (see pkg/clouds/pulumi/k3s/cluster.go),
	// because whether klipper-lb can have ports 80 and 443 depends on whether the
	// machine's own compose ingress already holds them.
	Disable []string `json:"disable,omitempty" yaml:"disable,omitempty"`

	// ExtraServerArgs is passed verbatim to the k3s server. The flags SC manages
	// itself are refused (managedServerArgs).
	ExtraServerArgs []string `json:"extraServerArgs,omitempty" yaml:"extraServerArgs,omitempty"`

	// InstallScriptURL overrides the installer location for a network that cannot
	// reach Rancher's (see DefaultInstallScriptURL).
	InstallScriptURL string `json:"installScriptUrl,omitempty" yaml:"installScriptUrl,omitempty"`

	api.Credentials `json:",inline" yaml:",inline"`
}

K3sClusterConfig is the resource config of `type: k3s-cluster`.

It describes the CLUSTER only. The machines it runs on are described by a `vm` resource in the same parent stack and referenced by name through VmResource — the same split as `gcp-gke-autopilot` referencing its own resources, and the reason the k3s path inherits cloud-init, SSH readiness, key handling and every future cloud driver from pkg/clouds/vm without a line of its own.

func (*K3sClusterConfig) CredentialsValue

func (r *K3sClusterConfig) CredentialsValue() string

CredentialsValue returns the resolved `${auth:...}` blob verbatim, "{}" when none was given. A k3s cluster authenticates to nothing: it rides the SSH key of the vm resource it names, and that key is never copied into this config.

func (*K3sClusterConfig) DependsOnResources

func (r *K3sClusterConfig) DependsOnResources() []api.ParentResourceDependency

DependsOnResources tells provision.go that this resource must be provisioned AFTER the vm resource it names, and makes the vm resource's output available as params.ResourceOutputs[VmResource] (pkg/clouds/pulumi/provision.go:145-172).

This is the whole mechanism by which a cluster learns its nodes' addresses and readiness gates without a StackReference back into the stack currently being provisioned — which is not merely slow, it is a cycle.

func (*K3sClusterConfig) EffectiveInstallScriptURL

func (r *K3sClusterConfig) EffectiveInstallScriptURL() string

func (*K3sClusterConfig) EffectiveServers

func (r *K3sClusterConfig) EffectiveServers() int

EffectiveServers is the control-plane node count, defaulting to 1.

The one place the default lives, so the provisioning tier never re-derives it: a resource that disagreed with itself about which hosts are servers would join a machine to a cluster as an agent on one apply and as an etcd member on the next, and the second of those is not reversible by changing the number back.

func (*K3sClusterConfig) EffectiveVersion

func (r *K3sClusterConfig) EffectiveVersion() string

func (*K3sClusterConfig) ProjectIdValue

func (r *K3sClusterConfig) ProjectIdValue() string

ProjectIdValue has no meaning for a cluster SC installs over SSH: there is no cloud project behind it. The k8s-style "n/a" keeps `${resource:...projectId}` from interpolating to something that looks real (same choice as vm.VmConfig).

func (*K3sClusterConfig) ProviderType

func (r *K3sClusterConfig) ProviderType() string

func (*K3sClusterConfig) Validate

func (r *K3sClusterConfig) Validate() error

Validate enforces the same rule as the rest of the VM path: either support a field or refuse it. It is called from the provisioning tier rather than from ReadK3sClusterConfig, because the config readers are also driven with an empty config by cmd/schema-gen, and a reader that rejects an empty config drops the whole resource out of the generated schema.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL