Documentation
¶
Overview ¶
Package ha implements HA control-plane join (`stratum server --server --token --certificate-key`): it securely distributes the shared CP certs, adds a live etcd member, brings up an additional apiserver, and fronts them with a per-node HAProxy.
Index ¶
- Constants
- Variables
- func AssembleNode(a NodeAssembly, req AssembleRequest) (host.NodeIPChoice, error)
- func AssembleNodeContext(ctx context.Context, a NodeAssembly, req AssembleRequest) (host.NodeIPChoice, error)
- func ClearJoinIntent() error
- func DowngradeEtcd(cpIP, target string) error
- func DownloadCerts(bootstrapKubeconfig, certKeyHex string, check func(bundleProfile) error) (bundleGrant, error)
- func DownloadCertsAdmin(certKeyHex string) error
- func EtcdClusterHealthy(cpIPs []string) error
- func GrantCertsReader(ctx context.Context, cs kubernetes.Interface) error
- func JoinIntentPath() string
- func JoinIntentSurvives() (bool, error)
- func JoinState(cpIP string) (members int, bundle bool, err error)
- func MemberAdd(cpIPs []string, newName, newPeerURL string) (string, error)
- func MemberClientIPs(cpIP string) ([]string, error)
- func MemberClientIPsVia(cpIPs []string) ([]string, error)
- func MemberCount(cpIPs []string) (int, error)
- func MemberHostsVia(cpIPs []string) ([]string, error)
- func MemberRemove(cpIPs []string, peerURL string) error
- func NarrowCertsReaderBinding(ctx context.Context, cs kubernetes.Interface) error
- func NewCertKey() (string, error)
- func NodeRepoint(w io.Writer, vip string) ([]string, error)
- func PrepareControlPlaneJoin(ctx context.Context, o JoinOptions) (*nodestate.NodeState, error)
- func PruneExpiredGrants(ctx context.Context, cs kubernetes.Interface) error
- func PublishedBundleCarriesKey(cs kubernetes.Interface, encryptionConfig []byte) (published, carries bool, err error)
- func ReUploadCerts(certKeyHex string) error
- func RejoinEtcd(ctx context.Context, w io.Writer, cp1IP string, force bool) error
- func RekeySharedCerts(certKeyHex string, confirm func(context.Context) error) error
- func RepointCilium(w io.Writer, vip string) ([]string, error)
- func RepointCiliumWith(ctx context.Context, w io.Writer, cs kubernetes.Interface, vip string) ([]string, error)
- func RepointKubeconfigs(server string) error
- func SharedCertsKeySetAt() (setAt time.Time, present bool, err error)
- func UploadCerts(certKeyHex string, owner *metav1.OwnerReference, ...) error
- type AssembleRequest
- type EtcdMemberHealth
- type EtcdStorage
- type JoinOptions
- type NodeAssembly
Constants ¶
const JoinBundleEtcdKey = "/registry/secrets/kube-system/" + certsSecret
JoinBundleEtcdKey is where the apiserver stores the stratum-certs Secret, the encrypted shared-cert bundle a control-plane join downloads. Its existence is what tells an etcd reader that a join can still be made with it.
This couples to the apiserver's storage layout, not to any API: "/registry" is kube-apiserver's default --etcd-prefix, which staticpod's apiserver args do not override, and "secrets/<namespace>/<name>" is how it keys a Secret under that prefix. A change to either one makes this key name nothing, and the bundle gate then reads every cluster as having no bundle published -- which is why internal/cluster checks it against a database written by the etcd this repo ships.
Variables ¶
var ErrDowngradeNotValidated = errors.New("etcd did not validate a downgrade")
ErrDowngradeNotValidated matches a DowngradeEtcd that failed at its validate step, before it asked etcd to change anything.
var ErrEtcdNoStatus = errors.New("etcd status")
ErrEtcdNoStatus matches an EtcdStorageAt that failed because the member did not answer its status request, as opposed to a failure to set up the client or to read the membership of a member that did answer.
var VIPServes = checkVIPServes
VIPServes reports whether the VIP serves this cluster's apiserver, as checkVIPServes does. Exported, and a var, so a test of a command that repoints, in this package or another, can stand in for a VIP that does or does not answer.
Functions ¶
func AssembleNode ¶
func AssembleNode(a NodeAssembly, req AssembleRequest) (host.NodeIPChoice, error)
AssembleNode runs the common node steps of controller bootstrap and control-plane join, in the order both paths' guards pin: host directories, prerequisites, the caller's network resolution, pod-CIDR trust, node-IP resolution, the caller's before-certificates hook, certificates, kubeconfigs, the profile step (a node with a profile only), the caller's before-runtime-configs hook, and the runtime configs on the caller's node state. Every fallible step returns before the steps that follow, and a hook's error is the error reported, so a caller's refusal text survives the shared assembly. The resolved node IP choice is returned for the caller's own use.
func AssembleNodeContext ¶
func AssembleNodeContext(ctx context.Context, a NodeAssembly, req AssembleRequest) (host.NodeIPChoice, error)
AssembleNodeContext stops between preparation steps when ctx ends. A step already in flight finishes and retains its own error; cancellation prevents the next step from starting, including mutations of local files or firewall.
func ClearJoinIntent ¶
func ClearJoinIntent() error
ClearJoinIntent ends a join's intent journal once its work is done: the node's local etcd data is confirmed on a later start, so the state between the join's state save and etcd's first WAL no longer needs a recovery record. Derived from version.K8sConfDir at call time — the same path JoinIntentSurvives reports on. A missing file is fine; any other error is returned to the caller.
func DowngradeEtcd ¶
DowngradeEtcd asks the etcd cluster at cpIP to downgrade to target, a major.minor version: etcd checks that it can, then lowers the cluster version and migrates every member's storage down to it, so the older etcd can open the data. It is a cluster-wide operation, and it ends once every member runs the target version.
func DownloadCerts ¶
func DownloadCerts(bootstrapKubeconfig, certKeyHex string, check func(bundleProfile) error) (bundleGrant, error)
DownloadCerts fetches + decrypts the shared certs (authenticating with the bootstrap-kubeconfig token), writes them to version.PKIDir, and returns the grant the bundle makes of the download: the etcd client addresses the cluster published, whether the bundle itself carried the at-rest encryption key — a fact about the download, not about the disk, where a key left by an earlier failed join would survive a keyless bundle — the cluster VIP the publishing control plane recorded, and the publication's generation. From a bundle in the v1 envelope the facts come from the ciphertext's metadata; from a legacy flat-map bundle — published before the envelope, or fetched with admin credentials by an already-joined control plane, which has no token ID — from the shapes that bundle always carried.
The entry decrypted is the grant's own: the bootstrap token in the kubeconfig names the certs-<token-id> entry the mint that printed this certificate-key wrote, so a second mint moving the legacy entry to a newer key does not break this join. The token is read from the kubeconfig's own auth info rather than the built client config: clientcmd merges user credentials into a config only when its transport is TLS, and the token ID must be known whatever endpoint serves the Secret. check is openGrant's.
func DownloadCertsAdmin ¶
DownloadCertsAdmin fetches + decrypts the shared certs using this node's admin.kubeconfig. For an already-joined control plane pulling freshly-staged CA-rotation material (it has no bootstrap token, but does have admin creds). With no token there is no grant entry of its own, so it decrypts the legacy certs entry — the newest mint's or the rotation re-upload's key.
func EtcdClusterHealthy ¶
EtcdClusterHealthy fails closed: every named member must answer, must know a leader, and every member must name the SAME leader.
The agreement check is the point. Per-member health is satisfied by a split brain — two partitions, each with a leader, each locally convinced it is fine — and a split brain is the one condition under which narrowing etcd's trust cluster-wide cannot be walked back, because the partition that loses will not accept the anchor the other half kept.
func GrantCertsReader ¶
func GrantCertsReader(ctx context.Context, cs kubernetes.Interface) error
GrantCertsReader writes the stratum:certs-reader grant every upload publishes before the Secret: the Role and the RoleBinding that let the control-plane bootstrap group get the stratum-certs Secret and hold the maintenance fence, and, before either, the admission policy that keeps that group to its own fence (ensureJoinFencePolicy). The grant waits for enforcement on every serving API server before writing the Role; policy storage alone does not synchronize the admission and RBAC caches.
func JoinIntentPath ¶
func JoinIntentPath() string
JoinIntentPath is the durable pre-add intent journal's path in this node's state directory, derived from version.K8sConfDir at call time — the path the caller-facing gate (JoinIntentSurvives) and the refusals that name the file report. The join's own reads, writes, and clear go through the package var joinIntentPath instead, which this package's tests redirect; the two name the same file because version.K8sConfDir is never reassigned outside tests.
func JoinIntentSurvives ¶
JoinIntentSurvives reports whether a join intent file survives in this node's state directory, whatever its contents. A join that added its member but never persisted the node state leaves the file there as the recovery record, so a bare `stratum up` that treated it as absent would bootstrap a separate single-member cluster over the recorded membership. An unreadable file is an error, never "absent" — readJoinIntent's rule.
func JoinState ¶
JoinState reports, from the etcd at cpIP, how many members (learners included) it has and whether a control-plane join bundle is published. A learner counts: it is a control plane that is joining, and it has whatever at-rest key the cluster had when it downloaded the shared certs. Both reads are linearizable (the client's default), so neither is a stale answer from a lagging member.
func MemberAdd ¶
MemberAdd adds newPeerURL to the cluster reachable at any of cpIPs' etcd and returns the ETCD_INITIAL_CLUSTER string the new member must boot with (state=existing).
It retries etcd's "unhealthy cluster" refusal, and only that. etcd's strict reconfiguration check refuses a MemberAdd until every member has been connected for a few seconds, so a join that follows another one closely, which is what a scripted three-node bring-up does, was refused outright and the node exited half-joined. The refusal means nothing was added, so retrying it is safe; any other error is returned at once.
func MemberClientIPs ¶
MemberClientIPs returns the host of every member's client URL (HAProxy backends).
func MemberClientIPsVia ¶
MemberClientIPsVia is MemberClientIPs through any of several endpoints.
func MemberCount ¶
MemberCount is how many members (learners included) the cluster at cpIPs' etcd has, as the answering member last saw it.
func MemberHostsVia ¶
MemberHostsVia returns one host per etcd member, learners included, read through any of several endpoints: the host of its first non-loopback client URL, or of its first peer URL when it has none. A member added but not yet started has published no client URL, and a caller accounting for every member must still see it. A member with neither is an error, since it cannot be placed on any node.
func MemberRemove ¶
MemberRemove drops a member by its peer URL (rollback on a failed join), through any of cpIPs' etcd. The join's rollback uses the same published endpoints as its MemberAdd: with the --join host being a VIP, removing through it would fail exactly when a rollback is needed.
func NarrowCertsReaderBinding ¶
func NarrowCertsReaderBinding(ctx context.Context, cs kubernetes.Interface) error
NarrowCertsReaderBinding rewrites an existing stratum:certs-reader RoleBinding to name only certsReaderSubjects, and leaves an absent one absent. Every token mint calls it, so an upgraded cluster drops the worker grant on an existing stratum-certs Secret the first time it mints any token, not only on the next upload.
func NewCertKey ¶
NewCertKey returns a fresh AES-256 certificate-key (hex) for upload-certs.
func NodeRepoint ¶
NodeRepoint moves an existing node's clients onto the cluster VIP (kube-vip's one address serves in-cluster and external clients): it rewrites this node's kubeconfigs to https://<vip>:6443 and restarts the local consumers. Used when a cluster that started as a single controller (clients on 127.0.0.1) grows into HA. There is no per-worker proxy; kube-vip's one address takes its place. It changes nothing unless the VIP already serves this cluster's apiserver with a certificate that covers it. A consumer that fails to restart comes back as a warning beside a nil error — the kubeconfigs were rewritten, so the repoint landed — never as a failure.
func PrepareControlPlaneJoin ¶
PrepareControlPlaneJoin readies this node as an additional control plane: payload, prereqs, shared certs, local leaves, live etcd member-add, and the persisted node state the supervisor renders etcd's flags from. The CALLER (stratum up --join) then starts the supervisor.
ctx is the run's interrupt context (internal/interrupt), created by runUp before preflight and threaded through preparation: a signal received before the live member-add stops the join at a step boundary, never performing the add. From the member-add to the persisted node state the first signal is deliberately deferred — the policy is documented at that window.
func PruneExpiredGrants ¶
func PruneExpiredGrants(ctx context.Context, cs kubernetes.Interface) error
PruneExpiredGrants drops the stratum-certs grant entries whose bootstrap token is gone (expiredGrantEntries), and leaves the legacy certs entry, every live grant's entry and every owner reference as read. The operator runs it on every pass, so expired grants go even when no one mints. It writes nothing when nothing expired: rewriting the cluster's CA-key ciphertext on every pass would bump its resourceVersion without changing anything. The write is the object as read minus the expired entries, carrying the resourceVersion it read, so a mint landing meanwhile conflicts, and the retry re-reads and prunes again from the fresh state. An absent Secret — nothing minted yet, or the bundle deleted deliberately — is left absent.
func PublishedBundleCarriesKey ¶
func PublishedBundleCarriesKey(cs kubernetes.Interface, encryptionConfig []byte) (published, carries bool, err error)
PublishedBundleCarriesKey answers, over the Secret API, whether a control-plane join bundle is published and whether it carries this node's CIS at-rest encryption key. It is the bundle owner's answer to the question the controller-start key gate (internal/cluster) asks, and it answers from the same Secret a join will actually download — not from the apiserver's storage key for it, which can trail what the API serves.
"Published" is the Secret's existence, never a judgment of its entries: the multi-entry grant Secret carries one ciphertext entry per live grant beside the legacy certs entry, and every such state is a bundle a join can be made with. "Carries the key" is a decryption, not a label: an upload whose bundle carries the at-rest key publishes the proof entry sealed under that key, so the open — not an annotation anyone with Secret write can forge — says whether this node's key is the one a joiner would receive. A proof that does not open under this node's key is the answer "unkeyed", not an error: it is exactly what a bundle uploaded under another key, a keyless upload, or one from a stratum that predated the proof looks like. A Secret that cannot be read is an error, never an answer, because either published answer it could fake is wrong on one side.
encryptionConfig is this node's encryption-provider config — the bytes of version.EncryptionConfigPath — or nil when this node holds no at-rest key yet. Nil is the first gate's case: there is nothing to match the proof against, so a published bundle can only answer unkeyed.
func ReUploadCerts ¶
ReUploadCerts re-encrypts the current on-disk shared material — including the trust bundles and any mid-rotation staged CAs — into the stratum-certs Secret, so a CP joining after a CA rotation receives the new material. It is UploadCerts re-run (create-or-update), except that an old signing key a rotation recorded as forced went ahead without is left out when its staged replacement is there. It passes no owner, so it rewrites the legacy certs entry under its key and keeps the owner references as read, and it removes every grant entry in the same update. A grant is the shared files sealed at its mint, under a key only its join command holds, so nothing can re-seal it with the rotated trust: kept, it would hand a joiner the pre-rotation set — during a halted rotation a joiner whose pin still matches the old root would join without the staged CA and strand the resumed run. A join command minted before the rotation therefore fails at decryption, before it changes anything, and a new one is minted from the current material. ReUploadCerts publishes under the CA rotation's own fence: the rotation holds and renews the lease for its whole run, and its phase machinery pins what each phase applied, so no per-write recheck here — the rotation engine owns the fence and its resumability.
func RejoinEtcd ¶
RejoinEtcd re-adds THIS control plane's etcd to a freshly-restored cluster. After `stratum snapshot restore` resets the seed node (cp1) to a NEW single-member cluster holding the restored data, every OTHER control plane runs this: it stops the local CP, wipes the stale (old-cluster) etcd data dir, adds this member to the seed's etcd as a learner, rewrites the etcd manifest as a joining member (state=existing), restarts, and promotes the learner once it has caught up. The node's PKI + kubeconfigs are already in place — this is a rejoin, not a fresh join. cp1IP is the seed's IP (from --server).
A re-run resumes rather than starting over: see planRejoin. force lifts the refusal to wipe an etcd whose cluster still has quorum; a path that does not wipe never needs it. The writer leads, after ctx: the periodic waiting report goes through it, so the CLI passes os.Stdout and the MCP handler the buffer its result is built from.
ctx bounds the wait for this node's etcd, which can run to rejoinMaxWait and holds the etcd data lock for all of it -- a non-blocking flock, so every other stratum command that wants it fails outright meanwhile.
RejoinEtcd takes the node maintenance lock (internal/nodelock) before the data lock and holds both for the whole run: it stops this node's control plane and wipes its etcd data dir, which no other maintenance operation -- an upgrade, a rollback, a CA rotation -- must race. While one holds the lock, the rejoin refuses, naming the holder, before anything is touched.
func RekeySharedCerts ¶
RekeySharedCerts replaces the shared bundle and removes all per-token ciphertexts in one Secret update, so no previous certificate-key can still decrypt a pending grant. Like rotation's ReUploadCerts, this explicit operator action intentionally invalidates previously issued join keys. The caller must hold the maintenance fence and pass the rekey guard first, and hand the fence's recheck in as confirm: the rekey makes several API calls between the take and the Secret write, and a lease lost while one of them stalled must refuse the publication.
func RepointCilium ¶
RepointCilium points every Cilium agent + operator at the VIP instead of a single control-plane IP, so the CNI survives any single control-plane failure. Central, run ONCE after the VIP exists. (Cilium's kube-proxy-free agents dial the apiserver directly and cannot use the in-cluster Service before they are up, so the address must be highly available itself.) It writes through this node's admin.conf. A rollout that does not finish is a warning beside a nil error: the env change is already applied.
func RepointCiliumWith ¶
func RepointCiliumWith(ctx context.Context, w io.Writer, cs kubernetes.Interface, vip string) ([]string, error)
RepointCiliumWith is RepointCilium through cs, so a caller with its own credential is bounded by that credential's RBAC. A rollout that does not finish in time is returned as a warning beside the nil error — the change is applied and live, so the repoint is not a failure — naming what each unfinished rollout last reported. Like NodeRepoint, it changes nothing unless the VIP serves this cluster's apiserver, and its progress goes through the writer it is handed first (after its context).
func RepointKubeconfigs ¶
RepointKubeconfigs rewrites every kubeconfig on this node to the given server URL, preserving embedded credentials, and records that endpoint in the node state's Server — the record every kubeconfig writer reads, so the next `stratum up` writes kubeconfigs from the repointed endpoint instead of the stale one and the repoint survives the restart. That includes bootstrap.conf: it is never deleted, workers hand it to the kubelet on every start, and a kubelet that has to re-bootstrap would otherwise dial the old server. A node with no state file has nothing to record (workers keep their join server); a state that cannot be read is an error, never a silent skip.
func SharedCertsKeySetAt ¶
SharedCertsKeySetAt reports when the stratum-certs certificate-key was last set — the annotation UploadCerts stamps on every re-key — falling back to the Secret's CreationTimestamp for a Secret written before that annotation existed. present is false when the Secret does not exist. `certs check` uses this to surface a key that has not moved since install.
func UploadCerts ¶
func UploadCerts(certKeyHex string, owner *metav1.OwnerReference, confirm func(context.Context) error) error
UploadCerts encrypts the shared CP certs into the stratum-certs Secret and grants the control-plane bootstrap group read access (so a joining CP can fetch them with its control-plane bootstrap token, then decrypt with the certificate-key out of band; a worker token cannot read them).
The Secret is the multi-entry grant Secret: ONE Secret, one ciphertext entry per live grant keyed by token ID (certs-<token-id>) beside the legacy certs entry. This upload APPENDS its own grant's entry — carrying the v1 envelope, the bundle's file set with the authenticated generation, at-rest-key, VIP and etcd-endpoint metadata — and re-encrypts the legacy certs entry under its own key as the flat map every supported joiner version decodes; it never replaces another live grant's entry, so minting a second control-plane join does not break the first — each pending grant decrypts its own entry with the certificate-key its mint printed while its bootstrap token is live, until a rotation, migration or rekey re-upload (ReUploadCerts, RekeySharedCerts) retires every grant. The legacy certs entry follows the newest mint's key, so a joiner running an older stratum release, which reads only that entry, can use only the most recently minted join command. The upload drops the entry of every grant whose bootstrap token is gone (expiredGrantEntries) in the same update that appends its own — the operator does the same on each pass (PruneExpiredGrants), but may be turned off — and refuses, before writing anything, a mint whose grant the Secret cannot hold (grantSecretBudget).
owner, when non-nil, is added (deduped by UID) to the Secret's OwnerReferences — the bootstrap-token Secret minted in the same `token create --control-plane` run. Garbage collection removes stratum-certs only once EVERY owning token is gone, so the shared-cert material is reaped when the last granting token expires but is not pulled out from under a second, still-valid control-plane join whose own token is still live. Owner refs are merged, never replaced, precisely so a later `token create` cannot shorten an earlier token's access. The CA-rotation re-upload path (ReUploadCerts) passes nil and preserves whatever owners the Secret carries. UploadCerts publishes the shared certificates under certKeyHex, tied to owner when provided. confirm is the caller's maintenance-lease recheck (nil when the caller holds no fence): it runs immediately before the Secret write, inside the conflict retry, so a lease lost while an earlier call stalled refuses the publication instead of overwriting the successor's bundle with this node's snapshot.
Types ¶
type AssembleRequest ¶
type AssembleRequest struct {
// Hostname is this node's name: the host the certificate and kubeconfig
// effects run for, and the join's etcd member name.
Hostname string
// NodeIPFlag is --node-ip from the command line ("" when not given);
// ConfigNodeIP is spec.network.nodeIP; SavedIP is the address a node with
// saved state was brought up with ("" on a fresh bring-up or join). The IP
// precedence is host.ResolveNodeIP's: flag, saved, config, $NODE_IP, then
// detection.
NodeIPFlag, ConfigNodeIP, SavedIP string
// VIPAddress is the cluster VIP ("" when none): the extra apiserver SAN,
// the kubeconfigs' VIP marker, and a controller's client-facing endpoint.
VIPAddress string
// APIServer is the endpoint every kubeconfig on this node targets: the
// node state's recorded endpoint (bring-up's rule — the VIP when
// configured, the local apiserver otherwise — which the join records
// too), not the join URL.
APIServer string
// PrereqOptions are the host-preparation settings the prerequisites check
// runs with (host.PrereqOptions): the node's profile, whose config files
// the assembly writes, the SELinux opt-out, and whether this node was
// brought up before.
PrereqOptions host.PrereqOptions
// State is the node state the runtime configs render from.
State *nodestate.NodeState
}
AssembleRequest is what a caller hands the shared node assembly: the inputs the common steps consume, and the node state the runtime configs render from. State may be handed in partially built — the caller's hooks fill the fields only it knows (the netconfig's CIDRs, the resolved node IP, the carried membership) as the assembly resolves them — so the runtime configs still render the completed state.
type EtcdMemberHealth ¶
EtcdMemberHealth is one member's answer. Err set means the member did not answer, which is never treated as healthy.
func EtcdMemberStatus ¶
func EtcdMemberStatus(cpIP string) EtcdMemberHealth
EtcdMemberStatus asks the etcd at cpIP how it is. A member with no leader is reported, not hidden: it is up, it is answering, and it is not part of a working quorum.
func (EtcdMemberHealth) Healthy ¶
func (m EtcdMemberHealth) Healthy() bool
Healthy reports whether this member answered, knows a leader, and raised no alarms of its own.
type EtcdStorage ¶
type EtcdStorage struct {
StorageVersion string
Members int
Downgrading bool
// DowngradeTarget is the version an enabled downgrade is taking the
// cluster to, "" when none is enabled.
DowngradeTarget string
}
EtcdStorage is the etcd at one address as a rollback needs it: the version its data is stored at, how many members its cluster has, and whether a downgrade is enabled.
func EtcdStorageAt ¶
func EtcdStorageAt(cpIP string) (EtcdStorage, error)
EtcdStorageAt reads the storage version of the etcd member at cpIP, the size of its cluster and whether a downgrade is enabled. Both reads are the member's own view: a member of a cluster that has lost quorum still answers them, so it is not mistaken for one that is down.
type JoinOptions ¶
type JoinOptions struct {
Server string
Token string
CAHash string
CertKey string
NodeIP string // --node-ip ("" when not given)
// ConfigNodeIP is spec.network.nodeIP. Kept apart from NodeIP so the join
// logs which one won (host.ResolveNodeIP has the whole precedence).
ConfigNodeIP string
Operator bool // run the stratum-operator host service on this CP too (leader-elected)
CertRenew bool // install the cert auto-renew timer on this CP too
// Profile is the node profile this control plane runs, by name; "" for
// none. CIS is its legacy spelling, true exactly for the cis profile, which
// the join's key agreement and config writers still read.
Profile string
CIS bool
// SELinuxOptOut is this control plane's spec.selinux: false.
SELinuxOptOut bool
VIP nodestate.VIP // the cluster VIP this CP joins (kube-vip election)
// PodCIDR/ServiceCIDR are spec.network.podCIDR and serviceCIDR, "" when
// unset. They cannot override the cluster's ranges, which ride in with the
// shared certs: a different one refuses the join (checkJoinNetwork).
PodCIDR string
ServiceCIDR string
// ClusterDomain is spec.network.clusterDomain, "" when unset. It cannot
// override the cluster's domain, which rides in with the shared certs: a
// different one refuses the join (checkJoinDomain).
ClusterDomain string
// Cilium is this control plane's spec.network.cilium. It must be the
// cluster's, which rides in with the shared certs: a different one refuses
// the join (checkJoinCilium).
Cilium addons.CiliumSettings
}
JoinOptions for `stratum server --server --token --certificate-key`.
type NodeAssembly ¶
type NodeAssembly struct {
// HostDirs creates the directory layout (host.Dirs).
HostDirs func() error
// Prereqs checks the host prerequisites for the settings in use
// (host.Prereqs).
Prereqs func(host.PrereqOptions) error
// ResolveNet resolves the cluster's network for this node: the
// controller's netconfig resolution, the join's shared-cert download and
// cluster gates. Every one of them can refuse.
ResolveNet func() (netcfg.Config, error)
// TrustPodCIDRs opens the host firewall for the pod interfaces
// (host.TrustPodCIDRs).
TrustPodCIDRs func(podCIDR string)
// ResolveNodeIP picks this node's address, falling back to detection
// (host.ResolveNodeIP, host.DetectNodeIPs).
ResolveNodeIP func(flag, config, state string, detect func() (string, error)) (host.NodeIPChoice, error)
DetectNodeIPs func(dualStack bool) (string, error)
// BeforeCerts runs between the resolved node IP and the certificates: the
// controller's IP-move refusal and node-state completion, the join's
// node-IP capture.
BeforeCerts func(choice host.NodeIPChoice) error
// IssueCerts mints this node's leaves from the shared CAs (pki.GenerateAll).
IssueCerts func(host, nodeIP string, sans *pki.APISANs) error
// WriteKubeconfigs writes the node's kubeconfigs (kubeconfig.GenerateAll).
WriteKubeconfigs func(host, server string, vip bool) error
// WriteProfileConfigs writes the config files the node's profile declares
// and its apiserver flags reference (profilefiles.Write).
// Runs only for a node with a profile, and before everything that
// follows: it can refuse.
WriteProfileConfigs func() error
// BeforeRuntimeConfigs runs between the profile step and the runtime configs.
// Nil runs nothing there; the join puts its retained-data re-check, etcd
// probe, and intent reconciliation in it, so they still refuse before
// anything that follows.
BeforeRuntimeConfigs func() error
// WriteRuntimeConfigs writes the kubelet and containerd configs
// (runtime.PrepareNodeConfigs) on the request's node state.
WriteRuntimeConfigs func(st *nodestate.NodeState) error
}
NodeAssembly is the shared node assembly's effects, as one struct parameter in the style of joinEffects and tokenSteps (internal/token/token.go): the safety order — every step that can refuse returns before the steps that follow — is otherwise held only by statement order, which no test can observe without touching the host. A struct rather than package variables keeps the tests that use a recording set safe to run in parallel.
ResolveNet hands the assembly the cluster's network for this node; its CIDRs and domain feed the pod-CIDR trust, the address detection, and the apiserver SANs. BeforeCerts receives the resolved node IP choice before any certificate is issued, so a caller can refuse a move or pin the address into its node state first.
Source Files