Documentation
¶
Overview ¶
Package kubernetes runs apps on a Kubernetes cluster (issue #72, PR 6, O-33).
Design: docs/design/notes-kubernetes-runtime-issue-72.md. In short:
- One namespace per app, "pando-<app id>", with a default-deny NetworkPolicy that admits only Pando's server pods and the app's own pods (R-023, R-025). Nothing in it is reachable from outside the cluster: no NodePort, LoadBalancer, Ingress, hostPort or hostNetwork (R-026).
- A headless Service per workload, so the workload's name resolves inside the namespace as it does on a Docker network, and so Pando's proxy can reach it by a name Upstream computes without asking the API.
- Bare pods with restartPolicy Never, owned by no controller. Nothing but Pando's reconciler starts a workload again, so a failed app stays failed (R-151).
- A PersistentVolumeClaim per volume, its PersistentVolume patched to Retain, kept when the app is destroyed unless the volumes are asked for too (R-204).
- The edge (R-174) in its own namespace, as a Deployment of at least two replicas with a disruption budget, behind the only LoadBalancer or NodePort Service Pando creates.
Core never learns any of this vocabulary (R-251): it sees a runtime adapter whose capabilities say what it can do (R-254).
Index ¶
- Constants
- func Info() api.KindInfo
- type Adapter
- func (a *Adapter) Apply(ctx context.Context, p api.BundlePlan) (api.BundleHandle, error)
- func (a *Adapter) ApplyEdge(ctx context.Context, p api.EdgePlan) error
- func (a *Adapter) Capabilities(ctx context.Context) (api.RuntimeCapabilities, error)
- func (a *Adapter) Capacity(ctx context.Context) (api.Capacity, error)
- func (a *Adapter) Category() api.Category
- func (a *Adapter) Configure(_ context.Context, raw json.RawMessage) error
- func (a *Adapter) CreateVolume(ctx context.Context, req api.VolumeRequest) (api.VolumeHandle, error)
- func (a *Adapter) Destroy(ctx context.Context, ref api.BundleRef, opts api.DestroyOptions) error
- func (a *Adapter) DestroyVolume(ctx context.Context, h api.VolumeHandle) error
- func (a *Adapter) EdgeVolumes(context.Context) ([]api.VolumeHandle, error)
- func (a *Adapter) Edges(ctx context.Context) ([]string, error)
- func (a *Adapter) Exec(ctx context.Context, ref api.WorkloadRef, req api.ExecRequest) (api.ExecSession, error)
- func (a *Adapter) HasImage(context.Context, string) (bool, error)
- func (a *Adapter) HealthCheck(ctx context.Context) error
- func (a *Adapter) ImportImage(context.Context, io.Reader) (string, error)
- func (a *Adapter) InUse(ctx context.Context) (api.InUse, error)
- func (a *Adapter) Kind() string
- func (a *Adapter) LargestFitFor(ctx context.Context, bundleID string) (*api.Fit, error)
- func (a *Adapter) Logs(ctx context.Context, ref api.WorkloadRef, opts api.LogOptions) (io.ReadCloser, error)
- func (a *Adapter) Observe(ctx context.Context, ref api.BundleRef) (api.ObservedBundle, error)
- func (a *Adapter) ObserveEdge(ctx context.Context, name string) (api.EdgeState, error)
- func (a *Adapter) RemoveEdge(ctx context.Context, name string) error
- func (a *Adapter) RestoreVolume(ctx context.Context, h api.VolumeHandle, src io.Reader) error
- func (a *Adapter) SnapshotVolume(ctx context.Context, h api.VolumeHandle, dst io.Writer) error
- func (a *Adapter) Stop(ctx context.Context, ref api.BundleRef) error
- func (a *Adapter) Trial(ctx context.Context, req api.TrialRequest) (api.TrialResult, error)
- func (a *Adapter) Upstream(_ context.Context, ref api.WorkloadRef, port int) (api.Upstream, error)
- func (a *Adapter) Usage(ctx context.Context, ref api.BundleRef) (api.BundleUsage, error)
- func (a *Adapter) WatchBundles(ctx context.Context, sink func(api.BundleEvent)) error
- type Config
Constants ¶
const Kind = "kubernetes"
Kind is the adapter's kind string.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Adapter ¶
type Adapter struct {
// contains filtered or unexported fields
}
Adapter runs workloads as pods on a Kubernetes cluster.
func (*Adapter) Apply ¶
func (a *Adapter) Apply(ctx context.Context, p api.BundlePlan) (api.BundleHandle, error)
Apply converges an app's namespace toward the plan.
Idempotent: each object is compared with what exists, and a workload's pod is recreated only when its digest differs or it has exited. Calling Apply with a satisfied plan writes nothing but the pull Secret, which is rewritten on every call because a registry password can expire (O-41).
func (*Adapter) Capabilities ¶
Capabilities reports what this adapter can do (R-254).
func (*Adapter) Capacity ¶
Capacity is adapter-reported (R-243): what the eligible nodes can hold, less what pods outside Pando's namespaces already ask for on them, since a cluster also runs things that are not Pando's. The planner takes Pando's own allocations off the totals itself.
LargestFit is the one node with the most room left once every pod's requests are counted, Pando's too, since the planner compares it as it is (R-242).
func (*Adapter) CreateVolume ¶
func (a *Adapter) CreateVolume(ctx context.Context, req api.VolumeRequest) (api.VolumeHandle, error)
CreateVolume makes a claim in the app's namespace.
func (*Adapter) Destroy ¶
Destroy removes an app.
With KeepVolumes, the default (R-204), the pods, Services, Secrets and ConfigMaps go and the namespace stays with its claims, its default-deny policy and Pando's role binding: a claim lives in a namespace. Without it, each volume goes as DestroyVolume removes one, and then the namespace.
func (*Adapter) DestroyVolume ¶
DestroyVolume deletes a volume and its data. The volume's reclaim policy is set back to Delete first: deleting a retained volume's object would leave the disk behind it, which is not what destroying a volume means.
func (*Adapter) EdgeVolumes ¶
EdgeVolumes: none. Edge replicas share nothing; certificates on this runtime are kept by Pando, not by the edge.
func (*Adapter) Exec ¶
func (a *Adapter) Exec(ctx context.Context, ref api.WorkloadRef, req api.ExecRequest) (api.ExecSession, error)
Exec opens a session in a workload's running pod. Core has already authorized it and written the audit event (R-084, R-228).
func (*Adapter) HasImage ¶ added in v0.5.0
ImportImage is not offered: every node pulls from a registry. HasImage is true: this runtime runs images from a registry, which is where they stay, and imports none of its own.
func (*Adapter) HealthCheck ¶
HealthCheck says whether the cluster can be used: the API answers, the configured RuntimeClass exists, and the network plugin enforces NetworkPolicy, which every isolation guarantee here rests on (O-43).
func (*Adapter) ImportImage ¶
func (*Adapter) LargestFitFor ¶
LargestFitFor is the roomiest node for one bundle: what each node has left, plus what the bundle's own pods ask for there, since a redeploy replaces them, so a redeploy that fits where the app runs is never refused (R-242). Nil when the cluster has no eligible node, and the totals check refuses.
func (*Adapter) Logs ¶
func (a *Adapter) Logs(ctx context.Context, ref api.WorkloadRef, opts api.LogOptions) (io.ReadCloser, error)
Logs streams a workload's newest pod's log, a finished one included, so the crash that sent an app to failed can still be read.
func (*Adapter) Observe ¶
Observe reports what exists. It never remediates (design 05 §2.1): a pod that exited is reported exited, and the reconciler decides.
func (*Adapter) ObserveEdge ¶
ObserveEdge reports whether an edge is serving, in words an operator can act on when it is not, and warns when its replicas share a node.
func (*Adapter) RemoveEdge ¶
RemoveEdge removes an edge and everything made for it. The proxy alias goes with the last edge.
func (*Adapter) RestoreVolume ¶
RestoreVolume replaces a volume's contents with a tar stream SnapshotVolume wrote. In place only (R-206): the claim is found by the app's identity in the handle, so it is the app recreated from the same spec.
func (*Adapter) SnapshotVolume ¶
SnapshotVolume writes a volume's contents as a tar stream, from a helper pod with the claim mounted read-only.
func (*Adapter) Stop ¶
Stop removes the app's running pods and keeps everything else: its Services, its volumes, its configuration, and any pod that has already finished. A finished pod uses nothing and is how the last crash's output stays readable: the reconciler stops an app as it gives up on it (R-150), and deleting every pod then took the log of the crash that sent it to failed with it — on Docker the stopped container keeps its log. A finished pod cannot be started again, so a stopped app is started by Apply creating new pods.
func (*Adapter) Trial ¶
func (a *Adapter) Trial(ctx context.Context, req api.TrialRequest) (api.TrialResult, error)
Trial starts a workload once in a namespace of its own and reports what it did (R-097). An observer container in the same pod shares its network namespace — what the Docker adapter's sidecar joins — and reads the listening sockets from /proc/net/tcp, so any image is observed, one with no shell included. Writes are not observed: nothing like a container diff exists in the API (SupportsWriteObservation is false).
The namespace denies every connection in, Pando's included: nothing needs to reach an app that is being watched. Where the rules the app would run with restrict anything, the trial is held to them as the app is (R-097, R-187): it reaches cluster DNS and its own namespace, and the outside only through an egress gateway in that namespace. It is removed at the end, on every path.
func (*Adapter) Upstream ¶
Upstream is the workload's Service name inside the cluster (R-023).
Computed, not looked up: it runs on every proxied request. A name rather than an address, so it survives the pod being recreated. Only Pando's server pods may connect to it (ensurePolicies), and nothing outside the cluster can resolve it (R-026).
func (*Adapter) WatchBundles ¶
WatchBundles watches every pod that is an app's workload, across the cluster, and reports what changed (O-52). It reports and never acts (R-148): with restartPolicy Never an exited pod stays exited until the reconciler, under backoff, makes another (R-151).
One watch for the cluster rather than one per app namespace: Pando's role already reads pods cluster-wide for capacity (deploy/kubernetes/rbac.yaml), and thousands of watches would be thousands of open streams. The pods are listed first and the watch starts from that list's version, so only changes after the start are reported; the caller looks at every app when Watching arrives. A watch the API server closes on its own timeout is reopened from the last version seen without returning; a watch that cannot be resumed — the version has expired — returns, and the caller lists and sweeps again.
type Config ¶
type Config struct {
// Kubeconfig is a kubeconfig file. Empty is the in-cluster configuration,
// which is how Pando runs: inside the cluster it deploys to, because its
// proxy reaches a workload by a cluster DNS name.
Kubeconfig string `json:"kubeconfig,omitempty"`
Context string `json:"context,omitempty"`
// PandoNamespace is where Pando's server pods run, and PandoService the
// Service in front of them. An app's namespace admits connections from
// pods in PandoNamespace carrying Pando's server labels, and nothing else
// from outside itself (R-023).
PandoNamespace string `json:"pando_namespace,omitempty"`
PandoService string `json:"pando_service,omitempty"`
ServiceAccount string `json:"service_account,omitempty"`
// ProxyPort is the port Pando's proxy listens on in its pods, which the
// edge is allowed to reach.
ProxyPort int `json:"proxy_port,omitempty"`
// AppRole is the ClusterRole bound to Pando's ServiceAccount inside each
// app namespace it creates (deploy/kubernetes/rbac.yaml).
AppRole string `json:"app_role,omitempty"`
ClusterDomain string `json:"cluster_domain,omitempty"`
// PodCIDR and ServiceCIDR are the cluster's address ranges. An app may
// open connections out of the cluster, never into these: the API does not
// report them reliably, so they are settings.
PodCIDR string `json:"pod_cidr"`
ServiceCIDR string `json:"service_cidr"`
// APIServerCIDR is the API server's address, which only the edge may
// reach, when it falls inside an excluded range. Empty when it does not.
APIServerCIDR string `json:"api_server_cidr,omitempty"`
// StorageClass makes volumes; empty is the cluster's default class.
StorageClass string `json:"storage_class,omitempty"`
// VolumeSizeBytes is a volume's size when the request names none.
VolumeSizeBytes int64 `json:"volume_size_bytes,omitempty"`
// RuntimeClass runs app pods under a RuntimeClass, which decides the
// isolation class reported (R-114, R-115): a handler of gVisor (runsc) or
// Kata is `sandboxed`.
RuntimeClass string `json:"runtime_class,omitempty"`
// NodeSelector limits app pods, and the capacity reported, to nodes with
// these labels, as "key=value,key=value".
NodeSelector string `json:"node_selector,omitempty"`
// EgressGatewayImage is an image with Pando's binary at
// /usr/local/bin/pando, run as a restricted app's egress gateway (R-187).
// Empty means egress cannot be restricted on this runtime (R-186).
EgressGatewayImage string `json:"egress_gateway_image,omitempty"`
// HelperImage runs the NetworkPolicy canary, the trial run's port
// observer and volume backups: any image with a POSIX shell, tar, wget
// and httpd, such as busybox.
HelperImage string `json:"helper_image,omitempty"`
// The edge (edge.go).
EdgeNamespace string `json:"edge_namespace,omitempty"`
EdgeReplicas int `json:"edge_replicas,omitempty"`
EdgeServiceType string `json:"edge_service_type,omitempty"`
EdgeHTTPNodePort int `json:"edge_http_node_port,omitempty"`
EdgeHTTPSNodePort int `json:"edge_https_node_port,omitempty"`
// APIQPS and APIBurst limit the requests a second this adapter makes to
// the cluster's API, and how many it may make at once above that. Unset,
// client-go allows 5 and 10, which observes about 18 apps in a
// 15-second pass (notes-kubernetes-scale-issue-72.md).
APIQPS int `json:"api_qps,omitempty"`
APIBurst int `json:"api_burst,omitempty"`
}
Config is the adapter's configuration.
Source Files