Help us shape the future of Go! Take the 2026 Go Developer Survey

multidocker

package
v0.17.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 2, 2026 License: AGPL-3.0 Imports: 32 Imported by: 0

Documentation

Overview

Package multidocker runs apps on several Docker hosts (notes-multi-host-docker-issue-72.md, issue #72 PR 7).

It is a router in front of one single-host Docker adapter per host: every operation on an app runs the existing Docker adapter's code against the app's host. What it adds is placement (R-256), the forwarding agent each host runs so Pando's proxy can reach an app on another host without routing around itself (O-45, design 06 §4), and capacity summed over hosts.

Like every adapter it does not authorize, audit or touch Pando's state (R-027). Where an app runs is recorded on the hosts themselves: the app's network and volumes carry its bundle label, and the adapter reads them back.

Index

Constants

View Source 
const (
	EnvAgentCert        = "PANDO_AGENT_CERT"
	EnvAgentKey         = "PANDO_AGENT_KEY"
	EnvAgentAuthorities = "PANDO_AGENT_AUTHORITIES"
	EnvAgentPool        = "PANDO_AGENT_NETWORK_POOL"
	EnvAgentListen      = "PANDO_AGENT_LISTEN"

	// EnvAgentRuntimeSocket is the host's own container runtime socket, on a
	// host that joined a cluster: the agent then carries a control host's
	// runtime calls to it (R-416). This adapter sets it only on the agent it
	// keeps on the install's own machine (carryRuntime).
	EnvAgentRuntimeSocket = "PANDO_AGENT_RUNTIME_SOCKET"
)

Environment the agent reads its certificate, key and the authorities from. Set on its container by this adapter; the agent's key is on its own host only, and it is a server key: it cannot open a connection through any agent.

View Source 
const AgentContainer = agentContainer

AgentContainer is the agent's container name, and what a Docker adapter serving it is given as ProxyContainer.

View Source 
const Kind = "docker-hosts"

Kind is the adapter's kind string.

Variables

This section is empty.

Functions

func Info 

func Info() api.KindInfo

Info describes this kind of adapter for the forms that configure one (api.KindInfo, R-261).

Types

type Adapter 

type Adapter struct {
	// contains filtered or unexported fields
}

Adapter runs workloads on several Docker hosts.

func New 

func New() *Adapter

New builds an unconfigured adapter for the install's own machine.

func NewOn added in v0.17.0 

func NewOn(machine Machine) *Adapter

NewOn builds an unconfigured adapter for a process on machine: a control host that joined the cluster passes its name, so it reaches its own entry directly and every other machine, the install's own included, through that machine's agent (R-416).

func (*Adapter) Apply 

func (a *Adapter) Apply(ctx context.Context, p api.BundlePlan) (api.BundleHandle, error)

Apply places the bundle if it has no host (placement.go), makes sure the host's agent is running, and applies the plan there with the single-host adapter's code. The handle names the host.

func (*Adapter) ApplyEdge 

func (a *Adapter) ApplyEdge(ctx context.Context, p api.EdgePlan) error

The edge runs on each control host beside its Pando, as on one host.

func (*Adapter) ApplyStandby added in v0.17.0 

func (a *Adapter) ApplyStandby(ctx context.Context, req api.StandbyRequest) (api.StandbyHandle, error)

ApplyStandby creates or refreshes the bundle's stopped standby on a host other than its own (R-419).

func (*Adapter) Capabilities 

func (a *Adapter) Capabilities(ctx context.Context) (api.RuntimeCapabilities, error)

Capabilities are the control host's, with what several hosts change (R-254).

func (*Adapter) Capacity 

func (a *Adapter) Capacity(ctx context.Context) (api.Capacity, error)

Capacity sums the hosts open to new apps, with each host's figures in Details and the roomiest one's free space as LargestFit (R-243).

func (*Adapter) Category 

func (a *Adapter) Category() api.Category

func (*Adapter) Configure 

func (a *Adapter) Configure(_ context.Context, raw json.RawMessage) error

Configure reads the host list and the credentials, makes one Docker client per host and issues Pando's client certificate for the agents, in memory. Nothing is contacted: a host that is down at start is reported by HealthCheck and Observe, not by refusing to configure.

func (*Adapter) CreateVolume 

func (a *Adapter) CreateVolume(ctx context.Context, req api.VolumeRequest) (api.VolumeHandle, error)

CreateVolume makes storage on the bundle's host, placing the bundle first if it has none: the volume is then what keeps it there.

func (*Adapter) Destroy 

func (a *Adapter) Destroy(ctx context.Context, ref api.BundleRef, opts api.DestroyOptions) error

Destroy tears the bundle down on its host. With its volumes kept, the host stays the app's home (R-204): a re-created app returns to its data.

func (*Adapter) DestroyVolume 

func (a *Adapter) DestroyVolume(ctx context.Context, vh api.VolumeHandle) error

func (*Adapter) EdgeVolumes 

func (a *Adapter) EdgeVolumes(ctx context.Context) ([]api.VolumeHandle, error)

func (*Adapter) Edges 

func (a *Adapter) Edges(ctx context.Context) ([]string, error)

func (*Adapter) Exec 

func (a *Adapter) Exec(ctx context.Context, ref api.WorkloadRef, req api.ExecRequest) (api.ExecSession, error)

func (*Adapter) HasImage added in v0.5.0 

func (a *Adapter) HasImage(context.Context, string) (bool, error)

ImportImage is not offered (ImageDelivery is registry only). HasImage is true: this runtime runs images from a registry, which is where they stay, and imports none of its own.

func (*Adapter) HealthCheck 

func (a *Adapter) HealthCheck(ctx context.Context) error

HealthCheck asks every host. The adapter is healthy while this machine's Docker and at least one host open to new apps answer, and every answering host runs the same platform; a host that does not answer makes its own apps unobservable (Observe), not every app undeployable — the install's own machine included, seen from another control host.

func (*Adapter) ImportImage 

func (a *Adapter) ImportImage(context.Context, io.Reader) (string, error)

func (*Adapter) InUse 

func (a *Adapter) InUse(ctx context.Context) (api.InUse, error)

InUse sums every host, at once: each takes about a second.

func (*Adapter) Kind 

func (a *Adapter) Kind() string

func (*Adapter) LargestFitFor 

func (a *Adapter) LargestFitFor(ctx context.Context, bundleID string) (*api.Fit, error)

LargestFitFor is the roomiest place this bundle may go, for the planner (R-242 per host). A bundle already on a host may go only there, and what it holds there counts as free. A new one may go to any host open to new apps that answers. Nil when its host does not answer: Observe reports that, and the planner checks the totals alone.

func (*Adapter) Logs 

func (a *Adapter) Logs(ctx context.Context, ref api.WorkloadRef, opts api.LogOptions) (io.ReadCloser, error)

func (*Adapter) Observe 

func (a *Adapter) Observe(ctx context.Context, ref api.BundleRef) (api.ObservedBundle, error)

Observe reports the bundle on its host. An app whose host does not answer is an error, which the reconciler records as unobservable — never as failed (R-151, design 05 §2) — and never as absent, which would have the reconciler create it again on another host (O-46).

func (*Adapter) ObserveEdge 

func (a *Adapter) ObserveEdge(ctx context.Context, name string) (api.EdgeState, error)

func (*Adapter) PromoteStandby added in v0.17.0 

func (a *Adapter) PromoteStandby(ctx context.Context, req api.PromoteRequest) (api.PromoteResult, error)

PromoteStandby starts the standby as the app (R-419), from the newest copy of its volumes, falling back to older ones while it does not come up healthy (R-420).

Until the app's plan is first applied on the standby's host nothing has started, and an error is a refusal: the result names no host, and what was restored there is removed again. From then on the app is on that host — its home mark is written first, so a restarted process and a lost host that returns both find it there — and an error keeps the result's host (R-153). A host that already holds the bundle's home mark had the app started on it by a promotion cut short: no copy is restored over the volume it runs on, and the promotion is reported done. Once the app is healthy its staging volumes on the host are removed: their disk is counted against the host (R-242, R-422), and the app's next standby is elsewhere.

func (*Adapter) PullImage 

func (a *Adapter) PullImage(ctx context.Context, ref string) error

func (*Adapter) ReclaimNetworks 

func (a *Adapter) ReclaimNetworks(ctx context.Context, owns func(string) bool) (int, error)

ReclaimNetworks removes each host's app networks nothing is attached to.

func (*Adapter) RejoinNetworks 

func (a *Adapter) RejoinNetworks(ctx context.Context, owns func(string) bool) (int, error)

RejoinNetworks keeps each host's agent running and joined to that host's app networks, and refreshes the placement map. Called at start and every fifteen seconds on every replica (cmd/pando); Pando's own replicas join no app network on this runtime.

func (*Adapter) RemoveEdge 

func (a *Adapter) RemoveEdge(ctx context.Context, name string) error

func (*Adapter) RemoveHelpers 

func (a *Adapter) RemoveHelpers(ctx context.Context) error

func (*Adapter) RemoveStaleCopy added in v0.17.0 

func (a *Adapter) RemoveStaleCopy(ctx context.Context, ref api.BundleRef, hostName string, opts api.StaleCopyOptions) error

RemoveStaleCopy removes the bundle from host, which it was failed over from (R-419). The app's home is refused: that copy is the app. A deleted app has no home, and its copy keeps its volumes when the delete kept its storage (R-204).

func (*Adapter) RemoveStandby added in v0.17.0 

func (a *Adapter) RemoveStandby(ctx context.Context, ref api.BundleRef) error

RemoveStandby removes the bundle's standby from every host. A host that does not answer is reported, so the caller tries again.

func (*Adapter) RestoreVolume 

func (a *Adapter) RestoreVolume(ctx context.Context, vh api.VolumeHandle, src io.Reader) error

func (*Adapter) Run added in v0.17.0 

func (a *Adapter) Run(ctx context.Context, ref api.WorkloadRef, req api.RunRequest) error

Run runs a command in a workload on whichever host has its app (R-423).

func (*Adapter) Self 

func (a *Adapter) Self(ctx context.Context) (api.SelfWorkload, error)

Self-upgrade replaces Pando's container on this control host, as on one host (R-359); core allows it only with a single replica.

func (*Adapter) ShipVolumes added in v0.17.0 

func (a *Adapter) ShipVolumes(ctx context.Context, req api.ShipRequest) (api.ShipResult, error)

ShipVolumes sends one copy of the bundle's volumes to its standby.

func (*Adapter) SnapshotVolume 

func (a *Adapter) SnapshotVolume(ctx context.Context, vh api.VolumeHandle, dst io.Writer) error

func (*Adapter) StartHelper 

func (a *Adapter) StartHelper(ctx context.Context, spec api.HelperSpec) (string, error)

func (*Adapter) Stop 

func (a *Adapter) Stop(ctx context.Context, ref api.BundleRef) error

func (*Adapter) Trial 

func (a *Adapter) Trial(ctx context.Context, req api.TrialRequest) (api.TrialResult, error)

Trial runs on the control host asking for it [P]: it is thrown away, so it needs no placement.

func (*Adapter) Upstream 

func (a *Adapter) Upstream(ctx context.Context, ref api.WorkloadRef, port int) (api.Upstream, error)

Upstream is the workload's container name on its host, as one host's adapter answers, and a Dial through that host's agent (O-45). The proxy decides the request before it calls Dial; the agent only carries it.

Answered from the placement map, which is in memory: this runs on every proxied request.

func (*Adapter) Usage 

func (a *Adapter) Usage(ctx context.Context, ref api.BundleRef) (api.BundleUsage, error)

func (*Adapter) WatchBundles 

func (a *Adapter) WatchBundles(ctx context.Context, sink func(api.BundleEvent)) error

WatchBundles follows every host's event stream at once, one stream per host (O-52), and reports what happened without acting on it (R-148).

A host whose stream drops is reconnected here, with backoff, while every other host's stream carries on: one host restarting its daemon must not make the reconciler look at every app on every host. When a host's stream opens again after a gap, BundleEventMissed says that something on it may have been missed. BundleEventWatching is sent once every host's first attempt has opened or failed — a host that is down at the start is caught up with BundleEventMissed when it comes back. It returns only when ctx ends.

type AgentKeeper added in v0.5.0 

type AgentKeeper struct {
	// contains filtered or unexported fields
}

AgentKeeper keeps a forwarding agent running on one Docker API. The Incus runtime runs one in each app's VM (notes-incus-runtime.md), with this adapter's code, so the two keep one agent and one protocol.

func NewAgentKeeper added in v0.5.0 

func NewAgentKeeper(cli *client.Client, host string, port int, pool netip.Prefix, authorities hostagent.Authorities, image func(context.Context) string, settings string) *AgentKeeper

NewAgentKeeper keeps an agent on cli whose certificate names host, listening on port and forwarding only into pool. settings is where the operator sets agent_image and agent_port, as a remedy names it.

func (*AgentKeeper) Ensure added in v0.5.0 

func (a *AgentKeeper) Ensure(ctx context.Context) (bool, error)

Ensure makes sure the agent runs, and reports whether it had to create it.

type Config 

type Config struct {
	// Hosts are the Docker hosts, in order. Exactly one is the control host:
	// the one Pando's replicas, the edge, BuildKit and the registry run on
	// (O-47). A JSON list, or a string holding one, which is how a form's
	// text area sends it.
	Hosts json.RawMessage `json:"hosts"`

	// NetworkPool, NetworkBlockBits and OCIRuntime are the single-host Docker
	// adapter's settings of the same names, applied on every host. Each host
	// carves its own app networks from the same range: a bridge network
	// exists on one host only.
	NetworkPool      string `json:"network_pool,omitempty"`
	NetworkBlockBits int    `json:"network_block_bits,omitempty"`
	OCIRuntime       string `json:"oci_runtime,omitempty"`

	// PidsLimit is the single-host Docker adapter's setting of the same name,
	// applied on every host. Carried over when a single-host install becomes
	// a multi-host one (R-418).
	PidsLimit int64 `json:"pids_limit,omitempty"`

	// AgentImage is the image each host's forwarding agent and each
	// restricted app's egress gateway run: an image of Pando that every host
	// can pull. Empty is the image Pando's own container on the control host
	// was started from.
	AgentImage string `json:"agent_image,omitempty"`

	// AgentPort is the host port each agent publishes. Zero is 7443.
	AgentPort int `json:"agent_port,omitempty"`

	// Credentials arrive from the encrypted adapter_credentials store and
	// are held in memory only (R-190).
	Credentials Credentials `json:"credentials"`
}

Config is the adapter's configuration.

type Credentials 

type Credentials struct {
	// AgentAuthority is the agents' certificate authority as PEM, each
	// certificate followed by its key, the one that issues first. Generated
	// by `pando host-agent new-authority`. Pando issues its own client
	// certificate and each agent's server certificate from it.
	AgentAuthority string `json:"agent_authority"`

	// DockerTLS is the client certificate, its key and the CA that signed the
	// daemons' certificates, as PEM, for hosts reached over tcp://.
	DockerTLS string `json:"docker_tls,omitempty"`

	// SSHKey is the private key for hosts reached over ssh://.
	SSHKey string `json:"ssh_key,omitempty"`
}

Credentials are the adapter's secrets.

type HostConfig 

type HostConfig struct {
	// Name is how the host is known in placement, the console, errors and
	// the agent's certificate: lower-case letters, digits and dashes.
	Name string `json:"name"`

	// Endpoint is the Docker API: unix:///var/run/docker.sock (the control
	// host, usually), tcp://host:2376 with docker_tls, or ssh://user@host
	// with ssh_key, or agent://host:port for a host that joined the cluster,
	// whose runtime is reached only through its agent (R-416). Empty is the
	// environment, as the single-host adapter.
	Endpoint string `json:"endpoint,omitempty"`

	// CertSHA256 is the SHA-256 of the certificate the host joined the
	// cluster with, hex: its agent must present that one. Pando sets it from
	// the host's record.
	CertSHA256 string `json:"cert_sha256,omitempty"`

	// SSHHostKey is the host's SSH public key, as in known_hosts or
	// authorized_keys, for an ssh:// endpoint. Required: an unverified SSH
	// host key would hand the root-equivalent Docker credential to whoever
	// answers.
	SSHHostKey string `json:"ssh_host_key,omitempty"`

	// AgentAddress is where Pando's proxy reaches this host's agent, as
	// host:port. Empty is the endpoint's host and the agent port.
	AgentAddress string `json:"agent_address,omitempty"`

	// Control marks the install's own machine: the host the install started
	// on, whose agent the runtime keeps. In a cluster of several control
	// hosts, each one runs the edge, trial runs and self-upgrade on its own
	// machine (Machine), not on this one.
	Control bool `json:"control,omitempty"`

	// NoPlacement keeps new apps off the host; apps already on it stay
	// (R-256). It is how a host is emptied before it is removed.
	NoPlacement bool `json:"no_placement,omitempty"`

	// TotalCPUMillis and TotalMemoryBytes tell Pando how much of the host it
	// may use, as on one host. Empty asks the daemon.
	TotalCPUMillis   int   `json:"total_cpu_millis,omitempty"`
	TotalMemoryBytes int64 `json:"total_memory_bytes,omitempty"`
	TotalDiskBytes   int64 `json:"total_disk_bytes,omitempty"`
}

HostConfig is one Docker host.

type Machine added in v0.17.0 

type Machine struct {
	// Name is this machine's name in the host list. Empty is the install's
	// own machine: the entry marked control. A name the list does not hold
	// is a control host with no app role: no entry is this machine.
	Name string

	// Endpoint is this machine's own Docker API, used for its entry when the
	// list reaches it through its agent, and for the edge, trial runs and
	// self-upgrade when the list has no entry for it. Empty is the
	// environment.
	Endpoint string
}

Machine is which machine the process running the adapter is on. Every control host of a cluster reads the same host list (R-414), so the list cannot say which entry is "here": the process is told, by the server that starts it, and reaches its own entry's Docker directly and every other machine's through that machine's agent (R-416).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.