server

package
v0.1.0-alpha.41 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 22, 2026 License: Apache-2.0 Imports: 59 Imported by: 0

Documentation

Overview

Package server implements the control plane: the gRPC API for the CLI and agents, and the reconciler that turns desired state into assignments.

Index

Constants

View Source
const DefaultNodeLostTimeout = 5 * time.Minute

DefaultNodeLostTimeout is how long a node may be disconnected before its replicas are rescheduled elsewhere.

View Source
const (
	MinNodeLostTimeout = 10 * time.Second
)

Bounds of the settings.

View Source
const TXTPrefix = "_armada-verify."

TXTPrefix is the label under which domain owners publish the verification token: TXT _armada-verify.<domain> = <token>.

Variables

This section is empty.

Functions

func CheckACMECA

func CheckACMECA(v string) error

CheckACMECA accepts letsencrypt, letsencrypt-staging, or an https:// URL.

func CheckDNSProvider

func CheckDNSProvider(v string) error

CheckDNSProvider accepts the providers Armada supports, or none.

func CheckDNSToken

func CheckDNSToken(token []byte) error

CheckDNSToken bounds a DNS token and refuses control characters.

func CheckEmail

func CheckEmail(v string) error

CheckEmail bounds the ACME email; ACME checks the rest.

func CheckLogMaxBytes

func CheckLogMaxBytes(n int64) error

CheckLogMaxBytes validates the log history size per node; 0 means the default.

func CheckLogRetention

func CheckLogRetention(d time.Duration) error

CheckLogRetention validates a log retention; 0 means the default.

func CheckNodeLostTimeout

func CheckNodeLostTimeout(d time.Duration) error

CheckNodeLostTimeout bounds the node-lost timeout.

func IdentityFrom

func IdentityFrom(ctx context.Context) (pki.Identity, bool)

IdentityFrom returns the authenticated caller stored by the interceptors.

func SaveSettings

func SaveSettings(ctx context.Context, st store.Store, s Settings) error

SaveSettings stores the cluster's settings.

Types

type Cluster

type Cluster interface {
	Members(ctx context.Context) ([]etcd.MemberInfo, error)
	// AddLearner adds a non-voting member and returns its ID and the
	// initial cluster for its first start.
	AddLearner(ctx context.Context, name string, addr netip.Addr, peerPort int) (uint64, string, error)
	// Promote turns caught-up learners into voting members.
	Promote(ctx context.Context) ([]string, error)
	// Reachable reports whether a member answers.
	Reachable(ctx context.Context, m etcd.MemberInfo) bool
	RemoveMember(ctx context.Context, id uint64) error
}

Cluster manages the etcd membership of the control plane nodes.

type Config

type Config struct {
	Store  store.Store
	TLS    *tls.Config
	Logger *slog.Logger
	// Resync is how often the reconciler runs without a trigger.
	Resync time.Duration
	// StreamRecheck is how often a user's or service's open AdminService
	// stream checks its access again (default streamRecheckEvery).
	StreamRecheck time.Duration

	// Certs manages ACME certificates. Nil disables TLS certificate
	// management (HAProxy then only serves its self-signed default).
	Certs certs.Manager
	// PlainHTTP is set when the cluster obtains no certificates: every
	// node's proxy serves the apps over plain HTTP on port 80 too, and
	// their URLs start with http://.
	PlainHTTP bool
	// ACMEAddr is the private ip:port where the server answers HTTP-01
	// challenges forwarded by every node's HAProxy. Required with Certs.
	ACMEAddr  string
	ACMEEmail string
	// ACMECA and DNSProvider are what the certificate manager uses, shown
	// by GetClusterConfig.
	ACMECA      string
	DNSProvider string
	// OnSettingsChanged is called after the cluster's server settings
	// changed (ADR 0031): the daemon restarts this term to apply them.
	OnSettingsChanged func()
	// Resolver looks up domain verification TXT records (default: system).
	Resolver       TXTResolver
	VerifyInterval time.Duration

	// Rollout timing; zero values use the defaults. DrainTimeout is how
	// long a replica keeps running after leaving the proxies, Warmup how
	// long a new replica must serve before an old one is drained, and
	// ProgressDeadline how long a rollout may go without a new replica
	// becoming ready (plus the time its probes may legitimately need).
	DrainTimeout     time.Duration
	Warmup           time.Duration
	ProgressDeadline time.Duration
	// NodeLostTimeout is how long a node may be disconnected before its
	// replicas are rescheduled on other nodes (default 5 minutes).
	NodeLostTimeout time.Duration
	// AutoscaleInterval is how often autoscaled apps are evaluated (ADR
	// 0020). Default 15s; tests shorten it.
	AutoscaleInterval time.Duration
	// BackupInterval is how often volume backups are checked for being due
	// (ADR 0041). Default 30s; tests shorten it.
	BackupInterval time.Duration
	// ControlPlaneBackup writes the files of a backup of this server's
	// control plane for the scheduled backups to a bucket (ADR 0051). Nil
	// disables them. ControlPlaneBuckets opens their bucket (tests; nil:
	// S3).
	ControlPlaneBackup  ControlPlaneBackup
	ControlPlaneBuckets func(volbackup.Target) (volbackup.Objects, error)
	// JobRetryBase is the delay before a job's first retry (ADR 0019); it
	// doubles with each failed attempt. Default 10s; tests shorten it.
	JobRetryBase time.Duration
	// Now returns the current time (default time.Now); tests replace it.
	Now func() time.Time

	// Secrets holds the master key that encrypts secrets at rest. Nil
	// disables secrets.
	Secrets *secrets.Keyring

	// CA signs node certificates for joining and renewing nodes. Nil
	// disables both.
	CA *pki.CA
	// JoinAddr is the private ip:port of the join listener, which serves
	// only JoinService with JoinTLS (no client certificates). Empty
	// disables joining.
	JoinAddr string
	// JoinListener, if set, is the join listener already listening on
	// JoinAddr (tests: no window in which another test takes the port).
	JoinListener net.Listener
	JoinTLS      *tls.Config
	// JoinPublic is a mesh cluster's join listener (ADR 0049): JoinAddr is
	// the leader's public endpoint, and the listener is open only while a
	// join token is open, answering only the addresses tokens name.
	JoinPublic bool
	// JoinSiteAddr is the leader's public endpoint and join port in a plain
	// or wireguard cluster (ADR 0050): a second join listener, open only
	// while a token for another site than the leader's is, answering only
	// the addresses those tokens name. Empty when the leader has no
	// endpoint. JoinSiteListener, if set, already listens for it (tests).
	JoinSiteAddr     string
	JoinSiteListener net.Listener
	// APIAddr is the control plane address given to joining agents.
	APIAddr string
	// NodeCertValidity is the lifetime of issued node certificates
	// (default pki.NodeValidity).
	NodeCertValidity time.Duration

	// MetricsAddr is the private ip:port of the metrics endpoint (ADR 0017),
	// served with TLS and only to metrics certificates and cluster-admin. Empty
	// disables it.
	MetricsAddr string
	// MetricsCache is how long metrics gathered for one scrape answer the
	// next ones (default 10s, metricsCacheFor).
	MetricsCache time.Duration

	// Name is this control plane node's name.
	Name string
	// JoinIdle closes idle connections to the join listener sooner than
	// its default of 15 seconds (tests).
	JoinIdle time.Duration
	// AlertHTTP, if set, sends notices to HTTP alert channels (tests;
	// ADR 0045).
	AlertHTTP *http.Client
	// Cluster manages the control plane's etcd membership. Nil disables
	// adding and removing servers.
	Cluster Cluster
}

Config configures a Server.

type ControlPlaneBackup

type ControlPlaneBackup func(ctx context.Context, fn func(dir string) error) error

ControlPlaneBackup writes the files of a backup of this server's control plane (what `armadad backup` writes) into a directory and calls fn with it, removing the directory afterwards.

type Server

type Server struct {
	armadav1.UnimplementedAdminServiceServer
	armadav1.UnimplementedAgentServiceServer
	// contains filtered or unexported fields
}

Server is the Armada control plane.

func New

func New(cfg Config) (*Server, error)

New builds a Server. TLS must require and verify client certificates.

func (*Server) BackupVolume

BackupVolume streams a gzipped tar of a volume from its node.

func (*Server) CancelJob

CancelJob ends a job's active runs.

func (*Server) CertificateEvent

func (s *Server) CertificateEvent(name string, err error)

CertificateEvent records that a certificate was obtained (err nil) or that obtaining it failed.

func (*Server) ClusterStatus

ClusterStatus summarizes the cluster: counts, and the apps and nodes that need attention, bounded however large the cluster is.

func (*Server) Connect

func (s *Server) Connect(stream armadav1.AgentService_ConnectServer) error

Connect handles an agent's control stream.

func (*Server) CordonNode

CordonNode marks a node (un)schedulable; replicas on a cordoned node are moved elsewhere.

func (*Server) CreateAlertChannel

func (s *Server) CreateAlertChannel(ctx context.Context, req *armadav1.AlertChannelRequest) (*armadav1.AlertChannel, error)

CreateAlertChannel adds a channel (cluster-admin only).

func (*Server) CreateAssignment

func (s *Server) CreateAssignment(ctx context.Context, req *armadav1.AssignmentRequest) (*armadav1.Principal, error)

CreateAssignment gives a principal a role.

func (*Server) CreateJoinToken

CreateJoinToken creates a single-use join token.

func (*Server) CreateService

func (s *Server) CreateService(ctx context.Context, req *armadav1.CreateServiceRequest) (*armadav1.Principal, error)

CreateService records a service with its first roles.

func (*Server) CreateSpace

func (s *Server) CreateSpace(ctx context.Context, req *armadav1.CreateSpaceRequest) (*armadav1.SpaceInfo, error)

CreateSpace records a new space (ADR 0026).

func (*Server) CreateUser

CreateUser records a user with their first roles and signs a certificate for the key in the CSR.

func (*Server) CreateVolumeSnapshot

CreateVolumeSnapshot backs a volume up now.

func (*Server) DeleteAlertChannel

DeleteAlertChannel removes a channel.

func (*Server) DeleteAssignment

func (s *Server) DeleteAssignment(ctx context.Context, req *armadav1.AssignmentRequest) (*armadav1.Principal, error)

DeleteAssignment takes a role back. A caller keeps its own cluster-admin.

func (*Server) DeleteJob

DeleteJob removes a job with its runs; their containers are removed.

func (*Server) DeletePrincipal

DeletePrincipal revokes every certificate of a user or service. Callers cannot delete themselves, so a cluster-admin is always left.

func (*Server) DeleteSecret

DeleteSecret removes every version of a key no app uses, or with whole every key of the secret.

func (*Server) DeleteSpace

DeleteSpace removes an empty space.

func (*Server) Deploy

Deploy validates a spec and records it as the app's active revision.

func (*Server) Events

Events streams events, newest last (ADR 0018).

func (*Server) Exec

func (s *Server) Exec(stream armadav1.AdminService_ExecServer) error

Exec runs a command in one replica of an app (ADR 0015): it picks the replica, asks its node's agent to open a session, and relays between the admin's stream and the agent's. The command's arguments are never logged or audited, only its name.

func (*Server) ExecSession

func (s *Server) ExecSession(stream armadav1.AgentService_ExecSessionServer) error

ExecSession receives the session stream for a pending exec request from the agent it was sent to, and holds it until the admin side is done.

func (*Server) GetAppSpec

GetAppSpec returns the spec an app or job runs (its target revision) as a file holds it, ready to import elsewhere (ADR 0032).

func (*Server) GetCertificates

GetCertificates hands an agent the proxy certificates its assignment names by reference. Only node certificates reach it (AgentService), and only names the last proxy configuration holds are answered: every node serves every one of them.

func (*Server) GetClusterConfig

GetClusterConfig returns cluster-wide settings.

func (*Server) IssueMetricsCertificate

IssueMetricsCertificate signs a metrics client certificate from the caller's CSR (ADR 0017) and records it, so it can be revoked. The key never reaches the server, and the certificate can only scrape metrics: the API refuses its identity.

func (*Server) IssueServiceCertificate

IssueServiceCertificate signs a certificate for an existing service from the tool's CSR.

func (*Server) IssueUserCertificate

IssueUserCertificate gives a user a new certificate; the ones issued earlier stop working.

func (*Server) LabelNode

LabelNode sets and removes a node's admin labels (ADR 0014). Replicas whose node no longer matches their app's placement are moved.

func (*Server) ListAlertChannels

ListAlertChannels lists the channels with how delivery went since the leader started.

func (*Server) ListApps

ListApps lists apps one summary each, in one space or every space the caller sees, a page at a time.

func (*Server) ListDomains

ListDomains returns custom domains with what their owners must configure.

func (*Server) ListJobRuns

ListJobRuns lists a job's runs, newest first.

func (*Server) ListJobs

ListJobs describes every job.

func (*Server) ListMetricsCredentials

ListMetricsCredentials lists the metrics scrapers with a certificate on record that has not expired.

func (*Server) ListPrincipals

ListPrincipals lists users and services with their roles.

func (*Server) ListRevisions

ListRevisions lists an app's retained revisions, newest first.

func (*Server) ListRoles

ListRoles lists the built-in roles and the methods each allows.

func (*Server) ListSecrets

ListSecrets lists a secret's keys without their values.

func (*Server) ListServers

ListServers lists the control plane nodes.

func (*Server) ListSpaceSecrets

ListSpaceSecrets lists the secrets of a space without their values.

func (*Server) ListSpaces

ListSpaces lists every space with the number of apps and jobs in it.

func (*Server) ListVolumeSnapshots

ListVolumeSnapshots lists the backups in a volume's bucket.

func (*Server) ListVolumes

ListVolumes lists volumes, where they are, and how much they hold.

func (*Server) Logs

Logs streams logs from an app's replicas by asking each replica's agent to push them.

func (*Server) PushLogs

func (s *Server) PushLogs(stream armadav1.AgentService_PushLogsServer) error

PushLogs receives a log stream for a pending request from the agent that was asked for it.

func (*Server) Reconcile

func (s *Server) Reconcile(ctx context.Context) error

Reconcile brings scheduled replicas in line with each app's desired state and pushes per-node assignments to connected agents.

func (*Server) RemoveApp

RemoveApp starts removing an app. The reconciler drains its replicas and then removes it, so requests in progress finish as in any other rollout.

func (*Server) RemoveNode

RemoveNode deletes a node, revokes its certificates, and disconnects its agent.

func (*Server) RemoveServer

RemoveServer removes a control plane node from the etcd cluster and forgets it. Its agent stays a node until `armada node delete`.

func (*Server) RemoveVolume

RemoveVolume deletes a volume's data on its node, then its record. A volume the app's spec still mounts is refused. When its node was removed from the cluster only the record goes.

func (*Server) RenewCertificate

RenewCertificate issues a new certificate for the calling node.

func (*Server) RenewOwnCertificate

RenewOwnCertificate gives the calling user or service a new certificate. A service's other certificates are revoked, as a node's are; a user's stay valid until they expire or `user credentials` replaces them. User renewals are rate limited per name, and recorded in the same transaction that checks the caller's certificate is still accepted.

func (*Server) RestartApp

RestartApp makes a new revision of an app's spec with the latest secret values and rolls it out (ADR 0052): the way an app picks up a changed secret, or starts every replica afresh.

func (*Server) RestoreVolume

func (s *Server) RestoreVolume(stream armadav1.AdminService_RestoreVolumeServer) error

RestoreVolume replaces a volume's contents with a gzipped tar from the admin, on its node or, with start.node, on another node it is then bound to. The agent stops the app's container on the node while it restores.

func (*Server) RestoreVolumeSnapshot

RestoreVolumeSnapshot replaces a volume's contents with a backup from its bucket, on its node or, while that node is not connected, on another.

func (*Server) RestrictNode

RestrictNode makes a node restricted (only apps whose placement selects one of its labels) or open.

func (*Server) RevokeMetricsCredentials

RevokeMetricsCredentials refuses every certificate of a metrics scraper from now on, including ones issued before they were recorded.

func (*Server) Rollback

Rollback rolls an app out to an earlier revision. Without a revision it cancels a rollout in progress, or else returns to the revision before the active one.

func (*Server) RotateNodeKey

RotateNodeKey asks a connected node's agent for a new WireGuard key (ADR 0025); the agent announces it, and every peer list follows.

func (*Server) RunJob

RunJob starts a run of a job now. With concurrency forbid it refuses while a run is active; with replace it cancels that run.

func (*Server) Serve

func (s *Server) Serve(ctx context.Context, lis net.Listener) error

Serve accepts connections on lis and runs the reconciler until ctx is cancelled. With several control plane nodes only the leader serves, for as long as its term lasts.

func (*Server) SetClusterConfig

SetClusterConfig updates the cluster-wide settings the request names: the root domain and the trusted proxies apply at once; the server settings (ADR 0031) are stored and the leader restarts its term to apply them.

func (*Server) SetNodeRelay

SetNodeRelay chooses a node as its site's relay between nodes behind NAT (ADR 0050), or drops the choice so the leader picks one itself.

func (*Server) SetNodeSite

SetNodeSite moves a node to another site (ADR 0050). The site decides how peers reach the node, so only admins set it; the node carries it as the armada.site label for placement.

func (*Server) SetSecret

SetSecret stores a new version of a secret's key, encrypted with the master key, and optionally rolls out every app that uses the key.

func (*Server) SetTraffic

SetTraffic changes how an app in multiple revision mode splits requests until its next deploy (ADR 0022).

func (*Server) Status

Status reports apps, replicas, and nodes. With an app it reports that app alone, with no nodes, and reads only what the app's answer needs: `armada app deploy --wait` asks every second.

func (*Server) TestAlertChannel

TestAlertChannel sends a test notice now and says how it went.

func (*Server) Top

Top reports what replicas use from the latest samples of their nodes (ADR 0016). A replica's usage only counts when it comes from the node the replica is assigned to.

func (*Server) Trigger

func (s *Server) Trigger()

Trigger requests a reconcile as soon as possible.

func (*Server) UpdateAlertChannel

func (s *Server) UpdateAlertChannel(ctx context.Context, req *armadav1.AlertChannelRequest) (*armadav1.AlertChannel, error)

UpdateAlertChannel changes the fields the request names.

func (*Server) UpdatePrincipal

func (s *Server) UpdatePrincipal(ctx context.Context, req *armadav1.UpdatePrincipalRequest) (*armadav1.Principal, error)

UpdatePrincipal replaces the node labels a principal's apps may require.

func (*Server) UpdateSpace

func (s *Server) UpdateSpace(ctx context.Context, req *armadav1.UpdateSpaceRequest) (*armadav1.SpaceInfo, error)

UpdateSpace changes the default space's network isolation (ADR 0026). cluster-admin only.

func (*Server) VerifyDomain

func (s *Server) VerifyDomain(ctx context.Context, req *armadav1.VerifyDomainRequest) (*armadav1.Domain, error)

VerifyDomain checks a domain's TXT record now.

func (*Server) VolumeSession

func (s *Server) VolumeSession(stream armadav1.AgentService_VolumeSessionServer) error

VolumeSession receives the session stream for a pending volume request from the agent it was sent to, and holds it until the admin side is done.

type Settings

type Settings struct {
	TLS         *bool  `json:"tls,omitempty"`
	ACMECA      string `json:"acmeCa,omitempty"`
	ACMEEmail   string `json:"acmeEmail,omitempty"`
	DNSProvider string `json:"dnsProvider,omitempty"`
	// DNSProviderSet records "none" chosen on purpose, which a flag must not
	// override.
	DNSProviderSet  bool            `json:"dnsProviderSet,omitempty"`
	DNSToken        *secrets.Sealed `json:"dnsToken,omitempty"`
	DNSTokenVersion int             `json:"dnsTokenVersion,omitempty"`
	MetricsPort     *int            `json:"metricsPort,omitempty"`
	// NodeLostSeconds is the node-lost timeout; zero means unset.
	NodeLostSeconds int64 `json:"nodeLostSeconds,omitempty"`
}

Settings are the server settings that must be the same on every server, because any of them can lead: certificates, the DNS provider and its token, the metrics port, and the node-lost timeout. Unset fields fall back to each server's flags, then the defaults.

func LoadSettings

func LoadSettings(ctx context.Context, st store.Store) (Settings, error)

LoadSettings reads the cluster's settings; none stored yields zero.

func (*Settings) SealToken

func (s *Settings) SealToken(keys *secrets.Keyring, token []byte) error

SealToken stores token as the DNS token, a new version of it.

func (Settings) Token

func (s Settings) Token(keys *secrets.Keyring) (v secrets.Value, ok bool, err error)

Token opens the DNS token; ok is false when none is stored.

type TXTResolver

type TXTResolver interface {
	LookupTXT(ctx context.Context, name string) ([]string, error)
}

TXTResolver looks up DNS TXT records.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL