config

package
v0.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 23, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Index

Constants

View Source
const (
	NodeRoleServer  = "server"
	NodeRoleWorker  = "worker"
	NodeRoleIngress = "ingress"
	NodeRoleMixed   = "mixed"
)

NodeRole values for SB_NODE_ROLE. The role partitions which components a cluster-mode daemon runs (Raft voter promotion, sandbox worker work, public ingress reconciler). Default is NodeRoleMixed — every node runs every component, matching pre-role behavior bit-for-bit. The split exists so a 200-worker cluster doesn't have 200 Raft voters and 200 nodes each holding a 10K-route public ingress table; see plans/data-plane-load-balancer.md.

SB_NODE_ROLE accepts either a single token or a comma-separated combination of base roles, e.g. "worker,ingress" for a data-plane edge node or "server,ingress" for a control + ingress node. NodeRoleMixed remains the shorthand for "server,worker,ingress" and may not be combined with anything else. Runtime topology validation keeps mixed and hybrid-role nodes as a small-cluster convenience only; clusters above 10 live nodes must use dedicated server, worker, and ingress roles.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	PATToken            string
	APIHost             string
	APIPort             int
	Domain              string
	PublicHost          string
	CaddyAdminURL       string
	CaddyServerID       string
	DBPath              string
	DockerNetwork       string
	ToolboxBinaryPath   string
	ToolboxMountPath    string
	ToolboxPort         int
	IdleTimeoutMinutes  int
	ContainerPrivileged bool
	ResourceLimitsOff   bool
	// Runtime is the host default container runtime for new sandboxes.
	// Per-sandbox CreateSandboxRequest.Runtime overrides it. Allowed values
	// are "docker" (default), "gvisor", or "kata"; validation lives in Load().
	Runtime                     string
	AutoReconcile               bool
	EnableCaddy                 bool
	EnableNetworkRules          bool
	EnableEventMonitor          bool
	EnableSSHGateway            bool
	SSHListenAddr               string
	SSHHostKeyPath              string
	CredentialEncryptionKey     string
	CredentialEncryptionKeyPath string
	MountsRootPath              string
	MountsCredentialsRuntimeDir string
	MountWaitTimeout            time.Duration
	LogLevel                    string
	ShutdownTimeout             time.Duration
	HTTPClientTimeout           time.Duration
	DockerRuntimeWaitTimeout    time.Duration
	ToolboxWaitTimeout          time.Duration
	ReconcileInterval           time.Duration
	NetstatsPollInterval        time.Duration
	UploadMaxBytes              int64
	// OTELMetricsEnabled starts a native OTLP/HTTP metric exporter that bridges
	// the daemon's aerolvm_* expvars into OpenTelemetry observations. It is
	// also enabled automatically when SB_OTEL_METRICS_ENDPOINT is set.
	OTELMetricsEnabled  bool
	OTELMetricsEndpoint string
	OTELMetricsInterval time.Duration
	// OTELTracesEnabled starts a native OTLP/HTTP trace exporter for API
	// request spans. It is also enabled automatically when
	// SB_OTEL_TRACES_ENDPOINT is set.
	OTELTracesEnabled     bool
	OTELTracesEndpoint    string
	OTELTracesSampleRatio float64
	OTELServiceName       string

	// Admission control. Admission is purely resource-math: CPU/memory
	// reservation ratios plus a live memory floor. There is no fixed sandbox
	// count cap — the host runs as many sandboxes as the math allows.
	CPUReservationRatio    float64
	MemoryReservationRatio float64
	MemoryFloorRatio       float64
	// CPUOverProvisionFactor and MemoryOverProvisionFactor multiply the
	// reservation budgets above. Docker --cpus is a CFS cap (not a hard
	// reservation) and Linux lazy-allocates memory pages, so a host with
	// mostly-idle sandboxes can safely accept far more reservations than its
	// nominal capacity. The live MemoryFloorRatio check is the backstop that
	// catches real pressure when reservations and reality diverge. 0 or <1
	// is clamped to 1.0 (no overcommit) — operators that want strict packing
	// should lower the reservation ratios instead.
	CPUOverProvisionFactor    float64
	MemoryOverProvisionFactor float64
	HostCPUCoresOverride      int
	HostMemoryMBOverride      int
	// DiskReservationRatio gates total per-sandbox disk reservations against
	// HostDiskGB, or the auto-detected host disk size when HostDiskGB is not
	// set. 0 disables disk admission while still reporting disk observability.
	// HostDiskGB remains the operator override for a stricter Docker-volume
	// budget when the host filesystem is larger than the sandbox pool.
	DiskReservationRatio float64
	HostDiskGB           int
	// HostGPUCount and HostGPUVendor describe the GPU inventory wired into
	// Docker via nvidia-container-runtime / amdgpu / etc. Used by placement
	// scheduling so a GPU sandbox is never forwarded to a GPU-less peer.
	HostGPUCount  int
	HostGPUVendor string
	// HostSupportedRuntimes is a comma-separated SB_HOST_RUNTIMES list
	// declaring which OCI runtimes this host has installed. Empty falls
	// back to ["docker"] in capacity.New so existing single-runtime hosts
	// don't need new env to keep accepting placements.
	HostSupportedRuntimes []string

	// L4PortRangeStart / L4PortRangeEnd bound the parent-host port pool that
	// raw-TCP sandbox exposures (caddy-l4) are allocated from. The allocator
	// picks a random candidate first; collisions fall back to a deterministic
	// scan. Both sides are inclusive.
	//
	// The default range [22000, 23000] sits ABOVE the Linux registered-ports
	// boundary (1024) and BELOW the default ephemeral-port range
	// (net.ipv4.ip_local_port_range, typically 32768-60999). Keeping the pool
	// out of the ephemeral range matters: if these ports overlapped, the
	// kernel could hand any of them to an unrelated outbound connection as a
	// source port, and the next L4 expose attempt to bind() that number would
	// race-fail with EADDRINUSE. 1000 slots is the deliberate concurrent-TCP-
	// exposure cap per host; raise it via the env vars if you need more, but
	// keep both bounds outside the host's ephemeral range.
	L4PortRangeStart int
	L4PortRangeEnd   int
	// L4TLSListen is the listen address for the shared TLS-SNI multiplexer.
	// Empty disables TLS-SNI exposure entirely (the daemon will reject
	// protocol="tls" requests). When set, caddy-l4 binds this address and
	// routes by SNI to per-sandbox subdomains.
	//
	// install.sh sets this to ":443" in domain mode (which always uses
	// DNS-01 wildcard issuance, so :443 is free of ACME traffic and caddy-l4
	// can own it). The HTTPS Caddy server is moved to 127.0.0.1:8443 in that
	// case. In IP/path mode (no --domain) this stays empty and caddy-l4
	// is never started.
	L4TLSListen string
	// L4TLSFallback is the local address caddy-l4 forwards a TLS connection
	// to when no per-sandbox SNI route matches — i.e. the regular HTTPS site
	// served by Caddy itself (sandbox API and the catch-all 404).
	// Required when L4TLSListen is non-empty; ignored otherwise. Default is
	// "127.0.0.1:8443" to match install.sh's relocated HTTPS listener.
	L4TLSFallback string

	// Cluster mode (Phase 1). When EnableCluster is false the daemon runs as
	// a standalone single-node sandbox runner — byte-identical to the legacy
	// behavior. When true, this node joins (or bootstraps) a Raft+gossip
	// cluster that owns the placement map (sandbox_id -> owner node). Each
	// sandbox is owned by exactly one node; the owner's local SQLite remains
	// the source of truth for sandbox state. Hot-path traffic (toolbox,
	// sessions, port forwards) is transparently reverse-proxied to the owner.
	EnableCluster bool
	// NodeRole partitions cluster-mode components across this daemon. Values
	// are one of NodeRoleServer / NodeRoleWorker / NodeRoleIngress /
	// NodeRoleMixed. Default NodeRoleMixed preserves pre-role behavior
	// (every component runs on every node). SB_NODE_ROLE. Ignored when
	// EnableCluster=false — single-node mode is implicitly "mixed".
	NodeRole string
	NodeID   string
	// NodeName is the operator-friendly label gossiped to peers and shown on
	// the dashboard (e.g. "node1", "wrk2"). Display-only; raft/gossip identity
	// stays on NodeID. SB_NODE_NAME; empty falls back to NodeID at display
	// time so single-node and pre-existing deployments need no config change.
	NodeName            string
	RaftBindAddr        string
	RaftAdvertiseAddr   string
	RaftDataDir         string
	GossipBindAddr      string
	GossipAdvertiseAddr string
	BootstrapPeers      []string
	ClusterBootstrap    bool
	SelfAPIAdvertiseURL string
	// DataPlaneAdvertiseHost is the host/IP other nodes use for sandbox
	// public ingress (HTTP/SNI passthrough and raw TCP proxying) when this
	// node owns a sandbox. It is intentionally separate from
	// SelfAPIAdvertiseURL: many deployments put API traffic behind a shared
	// load balancer or API-only DNS name that must not be used as the
	// owner-data-plane target. SB_DATA_PLANE_ADVERTISE_HOST.
	DataPlaneAdvertiseHost string
	// IngressAdvertiseHost is the *public* host the SDK and end users hit for
	// sandbox URLs in cluster mode. It is separate from PublicHost (which is
	// the local node's bind-or-NAT address) and from DataPlaneAdvertiseHost
	// (which is the peer-internal LAN address other nodes use). When set, the
	// Caddy client uses it as the hostname in composed sandbox URLs; when
	// empty, it falls back to PublicHost so single-node mode is unchanged.
	//
	// Operators point this at whatever the cluster's data-plane LB endpoint
	// is — a wildcard DNS record fronting the ingress nodes, a cloud NLB, a
	// MetalLB/BGP VIP, or DNS round-robin across ingress nodes. The build
	// itself does not run any load balancer; that's a deployment decision.
	// See plans/data-plane-load-balancer.md.
	// SB_INGRESS_ADVERTISE_HOST.
	IngressAdvertiseHost          string
	ClusterRaftCommitTimeout      time.Duration
	ClusterCapacityGossipInterval time.Duration
	// ClusterMaxAutoVoters caps gossip-driven Raft voter promotion. Additional
	// nodes are added as non-voters so they still receive the placement log
	// without increasing quorum size. 0 means unlimited, preserving the old
	// behavior for tests or intentionally small fully-voting clusters.
	ClusterMaxAutoVoters int
	// ClusterCreateMaxPendingPerWorker caps reservation-stage creates per
	// worker. This is a leader-side queue guard: when a burst tries to send
	// more than this many not-yet-promoted creates to one worker, the leader
	// rejects with Retry-After instead of letting that worker absorb an
	// unbounded image-pull/docker-create storm. 0 disables the cap.
	// SB_CLUSTER_CREATE_MAX_PENDING_PER_WORKER.
	ClusterCreateMaxPendingPerWorker int
	// ClusterDeadOwnerGrace is how long the leader waits after memberlist marks
	// a node dead before orphaning its placements and removing it from the
	// raft configuration. Long enough to absorb transient gossip flap
	// (network blips, GC pauses) but short enough that operators don't have
	// to wait minutes to recover.
	ClusterDeadOwnerGrace time.Duration
	// ClusterGossipSecretKey, when non-empty, enables AES gossip encryption +
	// authentication. Accepts a base64-encoded 16/24/32-byte key (AES-128/192/256).
	// Required in cluster mode (Load() refuses to start otherwise). The escape
	// hatch is ClusterInsecureGossip below. SB_GOSSIP_SECRET_KEY.
	ClusterGossipSecretKey string
	// ClusterInsecureGossip explicitly opts out of the gossip-key requirement
	// when SB_ENABLE_CLUSTER=true. Only safe on a fully isolated network where
	// every peer that can reach gossip+raft ports is trusted. Default false —
	// the daemon refuses to boot in cluster mode without a gossip key. Use only
	// for ephemeral test setups. SB_CLUSTER_INSECURE_GOSSIP.
	ClusterInsecureGossip bool
	// ClusterInsecureCredentials opts out of the shared-credential-key
	// requirement in cluster mode. Without a key shared across nodes, sealed
	// registry passwords and per-mount credentials replicated via raft cannot
	// be decrypted by a failover owner — recovered sandboxes lose access to
	// private registries and credentialed mounts. Default false: the daemon
	// refuses to boot in cluster mode unless either SB_CREDENTIAL_ENCRYPTION_KEY
	// is set explicitly or a key file already exists at
	// SB_CREDENTIAL_ENCRYPTION_KEY_PATH (the operator may have distributed it
	// out of band). Set true only for ephemeral test setups that don't use
	// sealed creds. SB_CLUSTER_INSECURE_CREDENTIALS.
	ClusterInsecureCredentials bool

	// Cluster-internal mTLS. When enabled, leader-forwarded raft applies (and
	// any other future cluster-internal RPC) ride over a separate HTTPS listener
	// that requires a client certificate signed by the cluster CA. Without TLS
	// the same RPCs ride over the public API URL with only the shared PAT for
	// auth — fine on a private overlay, but a client-cert pin is the right
	// default for any internet-adjacent deployment.
	//
	// ClusterTLSDir holds ca.crt, ca.key (only on the bootstrap node and any
	// joiner that received the bundle), node.crt, node.key. cluster-init.sh
	// generates the CA and a node cert; cluster-join.sh signs a fresh node
	// cert from the bundled CA. SB_CLUSTER_TLS_DIR.
	ClusterTLSDir string
	// ClusterInternalListenAddr is the bind address for the mTLS internal
	// listener. SB_CLUSTER_INTERNAL_LISTEN. Default 0.0.0.0:7002.
	ClusterInternalListenAddr string
	// ClusterInternalAdvertiseURL is the URL peers dial for cluster-internal
	// RPCs. Falls back to https://<derived-host>:<internal-port> when empty.
	// Must be HTTPS — plaintext defeats the purpose. SB_CLUSTER_INTERNAL_ADVERTISE.
	ClusterInternalAdvertiseURL string
	// ImageBuildContextEnabled is the operator opt-in for the contextHashes
	// upload path — image builds whose context includes caller-supplied
	// local files (COPY/ADD). Off by default because the resolution path
	// needs an object-store + registry combo to push the resulting layered
	// image somewhere the docker daemon can pull from on the next sandbox
	// start. With this disabled, builds that only RUN commands (no
	// caller-side context) still work — they execute against a tar
	// containing just the Dockerfile.
	//
	// NOTE: enabling this is necessary but not sufficient. The context
	// resolver itself is not yet wired, so requests with contextHashes will
	// still return HTTP 501 even when this flag is true. The flag exists
	// so operators can explicitly opt in to that codepath as soon as the
	// resolver lands, without a daemon redeploy.
	ImageBuildContextEnabled bool
	// ImageBuildTimeout caps a single `docker build` (or `docker push`)
	// call from any image-build path: the native POST /v1/images/build
	// handler and the Daytona facade's createSandbox build-on-create flow.
	// Build time is opaque (depends on the Dockerfile) so the default is
	// generous; we bound it only to keep a runaway build from permanently
	// parking the HTTP handler.
	ImageBuildTimeout time.Duration
	// ImageBuildGCEnabled toggles the periodic janitor that sweeps
	// locally-built images (BuiltImageNamespace, i.e. "aerolvm-build/*")
	// that are no longer referenced by any active sandbox AND were created
	// more than ImageBuildGCTTL ago. Without this, images produced by
	// standalone POST /v1/images/build calls or by builds whose followup
	// CreateSandbox failed accumulate forever — service.maybeRemoveImage
	// only runs on sandbox destroy and so can't see images that never had
	// a sandbox row.
	ImageBuildGCEnabled bool
	// ImageBuildGCInterval is how often the janitor ticker fires. Default
	// 10m: cheap enough (one filtered /images/json call + one indexed store
	// lookup per match) that running it more often would only matter if
	// builds were churning faster than the TTL — which would itself be a
	// signal something is wrong upstream.
	ImageBuildGCInterval time.Duration
	// ImageBuildGCTTL is the minimum age a built image must reach before
	// it becomes eligible for removal. Default 1h: comfortably longer than
	// any reasonable retry/network-blip between build and create, so a
	// transient hiccup doesn't have the janitor yanking an image a client
	// is about to use.
	ImageBuildGCTTL time.Duration
	// ImageDistributionAOCRHost is the registry host treated as the optional
	// managed AOCR image-distribution provider. Empty configs constructed in
	// tests fall back to the product default in the service layer.
	ImageDistributionAOCRHost string
	// ImagePullMaxConcurrent caps simultaneous Docker /images/create streams
	// per worker. Pulls for the same image/auth key are still deduplicated; this
	// cap protects the daemon and registry when many different cold images are
	// requested at once. 0 disables the cap.
	ImagePullMaxConcurrent int
	// ImagePullFailureBackoff suppresses immediate retries for the same
	// image/auth key after Docker or the registry returns an error. This keeps a
	// bad tag or rate-limit response from turning into a per-create retry storm.
	ImagePullFailureBackoff time.Duration

	// AutoImportEnabled gates the F21 post-pull auto-import path: after a
	// successful pull of a private upstream image, sandboxd asks AOCR to
	// re-mount the bytes under `cluster/<id>/_imported/...` so future
	// recreates on other nodes pull with the cluster PAT and survive an
	// upstream credential rotation. Off by default; flipping it on requires
	// AutoImportHooksBaseURL, AutoImportClusterID, and a non-empty PAT file.
	// SB_AUTO_IMPORT_ENABLED.
	AutoImportEnabled bool
	// AutoImportHooksBaseURL is the AOCR hooks service root (no trailing
	// slash). The importer appends `/v1/internal/imports`.
	// SB_AUTO_IMPORT_HOOKS_URL.
	AutoImportHooksBaseURL string
	// AutoImportClusterID identifies this sandboxd cluster to AOCR. Goes
	// into the import request body and constrains the cluster PAT's scope
	// on the AOCR side. SB_AUTO_IMPORT_CLUSTER_ID.
	AutoImportClusterID string
	// AutoImportClusterPATPath is the file path to the bearer token
	// presented to the AOCR ImportAPI. File-sourced (not env) so the secret
	// is rotatable without restarting and never appears in process listings.
	// SB_AUTO_IMPORT_CLUSTER_PAT_PATH.
	AutoImportClusterPATPath string
	// AutoImportRetentionSuffix is appended to the imported tag in the
	// cluster namespace (e.g. `--idle-90d`). Must begin with `--` to match
	// AOCR's retention parser; empty falls back to the server-side default.
	// SB_AUTO_IMPORT_RETENTION_SUFFIX.
	AutoImportRetentionSuffix string
	// AutoImportRequestTimeout caps a single ImportAPI call. The mount-from-
	// repo flow is normally sub-second; the timeout guards against an
	// upstream that hangs on layer enumeration.
	// SB_AUTO_IMPORT_REQUEST_TIMEOUT.
	AutoImportRequestTimeout time.Duration
	// AutoImportReconcileInterval is the period between retry sweeps for
	// rows the post-pull path left flagged `auto_import_pending`. Too short
	// and a flapping AOCR causes a fan-out storm; too long and a transient
	// outage leaves the failover path on the F17 wrap-creds fallback for
	// longer than necessary. Default 5m.
	// SB_AUTO_IMPORT_RECONCILE_INTERVAL.
	AutoImportReconcileInterval time.Duration
	// AutoImportMaxInFlight bounds per-tick fan-out so a recovery storm
	// (everything pending at once after a long outage) cannot saturate
	// AOCR or the local Docker socket. Default 4.
	// SB_AUTO_IMPORT_MAX_IN_FLIGHT.
	AutoImportMaxInFlight int

	// MirrorHost is the AOCR pull-through mirror vhost
	// (e.g. `mirror.aocr.aerol.ai`). When non-empty AND at least one
	// upstream is configured, the docker client rewrites matching image
	// refs through this host before pulling. Empty disables rewriting and
	// pulls hit the upstream registry directly — the safety hatch for
	// nodes not running with AOCR mirror enabled. SB_MIRROR_HOST.
	MirrorHost string
	// MirrorPushHost is the AOCR push vhost (e.g. `aocr.aerol.ai`). Used
	// only for idempotency: a ref that already points at the push vhost
	// (e.g. a cluster snapshot or a previously-imported image) is not
	// re-rewritten. Optional. SB_MIRROR_PUSH_HOST.
	MirrorPushHost string
	// MirrorUpstreams is the comma-separated host=shortname mapping
	// (`ghcr.io=ghcr,gcr.io=gcr,quay.io=quay,registry.k8s.io=k8s`). Docker
	// Hub is intentionally absent — it's mirrored via the Docker daemon's
	// `registry-mirrors` daemon.json setting, not client-side rewriting.
	// SB_MIRROR_UPSTREAMS.
	MirrorUpstreams []MirrorUpstreamMapping
	// UpstreamWrapKeyPath is the file path to the per-cluster AES-GCM wrap
	// key used to seal docker `RegistryAuth` payloads before they reach
	// the mirror, so the mirror vhost never sees raw upstream PATs.
	// File-sourced (not env) with required mode 0400. When empty or
	// unreadable, the mirror falls back to anonymous pulls — fine for
	// public images, but private images will 401. SB_UPSTREAM_WRAP_KEY_PATH.
	UpstreamWrapKeyPath string
}

func Load

func Load() (Config, error)

func (Config) DomainMode

func (c Config) DomainMode() bool

func (Config) EffectivePublicHost added in v0.2.1

func (c Config) EffectivePublicHost() string

EffectivePublicHost returns the hostname or IP that should appear in SDK-facing sandbox URLs. In cluster mode, operators set SB_INGRESS_ADVERTISE_HOST to whatever fronts the ingress tier (cloud NLB, MetalLB VIP, wildcard DNS, etc.); single-node mode and clusters that haven't configured an ingress host fall back to PublicHost, matching the pre-cluster behavior.

func (Config) IdleTimeout

func (c Config) IdleTimeout() time.Duration

func (Config) IsIngress added in v0.2.1

func (c Config) IsIngress() bool

IsIngress reports whether this daemon installs owner-aware Caddy routes for remote sandboxes. Pure server/worker nodes return false; mixed and any hybrid that includes "ingress" return true.

func (Config) IsServer added in v0.2.1

func (c Config) IsServer() bool

IsServer reports whether this daemon should run server-only responsibilities (Raft voter promotion, FSM ownership). Mixed nodes and any hybrid that includes "server" return true; pure worker, pure ingress, and the worker+ingress combo return false.

func (Config) IsWorker added in v0.2.1

func (c Config) IsWorker() bool

IsWorker reports whether this daemon owns sandboxes locally (Docker, lifecycle sweep, image GC, reservation replay). Pure ingress/server nodes return false; mixed and any hybrid that includes "worker" return true.

func (Config) ListenAddr

func (c Config) ListenAddr() string

func (Config) Roles added in v0.2.1

func (c Config) Roles() []string

Roles returns the base-role set this daemon should run. NodeRoleMixed expands to [server, worker, ingress]; hybrid comma-separated values ("worker,ingress") are split into their constituent base roles. The empty string maps to mixed for safety (callers that reach here via the validated config never see an empty string, but consumers that build a Config{} by hand in tests get the same default Load() applies). The returned slice is sorted and deduped.

type MirrorUpstreamMapping added in v0.2.3

type MirrorUpstreamMapping struct {
	Host      string
	Shortname string
}

MirrorUpstreamMapping is a single host=shortname pair parsed from SB_MIRROR_UPSTREAMS. Kept here (not in pkg/docker) so the config layer has no dependency on the docker package — main.go translates to docker.MirrorUpstream at wiring time.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL