Documentation
¶
Index ¶
Constants ¶
const ( NodeRoleServer = "server" NodeRoleWorker = "worker" NodeRoleIngress = "ingress" NodeRoleMixed = "mixed" )
NodeRole values for SB_NODE_ROLE. The role partitions which components a cluster-mode daemon runs (Raft voter promotion, sandbox worker work, public ingress reconciler). Default is NodeRoleMixed — every node runs every component, matching pre-role behavior bit-for-bit. The split exists so a 200-worker cluster doesn't have 200 Raft voters and 200 nodes each holding a 10K-route public ingress table; see plans/data-plane-load-balancer.md.
SB_NODE_ROLE accepts either a single token or a comma-separated combination of base roles, e.g. "worker,ingress" for a data-plane edge node or "server,ingress" for a control + ingress node. NodeRoleMixed remains the shorthand for "server,worker,ingress" and may not be combined with anything else. Runtime topology validation keeps mixed and hybrid-role nodes as a small-cluster convenience only; clusters above 10 live nodes must use dedicated server, worker, and ingress roles.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
PATToken string
APIHost string
APIPort int
Domain string
PublicHost string
CaddyAdminURL string
CaddyServerID string
DBPath string
DockerNetwork string
ToolboxBinaryPath string
ToolboxMountPath string
ToolboxPort int
IdleTimeoutMinutes int
ContainerPrivileged bool
ResourceLimitsOff bool
// Runtime is the host default container runtime for new sandboxes.
// Per-sandbox CreateSandboxRequest.Runtime overrides it. Allowed values
// are "docker" (default), "gvisor", or "kata"; validation lives in Load().
Runtime string
AutoReconcile bool
EnableCaddy bool
EnableNetworkRules bool
EnableEventMonitor bool
EnableSSHGateway bool
SSHListenAddr string
SSHHostKeyPath string
CredentialEncryptionKey string
CredentialEncryptionKeyPath string
MountsRootPath string
MountsCredentialsRuntimeDir string
MountWaitTimeout time.Duration
LogLevel string
ShutdownTimeout time.Duration
HTTPClientTimeout time.Duration
DockerRuntimeWaitTimeout time.Duration
ToolboxWaitTimeout time.Duration
ReconcileInterval time.Duration
NetstatsPollInterval time.Duration
UploadMaxBytes int64
// OTELMetricsEnabled starts a native OTLP/HTTP metric exporter that bridges
// the daemon's aerolvm_* expvars into OpenTelemetry observations. It is
// also enabled automatically when SB_OTEL_METRICS_ENDPOINT is set.
OTELMetricsEnabled bool
OTELMetricsEndpoint string
OTELMetricsInterval time.Duration
// OTELTracesEnabled starts a native OTLP/HTTP trace exporter for API
// request spans. It is also enabled automatically when
// SB_OTEL_TRACES_ENDPOINT is set.
OTELTracesEnabled bool
OTELTracesEndpoint string
OTELTracesSampleRatio float64
OTELServiceName string
// Admission control. Admission is purely resource-math: CPU/memory
// reservation ratios plus a live memory floor. There is no fixed sandbox
// count cap — the host runs as many sandboxes as the math allows.
CPUReservationRatio float64
MemoryReservationRatio float64
MemoryFloorRatio float64
// CPUOverProvisionFactor and MemoryOverProvisionFactor multiply the
// reservation budgets above. Docker --cpus is a CFS cap (not a hard
// reservation) and Linux lazy-allocates memory pages, so a host with
// mostly-idle sandboxes can safely accept far more reservations than its
// nominal capacity. The live MemoryFloorRatio check is the backstop that
// catches real pressure when reservations and reality diverge. 0 or <1
// is clamped to 1.0 (no overcommit) — operators that want strict packing
// should lower the reservation ratios instead.
CPUOverProvisionFactor float64
MemoryOverProvisionFactor float64
HostCPUCoresOverride int
HostMemoryMBOverride int
// DiskReservationRatio gates total per-sandbox disk reservations against
// HostDiskGB, or the auto-detected host disk size when HostDiskGB is not
// set. 0 disables disk admission while still reporting disk observability.
// HostDiskGB remains the operator override for a stricter Docker-volume
// budget when the host filesystem is larger than the sandbox pool.
DiskReservationRatio float64
HostDiskGB int
// HostGPUCount and HostGPUVendor describe the GPU inventory wired into
// Docker via nvidia-container-runtime / amdgpu / etc. Used by placement
// scheduling so a GPU sandbox is never forwarded to a GPU-less peer.
HostGPUCount int
HostGPUVendor string
// HostSupportedRuntimes is a comma-separated SB_HOST_RUNTIMES list
// declaring which OCI runtimes this host has installed. Empty falls
// back to ["docker"] in capacity.New so existing single-runtime hosts
// don't need new env to keep accepting placements.
HostSupportedRuntimes []string
// L4PortRangeStart / L4PortRangeEnd bound the parent-host port pool that
// raw-TCP sandbox exposures (caddy-l4) are allocated from. The allocator
// picks a random candidate first; collisions fall back to a deterministic
// scan. Both sides are inclusive.
//
// The default range [22000, 23000] sits ABOVE the Linux registered-ports
// boundary (1024) and BELOW the default ephemeral-port range
// (net.ipv4.ip_local_port_range, typically 32768-60999). Keeping the pool
// out of the ephemeral range matters: if these ports overlapped, the
// kernel could hand any of them to an unrelated outbound connection as a
// source port, and the next L4 expose attempt to bind() that number would
// race-fail with EADDRINUSE. 1000 slots is the deliberate concurrent-TCP-
// exposure cap per host; raise it via the env vars if you need more, but
// keep both bounds outside the host's ephemeral range.
L4PortRangeStart int
L4PortRangeEnd int
// L4TLSListen is the listen address for the shared TLS-SNI multiplexer.
// Empty disables TLS-SNI exposure entirely (the daemon will reject
// protocol="tls" requests). When set, caddy-l4 binds this address and
// routes by SNI to per-sandbox subdomains.
//
// install.sh sets this to ":443" in domain mode (which always uses
// DNS-01 wildcard issuance, so :443 is free of ACME traffic and caddy-l4
// can own it). The HTTPS Caddy server is moved to 127.0.0.1:8443 in that
// case. In IP/path mode (no --domain) this stays empty and caddy-l4
// is never started.
L4TLSListen string
// L4TLSFallback is the local address caddy-l4 forwards a TLS connection
// to when no per-sandbox SNI route matches — i.e. the regular HTTPS site
// served by Caddy itself (sandbox API and the catch-all 404).
// Required when L4TLSListen is non-empty; ignored otherwise. Default is
// "127.0.0.1:8443" to match install.sh's relocated HTTPS listener.
L4TLSFallback string
// Cluster mode (Phase 1). When EnableCluster is false the daemon runs as
// a standalone single-node sandbox runner — byte-identical to the legacy
// behavior. When true, this node joins (or bootstraps) a Raft+gossip
// cluster that owns the placement map (sandbox_id -> owner node). Each
// sandbox is owned by exactly one node; the owner's local SQLite remains
// the source of truth for sandbox state. Hot-path traffic (toolbox,
// sessions, port forwards) is transparently reverse-proxied to the owner.
EnableCluster bool
// NodeRole partitions cluster-mode components across this daemon. Values
// are one of NodeRoleServer / NodeRoleWorker / NodeRoleIngress /
// NodeRoleMixed. Default NodeRoleMixed preserves pre-role behavior
// (every component runs on every node). SB_NODE_ROLE. Ignored when
// EnableCluster=false — single-node mode is implicitly "mixed".
NodeRole string
NodeID string
// NodeName is the operator-friendly label gossiped to peers and shown on
// the dashboard (e.g. "node1", "wrk2"). Display-only; raft/gossip identity
// stays on NodeID. SB_NODE_NAME; empty falls back to NodeID at display
// time so single-node and pre-existing deployments need no config change.
NodeName string
RaftBindAddr string
RaftAdvertiseAddr string
RaftDataDir string
GossipBindAddr string
GossipAdvertiseAddr string
BootstrapPeers []string
ClusterBootstrap bool
SelfAPIAdvertiseURL string
// DataPlaneAdvertiseHost is the host/IP other nodes use for sandbox
// public ingress (HTTP/SNI passthrough and raw TCP proxying) when this
// node owns a sandbox. It is intentionally separate from
// SelfAPIAdvertiseURL: many deployments put API traffic behind a shared
// load balancer or API-only DNS name that must not be used as the
// owner-data-plane target. SB_DATA_PLANE_ADVERTISE_HOST.
DataPlaneAdvertiseHost string
// IngressAdvertiseHost is the *public* host the SDK and end users hit for
// sandbox URLs in cluster mode. It is separate from PublicHost (which is
// the local node's bind-or-NAT address) and from DataPlaneAdvertiseHost
// (which is the peer-internal LAN address other nodes use). When set, the
// Caddy client uses it as the hostname in composed sandbox URLs; when
// empty, it falls back to PublicHost so single-node mode is unchanged.
//
// Operators point this at whatever the cluster's data-plane LB endpoint
// is — a wildcard DNS record fronting the ingress nodes, a cloud NLB, a
// MetalLB/BGP VIP, or DNS round-robin across ingress nodes. The build
// itself does not run any load balancer; that's a deployment decision.
// See plans/data-plane-load-balancer.md.
// SB_INGRESS_ADVERTISE_HOST.
IngressAdvertiseHost string
ClusterRaftCommitTimeout time.Duration
ClusterCapacityGossipInterval time.Duration
// ClusterMaxAutoVoters caps gossip-driven Raft voter promotion. Additional
// nodes are added as non-voters so they still receive the placement log
// without increasing quorum size. 0 means unlimited, preserving the old
// behavior for tests or intentionally small fully-voting clusters.
ClusterMaxAutoVoters int
// ClusterCreateMaxPendingPerWorker caps reservation-stage creates per
// worker. This is a leader-side queue guard: when a burst tries to send
// more than this many not-yet-promoted creates to one worker, the leader
// rejects with Retry-After instead of letting that worker absorb an
// unbounded image-pull/docker-create storm. 0 disables the cap.
// SB_CLUSTER_CREATE_MAX_PENDING_PER_WORKER.
ClusterCreateMaxPendingPerWorker int
// ClusterDeadOwnerGrace is how long the leader waits after memberlist marks
// a node dead before orphaning its placements and removing it from the
// raft configuration. Long enough to absorb transient gossip flap
// (network blips, GC pauses) but short enough that operators don't have
// to wait minutes to recover.
ClusterDeadOwnerGrace time.Duration
// ClusterGossipSecretKey, when non-empty, enables AES gossip encryption +
// authentication. Accepts a base64-encoded 16/24/32-byte key (AES-128/192/256).
// Required in cluster mode (Load() refuses to start otherwise). The escape
// hatch is ClusterInsecureGossip below. SB_GOSSIP_SECRET_KEY.
ClusterGossipSecretKey string
// ClusterInsecureGossip explicitly opts out of the gossip-key requirement
// when SB_ENABLE_CLUSTER=true. Only safe on a fully isolated network where
// every peer that can reach gossip+raft ports is trusted. Default false —
// the daemon refuses to boot in cluster mode without a gossip key. Use only
// for ephemeral test setups. SB_CLUSTER_INSECURE_GOSSIP.
ClusterInsecureGossip bool
// ClusterInsecureCredentials opts out of the shared-credential-key
// requirement in cluster mode. Without a key shared across nodes, sealed
// registry passwords and per-mount credentials replicated via raft cannot
// be decrypted by a failover owner — recovered sandboxes lose access to
// private registries and credentialed mounts. Default false: the daemon
// refuses to boot in cluster mode unless either SB_CREDENTIAL_ENCRYPTION_KEY
// is set explicitly or a key file already exists at
// SB_CREDENTIAL_ENCRYPTION_KEY_PATH (the operator may have distributed it
// out of band). Set true only for ephemeral test setups that don't use
// sealed creds. SB_CLUSTER_INSECURE_CREDENTIALS.
ClusterInsecureCredentials bool
// Cluster-internal mTLS. When enabled, leader-forwarded raft applies (and
// any other future cluster-internal RPC) ride over a separate HTTPS listener
// that requires a client certificate signed by the cluster CA. Without TLS
// the same RPCs ride over the public API URL with only the shared PAT for
// auth — fine on a private overlay, but a client-cert pin is the right
// default for any internet-adjacent deployment.
//
// ClusterTLSDir holds ca.crt, ca.key (only on the bootstrap node and any
// joiner that received the bundle), node.crt, node.key. cluster-init.sh
// generates the CA and a node cert; cluster-join.sh signs a fresh node
// cert from the bundled CA. SB_CLUSTER_TLS_DIR.
ClusterTLSDir string
// ClusterInternalListenAddr is the bind address for the mTLS internal
// listener. SB_CLUSTER_INTERNAL_LISTEN. Default 0.0.0.0:7002.
ClusterInternalListenAddr string
// ClusterInternalAdvertiseURL is the URL peers dial for cluster-internal
// RPCs. Falls back to https://<derived-host>:<internal-port> when empty.
// Must be HTTPS — plaintext defeats the purpose. SB_CLUSTER_INTERNAL_ADVERTISE.
ClusterInternalAdvertiseURL string
// ImageBuildContextEnabled is the operator opt-in for the contextHashes
// upload path — image builds whose context includes caller-supplied
// local files (COPY/ADD). Off by default because the resolution path
// needs an object-store + registry combo to push the resulting layered
// image somewhere the docker daemon can pull from on the next sandbox
// start. With this disabled, builds that only RUN commands (no
// caller-side context) still work — they execute against a tar
// containing just the Dockerfile.
//
// NOTE: enabling this is necessary but not sufficient. The context
// resolver itself is not yet wired, so requests with contextHashes will
// still return HTTP 501 even when this flag is true. The flag exists
// so operators can explicitly opt in to that codepath as soon as the
// resolver lands, without a daemon redeploy.
ImageBuildContextEnabled bool
// ImageBuildTimeout caps a single `docker build` (or `docker push`)
// call from any image-build path: the native POST /v1/images/build
// handler and the Daytona facade's createSandbox build-on-create flow.
// Build time is opaque (depends on the Dockerfile) so the default is
// generous; we bound it only to keep a runaway build from permanently
// parking the HTTP handler.
ImageBuildTimeout time.Duration
// ImageBuildGCEnabled toggles the periodic janitor that sweeps
// locally-built images (BuiltImageNamespace, i.e. "aerolvm-build/*")
// that are no longer referenced by any active sandbox AND were created
// more than ImageBuildGCTTL ago. Without this, images produced by
// standalone POST /v1/images/build calls or by builds whose followup
// CreateSandbox failed accumulate forever — service.maybeRemoveImage
// only runs on sandbox destroy and so can't see images that never had
// a sandbox row.
ImageBuildGCEnabled bool
// ImageBuildGCInterval is how often the janitor ticker fires. Default
// 10m: cheap enough (one filtered /images/json call + one indexed store
// lookup per match) that running it more often would only matter if
// builds were churning faster than the TTL — which would itself be a
// signal something is wrong upstream.
ImageBuildGCInterval time.Duration
// ImageBuildGCTTL is the minimum age a built image must reach before
// it becomes eligible for removal. Default 1h: comfortably longer than
// any reasonable retry/network-blip between build and create, so a
// transient hiccup doesn't have the janitor yanking an image a client
// is about to use.
ImageBuildGCTTL time.Duration
// ImageDistributionAOCRHost is the registry host treated as the optional
// managed AOCR image-distribution provider. Empty configs constructed in
// tests fall back to the product default in the service layer.
ImageDistributionAOCRHost string
// ImagePullMaxConcurrent caps simultaneous Docker /images/create streams
// per worker. Pulls for the same image/auth key are still deduplicated; this
// cap protects the daemon and registry when many different cold images are
// requested at once. 0 disables the cap.
ImagePullMaxConcurrent int
// ImagePullFailureBackoff suppresses immediate retries for the same
// image/auth key after Docker or the registry returns an error. This keeps a
// bad tag or rate-limit response from turning into a per-create retry storm.
ImagePullFailureBackoff time.Duration
// AutoImportEnabled gates the F21 post-pull auto-import path: after a
// successful pull of a private upstream image, sandboxd asks AOCR to
// re-mount the bytes under `cluster/<id>/_imported/...` so future
// recreates on other nodes pull with the cluster PAT and survive an
// upstream credential rotation. Off by default; flipping it on requires
// AutoImportHooksBaseURL, AutoImportClusterID, and a non-empty PAT file.
// SB_AUTO_IMPORT_ENABLED.
AutoImportEnabled bool
// AutoImportHooksBaseURL is the AOCR hooks service root (no trailing
// slash). The importer appends `/v1/internal/imports`.
// SB_AUTO_IMPORT_HOOKS_URL.
AutoImportHooksBaseURL string
// AutoImportClusterID identifies this sandboxd cluster to AOCR. Goes
// into the import request body and constrains the cluster PAT's scope
// on the AOCR side. SB_AUTO_IMPORT_CLUSTER_ID.
AutoImportClusterID string
// AutoImportClusterPATPath is the file path to the bearer token
// presented to the AOCR ImportAPI. File-sourced (not env) so the secret
// is rotatable without restarting and never appears in process listings.
// SB_AUTO_IMPORT_CLUSTER_PAT_PATH.
AutoImportClusterPATPath string
// AutoImportRetentionSuffix is appended to the imported tag in the
// cluster namespace (e.g. `--idle-90d`). Must begin with `--` to match
// AOCR's retention parser; empty falls back to the server-side default.
// SB_AUTO_IMPORT_RETENTION_SUFFIX.
AutoImportRetentionSuffix string
// AutoImportRequestTimeout caps a single ImportAPI call. The mount-from-
// repo flow is normally sub-second; the timeout guards against an
// upstream that hangs on layer enumeration.
// SB_AUTO_IMPORT_REQUEST_TIMEOUT.
AutoImportRequestTimeout time.Duration
// AutoImportReconcileInterval is the period between retry sweeps for
// rows the post-pull path left flagged `auto_import_pending`. Too short
// and a flapping AOCR causes a fan-out storm; too long and a transient
// outage leaves the failover path on the F17 wrap-creds fallback for
// longer than necessary. Default 5m.
// SB_AUTO_IMPORT_RECONCILE_INTERVAL.
AutoImportReconcileInterval time.Duration
// AutoImportMaxInFlight bounds per-tick fan-out so a recovery storm
// (everything pending at once after a long outage) cannot saturate
// AOCR or the local Docker socket. Default 4.
// SB_AUTO_IMPORT_MAX_IN_FLIGHT.
AutoImportMaxInFlight int
// MirrorHost is the AOCR pull-through mirror vhost
// (e.g. `mirror.aocr.aerol.ai`). When non-empty AND at least one
// upstream is configured, the docker client rewrites matching image
// refs through this host before pulling. Empty disables rewriting and
// pulls hit the upstream registry directly — the safety hatch for
// nodes not running with AOCR mirror enabled. SB_MIRROR_HOST.
MirrorHost string
// MirrorPushHost is the AOCR push vhost (e.g. `aocr.aerol.ai`). Used
// only for idempotency: a ref that already points at the push vhost
// (e.g. a cluster snapshot or a previously-imported image) is not
// re-rewritten. Optional. SB_MIRROR_PUSH_HOST.
MirrorPushHost string
// MirrorUpstreams is the comma-separated host=shortname mapping
// (`ghcr.io=ghcr,gcr.io=gcr,quay.io=quay,registry.k8s.io=k8s`). Docker
// Hub is intentionally absent — it's mirrored via the Docker daemon's
// `registry-mirrors` daemon.json setting, not client-side rewriting.
// SB_MIRROR_UPSTREAMS.
MirrorUpstreams []MirrorUpstreamMapping
// UpstreamWrapKeyPath is the file path to the per-cluster AES-GCM wrap
// key used to seal docker `RegistryAuth` payloads before they reach
// the mirror, so the mirror vhost never sees raw upstream PATs.
// File-sourced (not env) with required mode 0400. When empty or
// unreadable, the mirror falls back to anonymous pulls — fine for
// public images, but private images will 401. SB_UPSTREAM_WRAP_KEY_PATH.
UpstreamWrapKeyPath string
}
func (Config) DomainMode ¶
func (Config) EffectivePublicHost ¶ added in v0.2.1
EffectivePublicHost returns the hostname or IP that should appear in SDK-facing sandbox URLs. In cluster mode, operators set SB_INGRESS_ADVERTISE_HOST to whatever fronts the ingress tier (cloud NLB, MetalLB VIP, wildcard DNS, etc.); single-node mode and clusters that haven't configured an ingress host fall back to PublicHost, matching the pre-cluster behavior.
func (Config) IdleTimeout ¶
func (Config) IsIngress ¶ added in v0.2.1
IsIngress reports whether this daemon installs owner-aware Caddy routes for remote sandboxes. Pure server/worker nodes return false; mixed and any hybrid that includes "ingress" return true.
func (Config) IsServer ¶ added in v0.2.1
IsServer reports whether this daemon should run server-only responsibilities (Raft voter promotion, FSM ownership). Mixed nodes and any hybrid that includes "server" return true; pure worker, pure ingress, and the worker+ingress combo return false.
func (Config) IsWorker ¶ added in v0.2.1
IsWorker reports whether this daemon owns sandboxes locally (Docker, lifecycle sweep, image GC, reservation replay). Pure ingress/server nodes return false; mixed and any hybrid that includes "worker" return true.
func (Config) ListenAddr ¶
func (Config) Roles ¶ added in v0.2.1
Roles returns the base-role set this daemon should run. NodeRoleMixed expands to [server, worker, ingress]; hybrid comma-separated values ("worker,ingress") are split into their constituent base roles. The empty string maps to mixed for safety (callers that reach here via the validated config never see an empty string, but consumers that build a Config{} by hand in tests get the same default Load() applies). The returned slice is sorted and deduped.
type MirrorUpstreamMapping ¶ added in v0.2.3
MirrorUpstreamMapping is a single host=shortname pair parsed from SB_MIRROR_UPSTREAMS. Kept here (not in pkg/docker) so the config layer has no dependency on the docker package — main.go translates to docker.MirrorUpstream at wiring time.