cluster

package
v0.6.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: MIT Imports: 54 Imported by: 0

Documentation

Overview

Package cluster implements Phase-1 cluster mode for sandboxd.

Architecture (Phase 1):

  • Membership: SWIM gossip via hashicorp/memberlist. Gossip carries identity and role metadata only so memberlist's 512-byte NodeMeta limit cannot strip Raft addresses.

  • Capacity heartbeats: server-role nodes fetch authenticated capacity.Snapshot payloads from worker-capable peers and require a fresh heartbeat before placement can target that worker.

  • Placement map: server-role nodes run a small Raft FSM (hashicorp/raft) holding sandbox_id -> owner_node_id plus replicated recovery metadata. Mutations happen on the leader; server reads are local from the FSM. Worker/ingress-only nodes run Agent instead: they gossip identity and receive owner API forwards, but all placement reads/writes go to the server quorum over authenticated RPC. They do not store the FSM and do not join Raft as non-voters.

  • Owner-sharded execution: once a sandbox is placed on node N, all of its state and lifecycle stays on N. The local SQLite store is unchanged. Cross-node API calls (toolbox, sessions, port forwards) are transparently reverse-proxied to the owner via internal/cluster.ForwardHTTP.

  • No central control plane: any node can accept any request. Mutating requests for sandbox X are forwarded to X's owner; CreateSandbox forwards to the placement target chosen by power-of-two-choices.

Single-node mode (cfg.EnableCluster = false) uses Noop so that callsites can stay unconditional.

Index

Constants

View Source
const (
	PublicInternalApplyPath           = "/v1/cluster/internal/apply"
	PublicInternalPlacementPath       = "/v1/cluster/internal/placement/"
	PublicInternalPlacementByNamePath = "/v1/cluster/internal/placement-by-name/"
	PublicInternalPlacementsPath      = "/v1/cluster/internal/placements"
	PublicInternalPlacementsQueryPath = "/v1/cluster/internal/placements/query"
	PublicInternalPlacementsPagePath  = "/v1/cluster/internal/placements/page"
	PublicInternalPlacementsByIDsPath = "/v1/cluster/internal/placements-by-ids"
	// PublicInternalOwnedRecoveryPath serves a node its OWN failover-recreate
	// placements (spec + secret handle). It is how a dedicated worker — which
	// has no FSM — learns that a placement was reassigned to it.
	PublicInternalOwnedRecoveryPath = "/v1/cluster/internal/placements/owned-recovery"
	// PublicInternalReassignStuckPath lets an owner that keeps failing to
	// recreate a sandbox ask the leader to move it. Target selection stays on
	// the FSM side.
	PublicInternalReassignStuckPath   = "/v1/cluster/internal/placements/reassign-stuck"
	PublicInternalRecoveryPath        = "/v1/cluster/internal/recovery/"
	PublicInternalSelectPlacementPath = "/v1/cluster/internal/select-placement"
	PublicInternalVolumePath          = "/v1/cluster/internal/volume"
	PublicInternalSnapshotNodePath    = "/v1/cluster/internal/snapshot-node"
	PublicInternalDrainStatePath      = "/v1/cluster/internal/drain/"
	PublicInternalAuditACLPath        = "/v1/cluster/internal/audit-acl/"
	PublicInternalClusterLeaderPath   = "/v1/cluster/leader"
	// PublicInternalSecretPath receives peer fan-out of sealed secret blobs
	// (POST upsert) and delete-fanout (DELETE .../{sandboxID}). Auth is PAT +
	// d.Auth like every other /v1/cluster/internal/... route.
	PublicInternalSecretPath = "/v1/cluster/internal/secrets"
	// PublicInternalSandboxAuditPath is the prefix for peer-local secret audit
	// reads. Full path: .../sandboxes/{id}/audit
	PublicInternalSandboxAuditPath = "/v1/cluster/internal/sandboxes/"
	// PublicInternalNodeStorageRetirementsPath serves the replicated operator
	// attestations that a node's storage was destroyed. Obligation owners are
	// workers, which hold no FSM, so they read the set from the server tier
	// rather than from a table on whichever node the operator called.
	PublicInternalNodeStorageRetirementsPath = "/v1/cluster/internal/node-storage-retirements"
	// PublicInternalArtifactCatalogPath serves and accepts the replicated
	// template / JS-bundle metadata catalogue.
	PublicInternalArtifactCatalogPath = "/v1/cluster/internal/artifact-catalog"
	// PublicInternalArtifactCatalogEpochPath allocates ONE publisher its
	// fencing token. It is deliberately separate from the catalogue read: a
	// publisher needs a single number, and serving it from the general page
	// would hand every publisher the fleet's whole coverage list and scan
	// every node's inventory under the FSM lock to produce it.
	PublicInternalArtifactCatalogEpochPath = "/v1/cluster/internal/artifact-catalog/epoch"
)
View Source
const (
	PublicInternalAPITokenWritePath   = "/v1/cluster/internal/tokens/write"
	PublicInternalAPITokenChangesPath = "/v1/cluster/internal/tokens"
	PublicInternalAPITokenPagePath    = "/v1/cluster/internal/tokens/page"
)

API tokens over the cluster (plans/identity-and-access.md §5.4).

create/revoke on any node ─► server tier ─► leader: fv gate ─► opPutAPIToken / opDeleteAPIToken
worker or ingress node ────► TokenMirror polls GET tokens?since=N (304 | delta | resnapshot)
                             └► page walk: GET tokens/page?after=…
server node ───────────────► reads its own FSM (always current)

Writes use a dedicated internal endpoint, like egress profiles, for two reasons: the caller needs the stored record back, and only a leader that has checked every Raft server's build may apply a token op, so an older leader can never commit an op its followers would apply and it wouldn't (or the reverse).

View Source
const (
	// ArtifactKindTemplate / ArtifactKindJSBundle name the two catalogues.
	// Templates are not tenant-scoped, so they publish under the empty
	// tenant; JS bundles carry their owner.
	ArtifactKindTemplate = "template"
	ArtifactKindJSBundle = "js-bundle"

	// MaxArtifactCatalogChunkRows / MaxArtifactCatalogChunkBytes bound ONE
	// raft command. The apply transport caps a request at 1 MiB and the
	// payload is base64-encoded inside the command, so a publisher that sent
	// its whole inventory at once would be refused for an ordinary inventory
	// of small rows — 4,096 of them encode to ~1.6 MB.
	MaxArtifactCatalogChunkRows  = 256
	MaxArtifactCatalogChunkBytes = 512 << 10

	// MaxArtifactCatalogPageRows / artifactCatalogPageByteBudget bound one
	// READ. The cursor carries the rest, so a large catalogue is walked
	// rather than truncated.
	MaxArtifactCatalogPageRows = 2048
)
View Source
const (
	// CreateBackpressureRetryAfterSeconds is the public Retry-After hint for
	// queue/concurrency backpressure. Keep it short: these rejects are caused by
	// transient in-flight create fan-in, not by a long operator action.
	CreateBackpressureRetryAfterSeconds = 5
	CapacityRetryAfterSeconds           = 30
	// InvalidTopologyRetryAfterSeconds is the hint for ErrInvalidTopology
	// rejects, which only clear when an operator reshapes the cluster
	// (promoting servers or adding workers). A long back-off keeps clients
	// from busy-retrying while the human change is in flight.
	InvalidTopologyRetryAfterSeconds = 300
	// LeaderChangeRetryAfterSeconds is the hint for a write that found no
	// reachable leader (IsLeaderUnavailable). An election settles within a
	// few heartbeats, so a short back-off.
	LeaderChangeRetryAfterSeconds = 2
)
View Source
const (
	PublicInternalEgressProfileWritePath = "/v1/cluster/internal/egress-profiles/write"
	PublicInternalEgressProfileReadPath  = "/v1/cluster/internal/egress-profiles/read"
)

Named egress profiles over the cluster (plans/egress-domain-filtering.md D21, CEO D19). Writes go through a dedicated internal endpoint rather than the generic forwarded apply, for two reasons: the caller needs the stored record back (its generation), and a write may only be applied by a leader that has checked every Raft server's build. Only builds with this endpoint serve it, so an older leader can never commit a profile op its followers would apply and it wouldn't.

View Source
const (
	// HeaderForwarded marks a request a peer forwarded on a user's behalf.
	// The receiver runs it locally instead of routing it again, and the
	// auth edge names a forwarded PAT caller "operator via <edge>" instead
	// of "node:<edge>". Like every X-Cluster-* header it is honoured only
	// from a live mTLS peer (pkg/api's clusterControlHeaderGuard).
	HeaderForwarded = "X-Cluster-Forwarded"
	// HeaderTeam selects which of a credential's teams a request acts for
	// (CEO review D13); pkg/api.HeaderTeam is this constant.
	HeaderTeam = "X-AerolVM-Team"
	// HeaderCorrelationID ties the records one API call writes on every
	// node together (H3).
	HeaderCorrelationID = "X-Correlation-ID"
	// HeaderPATOrigin says where the edge accepted a forwarded operator
	// token: PATOriginHost (a loopback peer, or a cluster peer acting for
	// itself) or PATOriginPublic (anything else). An owner under
	// SB_PAT_SCOPE=host refuses a forwarded PAT unless it says host, so one
	// node left on scope "any" cannot launder a remote PAT through a
	// forward (plans/identity-and-access.md §5.11). Like every X-Cluster-*
	// header it is honoured only from a live mTLS peer.
	HeaderPATOrigin = "X-Cluster-PAT-Origin"
	// HeaderSurface marks a forward of a remote MCP tool's inner API call
	// (§5.14). Its one value is auditlog.SurfaceMCP. On that forward, and
	// only there, the owner also accepts the caller's MCP-only access token
	// (an mcp_audiences audience), and records the call with surface mcp.
	// Like every X-Cluster-* header it is honoured only from a live mTLS
	// peer, and it widens where a verified credential is accepted; it never
	// supplies an identity.
	HeaderSurface = "X-Cluster-Surface"
	// HeaderSSHGateway marks a forward of a call the SSH gateway bridged
	// for the holder of a sandbox's key (auditlog.SSHGatewayCall.Encode:
	// the sandbox id, plus the open marker on the session's first call).
	// The owner then names sandbox_key:<id> as the caller, via the edge,
	// instead of the operator whose token the gateway's loopback hop
	// carries, and writes the session's ssh record. Like every X-Cluster-*
	// header it is honoured only from a live mTLS peer, and pkg/api honours
	// it only with the operator token, on the sandbox's sessions routes,
	// for the sandbox the path names.
	HeaderSSHGateway = "X-Cluster-Ssh-Gateway"
	// HeaderRouteHint marks a forward whose target came from the edge's
	// owner cache (owner_cache.go), not from a lookup the edge just made. Its
	// value is a random token the edge made for this one request. The
	// receiver serves the request only when its own lookup says it owns the
	// sandbox; otherwise it answers 421 with HeaderNotOwner set to the same
	// token. Like every X-Cluster-* header it is honoured only from a live
	// mTLS peer, and StampForward sets it only from the request context
	// (ContextWithRouteHint), never from a header that came in.
	HeaderRouteHint = "X-Cluster-Route-Hint"
	// HeaderNotOwner is the response header of that 421 ("not here"). It
	// echoes the request's HeaderRouteHint token. The edge routes the call
	// again only on a 421 carrying its own token, so a 421 from anything
	// else (the sandbox's own server behind a toolbox route, a node on an
	// older build) reaches the caller as it is and never causes a second
	// send. The receiver drops the hint header before any handler runs, so
	// nothing behind it can learn the token.
	HeaderNotOwner = "X-Cluster-Not-Owner"
	// HeaderOwnerEpoch carries the placement's OwnerEpoch the edge read when
	// it chose the target (invariant I2, plan C4 part 4): from a lookup it
	// just made, or from its owner-cache entry. The owner compares it with
	// its own reading and turns down a cache-routed forward that is older
	// (HeaderNotOwner). Like every X-Cluster-* header it is honoured only
	// from a live mTLS peer, and StampForward sets it only from the request
	// context (ContextWithOwnerEpoch).
	HeaderOwnerEpoch = "X-Cluster-Owner-Epoch"
)

Headers a forward of a user's request carries to the peer that will run it (plans/identity-and-access.md §5.2, eng review finding 1.2 and E6).

View Source
const (
	PATOriginHost   = "host"
	PATOriginPublic = "public"
)

PAT origins carried in HeaderPATOrigin.

View Source
const (
	PublicInternalIdentityWritePath = "/v1/cluster/internal/identity/write"
	PublicInternalIdentityReadPath  = "/v1/cluster/internal/identity/read"
)

Cloud identity over the cluster (plans/sandbox-cloud-identity.md D1.3, D1.5). Writes go through a dedicated internal endpoint, like egress profiles, so the caller gets the stored state back and only a leader that checked every Raft server's build applies them. Reads answer from a server's FSM; a worker (no FSM) reads through the server tier.

View Source
const (
	RouteHopEdge  = "edge"
	RouteHopOwner = "owner"
)

Route hops for RecordRouteLookup.

View Source
const (
	DefaultPlacementPageLimit = 1000
	MaxPlacementPageLimit     = 5000
	// MaxReplicatedIngressRouteNodes is the largest ingress tier where every
	// ingress-capable node keeps the full public route table. Ordinary DNS
	// round-robin / TCP load balancers can only work if any advertised ingress
	// node can answer for any sandbox.
	MaxReplicatedIngressRouteNodes = 10
)
View Source
const (
	PublicInternalTeamQuotaWritePath = "/v1/cluster/internal/team-quotas/write"
	PublicInternalTeamQuotaReadPath  = "/v1/cluster/internal/team-quotas/read"
)

Per-team quotas over the cluster (plan item P2). Writes go through a dedicated internal endpoint, like egress profiles: the caller needs the stored record back, and only a leader that has checked every Raft server's build may apply one.

View Source
const (
	PublicInternalWarmPoolWritePath = "/v1/cluster/internal/warm-pools/write"
	PublicInternalWarmPoolReadPath  = "/v1/cluster/internal/warm-pools/read"
)

API warm-pool entries over the cluster (plans/agent-sandbox-feature- parity.md W1).

PUT/DELETE on any node ─► server tier ─► leader: fv gate ─► opPutWarmPool / opDeleteWarmPool
worker with a warm pool ─► polls POST warm-pools/read {since} (unchanged | full list)
server node ─────────────► reads its own FSM

Writes use a dedicated internal endpoint, like egress profiles and API tokens: the caller needs the stored record back, and only a leader that has checked every Raft server's build may apply the op, so an older leader can never commit an op its followers would apply and it wouldn't (or the reverse).

View Source
const (
	// PublicWasmMigratePath is the operator-facing live-migration endpoint (§4.4).
	PublicWasmMigratePath = "/v1/cluster/wasm-migrate"
	// PublicInternalWasmMigratePath prefixes per-sandbox internal export/import routes.
	PublicInternalWasmMigratePath = "/v1/cluster/internal/wasm-migrate/"

	// WasmMigrateCloneGenHeader carries the §4.8 clone-generation token on export.
	WasmMigrateCloneGenHeader = "X-Aerol-Clone-Generation"
	// WasmMigrateTarMediaType is the streamed mem.snap tarball content type.
	WasmMigrateTarMediaType = "application/vnd.aerolvm.wasm-migrate.v1+tar"
)
View Source
const CatchUpRetryAfterSeconds = 1

CatchUpRetryAfterSeconds is the Retry-After for ErrNotCaughtUp. Catch-up is one leader round trip plus the log replay; on the live 3-node cluster the restarted node's listeners were up ~0.5s before its view was.

View Source
const DefaultPlacementShardCount = 16384

DefaultPlacementShardCount is the stable shard space for placement-index reads. The value intentionally stays independent from the number of ingress nodes: ingress ownership changes remap shard IDs to nodes, but a sandbox's shard ID remains stable as the fleet scales up and down.

View Source
const FailoverRefusalReasonPlacement = "placement_unsatisfiable"

FailoverRefusalReasonPlacement names a failover refused because the sandbox's placement rule excludes every live candidate.

View Source
const InternalAPIPath = "/internal/apply"

InternalAPIPath is the URL path served by the cluster-internal mTLS listener. Distinct from the public /v1/cluster/internal/apply route so a misconfigured load balancer can't accidentally tunnel public traffic into the cluster-internal channel.

View Source
const InternalReadIndexPath = "/internal/read-index"

InternalReadIndexPath is served on the cluster-internal mTLS listener: the leader half of the read index. Never 404 on a current build, so a follower can tell a leader that predates it (404) from one that cannot answer yet.

View Source
const MaxMixedClusterNodes = 10

MaxMixedClusterNodes is the largest live cluster size allowed to keep the legacy mixed/hybrid convenience topology. Above this, production clusters must run dedicated server, worker, and ingress tiers.

View Source
const MaxPlacementChangesWait = 25 * time.Second

MaxPlacementChangesWait bounds one long-poll. It stays under the Agent's control-plane request timeout plus slack, so a parked poll is never cut off by the caller's own deadline.

View Source
const MaxServerTierNodes = 7

MaxServerTierNodes caps the live server-role tier. The voter cap (SB_CLUSTER_MAX_AUTO_VOTERS, default 5) bounds how many nodes *vote*, not how many hold state: a surplus server-role node is added as a Raft non-voter and still receives the full log and FSM — at 100k sandboxes that is ~72 MB of placement state plus a leader replication stream each. Scaling out belongs in the worker and ingress tiers, which run Agent and hold no FSM at all, so the control plane stays a small fixed set however large the fleet grows.

7 = the default 5 voters plus two slots, so a rolling replacement can bring a new server up before retiring the old one without tripping the gate. Deliberately a constant, like MaxMixedClusterNodes: an env override here would just re-open the foot-gun it exists to close.

View Source
const PeerNodeIDHeader = "X-Cluster-Peer-Node-ID"

PeerNodeIDHeader is set by production peer callers so the receiver can bind the mTLS leaf to the claimed gossip node id.

View Source
const PeerNotInMembershipMessage = "cluster peer not in membership"

PeerNotInMembershipMessage is the 403 body every peer check (the internal listener, the v1 apply route, the peer middleware) answers a node it does not see alive in gossip. The callers match it to restore ErrMembershipPending, so it is one constant, never retyped.

View Source
const PublicInternalDrainedNodesPath = "/v1/cluster/internal/drained-nodes"

PublicInternalDrainedNodesPath returns the whole drained set in one call.

The per-node route (PublicInternalDrainStatePath) is one control-plane round trip per question. The secret reseal asks about every current holder of every HA sandbox an owner has, on a 30s tick, on every node — at 2,000 workers that is thousands of round trips a second for a set that is almost always empty and changes only when an operator drains something.

View Source
const PublicInternalPlacementChangesPath = "/v1/cluster/internal/placements/changes"

PublicInternalPlacementChangesPath is the long-poll delta feed endpoint.

View Source
const PublicInternalStorageObligationsPath = "/v1/cluster/internal/storage-obligations"

PublicInternalStorageObligationsPath serves the job view to agents (an ingress or worker has no FSM). One call to a server — never a fan-out.

View Source
const (

	// ServerKeyRingFSMOpsVersion: the server key is a ring and sealed blobs
	// may name their key (ServerKeyRingAllowed). No FSM op: the version
	// gates `sandboxd key rotate --stage`, which writes a ring file older
	// builds refuse to boot from.
	ServerKeyRingFSMOpsVersion = 4
)

fsmOpsVersion is the FSM op set this build applies, advertised in gossip meta ("fv"). An old build fails an op it doesn't know, so a leader accepts a write that needs a newer op only once every Raft server reports at least that version (egressProfileWritesAllowed).

fv is also the build's cross-node format version: a feature that writes something an older build cannot read back gates on it the same way. Bump it once per release that adds such a feature, and give the feature its own constant naming the version that introduced it.

Variables

View Source
var (
	// ErrEgressProfileInUse is a delete of a profile placements reference.
	ErrEgressProfileInUse = errors.New("cluster: egress profile is referenced by sandboxes")
	// ErrEgressProfileCapExceeded is a profile change that would push a
	// referencing sandbox past the union hostname cap.
	ErrEgressProfileCapExceeded = errors.New("cluster: egress profile change would exceed a referencing sandbox's hostname cap")
	// ErrClusterVersion is a write that needs an FSM op some Raft server's
	// build doesn't apply yet; it succeeds once the rolling upgrade is done.
	ErrClusterVersion = errors.New("cluster: not every server runs a build that supports this write yet")
)
View Source
var (
	// ErrIdentityKeysConflict is a key-set write whose expected generation
	// is stale: another node wrote first. Re-read and decide again.
	ErrIdentityKeysConflict = errors.New("cluster: identity signing keys changed concurrently")
	// ErrIdentityInvalid is a key set or role the FSM refuses.
	ErrIdentityInvalid = errors.New("cluster: invalid identity write")
)
View Source
var (
	// ErrAPITokenNameTaken is a create whose name another token in the same
	// team already holds (409).
	ErrAPITokenNameTaken = errors.New("cluster: API token name already used in this team")
	// ErrAPITokenIDTaken is a create whose random id collides with another
	// token's: the proposer mints a new id and retries.
	ErrAPITokenIDTaken = errors.New("cluster: API token id already exists")
)
View Source
var (
	// ErrVolumeQuotaExceeded is returned by opUpsertVolume when inserting a new
	// row would exceed the tenant's configured volume-count cap.
	ErrVolumeQuotaExceeded = errors.New("cluster: tenant volume quota exceeded")
	// ErrUnknownVolume is returned by volume reads/deletes when no replicated row
	// exists for the (tenant, id).
	ErrUnknownVolume = errors.New("cluster: unknown volume")
	// ErrVolumeInUse is returned when a delete races with or follows a sandbox
	// attachment that still points at the volume.
	ErrVolumeInUse = errors.New("cluster: volume is still attached")
)
View Source
var ErrArtifactCatalogSuperseded = errors.New("cluster: artifact catalogue publication is superseded")

ErrArtifactCatalogSuperseded reports a publication from a process the catalogue has already moved past. The publisher re-seeds its epoch from the authority and republishes rather than believing it is clean.

View Source
var ErrArtifactNodeUnavailable = fmt.Errorf("cluster: the node holding this artifact is unavailable: %w", ErrNoPlacementTarget)

ErrArtifactNodeUnavailable is ErrNoPlacementTarget's specific form for a request pinned to one node by an artifact it holds (a node-bound js-bundle or built image) when that node is not a live, capacity-reporting member. errors.Is(err, ErrNoPlacementTarget) stays true; the API adds a code so a client can tell "re-create the artifact" from "wait for capacity".

View Source
var ErrCapacityExceeded = errors.New("cluster: target capacity exceeded after pending reservations")

ErrCapacityExceeded is returned when an opReserve apply finds that the chosen target no longer has headroom for the request once concurrent in-flight reservations are added to its gossiped reserved totals. Routers translate this to 503 so the client retries; the next SelectPlacement will see the new reservation and pick a different node.

View Source
var ErrClusterAtCapacity = fmt.Errorf("%w: every worker that can run this sandbox is at capacity (cpu, memory, disk or network slots)", ErrNoPlacementTarget)

ErrClusterAtCapacity is ErrNoPlacementTarget's form when every worker that could run the request is full: its CPU, memory or disk budget is spent, or its network-slot pool for the runtime has no free slot. It stays an ErrNoPlacementTarget so every existing 503 + Retry-After mapping applies; its message says "capacity", so a client reading it can tell a full fleet from a fleet that cannot run the request at all.

View Source
var ErrCreateBackpressure = errors.New("cluster: create backpressure")

ErrCreateBackpressure is returned when the leader-side create queue for a worker is full. This is intentionally distinct from ErrCapacityExceeded: capacity means "pick another worker or wait for resources"; backpressure means "too many creates are already in-flight to this worker, retry shortly."

View Source
var ErrCustomHostnameConflict = errors.New("cluster: custom hostname already in use")

ErrCustomHostnameConflict is returned when an opAddCustomDomain entry asks the FSM to claim a hostname already held by a different sandbox. Maps to the same 409 the local SQLite custom-domains insert returns — the FSM is the cluster-wide tiebreaker that catches the race where two sandboxes on different owners try to claim the same hostname concurrently. The API/service layer surfaces this as models.ErrCustomDomainConflict after reading the raft Apply result.

View Source
var ErrHostPortReserved = errors.New("cluster: tcp host port already reserved")

ErrHostPortReserved is returned when the FSM rejects a raw-TCP exposure because another placement already owns the requested cluster-wide host port. Callers can retry with another candidate instead of surfacing a random expose failure to the user.

View Source
var ErrIncarnationConflict = errors.New("cluster: sandbox incarnation conflict")

ErrIncarnationConflict fences a placement update carrying secret state for a different lifetime of the same sandbox ID.

View Source
var ErrInvalidSecretHandle = errors.New("cluster: invalid secret handle")

ErrInvalidSecretHandle rejects partial, generation-less, or cross-lifecycle provider handles before they can enter replicated placement state.

View Source
var ErrInvalidTopology = errors.New("cluster: invalid topology")

ErrInvalidTopology is returned when the live cluster shape violates a production topology invariant. API layers translate this to 503 so clients retry after the operator fixes membership instead of treating it as a malformed request.

View Source
var ErrLastVoter = errors.New("cluster: cannot remove last raft voter")

ErrLastVoter prevents an explicit removal from deleting the last voting raft server and leaving the cluster with no quorum path.

View Source
var ErrMemberStillAlive = errors.New("cluster: raft member is still alive")

ErrMemberStillAlive protects operators from accidentally removing a live control-plane node. Stop the node first, or pass force through the public lifecycle API when intentionally retiring a live member.

View Source
var ErrMembershipPending = errors.New("cluster: peer membership not yet recognised by the control plane")

ErrMembershipPending means a control-plane server refused this node only because its gossip view does not (yet) list the node as alive.

A restarting node hits this on every boot: its clean shutdown broadcast a gossip Leave, and its first control-plane call lands ~100ms after start — before the server has processed the rejoin. The live hetero run (T18, 2026-09-27) lost a worker permanently to it: enterprise treated the 403 as fatal, each restart re-broadcast the Leave and re-lost the same race, and systemd's restart limit made it final.

View Source
var ErrNameConflict = errors.New("cluster: sandbox name already in use")

ErrNameConflict is returned when the placement FSM rejects an opPlace / opUpsertSpec because a different sandbox already owns the requested Name. Sandbox names are unique cluster-wide; without this check, two concurrent creates landing on different owners could each succeed locally and present ambiguous name-based lookups to facades like Daytona that resolve sandboxes by name. Callers handle this by rolling back the local create and surfacing 409 Conflict to the user.

View Source
var ErrNoEgressGatewayTarget = fmt.Errorf("%w: no node has a ready egress gateway for hostname egress rules", ErrNoPlacementTarget)

ErrNoEgressGatewayTarget is ErrNoPlacementTarget's form for a gateway-mode create (hostname egress rules) when no candidate node has a ready egress gateway (plans/egress-domain-filtering.md CEO D20). It stays an ErrNoPlacementTarget so every existing 503 mapping still applies.

View Source
var ErrNoLeader = errors.New("cluster: timed out waiting for leader")

ErrNoLeader is returned when no leader is seated yet — an election is in flight, or this node has just started and has not learned who the leader is. Distinct from ErrNotLeader ("someone else is the leader"), but in the same class for callers: both mean "ask again shortly", neither means the data is bad.

View Source
var ErrNoPlacementTarget = errors.New("cluster: no worker placement target available")

ErrNoPlacementTarget is returned when no alive worker-capable node can accept a new sandbox. This is distinct from "self wins": a pure server or ingress node must not silently fall back to local Docker ownership.

View Source
var ErrNoReassignTarget = errors.New("cluster: no alternate node available for stuck placement")

ErrNoReassignTarget reports that no live node other than the failing owner can host the placement. The caller keeps retrying where it is rather than orphaning a recoverable sandbox — but it must not be told the placement moved, or it resets the failure counter that drives the escalation.

View Source
var ErrNotCaughtUp = errors.New("cluster: placement view is still catching up after a restart")

ErrNotCaughtUp means this node restarted and its placement view has not yet applied the cluster's read index, so a sandbox missing from it may still exist. Callers answer 503 + Retry-After; it is never evidence of absence.

View Source
var ErrNotLeader = errors.New("cluster: not raft leader")

ErrNotLeader is returned by mutating Cluster operations when this node is not the Raft leader. Callers should retry against the leader (which Client resolves automatically by following raft leadership).

View Source
var ErrOrphanClaimConflict = errors.New("cluster: orphaned placement belongs to a different previous owner")

ErrOrphanClaimConflict is returned when a node tries to reclaim an orphaned placement that was explicitly orphaned from a different previous owner. This prevents a returning node from stealing another dead node's sandbox just because it has a stale local row with the same ID.

View Source
var ErrOrphaned = errors.New("cluster: sandbox owner is dead, placement orphaned")

ErrOrphaned is returned by OwnerOf when a placement exists but its owner has been auto-evicted and the dead-owner reconciler has cleared the pointer. This is the terminal state for default non-HA sandboxes; callers should surface 410 Gone. Sandboxes that opted into failover.policy=recreate are reassigned before the remaining dead-owner placements are orphaned.

View Source
var ErrPeerInternalURLInvalid = errors.New("cluster: peer InternalURL must be an absolute https URL")

ErrPeerInternalURLInvalid is returned before any credentials are attached when gossip advertises a non-HTTPS or malformed internal endpoint.

View Source
var ErrPeerInternalURLRequired = errors.New("cluster: peer InternalURL required (mTLS fail-closed)")

ErrPeerInternalURLRequired is returned when local TLS is configured but the peer has no InternalURL. Callers must not fall back to public APIURL+PAT.

View Source
var ErrPlacementDeleted = errors.New("cluster: sandbox lifecycle was deleted")

ErrPlacementDeleted refuses an opPlace that would write back the placement of a lifetime (sandbox ID + incarnation) the cluster already deleted. It is what a node that was cut off gets when its ownership replay tries to place a sandbox someone deleted meanwhile; that node must destroy its copy.

View Source
var ErrPlacementRulesUnsupported = fmt.Errorf("%w: the control plane runs a build that cannot apply placement rules yet", ErrNoPlacementTarget)

ErrPlacementRulesUnsupported is a create with a placement rule that reached a control plane too old to filter on labels (it answered without saying it checked them). The agent fails closed rather than trust a target picked without the rule; it stays an ErrNoPlacementTarget, so the create is a retryable 503 that succeeds once the server tier is upgraded.

View Source
var ErrPlacementUnsatisfiable = models.ErrPlacementUnsatisfiable

ErrPlacementUnsatisfiable is a create whose placement rule (zone and node labels, plans/placement-rules.md) no live worker's heartbeat satisfies. It is deliberately NOT an ErrNoPlacementTarget: that is a retryable 503, and waiting does not make a node grow a label. The API answers 409 placement_unsatisfiable naming the rule.

View Source
var ErrRecoveryPayloadTooLarge = errors.New("cluster: recovery payload exceeds inline size limit")

ErrRecoveryPayloadTooLarge is returned when a redacted spec + secret handle encodes past inlineRecoveryMaxBytes. It surfaces at create validation (the user-facing 400) and again defensively at command encode time — hitting the latter without the former means a service-side mutation grew the spec after validation, which is a bug, not a routing case.

View Source
var ErrReservationConflict = errors.New("cluster: sandbox already placed or reserved")

ErrReservationConflict is returned when opReserve tries to reserve a sandbox ID that already has a non-expired placement (placed or actively reserved by a different owner). Indicates either a router racing a completed sandbox or a router with a stale view.

View Source
var ErrSecretRecipientsCASMismatch = errors.New("cluster: secret recipients CAS mismatch")

ErrSecretRecipientsCASMismatch is returned when opUpdateSecretRecipients carries expected incarnation / seal-generation pretenses that no longer match the live placement (stale reseal loser).

View Source
var ErrStalePlacementView = errors.New("cluster: this server has lost contact with the raft leader; its placement view may be stale")

ErrStalePlacementView means this server is a follower that has not heard from the raft leader within staleReadBound, so its placement view may be behind what the leader has committed. Callers answer 503 + Retry-After.

View Source
var ErrStuckReassignNotOwner = errors.New("cluster: reassign requester is not the current placement owner")

ErrStuckReassignNotOwner rejects a reassignment request from a node that is not the placement's current owner (or names a superseded incarnation). A stale request from a previous owner must not bounce a live sandbox off the node that has just taken it over.

View Source
var ErrTeamQuotaExceeded = models.ErrTeamQuotaExceeded

ErrTeamQuotaExceeded is a reservation, promote or spec change the leader refused because it would take a team past its quota.

View Source
var ErrUnknownMember = errors.New("cluster: unknown raft member")

ErrUnknownMember is returned when an operator asks to remove a node that is not present in the current raft configuration.

View Source
var ErrUnknownSandbox = errors.New("cluster: unknown sandbox placement")

ErrUnknownSandbox is returned by OwnerOf when no placement record exists for the given sandbox ID. Callers should treat this as "owned locally" only when they have just-created the sandbox and not yet committed its placement.

View Source
var ErrVolumeReclaimMoved = errors.New("cluster: volume reclaim was acked or taken by another node")

ErrVolumeReclaimMoved is a claim of an entry that is gone or no longer held by the node the claimer saw: another node took it, or it was acked.

View Source
var ErrWarmPoolLimit = errors.New("cluster: warm pool limit reached")

ErrWarmPoolLimit is a put of a new entry when models.MaxWarmPools already exist. Decided at apply time, so racing creates resolve the same way on every replica.

Functions

func ApplyErrorRetryAfterSeconds added in v0.6.1

func ApplyErrorRetryAfterSeconds(err error) int

ApplyErrorRetryAfterSeconds is the Retry-After an apply listener sets for a failure that is worth retrying on a delay; 0 means none.

func ApplyErrorStatus added in v0.6.1

func ApplyErrorStatus(err error) int

ApplyErrorStatus is the HTTP status an apply listener answers a failed apply with. It is exported because the v1 apply handler is the other listener and has to answer a given verdict identically.

func AuthenticatedPeerNodeID added in v0.6.1

func AuthenticatedPeerNodeID(r *http.Request) (string, error)

AuthenticatedPeerNodeID returns the node identity authenticated by the request's verified mTLS leaf. Cluster-control headers are not credentials: callers must first prove possession of a CA-signed node certificate, and an optional explicit node claim must match that certificate.

func CanOwnSandboxRole

func CanOwnSandboxRole(role string) bool

CanOwnSandboxRole reports whether a gossiped node role may own sandboxes. Empty is treated as worker-capable for rolling upgrades from builds that did not advertise role metadata.

func CanServeControlPlaneRole

func CanServeControlPlaneRole(role string) bool

CanServeControlPlaneRole reports whether a gossiped node role may host the authoritative Raft/FSM control plane. Empty is treated as server-capable for rolling upgrades from builds that did not advertise role metadata.

func CanServeIngressRole

func CanServeIngressRole(role string) bool

CanServeIngressRole reports whether a gossiped node role may host public ingress routes. Empty is treated as ingress-capable for rolling upgrades from builds that did not advertise role metadata.

func CaughtUp added in v0.6.9

func CaughtUp(c Client) bool

CaughtUp reports whether c may treat a placement missing from its view as absent. Clients without a local replica report true: an Agent's reads go to a server, which applies this gate itself and answers 503 (the Agent then asks the next server); Noop is the whole truth.

func ClientForPeer added in v0.6.1

func ClientForPeer(base *http.Client, expectedNodeID string) *http.Client

ClientForPeer returns an HTTP client that verifies the dialed peer presents DNS SAN node:<expectedNodeID> (in addition to the shared clusterServerName hostname check). Empty expected or a non-TLS base is a no-op.

func ContextWithOwnerEpoch added in v0.6.12

func ContextWithOwnerEpoch(ctx context.Context, epoch uint64) context.Context

ContextWithOwnerEpoch records the owner epoch the edge read for the forwards made under ctx. 0 records nothing.

func ContextWithPATOrigin added in v0.6.8

func ContextWithPATOrigin(ctx context.Context, origin string) context.Context

ContextWithPATOrigin records where the API edge accepted the operator token of this request, for StampForward to carry to the peer.

func ContextWithRouteHint added in v0.6.12

func ContextWithRouteHint(ctx context.Context, token string) context.Context

ContextWithRouteHint marks the forwards made under ctx as routed from the owner cache, with token as the request's HeaderRouteHint.

func DeadOwnerGrace added in v0.6.12

func DeadOwnerGrace(cfg config.Config) time.Duration

DeadOwnerGrace is how long the leader waits, after it first sees a node dead, before it moves or orphans that node's sandboxes (SB_DEAD_OWNER_GRACE).

func DecodeSandboxNameKey added in v0.6.5

func DecodeSandboxNameKey(key string) (ownerRef, name string, qualified bool)

DecodeSandboxNameKey splits a nameIndex key into its owner and user name. qualified is false for a plain key. A key that starts with the prefix but isn't exactly what QualifiedSandboxName would produce (a legacy name that began with "owner:" before the prefix was reserved) also reports false and decodes to itself: decoding never fails, it only declines to rewrite.

func DecodeSpecName added in v0.6.5

DecodeSpecName returns spec with an owner-qualified Name turned back into the user's name. This is the choke point for recreating a local sandbox from a replicated spec: the local store keeps user names, unique per owner.

func ExportWasmMigrateLocal added in v0.5.3

func ExportWasmMigrateLocal(export func(w io.Writer) (string, error)) ([]byte, string, error)

ExportWasmMigrateLocal buffers export output for tests.

func ExposedPortRoutesForPlacement

func ExposedPortRoutesForPlacement(p Placement) map[int]ExposedPortRoute

func ExtractPeerNodeID added in v0.6.1

func ExtractPeerNodeID(cert *x509.Certificate) string

ExtractPeerNodeID returns the node-bound identity from a peer certificate. Only DNS/URI SAN `node:<id>` is authoritative; common-name and shared-SAN fallbacks are intentionally unsupported.

func ForwardedOwnerEpoch added in v0.6.12

func ForwardedOwnerEpoch(h http.Header) uint64

ForwardedOwnerEpoch reads HeaderOwnerEpoch from a forward. 0: absent or malformed (an edge on an older build), which the owner treats as unknown.

func GossipPATScope added in v0.6.8

func GossipPATScope(scope string) string

GossipPATScope is the value a node gossips for its SB_PAT_SCOPE: "host", or "" for anything else (the default "any").

func IngressRingVersion added in v0.6.1

func IngressRingVersion(members []Member) string

IngressRingVersion fingerprints the ordered set of ingress-eligible node IDs a ring was computed from. Equal versions mean two nodes will compute the same owners for every sandbox; different versions mean they are mid- convergence and their answers may disagree.

func IsControlPlaneUnavailable added in v0.6.1

func IsControlPlaneUnavailable(err error) bool

IsControlPlaneUnavailable reports whether err means only that the control plane could not serve this node at this instant — no leader seated, or the server not yet seeing this node in gossip. Neither says anything about the data being validated, so neither may be treated as fatal at boot.

func IsLeaderUnavailable added in v0.6.1

func IsLeaderUnavailable(err error) bool

IsLeaderUnavailable reports whether an error means only that leadership was not reachable at this instant.

This is the distinction that decides whether a caller may treat a failure as fatal. A boot-time validation that exits(1) on ErrNoLeader turns a routine election into a node that never starts — and with systemd's restart limit, never starts AGAIN. Leadership being momentarily unsettled is the normal state of a cluster that is starting up; it is not evidence that anything is wrong with what was being validated.

func IsLivePeer added in v0.6.1

func IsLivePeer(c Client, nodeID string) bool

IsLivePeer reports whether nodeID is present and alive in the caller's current gossip view. It deliberately uses the local view: authenticating a forwarded request must not recurse through another cluster HTTP request.

func IsMixedArchitectureRole added in v0.2.2

func IsMixedArchitectureRole(role string) bool

IsMixedArchitectureRole reports whether a role value runs more than one production tier. Empty legacy metadata is treated as mixed because old nodes behaved as server+worker+ingress.

func LargeClusterTopologyError added in v0.2.2

func LargeClusterTopologyError(members []Member) error

LargeClusterTopologyError enforces the production topology contract: clusters above MaxMixedClusterNodes must have dedicated server, worker, and ingress tiers. Small clusters keep the legacy convenience behavior.

func LiveMemberCount added in v0.2.2

func LiveMemberCount(members []Member) int

LiveMemberCount counts live members with a usable node ID.

func MaxArtifactCatalogRowsPerNode added in v0.6.1

func MaxArtifactCatalogRowsPerNode() int

MaxArtifactCatalogRowsPerNode is the publisher-side cap on one node's whole inventory of a kind. A node over it does not publish at all: it stays uncovered and the aggregator keeps asking it, which is slower but never wrong.

func MintIncarnationID added in v0.6.1

func MintIncarnationID() (string, error)

MintIncarnationID returns a 16-byte random hex string that uniquely tags a sandbox placement lifetime. Reassign preserves the value; every fresh reserve or unreserved placement mints a new one.

func OwnerEpochFromContext added in v0.6.12

func OwnerEpochFromContext(ctx context.Context) uint64

OwnerEpochFromContext is the epoch ContextWithOwnerEpoch recorded, or 0.

func OwnershipLease added in v0.6.12

func OwnershipLease(cfg config.Config) time.Duration

OwnershipLease is how long a node may keep acting on placement or ownership information it has not confirmed with the control plane: half the dead-owner grace. The leader cannot move a sandbox until a node has looked dead for the whole grace, so a node that stops trusting what it last heard at half the grace stops before any move it has not heard about. The other half is the margin for clocks, loop ticks and the time gossip takes to call a node dead.

func PATOriginFromContext added in v0.6.8

func PATOriginFromContext(ctx context.Context) string

PATOriginFromContext is the origin ContextWithPATOrigin recorded, or "".

func PeerDial added in v0.6.1

func PeerDial(m Member, internalClient *http.Client) (client *http.Client, base string, err error)

PeerDial selects the peer HTTP client and internal URL. Cluster peer RPCs are always mTLS and fail closed when either side is missing.

Prefer PeerDialCached (or Cluster/Agent.PeerDialMember) so mTLS VerifyPeer transport clones are reused per node.

func PeerDialCached added in v0.6.1

func PeerDialCached(m Member, internalClient, cached *http.Client) (client *http.Client, base string, err error)

PeerDialCached is PeerDial with an optional prebuilt per-peer mTLS client. When cached is non-nil it is returned as-is (no transport clone). Otherwise a VerifyPeerCertificate client is built via ClientForPeer.

func PeerDialPath added in v0.6.1

func PeerDialPath(m Member, internalClient *http.Client, path string) (*http.Client, string, error)

PeerDialPath is PeerDial plus a path suffix (e.g. /v1/cluster/internal/secrets).

func PlacementShardForSandbox

func PlacementShardForSandbox(sandboxID string, count int) int

PlacementShardForSandbox maps sandboxID to a stable shard ID in [0, count).

func PlacementViewFresh added in v0.6.12

func PlacementViewFresh(c Client) bool

PlacementViewFresh reports whether c may answer a placement read from its local view. Clients without a local replica report true: an Agent's reads go to a server, which applies this gate itself; Noop is the whole truth.

func PostWasmMigrateImport added in v0.5.3

func PostWasmMigrateImport(ctx context.Context, c Client, target Member, sandboxID, cloneGen string, body io.Reader) error

PostWasmMigrateImport pushes a mem.snap tarball to target.

func QualifiedSandboxName added in v0.6.5

func QualifiedSandboxName(ownerRef, name string) string

QualifiedSandboxName returns the nameIndex key for a sandbox called name owned by ownerRef. Operator sandboxes (ownerRef "") and unnamed sandboxes keep their plain name. A name that already carries the reserved prefix is returned unchanged, which makes the encoder idempotent: a spec read back from the FSM (SpecOf) and replayed through UpsertSpec keeps its key, and a legacy name that predates the reservation keeps the key it was indexed under.

func QualifySpecName added in v0.6.5

func QualifySpecName(spec *models.CreateSandboxRequest, ownerRef string) *models.CreateSandboxRequest

QualifySpecName returns spec with Name rewritten to its nameIndex key for ownerRef. It returns spec itself when nothing changes (operator, unnamed, already qualified), so the operator create path allocates nothing; when it rewrites, it returns a shallow copy and never mutates the caller's spec.

func RecordMTLSUnknownPeer added in v0.6.1

func RecordMTLSUnknownPeer()

RecordMTLSUnknownPeer increments when an inbound mTLS peer id is not an Alive gossip member. Soft mode warns via this metric; enterprise rejects.

func RecordNotCaughtUpReject added in v0.6.9

func RecordNotCaughtUpReject()

RecordNotCaughtUpReject counts one request answered 503 because this node's placement view was still catching up (it would have been a false 404).

func RecordOwnerCacheHit added in v0.6.12

func RecordOwnerCacheHit()

RecordOwnerCacheHit counts one forward routed from an owner-cache entry.

func RecordOwnerCacheMiss added in v0.6.12

func RecordOwnerCacheMiss()

RecordOwnerCacheMiss counts one routed call that found no usable entry.

func RecordOwnerCacheNotOwner added in v0.6.12

func RecordOwnerCacheNotOwner()

RecordOwnerCacheNotOwner counts one cache-routed forward this node turned down because its own lookup does not say it owns the sandbox.

func RecordOwnerCacheRetry added in v0.6.12

func RecordOwnerCacheRetry()

RecordOwnerCacheRetry counts one cache-routed call the target turned down and the edge routed again from a fresh lookup.

func RecordOwnerForwardStale

func RecordOwnerForwardStale()

func RecordOwnerLeaseServe added in v0.6.12

func RecordOwnerLeaseServe()

RecordOwnerLeaseServe counts one forward the owner served from its lease reading, with no lookup (E3b).

func RecordRouteLookup added in v0.6.12

func RecordRouteLookup(hop string)

RecordRouteLookup counts one control-plane read the router made on hop.

func RecordRouteRequest added in v0.6.12

func RecordRouteRequest()

RecordRouteRequest counts one per-sandbox call a client sent to this node.

func RecordStalePlacementReadReject added in v0.6.12

func RecordStalePlacementReadReject()

RecordStalePlacementReadReject counts one internal placement read this server answered 503 because it had lost contact with the leader.

func RouteHintFromContext added in v0.6.12

func RouteHintFromContext(ctx context.Context) string

RouteHintFromContext is the token ContextWithRouteHint recorded, or "".

func SandboxNameFromKey added in v0.6.5

func SandboxNameFromKey(key string) string

SandboxNameFromKey returns the user-visible name for a nameIndex key.

func SelectSecretRecipients added in v0.6.1

func SelectSecretRecipients(sandboxID string, candidates []Member, ownerID string, maxBackups int) []string

SelectSecretRecipients builds the seal recipient set: owner first, then up to maxBackups other candidates chosen by deterministic rendezvous hashing over (sandboxID, nodeID). Sorting by NodeID alone hotspot-loads the lexicographically first workers; HRW spreads backup rows evenly while remaining stable for a given sandbox. When the cluster is smaller than owner+maxBackups, every eligible candidate is included. Empty ownerID or candidates yields nil.

func SetPeerNodeIDHeader added in v0.6.1

func SetPeerNodeIDHeader(req *http.Request, selfNodeID string)

SetPeerNodeIDHeader attaches this node's id for inbound withInternalMTLS checks.

func SnapshotHomeKey added in v0.6.8

func SnapshotHomeKey(owner, name string) string

SnapshotHomeKey is the key under which a snapshot's home node is recorded in the cluster index. It folds in the owning tenant so one tenant registering a same-named snapshot on another node cannot overwrite another tenant's entry and hijack its routing (cross-tenant #11). An empty owner (operator/legacy snapshots) uses the bare name, so existing entries and operator snapshots are unchanged. The index stores it as an opaque string, so the FSM is untouched.

func SnapshotImageHomeKey added in v0.6.9

func SnapshotImageHomeKey(image string) string

SnapshotImageHomeKey is the second key a committed snapshot's home is recorded under: its committed image tag (docker.SnapshotImageTagForOwner), in canonical spelling. A create that names the tag rather than the snapshot (an SDK passing snap.Image) carries no snapshot name to build SnapshotHomeKey from, so this key is how a node that does not hold the snapshot finds the one that does.

The tag's hash already folds in the owner, so the key is tenant-scoped as it stands. It never collides with a SnapshotHomeKey: a snapshot may not be named inside the tag namespace (the service refuses it), and an owner-qualified key carries a NUL that a tag never does. "" for a ref outside the namespace.

func SnapshotLocalOnlyKey added in v0.6.12

func SnapshotLocalOnlyKey(owner, name string) string

SnapshotLocalOnlyKey is the key a snapshot's home is also recorded under while its image exists only on that node: its registry push has not landed yet (pending, running or failing), or the cluster does not push snapshots. A create by snapshot name that reaches another node must then run on the home, because the registry copy that node would pull is not there. The home drops the key once the push lands, and creates spread through the registry copy again.

The leading NUL keeps it apart from every other key: a name key starts with the owner or the name, and a tag key carries no NUL. "" for an empty name.

func SoleRaftServer added in v0.6.12

func SoleRaftServer(c Client) bool

SoleRaftServer reports whether c's node is the only server in its Raft configuration. Nothing commits without such a node, so no delete can have happened while it was away. Clients without a local Raft report false.

func StampForward added in v0.6.8

func StampForward(out *http.Request, access controlplane.Access)

StampForward marks out, an outbound request that carries a user's request to a peer, as a forward, and copies what the peer needs to treat it as the same call. Every path that forwards a user's request calls it: the reverse-proxy Director behind ForwardHTTP (clusterroute, clustercreate, the runloop forwardToNode, js-bundle and template routing) and every fan-out client that builds a fresh request (the clusterlist merge, image build routing, template item routing, the per-worker catalogue lists). A call-site test in pkg/api fails the build on a new peer request that skips it.

It never supplies an identity. The peer re-authenticates the original credential, which travels with the request, and everything stamped here only narrows or annotates what that credential already allows:

  • HeaderForwarded, so the peer runs the request locally and records the edge as Via (from the verified mTLS peer, not from this header).
  • HeaderTeam, for a team-bound credential (an avm_ token or an OIDC token): the team the edge resolved, taken from the verified access rather than copied from the raw header. A fresh fan-out request dropped the header, so a multi-team caller's list was answered for its first team on every peer (E6). The peer re-applies the selection and refuses a team the credential is not bound to.
  • HeaderCorrelationID, from the request context the API edge filled, so the peer's records share the edge's id instead of minting their own.
  • HeaderPATOrigin, for an operator-token caller: where the edge accepted the PAT, from ContextWithPATOrigin. A PAT forward with no recorded origin says public, the value an owner under host scope refuses: an unknown origin must never pass as the host.
  • HeaderSurface, when the request is a remote MCP tool's inner call (the in-process transport marked its context with auditlog.SurfaceMCP; nothing on the wire can). Without it the owner would refuse an MCP-only access token and record the call without surface mcp. Any other request has the header removed, so the marker always follows the context and never a header that came in with the request.
  • HeaderSSHGateway, when the API edge found the request to be a call the SSH gateway bridged for a sandbox key holder (the context carries auditlog.SSHGatewayCall). Removed from every other request, the same way as HeaderSurface.
  • HeaderRouteHint, when the edge routed the request from its owner cache (ContextWithRouteHint). Removed from every other request, the same way as HeaderSurface.
  • HeaderOwnerEpoch, the owner epoch the edge read for its target (ContextWithOwnerEpoch). Removed from every other request.

It also marks the edge's api record as forwarded (auditlog.MarkForwarded): only the node that runs the handler records the call, so the peer writes the one record and the edge writes none (§5.9, "Forwarding").

A new forward marker is added here, so no forward path can miss it.

func StreamWasmMigrateExport added in v0.5.3

func StreamWasmMigrateExport(ctx context.Context, c Client, owner OwnerInfo, sandboxID string, w io.Writer) (cloneGen string, err error)

StreamWasmMigrateExport pulls a mem.snap tarball from owner into w.

func ValidateRecoveryPayloadSize added in v0.5.6

func ValidateRecoveryPayloadSize(sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

ValidateRecoveryPayloadSize reports whether the recovery record for (sandboxID, spec, secrets) fits in a raft log entry. Exported so the service layer can reject oversized specs with a clean validation error before any admission/container work happens. Encodes the exact bytes the FSM's recovery store persists, so validation and apply cannot disagree.

func VerifyPeerNodeID added in v0.6.1

func VerifyPeerNodeID(cert *x509.Certificate, expected string) bool

VerifyPeerNodeID reports whether expected matches the peer cert's node-bound identity. Empty expected is a no-op (returns true). Used by withInternalMTLS when the caller supplies an expected peer id.

func WithTeamQuotaAdmission added in v0.6.12

func WithTeamQuotaAdmission(ctx context.Context) context.Context

WithTeamQuotaAdmission marks ctx so the next opPlace it carries is admitted against the sandbox's team quota (QuotaAdmit): the create path sets it for a promote that had no reservation. The flag rides the command, so it reaches the leader through any number of forwards. A spec write is checked without it; the resize path still sets it, which changes nothing.

Types

type APITokenWriteRequest added in v0.6.8

type APITokenWriteRequest struct {
	Delete bool                 `json:"delete,omitempty"`
	Token  identity.TokenRecord `json:"token"`
	ID     string               `json:"id,omitempty"`
}

APITokenWriteRequest creates a token (Delete false, Token set) or revokes one (Delete true, ID set). A create carries the secret's hash only.

type APITokenWriteResponse added in v0.6.8

type APITokenWriteResponse struct {
	Token   identity.TokenRecord `json:"token"`
	Code    string               `json:"code,omitempty"`
	Message string               `json:"message,omitempty"`
}

APITokenWriteResponse is the outcome. Code is set, and Token empty, when the FSM or the upgrade gate refused the write.

func APITokenWriteOutcome added in v0.6.8

func APITokenWriteOutcome(err error) APITokenWriteResponse

APITokenWriteOutcome turns a write error into the body a peer gets, and err back into the same error, so sentinels survive forwarding.

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent is the worker/ingress-side cluster client. It deliberately does not start Raft, does not create a placement FSM, and never joins the raft configuration as a non-voter. It gossips identity/addresses, serves local capacity heartbeats, and delegates every authoritative placement read/write to server-role nodes.

func NewAgent

func NewAgent(cfg config.Config, logger *slog.Logger, admitter *capacity.Admitter) (*Agent, error)

func (*Agent) AckVolumeReclaim added in v0.6.12

func (a *Agent) AckVolumeReclaim(ctx context.Context, r VolumeReclaim) error

func (*Agent) AddCustomDomain added in v0.3.1

func (a *Agent) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

AddCustomDomain forwards to the cluster Apply pipe. hostname is canonicalized here (the public Cluster wrapper does the same on the other path); the FSM then enforces cluster-wide uniqueness.

func (*Agent) AddExposedPort

func (a *Agent) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

func (*Agent) AllocateArtifactCatalogEpoch added in v0.6.1

func (a *Agent) AllocateArtifactCatalogEpoch(ctx context.Context, kind, _ string, holder string) (int64, error)

AllocateArtifactCatalogEpoch asks the control plane to issue THIS node's fencing token. The server fills the node from the authenticated peer identity, so a node can only ever be issued its own.

It deliberately does not go through the catalogue page: a publisher needs one number, and the page answers with every publisher's coverage (tens of kilobytes at fleet scale) after scanning every node's inventory under the FSM read lock.

func (*Agent) ApplyEncoded

func (a *Agent) ApplyEncoded(ctx context.Context, payload []byte) error

func (*Agent) ArtifactCatalog added in v0.6.1

func (a *Agent) ArtifactCatalog(ctx context.Context, req ArtifactCatalogRequest) (ArtifactCatalogPage, error)

ArtifactCatalog reads one page from the server tier. An unreachable control plane is an error: the caller falls back to the fan-out rather than reporting a tenant's artifacts as absent.

func (*Agent) AssertOwnership

func (a *Agent) AssertOwnership(ctx context.Context, local []LocalSandboxState) error

func (*Agent) AttachInternalHandler

func (a *Agent) AttachInternalHandler(h http.Handler)

func (*Agent) AttachRecreator added in v0.6.1

func (a *Agent) AttachRecreator(r SandboxRecreator)

AttachRecreator wires the service-layer recreate hook used by the worker owner watcher. It mirrors Cluster.AttachRecreator so pkg/daemon attaches the same hook whatever role the node runs — a dedicated worker previously had no such method, so the daemon's interface probe silently skipped it and the only automatic recreation loop in the product belonged to *Cluster.

func (*Agent) AuditACLForSandbox added in v0.6.1

func (a *Agent) AuditACLForSandbox(ctx context.Context, sandboxID, incarnationID string) (AuditACL, bool, error)

func (*Agent) AuditCallerWritesAllowed added in v0.6.8

func (a *Agent) AuditCallerWritesAllowed() bool

AuditCallerWritesAllowed reports whether this node may write the audit caller field. Workers and ingress nodes gate on the same gossip view.

func (*Agent) AuditOwnerRef added in v0.6.1

func (a *Agent) AuditOwnerRef(ctx context.Context, sandboxID string) (string, bool, error)

func (*Agent) AuthoritativeNodeStorageRetirements added in v0.6.1

func (a *Agent) AuthoritativeNodeStorageRetirements(ctx context.Context) ([]NodeStorageRetirement, error)

AuthoritativeNodeStorageRetirements asks the control plane for the LEADER's answer. A worker discharging an obligation without an ACK is doing something irreversible, so it must not be authorized by a server whose FSM has not yet applied the operator's revoke.

func (*Agent) AuthoritativePlacementsByIDs added in v0.6.1

func (a *Agent) AuthoritativePlacementsByIDs(ctx context.Context, ids []string) (map[string]Placement, error)

func (*Agent) BeginDeletePlacementExact added in v0.6.1

func (a *Agent) BeginDeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Agent) CancelReservation

func (a *Agent) CancelReservation(ctx context.Context, sandboxID string) error

CancelReservation releases a pending reservation, fenced by its incarnation. The read that decides it goes to the leader. A plain lookup reaches a control-plane member chosen at random, and a follower answers it from its own FSM, which may not have applied a reservation proposed a few milliseconds earlier: a worker whose forwarded create failed fast (400 "image is required") then read "no such reservation", cancelled nothing, and the name stayed held for ReservationTTL, so every retry got "409 sandbox name already in use cluster-wide". Cluster.CancelReservation fixed the same race for raft members; an agent has no FSM of its own, so every cancel takes the authoritative read. If the leader cannot be asked, the error is returned and the caller's retract path logs it; the expiry sweep remains the safety net.

func (*Agent) ClaimOrphan

func (a *Agent) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Agent) ClaimVolumeReclaim added in v0.6.12

func (a *Agent) ClaimVolumeReclaim(ctx context.Context, r VolumeReclaim, nodeID string) error

func (*Agent) ClientForPeer added in v0.6.1

func (a *Agent) ClientForPeer(nodeID string) *http.Client

ClientForPeer returns a cached mTLS HTTP client that verifies DNS SAN node:<nodeID>. Legacy shared-SAN-only peer certs are rejected.

func (*Agent) Close

func (a *Agent) Close() error

func (*Agent) CustomDomainsOf added in v0.3.1

func (a *Agent) CustomDomainsOf(sandboxID string) []string

CustomDomainsOf returns the hostnames bound to sandboxID. Agent doesn't run the placement FSM locally, so this rides the same remote placement lookup the existing ExposedPortsOf path uses — failure modes match: a transient network blip yields nil, which the ingress reconciler treats as "no custom matchers right now" until the placement subscription wakes a re-read.

func (*Agent) DeletePlacement

func (a *Agent) DeletePlacement(ctx context.Context, sandboxID string) error

func (*Agent) DeletePlacementExact added in v0.6.1

func (a *Agent) DeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Agent) DeleteSecretOnPeers added in v0.6.1

func (a *Agent) DeleteSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, generation int64) (acked []string, err error)

func (*Agent) DeleteVolumeAttachmentsForSandbox added in v0.5.4

func (a *Agent) DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID, incarnationID string) error

func (*Agent) DestroyAtAllowed added in v0.6.12

func (a *Agent) DestroyAtAllowed() bool

DestroyAtAllowed is the same gate on a worker or ingress node.

func (*Agent) DrainedNodes added in v0.6.1

func (a *Agent) DrainedNodes() map[string]bool

DrainedNodes returns the control plane's drained set, cached for drainedNodesTTL. On a failed refresh it keeps serving the last good set: forgetting a drain because one request failed would put a secret copy straight back onto the node being evacuated.

func (*Agent) ExposedPortsOf

func (a *Agent) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

func (*Agent) FetchSandboxAuditFromPeer added in v0.6.1

func (a *Agent) FetchSandboxAuditFromPeer(ctx context.Context, nodeID, sandboxID string, limit int, cursor, kind, incarnationID, caller string) (AuditPeerPage, error)

FetchSandboxAuditFromPeer GETs a peer's local audit slice (worker agent).

func (*Agent) FetchTokenChanges added in v0.6.8

func (a *Agent) FetchTokenChanges(ctx context.Context, since uint64) ([]byte, error)

FetchTokenChanges is the mirror's poll (identity.TokenFeed), on the existing control-plane client: any live server, retried on the next one if a server answers 503. A 304 is identity.ErrTokensUnchanged.

func (*Agent) FetchTokenPage added in v0.6.8

func (a *Agent) FetchTokenPage(ctx context.Context, after string, limit int) ([]byte, error)

FetchTokenPage is one page of the mirror's page walk (identity.TokenFeed).

func (*Agent) ForgetSnapshotNode added in v0.6.7

func (a *Agent) ForgetSnapshotNode(ctx context.Context, snapshotName string) error

func (*Agent) ForwardHTTP

func (a *Agent) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

func (*Agent) IdentityAllowed added in v0.6.12

func (a *Agent) IdentityAllowed() bool

IdentityAllowed is the same gate on a worker or ingress node.

func (*Agent) IngressTargets added in v0.3.1

func (a *Agent) IngressTargets() models.IngressTarget

IngressTargets aggregates live ingress-role members' PublicHost values. Identity and role are gossip facts, so this reads the local view and only falls back to the control plane when gossip has nothing yet — the aggregate is a handful of hostnames, not a reason to redistribute capacity snapshots for the whole fleet.

func (*Agent) IsNodeDrained

func (a *Agent) IsNodeDrained(nodeID string) bool

func (*Agent) Leader

func (a *Agent) Leader() string

func (*Agent) ListAPITokens added in v0.6.8

func (a *Agent) ListAPITokens(ctx context.Context) ([]identity.TokenRecord, error)

ListAPITokens page-walks the server tier, so a list right after a create on this node shows it (the local mirror may be one poll behind).

func (*Agent) LocalMembers added in v0.6.1

func (a *Agent) LocalMembers() []Member

LocalMembers returns the gossip view only — never the control-plane HTTP members call. Used by list failover_ready so a page of sandboxes does not trigger N membership RPCs.

func (*Agent) LookupMember added in v0.6.1

func (a *Agent) LookupMember(id string) (Member, bool)

LookupMember resolves one peer by ID via the local gossip index (O(1)), falling back to a membership scan only when the index misses.

func (*Agent) Members

func (a *Agent) Members() []Member

func (*Agent) NodeStorageRetirements added in v0.6.1

func (a *Agent) NodeStorageRetirements(ctx context.Context) ([]NodeStorageRetirement, error)

NodeStorageRetirements reads the replicated set from the server tier. An unavailable control plane is an ERROR, never an empty set: discharging an obligation is irreversible, and so is failing to discharge one that the operator attested.

This is the DISCOVERY read, answerable by any server. A caller about to act irreversibly uses AuthoritativeNodeStorageRetirements.

func (*Agent) OwnerCache added in v0.6.12

func (a *Agent) OwnerCache() *OwnerCache

OwnerCache returns this worker or ingress node's owner cache, or nil when SB_CLUSTER_OWNER_CACHE is off.

func (*Agent) OwnerOf

func (a *Agent) OwnerOf(sandboxID string) (OwnerInfo, error)

func (*Agent) OwnerOfName

func (a *Agent) OwnerOfName(ownerRef, name string) (string, OwnerInfo, error)

OwnerOfName runs the per-owner name lookup (name_key.go) over the control plane's raw placement-by-name endpoint: the owner-qualified key first, then the plain key, accepted only when the placement belongs to ownerRef. The endpoint's raw-key contract is unchanged, so a worker on this binary gets the same answer from a control-plane voter that hasn't upgraded yet.

func (*Agent) OwnerOfNameKey added in v0.6.5

func (a *Agent) OwnerOfNameKey(key string) (string, OwnerInfo, error)

OwnerOfNameKey resolves one raw nameIndex key through the control plane.

func (*Agent) PeerDialMember added in v0.6.1

func (a *Agent) PeerDialMember(m Member) (*http.Client, string, error)

PeerDialMember selects the peer client/URL using the per-node mTLS cache.

func (*Agent) PeerInternalHTTPClient added in v0.6.1

func (a *Agent) PeerInternalHTTPClient() *http.Client

PeerInternalHTTPClient exposes the cert-pinned client for cluster list fan-out.

func (*Agent) PlacementOf

func (a *Agent) PlacementOf(sandboxID string) (Placement, bool)

func (*Agent) PlacementPage

func (a *Agent) PlacementPage(req PlacementPageRequest) PlacementPageResponse

PlacementPage reads one page of the placement index from the control plane. Concurrent identical reads share one control-plane request (freshFlight): every GET /v1/sandboxes on an ingress or worker makes one, and on the bank flagship 512 concurrent lists on one ingress made 512 concurrent mTLS requests to the servers, whose handshakes starved the node. A shared answer still comes from a read that began after the caller's own request, so coalescing adds no staleness. Each caller gets its own copy of the rows slice (some redact fields in place); the rows' own slices are shared and must not be modified.

func (*Agent) PlacementVersion

func (a *Agent) PlacementVersion() uint64

func (*Agent) Placements

func (a *Agent) Placements() []Placement

func (*Agent) PlacementsByIDs added in v0.6.1

func (a *Agent) PlacementsByIDs(ids []string) map[string]Placement

PlacementsByIDs batch-looks up IDs via a single control-plane POST. Prefer this over Placements() when only a page of IDs is needed.

func (*Agent) PlacementsForShards

func (a *Agent) PlacementsForShards(filter PlacementShardFilter) []Placement

PlacementsForShards reads this node's slice of the placement view, PAGED.

It used to be one unbounded request. A minimal 100k-placement answer encodes to ~28 MB, well past the agent's 16 MB JSON response ceiling, so the read failed and a cold agent fell back to an empty view — and the unfiltered endpoint does have a production caller: an ingress tier at or below MaxReplicatedIngressRouteNodes reaches it through an all-shards filter. Paging the page endpoint keeps every response bounded by MaxPlacementPageLimit rows instead of raising the ceiling.

func (*Agent) ProbeSecretOnPeers added in v0.6.1

func (a *Agent) ProbeSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, minGeneration int64) (holding []string, err error)

func (*Agent) PruneAuditACL added in v0.6.1

func (a *Agent) PruneAuditACL(context.Context, time.Time) error

func (*Agent) PublishArtifactCatalog added in v0.6.1

func (a *Agent) PublishArtifactCatalog(ctx context.Context, chunk ArtifactCatalogSnapshot) error

PublishArtifactCatalog forwards one chunk to the control plane.

func (*Agent) PushSecretBlobToAnyPeer added in v0.6.1

func (a *Agent) PushSecretBlobToAnyPeer(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

func (*Agent) PushSecretBlobToPeers added in v0.6.1

func (a *Agent) PushSecretBlobToPeers(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

Agent mirrors for worker nodes that seal locally and need to fan out.

func (*Agent) PutVolumeAttachments added in v0.5.4

func (a *Agent) PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error

func (*Agent) ReadEgressProfiles added in v0.6.6

func (a *Agent) ReadEgressProfiles(ctx context.Context, req EgressProfileReadRequest) (EgressProfileReadResponse, error)

ReadEgressProfiles reads profiles from the server tier. A dedicated worker holds no FSM, so point reads go through a cache refreshed every egressProfileCacheTTL (the poll the plan describes, done on demand by the re-apply pass and creates): a narrowed profile reaches a worker's sandboxes within egressProfileCacheTTL plus one re-apply interval. If the server tier can't be reached, entries up to egressProfileCacheMaxStale old still answer; past that the read fails and the caller fails closed.

func (*Agent) ReadIdentity added in v0.6.12

func (a *Agent) ReadIdentity(ctx context.Context, req IdentityReadRequest) (IdentityReadResponse, error)

ReadIdentity reads from the server tier. Point reads of roles (the create check and the refresher) go through a cache refreshed every identityRoleCacheTTL, like egress profiles: a tag an admin removed stops new creates and refreshes on a worker within that TTL. Past identityRoleCacheMaxStale with the server tier unreachable, the read fails and the caller fails closed. Key reads are not cached here: the service caches the parsed signer.

func (*Agent) ReadTeamQuota added in v0.6.12

func (a *Agent) ReadTeamQuota(ctx context.Context, req TeamQuotaReadRequest) (TeamQuotaReadResponse, error)

ReadTeamQuota reads a team's quota and usage from the server tier. A worker holds no FSM, so a start check (the hot caller) is answered from a short cache; the decisions that must be exact are the leader's.

func (*Agent) ReadWarmPools added in v0.6.12

func (a *Agent) ReadWarmPools(ctx context.Context, req WarmPoolReadRequest) (WarmPoolReadResponse, error)

ReadWarmPools reads the entries from the server tier. A dedicated worker holds no FSM; it polls this with the last Version it applied, so a quiet fleet costs each poll one small "unchanged" answer.

func (*Agent) ReassignPlacement added in v0.5.3

func (a *Agent) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

func (*Agent) RecordPlacement

func (a *Agent) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Agent) RegisterSnapshotNode added in v0.6.7

func (a *Agent) RegisterSnapshotNode(ctx context.Context, snapshotName, nodeID string) error

func (*Agent) RemoveCustomDomain added in v0.3.1

func (a *Agent) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

func (*Agent) RemoveExposedPort

func (a *Agent) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

func (*Agent) RemoveMember added in v0.2.2

func (a *Agent) RemoveMember(ctx context.Context, nodeID string, force bool) error

func (*Agent) ReportStorageObligations added in v0.6.1

func (a *Agent) ReportStorageObligations(ctx context.Context, report StorageObligationReport) error

ReportStorageObligations submits one owner report through the control plane.

func (*Agent) ReserveOnTarget

func (a *Agent) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

func (*Agent) ResolveCustomDomain added in v0.3.1

func (a *Agent) ResolveCustomDomain(hostname string) (string, bool)

ResolveCustomDomain returns ("", false) on agent nodes. A reverse-by-hostname lookup endpoint isn't wired through yet (the TLS-ask handler is intended to live alongside a node that runs raft locally — server/ingress roles). See task #17 for adding a remote lookup if agent-role ingress becomes a supported topology.

func (*Agent) RetireNodeStorage added in v0.6.1

func (a *Agent) RetireNodeStorage(ctx context.Context, nodeID, actor, reason string, attestedAt time.Time) error

RetireNodeStorage forwards the attestation to the control plane. Agents hold no FSM, so the command rides the same leader-forwarding path as every other write an agent issues.

func (*Agent) RevokeNodeStorageRetirement added in v0.6.1

func (a *Agent) RevokeNodeStorageRetirement(ctx context.Context, nodeID string) error

RevokeNodeStorageRetirement withdraws the attestation cluster-wide.

func (*Agent) SSHSessionOwnerRecordAllowed added in v0.6.9

func (a *Agent) SSHSessionOwnerRecordAllowed() bool

SSHSessionOwnerRecordAllowed is the same gate on a worker or ingress node, which is where most gateways run.

func (*Agent) SecretsOf

func (a *Agent) SecretsOf(sandboxID string) PlacementSecrets

func (*Agent) SelectPlacement

func (a *Agent) SelectPlacement(req capacity.Request) (PlacementTarget, error)

SelectPlacement asks the control plane for a target and nothing else.

func (*Agent) SelectPlacementForCreate added in v0.6.1

func (a *Agent) SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

SelectPlacementForCreate picks a target and the sandbox's seal recipients in one control-plane round trip, returning the bounded recipient ids instead of the candidate fleet.

func (*Agent) SelectPlacementWithCandidates added in v0.6.1

func (a *Agent) SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

func (*Agent) SelfAPIURL

func (a *Agent) SelfAPIURL() string

func (*Agent) SelfNodeID

func (a *Agent) SelfNodeID() string

func (*Agent) ServerKeyRingAllowed added in v0.6.9

func (a *Agent) ServerKeyRingAllowed() bool

ServerKeyRingAllowed is the same gate on a worker or ingress node.

func (*Agent) SetNodeDrainState

func (a *Agent) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

func (*Agent) SnapshotHomeNode added in v0.6.7

func (a *Agent) SnapshotHomeNode(ctx context.Context, snapshotName string) (string, bool, error)

func (*Agent) SpecOf

func (a *Agent) SpecOf(sandboxID string) *models.CreateSandboxRequest

func (*Agent) StorageObligations added in v0.6.1

func (a *Agent) StorageObligations(ctx context.Context) ([]StorageObligationView, error)

StorageObligations asks one control-plane server for the view.

func (*Agent) SubscribePlacement

func (a *Agent) SubscribePlacement(ctx context.Context) <-chan struct{}

SubscribePlacement wakes the ingress reconciler on placement changes. Only the delta feed can provide that on an Agent; without it (flag off), this is nil and the reconciler runs on its timer, as before.

func (*Agent) TokenPortsAllowed added in v0.6.12

func (a *Agent) TokenPortsAllowed() bool

TokenPortsAllowed is the same gate on a worker or ingress node.

func (*Agent) UpdatePlacementSecretRecipients added in v0.6.1

func (a *Agent) UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

func (*Agent) UpsertSpec

func (a *Agent) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Agent) VolumeAttachmentCount added in v0.5.4

func (a *Agent) VolumeAttachmentCount(ctx context.Context, tenant, id string) (int, error)

func (*Agent) VolumeAttachmentsForSandbox added in v0.6.12

func (a *Agent) VolumeAttachmentsForSandbox(ctx context.Context, sandboxID string) ([]models.VolumeAttachment, error)

func (*Agent) VolumeByID added in v0.5.4

func (a *Agent) VolumeByID(ctx context.Context, tenant, id string) (models.Volume, error)

func (*Agent) VolumeByName added in v0.5.4

func (a *Agent) VolumeByName(ctx context.Context, tenant, name string) (models.Volume, error)

func (*Agent) VolumeDelete added in v0.5.4

func (a *Agent) VolumeDelete(ctx context.Context, tenant, id string) error

func (*Agent) VolumeDeleteOwingBytes added in v0.6.12

func (a *Agent) VolumeDeleteOwingBytes(ctx context.Context, tenant, id, source string) error

func (*Agent) VolumeExistsForSource added in v0.5.4

func (a *Agent) VolumeExistsForSource(ctx context.Context, source string) (bool, error)

func (*Agent) VolumeReclaimsFor added in v0.6.12

func (a *Agent) VolumeReclaimsFor(ctx context.Context, nodeID string, limit int) ([]VolumeReclaim, error)

VolumeReclaimsFor asks a server, which answers from its replica.

func (*Agent) VolumeReclaimsReplicated added in v0.6.12

func (a *Agent) VolumeReclaimsReplicated() bool

VolumeReclaimsReplicated on a worker or ingress node: every live member, any role, runs a build with the ops (the agent cannot read the Raft configuration, so it asks the stricter question).

func (*Agent) VolumeUpsert added in v0.5.4

func (a *Agent) VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)

func (*Agent) VolumesForTenant added in v0.5.4

func (a *Agent) VolumesForTenant(ctx context.Context, tenant string) ([]models.Volume, error)

func (*Agent) WatchPlacementChanges added in v0.6.1

func (a *Agent) WatchPlacementChanges(ctx context.Context, fn PlacementChangeHandler) bool

WatchPlacementChanges registers fn with the delta feed. It returns false when the feed is off (flag off). fn first gets the current full view, if already synced.

func (*Agent) WriteAPIToken added in v0.6.8

func (a *Agent) WriteAPIToken(ctx context.Context, req APITokenWriteRequest) (APITokenWriteResponse, error)

WriteAPIToken sends a token write to the server tier; any server takes it and forwards it to the leader.

func (*Agent) WriteEgressProfile added in v0.6.6

WriteEgressProfile sends a profile write to the server tier; any server takes it and forwards it to the leader.

func (*Agent) WriteIdentity added in v0.6.12

func (a *Agent) WriteIdentity(ctx context.Context, req IdentityWriteRequest) (IdentityWriteResponse, error)

WriteIdentity sends an identity write to the server tier; any server takes it and forwards it to the leader.

func (*Agent) WriteTeamQuota added in v0.6.12

func (a *Agent) WriteTeamQuota(ctx context.Context, req TeamQuotaWriteRequest) (TeamQuotaWriteResponse, error)

WriteTeamQuota sends a quota write to the server tier; any server takes it and forwards it to the leader.

func (*Agent) WriteWarmPool added in v0.6.12

func (a *Agent) WriteWarmPool(ctx context.Context, req WarmPoolWriteRequest) (WarmPoolWriteResponse, error)

WriteWarmPool sends an entry write to the server tier; any server takes it and forwards it to the leader.

type ArtifactCatalogEpochRequest added in v0.6.1

type ArtifactCatalogEpochRequest struct {
	Kind string `json:"kind"`
	// Holder identifies the requesting process. A retry from the same holder
	// is answered with the token it already has rather than a new one.
	Holder string `json:"holder"`
}

ArtifactCatalogEpochRequest asks the authority for one publisher token. The node is NOT a body field: the server fills it from the authenticated peer identity, so a node can only ever be issued its own.

type ArtifactCatalogEpochResponse added in v0.6.1

type ArtifactCatalogEpochResponse struct {
	Epoch int64 `json:"epoch"`
}

ArtifactCatalogEpochResponse carries exactly the issued token.

type ArtifactCatalogPage added in v0.6.1

type ArtifactCatalogPage struct {
	Rows []ArtifactCatalogRow `json:"rows"`
	// Publishers are the nodes whose inventory of this KIND the catalogue
	// holds in full. The aggregator asks everyone else. It does not depend on
	// the tenant being read: a node that published an inventory containing
	// nothing for this tenant has still answered for it.
	Publishers []string `json:"publishers"`
	// NextPageToken continues the walk. Publishers are complete on the first
	// page; rows are not, so a caller that needs every row pages to the end.
	NextPageToken string `json:"next_page_token,omitempty"`
	// Authoritative distinguishes "nothing published yet" from "could not
	// ask". A non-authoritative answer must fall back to the fan-out.
	Authoritative bool `json:"authoritative,omitempty"`
	// PublisherEpoch is the epoch the catalogue holds for the node that
	// ASKED. A publisher takes the next one, which is how a restarted process
	// outranks requests its predecessor may still have in flight. It is
	// filled from the authenticated peer identity on the peer path, never
	// from the request body.
	PublisherEpoch int64 `json:"publisher_epoch,omitempty"`
}

ArtifactCatalogPage is what a reader gets back.

type ArtifactCatalogRequest added in v0.6.1

type ArtifactCatalogRequest struct {
	Kind      string `json:"kind"`
	Tenant    string `json:"tenant,omitempty"`
	Limit     int    `json:"limit,omitempty"`
	PageToken string `json:"page_token,omitempty"`
	// ForNodeID asks for that node's publisher epoch alongside the page. The
	// peer handler overrides it with the authenticated identity; a node may
	// only ever ask for its own.
	ForNodeID string `json:"for_node_id,omitempty"`
}

ArtifactCatalogRequest is the agent-facing read.

type ArtifactCatalogRow added in v0.6.1

type ArtifactCatalogRow struct {
	ID string `json:"id"`
	// Tenant scopes the row inside its kind. Empty means "not tenant-scoped"
	// (templates). It lives on the row, not on the publication, because
	// coverage has to be answerable for a tenant the publisher holds nothing
	// for.
	Tenant  string `json:"tenant,omitempty"`
	Payload []byte `json:"payload"`
}

ArtifactCatalogRow is one artifact's metadata as its holder serialized it. The FSM does not interpret Payload — the API layer owns the wire type — so adding a field to models.Template cannot require an FSM change.

type ArtifactCatalogSnapshot added in v0.6.1

type ArtifactCatalogSnapshot struct {
	Kind string `json:"kind"`
	// NodeID is the publisher. Callers never supply another node's id: the
	// peer-facing path takes it from the authenticated identity.
	NodeID string `json:"node_id"`
	// Epoch is the AUTHORITY-ISSUED fencing token for the publishing process:
	// a node reads the catalogue's committed epoch at boot and takes the next
	// one. A process identifier alone (a UUID) identifies a process without
	// ordering processes, so a request delayed in transport from a replaced
	// process could take ownership back and republish an obsolete inventory.
	Epoch int64 `json:"epoch"`
	// Revision orders publications WITHIN one epoch.
	Revision int64                `json:"revision"`
	Rows     []ArtifactCatalogRow `json:"rows"`
	// First starts a new snapshot (discarding any half-delivered one) and
	// Final commits it. A single-chunk snapshot sets both.
	First bool `json:"first,omitempty"`
	Final bool `json:"final,omitempty"`
	// Withdraw removes this node's coverage instead of replacing it. A node
	// whose inventory cannot be represented — over the per-node cap — must
	// say so: the aggregator skips a node it believes it covers, so leaving
	// the old coverage in place advertises an inventory the node no longer
	// has and nothing ever asks it again.
	Withdraw bool `json:"withdraw,omitempty"`
}

ArtifactCatalogSnapshot is one node's whole inventory of one kind, as the publisher wants it to become. It is delivered as a sequence of chunks and applied all-or-nothing.

func ChunkArtifactCatalogSnapshot added in v0.6.1

func ChunkArtifactCatalogSnapshot(kind, nodeID string, epoch, revision int64, rows []ArtifactCatalogRow) []ArtifactCatalogSnapshot

ChunkArtifactCatalogSnapshot splits an inventory into commands that fit the apply transport. The sequence is always non-empty: an EMPTY inventory is a real statement ("this node holds nothing of this kind"), and losing it is what keeps a tenant's list asking every worker forever.

func WithdrawArtifactCatalogCoverage added in v0.6.1

func WithdrawArtifactCatalogCoverage(kind, nodeID string, epoch, revision int64) ArtifactCatalogSnapshot

WithdrawArtifactCatalogCoverage builds the publication that removes this node's coverage of a kind.

type AuditACL added in v0.6.1

type AuditACL struct {
	SandboxID           string   `json:"sandbox_id"`
	IncarnationID       string   `json:"incarnation_id,omitempty"`
	OwnerRef            string   `json:"owner_ref"`
	AuditNodeIDs        []string `json:"audit_node_ids,omitempty"`
	AuditNodesTruncated bool     `json:"audit_nodes_truncated,omitempty"`
	ExpiresUnix         int64    `json:"expires_unix,omitempty"`
	// RetainedVersion is the Raft log index of the delete that created this
	// row. It provides a deterministic latest-lifecycle ordering without wall
	// clocks or a second replicated table.
	RetainedVersion uint64 `json:"retained_version"`
}

AuditACL is the minimal post-delete existence/authorization and evidence-node record retained in Raft. OwnerRef may be empty for operator-owned local workflows. It deliberately carries no spec, routes, secret refs, or events.

type AuditACLResponse added in v0.6.1

type AuditACLResponse struct {
	ACL    AuditACL `json:"acl"`
	Exists bool     `json:"exists"`
}

type AuditCallerGate added in v0.6.8

type AuditCallerGate interface {
	AuditCallerWritesAllowed() bool
}

AuditCallerGate is implemented by cluster clients that can tell whether every member can read the audit caller field. The service layer asserts it rather than adding it to Client: the single-node Noop does not implement it, so a cluster-mode service that has not attached its real client yet stays closed.

type AuditEventDTO added in v0.6.1

type AuditEventDTO = auditlog.Event

AuditEventDTO is the wire shape of one secret-audit event on the peer internal endpoint. Kept in cluster to avoid an import cycle with service.

type AuditPeerFetcher added in v0.6.1

type AuditPeerFetcher interface {
	FetchSandboxAuditFromPeer(ctx context.Context, nodeID, sandboxID string, limit int, cursor, kind, incarnationID, caller string) (AuditPeerPage, error)
}

AuditPeerFetcher is the narrow seam Service uses for audit fan-out so tests can inject a fake without a full Cluster.

type AuditPeerPage added in v0.6.1

type AuditPeerPage struct {
	Events     []AuditEventDTO `json:"events"`
	NextCursor string          `json:"next_cursor,omitempty"`
}

AuditPeerPage is the JSON body returned by the internal audit endpoint.

type CatchUpReporter added in v0.6.9

type CatchUpReporter interface {
	PlacementViewCaughtUp() bool
}

CatchUpReporter is implemented by clients that answer placement reads from a local replica that can lag the cluster after a restart (*Cluster).

type Client

type Client interface {
	// SelfNodeID returns this node's stable cluster identifier.
	SelfNodeID() string
	// SelfAPIURL returns this node's externally-reachable API base URL.
	SelfAPIURL() string

	// OwnerOf returns the node currently owning sandboxID, or
	// ErrUnknownSandbox if no placement record exists.
	OwnerOf(sandboxID string) (OwnerInfo, error)

	// OwnerOfName resolves a sandbox name within ownerRef's namespace to its
	// sandbox ID and current owner. Names are unique per owner and indexed
	// from replicated specs (see name_key.go); ownerRef "" is the operator
	// namespace. Returns ErrUnknownSandbox if no placement in that namespace
	// claims name.
	OwnerOfName(ownerRef, name string) (string, OwnerInfo, error)

	// OwnerOfNameKey resolves one raw nameIndex key with no owner filtering.
	// Only the cluster-internal placement-by-name endpoint uses it; callers
	// that act for a user go through OwnerOfName.
	OwnerOfNameKey(key string) (string, OwnerInfo, error)

	// SelectPlacement chooses a node to host a new sandbox with the given
	// resource request. In single-node mode it always returns self.
	SelectPlacement(req capacity.Request) (PlacementTarget, error)

	// SelectPlacementWithCandidates is SelectPlacement plus the filtered
	// candidate slice used to build the secret recipient set at reserve time.
	// Callers that only need the winner keep using SelectPlacement.
	SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

	// RecordPlacement commits sandboxID -> self into the FSM along with the
	// (optional) creation spec used to re-materialize the sandbox after a
	// failover. Idempotent — re-recording with the same owner is a no-op, and
	// passing spec=nil preserves any spec that was previously replicated for
	// this id (so a boot-time replay can't erase a richer record written by
	// the original CreateSandbox call).
	//
	// spec MUST be redacted (no plaintext credentials) before being passed in;
	// secrets is the provider handle produced by the service layer. Passing an
	// empty handle preserves any previously replicated ref (mirrors the
	// spec-preservation rule — a boot-time replay that has the spec but not
	// the secrets can't erase the original create's recoverability handle).
	RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

	// ClaimOrphan promotes an orphaned placement back to self. It succeeds
	// only when the placement is currently orphaned and either has no recorded
	// previous owner (legacy row) or was orphaned from this node. It preserves
	// any replicated spec/secrets when the caller passes nil.
	ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

	// UpsertSpec replaces the replicated spec for sandboxID without touching
	// ownership. Mutating handlers (resize, lifecycle) call this after a
	// successful local mutation so the FSM stays current — otherwise a
	// failover-recreated sandbox would revert to its create-time shape.
	//
	// spec MUST be redacted. An empty secrets handle preserves the previously
	// replicated ref — resize/lifecycle never touch credentials, so passing an
	// empty handle is the right choice for those callers; the "real" secrets
	// are only re-shipped when the user re-runs create or rotates them
	// explicitly.
	UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

	// SelectPlacementForCreate is SelectPlacement plus the seal recipients for
	// sandboxID, chosen where the membership already lives. It exists so a
	// create never has to ship the candidate fleet to the caller: at 2,000
	// nodes SelectPlacementWithCandidates makes every create's answer O(fleet)
	// in bytes and allocations to produce at most a handful of node ids.
	// recipientBackups <= 0 means the create wants no fan-out and skips the
	// selection entirely.
	SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

	// UpdatePlacementSecretRecipients replaces Placement.SecretRecipients
	// (and optionally the seal handle after a reseal) without touching
	// ownership or IncarnationID. expectedIncarnationID / expectedOwnerNodeID /
	// expectedSealGeneration are CAS pretenses: when the live placement
	// differs, the FSM rejects with ErrSecretRecipientsCASMismatch.
	//
	// expectedOwnerNodeID is not redundant with the other two. opReassign
	// changes the owner while preserving both the incarnation and the seal
	// generation, so without it a node that has just lost the lifecycle can
	// still land a reseal it started as owner. Pass the node this caller
	// believes coordinates the secret: itself when it owns the placement, or
	// "" for the leader-coordinated ownerless case (the empty expectation is
	// still an expectation — the FSM compares it).
	UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

	// SpecOf returns the most-recently-replicated CreateSandboxRequest for
	// sandboxID, or nil if no spec is recorded (pre-cluster sandbox, or no
	// placement). The returned spec is REDACTED — no plaintext secrets. Use
	// SecretsOf to retrieve the matching provider handle if you need to
	// reconstruct the full spec for a recreate.
	SpecOf(sandboxID string) *models.CreateSandboxRequest

	// SecretsOf returns the secret provider handle paired with SpecOf's spec.
	// Empty when the sandbox was created without any private-registry / mount
	// credentials, or when the placement predates secret replication.
	SecretsOf(sandboxID string) PlacementSecrets

	// AddExposedPort records intent that sandboxID has port exposed. Raw TCP
	// routes include HostPort so every ingress node can bind/proxy the same
	// cluster-wide endpoint. Idempotent when the route metadata is unchanged.
	AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

	// RemoveExposedPort drops a port intent from the placement. Idempotent.
	RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

	// ExposedPortsOf returns a copy of the replicated port route map for
	// sandboxID, or nil if none. Used by the recreator and ingress reconciler.
	ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

	// AddCustomDomain records intent that sandboxID owns hostname (already
	// canonicalized lower-case) cluster-wide. Returns ErrCustomHostnameConflict
	// when a different sandbox already holds it. Idempotent for the same
	// (sandbox, hostname) pair so retries are safe.
	AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

	// RemoveCustomDomain drops hostname from sandboxID's set. No-op when the
	// hostname isn't claimed by this sandbox (the local row may already be
	// gone after a Delete) so callers can retry without special-casing.
	RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

	// CustomDomainsOf returns a copy of the replicated hostname set for
	// sandboxID, or nil when no placement exists or the set is empty. The
	// ingress reconciler uses it to populate Caddy matchers on every node.
	CustomDomainsOf(sandboxID string) []string

	// ResolveCustomDomain answers "which sandbox owns this hostname?" from
	// the local FSM in O(1). Used by the TLS-ask handler on ingress nodes
	// that may not own the sandbox row themselves. hostname is matched
	// case-insensitively. Returns sandboxID, true when known.
	ResolveCustomDomain(hostname string) (string, bool)

	// DeletePlacement removes sandboxID from the FSM. Idempotent.
	DeletePlacement(ctx context.Context, sandboxID string) error
	// DeletePlacementExact removes only the placement lifecycle still owned by
	// expectedOwnerNodeID. It is the destroy/finalizer primitive: a delayed
	// cleanup must not erase an ID-reused or concurrently reassigned placement.
	DeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error
	// BeginDeletePlacementExact atomically freezes an exact owner/lifecycle
	// before irreversible secret and external-artifact cleanup. It is
	// idempotent for the same lifecycle and rejects reassignment races.
	BeginDeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error
	// AuditOwnerRef resolves the bounded, short-grace routing stub Raft keeps
	// for a deleted sandbox (owner_ref + evidence nodes). The stub exists only
	// for SB_AUDIT_DELETED_GRACE and under SB_AUDIT_DELETED_INDEX_MAX; the
	// durable authorization record is the evidence node's sandbox_audit_acl
	// row, and long-term history is the export backend's. PruneAuditACL
	// removes expired stubs through a deterministic Raft command.
	AuditOwnerRef(ctx context.Context, sandboxID string) (string, bool, error)
	// AuditACLForSandbox resolves an exact retained lifecycle when
	// incarnationID is non-empty, or the newest retained lifecycle otherwise.
	AuditACLForSandbox(ctx context.Context, sandboxID, incarnationID string) (AuditACL, bool, error)
	PruneAuditACL(ctx context.Context, cutoff time.Time) error

	// ReserveOnTarget writes opReserve into the FSM holding capacity + name
	// for target before the body is forwarded to it. The reservation carries
	// the redacted spec + secret ref so the target's RecordPlacement promote
	// step can run with nil Spec/empty Secrets and inherit them
	// atomically. ttl bounds how long the reservation can hold capacity if
	// the target never promotes (the leader's GC sweep cancels expired rows
	// at ~5s tick cadence). Spec MUST already be redacted — the raft log
	// must NOT carry plaintext credentials.
	ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

	// CancelReservation drops a pending reservation from the FSM. No-op on
	// missing or already-promoted (Placed) rows, so router rollback / TTL GC
	// / late successful promote can race harmlessly.
	CancelReservation(ctx context.Context, sandboxID string) error

	// SetNodeDrainState marks nodeID as drained (excluded from
	// SelectPlacement) or restores it to the candidate pool. Idempotent on
	// both edges. The mark lives in the FSM and survives the drained node
	// going away — the operator's intent must outlast the process they're
	// about to stop.
	SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

	// ReassignPlacement moves sandboxID to target without touching the
	// replicated spec or port intents. Used by WASM live migration once the
	// receiving node has imported a §4.8.1 checkpoint.
	ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

	// Platform-volume metadata replication. These keep a Daytona volume's
	// id/name/source consistent cluster-wide so get/list/delete-by-id survive the
	// tenant's API ownership moving between nodes. The data itself is
	// deterministic in S3/NFS and is never replicated.
	//
	// VolumeUpsert is an idempotent get-or-create: a duplicate (tenant, name)
	// converges on the existing row. It returns the canonical row and whether
	// this call created it, or ErrVolumeQuotaExceeded.
	VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)
	// VolumeDelete removes the (tenant, id) row, or returns ErrUnknownVolume.
	VolumeDelete(ctx context.Context, tenant, id string) error
	// VolumeByID / VolumeByName resolve a single row or return ErrUnknownVolume.
	VolumeByID(ctx context.Context, tenant, id string) (models.Volume, error)
	VolumeByName(ctx context.Context, tenant, name string) (models.Volume, error)
	// VolumesForTenant lists the tenant's rows, newest first.
	VolumesForTenant(ctx context.Context, tenant string) ([]models.Volume, error)
	// VolumeExistsForSource reports whether any replicated row points at source
	// (the reclaim worker's cluster-wide live-data guard).
	VolumeExistsForSource(ctx context.Context, source string) (bool, error)
	// VolumeAttachmentCount returns the cluster-wide live attachment count for a
	// volume. Put/Delete keep this index in the same raft log as volume metadata
	// so Daytona delete cannot miss a sandbox that lives on another worker.
	VolumeAttachmentCount(ctx context.Context, tenant, id string) (int, error)
	// VolumeAttachmentsForSandbox lists the attachments one sandbox holds
	// (a destroy finds its delete_with_sandbox volumes with it). Read-only;
	// an agent asks a server, which answers from its replicated FSM.
	VolumeAttachmentsForSandbox(ctx context.Context, sandboxID string) ([]models.VolumeAttachment, error)
	PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error
	DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID, incarnationID string) error

	// RegisterSnapshotNode records which node holds a snapshot's image, so a
	// snapshot-by-id GET/DELETE that lands on another node can be routed to it.
	// Idempotent and move-safe (re-register overwrites). ForgetSnapshotNode
	// drops the entry on delete. SnapshotHomeNode resolves the holder, or
	// returns ok=false when there is no entry (serve locally). Snapshots live in
	// per-node SQLite with no node column, so this replicated index is the only
	// cluster-wide home lookup.
	RegisterSnapshotNode(ctx context.Context, snapshotName, nodeID string) error
	ForgetSnapshotNode(ctx context.Context, snapshotName string) error
	SnapshotHomeNode(ctx context.Context, snapshotName string) (string, bool, error)

	// RemoveMember explicitly removes nodeID from the raft configuration after
	// marking it drained and orphaning any placements it owned. Unknown raft
	// members return ErrUnknownMember. Live members require force=true so an
	// operator cannot accidentally cut out a healthy server by typo.
	RemoveMember(ctx context.Context, nodeID string, force bool) error

	// IsNodeDrained reports whether nodeID is currently marked drained.
	// Reads the local FSM (no network hop). Used by observability endpoints
	// — placement scoring uses an internal accessor that takes one lock for
	// the whole sweep.
	IsNodeDrained(nodeID string) bool

	// ApplyEncoded is the receiving end of leader-forwarded raft writes. The
	// internal API endpoint pipes the request body through here on the leader
	// so any owner-side mutating call (Record/Upsert/Add/Remove/Delete) made on
	// a follower can transparently land on the leader's raft. Returns
	// ErrNotLeader if leadership has shifted; the forwarder retries.
	ApplyEncoded(ctx context.Context, payload []byte) error

	// AssertOwnership ensures the FSM lists self as owner for every entry in
	// local, and backfills any missing Spec / ExposedPorts so failover-recreate
	// works for sandboxes that pre-date the spec-replication features. Used at
	// boot. Idempotent.
	AssertOwnership(ctx context.Context, local []LocalSandboxState) error

	// ForwardHTTP reverse-proxies r to the given peer. Target NodeID and
	// InternalURL are required and the TLS leaf is pinned to node:<NodeID>.
	ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

	// AttachInternalHandler wires the API server's HTTP handler into the
	// cluster-internal mTLS listener so peers can reverse-proxy owner API calls
	// over the cert-pinned channel (not just leader-forwarded raft applies).
	// No-op for Noop and for Cluster instances with SB_CLUSTER_TLS_DIR unset.
	// Safe to call exactly once after construction; subsequent calls overwrite.
	AttachInternalHandler(h http.Handler)

	// Members returns a snapshot of all known cluster members.
	Members() []Member

	// LocalMembers returns the local gossip membership view without a
	// control-plane HTTP round-trip. Hot paths (list failover_ready) prefer
	// this over Members() on agent nodes so a page of sandboxes does not
	// fan out N membership RPCs.
	LocalMembers() []Member

	// IngressTargets aggregates live ingress-role nodes' gossiped PublicHost
	// values into the set of public addresses users must point DNS at for
	// custom domains. Hostnames and raw IPs are partitioned via net.ParseIP;
	// duplicates are removed; output ordering is stable so the API response
	// is byte-identical across calls when membership is unchanged. Source
	// reflects whether the cluster published a hostname, IPs, both, or
	// nothing usable (see models.IngressTargetSource*). Service-layer
	// IngressDNSTarget wraps this; clients hit GET /v1/ingress/dns.
	IngressTargets() models.IngressTarget

	// Placements returns the local FSM's hot placement snapshot. Recovery
	// payloads (Spec/secrets) are omitted; use PlacementOf/SpecOf for point
	// lookups that need them.
	Placements() []Placement

	// PlacementsForShards returns only placements whose sandbox ID belongs to
	// one of the requested placement shards. Ingress nodes use this instead of
	// pulling the full global placement map; server nodes serve it from the
	// FSM shard index and agent nodes delegate it to a server-role control
	// plane peer. Empty Shards means "all placements." Returned rows are hot
	// rows without Spec/secrets.
	PlacementsForShards(filter PlacementShardFilter) []Placement

	// PlacementPage returns a bounded global placement-index page sorted by
	// sandbox ID. It is the scalable read path for operators/control planes
	// that need to enumerate large clusters without asking every worker for
	// its local DB. Returned rows are hot rows without Spec/secrets.
	PlacementPage(req PlacementPageRequest) PlacementPageResponse

	// PlacementOf returns the full Placement record for sandboxID and true,
	// or a zero Placement and false if no record exists. Operator/debug
	// endpoints use this for convergence-status reads where the per-aspect
	// getters (OwnerOf/ExposedPortsOf) would lose the placement.Version
	// needed to compute "is this sandbox's route installed yet on this node".
	PlacementOf(sandboxID string) (Placement, bool)

	// PlacementsByIDs returns hot placement rows for the requested sandbox IDs
	// without dumping the full Placements() view. Missing IDs are omitted.
	// Used by list/get failover_ready batching so a page of N sandboxes does
	// not pay for a full FSM scan.
	PlacementsByIDs(ids []string) map[string]Placement

	// AuthoritativePlacementsByIDs returns the same bounded point set from the
	// current Raft leader. Unlike PlacementsByIDs, an unavailable or changing
	// leader is an error rather than an empty result. Destructive reconcilers
	// use this so a stale follower/control-plane outage cannot masquerade as
	// placement absence and retire live lifecycle data.
	AuthoritativePlacementsByIDs(ctx context.Context, ids []string) (map[string]Placement, error)

	// PlacementVersion is the FSM's monotonic apply counter — bumps on every
	// raft log entry the FSM applied. Exposed for metrics/observability and
	// as a tie-breaker for tests; the ingress reconciler now uses
	// SubscribePlacement to wake on apply rather than polling this counter.
	// Zero means "no version data" (Noop or fresh cluster).
	PlacementVersion() uint64

	// SubscribePlacement returns a buffered (cap=1) channel that receives a
	// signal after every FSM apply on this node. The channel is fed directly
	// from FSM.Apply, so a leader-side commit reaches every node's
	// subscribers as soon as raft delivers the log entry — no poll interval.
	//
	// Multiple applies between reads collapse into one wake (cap=1, drops on
	// full). Cancel ctx (or the returned cancel func) to deregister; both
	// are safe to call multiple times. Callers MUST tolerate spurious wakes —
	// the channel says "something changed in the FSM," not "the placement
	// you care about changed."
	//
	// In single-node mode (Noop) the returned channel never fires; this is
	// safe to use in a select{} alongside a ticker because Go's select treats
	// a never-firing nil channel as permanently un-ready, not an error.
	SubscribePlacement(ctx context.Context) <-chan struct{}

	// Leader returns the node ID of the current Raft leader, empty if none.
	Leader() string

	// Close shuts down the cluster cleanly.
	Close() error
}

Client is the surface the rest of the daemon (Service, API handlers) interacts with. Both *Cluster and *Noop satisfy it so callsites stay unconditional.

type Cluster

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster is the multi-node Client implementation. Construct with New.

func New

func New(cfg config.Config, logger *slog.Logger, admitter *capacity.Admitter) (*Cluster, error)

New constructs the server-role Cluster for cfg.EnableCluster=true. Caller takes ownership of Close. For worker/ingress-only roles call NewAgent; for cfg.EnableCluster=false call NewNoop.

func (*Cluster) APITokenChanges added in v0.6.8

func (c *Cluster) APITokenChanges(since uint64) (identity.TokenChanges, bool)

APITokenChanges serves a mirror's poll. changed is false when the cursor is current: the handler answers 304 with no body.

func (*Cluster) APITokenPage added in v0.6.8

func (c *Cluster) APITokenPage(after string, limit int) identity.TokenPage

APITokenPage serves one page of a mirror's page walk.

func (*Cluster) AckVolumeReclaim added in v0.6.12

func (c *Cluster) AckVolumeReclaim(ctx context.Context, r VolumeReclaim) error

func (*Cluster) AddCustomDomain added in v0.3.1

func (c *Cluster) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

AddCustomDomain replicates a sandbox→hostname binding through raft so every node can answer the TLS-ask probe and install the matching Caddy matcher. hostname is canonicalized (trim + lower) here so callers don't have to remember to do it themselves — the local SQLite layer already trims. Idempotent for the same (sandbox, hostname) pair; returns ErrCustomHostnameConflict when the hostname is held by a different sandbox.

func (*Cluster) AddExposedPort

func (c *Cluster) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

AddExposedPort replicates a port-exposure intent. Idempotent when the route metadata is unchanged. Safe to call from any node.

func (*Cluster) AllocateArtifactCatalogEpoch added in v0.6.1

func (c *Cluster) AllocateArtifactCatalogEpoch(ctx context.Context, kind, nodeID, holder string) (int64, error)

AllocateArtifactCatalogEpoch issues this process its fencing token through the replicated log.

Why an allocation and not a read: reading the committed epoch and locally choosing "one more" claims nothing. Two processes for the same node — a restart overlapping its predecessor, or a node rejoining while its old process drains — both read the same value, both pick the same successor, and the catalogue can no longer order them: whichever has the higher REVISION wins, which is the opposite of what the token is for. Allocating through the log makes every token distinct and monotonic.

func (*Cluster) ApplyEncoded

func (c *Cluster) ApplyEncoded(ctx context.Context, payload []byte) error

ApplyEncoded is the receiving side of leader-forwarded raft writes. It decodes-validates the payload (so a malformed body never reaches the FSM) and applies it locally. Returns ErrNotLeader if leadership has changed since the forwarder picked us — the caller is expected to retry.

func (*Cluster) ArtifactCatalog added in v0.6.1

ArtifactCatalog reads one page of one catalogue from the local FSM.

func (*Cluster) ArtifactCatalogForPeer added in v0.6.1

func (c *Cluster) ArtifactCatalogForPeer(req ArtifactCatalogRequest) ArtifactCatalogPage

ArtifactCatalogForPeer answers the agent-facing read.

func (*Cluster) ArtifactCatalogHolders added in v0.6.1

func (c *Cluster) ArtifactCatalogHolders(kind, tenant, id string) []string

ArtifactCatalogHolders returns the nodes whose committed inventory of kind holds (tenant, id), sorted. A point read for routing one artifact request: the gossip inventory that item routes use first trails a create by a capacity heartbeat, while a creator publishes here before it answers, so "created, then read at once" must fall back to this or 404.

func (*Cluster) AssertOwnership

func (c *Cluster) AssertOwnership(ctx context.Context, local []LocalSandboxState) error

AssertOwnership reconciles local sandbox state against the cluster FSM at boot. Used at boot. Idempotent. Best-effort callers should log errors and retry later.

Three-way decision per local row, with the FSM as the source of truth for ownership (never overwrite an existing non-self owner):

  • **No FSM placement.** Write opPlace claiming self as owner, with spec + secret ref so the new placement is born recoverable. The service only hands over such a row when it is provably not a deleted sandbox (service.settleUnplacedRow destroys the others), and the FSM refuses one whose lifecycle has a post-delete record (ErrPlacementDeleted). This is the case for sandboxes created before AssertOwnership last ran, e.g. on a single node turned into a cluster.

  • **FSM owner == self.** No ownership change. Backfill missing Spec / secret ref via opUpsertSpec (closes the pre-spec-replication gap) and replay ExposedPort intents via opAddExposedPort (no-ops when already recorded).

  • **FSM orphaned from self.** Reclaim via opClaimOrphan, then replay the same spec/port backfill. This is the false-positive dead-owner recovery path: gossip/Raft marked us dead, but the local sandbox never left.

  • **FSM owner != self, or orphaned from another owner.** **Do NOT reclaim.** This is the failover-recovery/stale-local-row case. Calling RecordPlacement here would overwrite another owner or steal a different dead node's orphan. Instead: log loudly and leave the FSM alone. service.reconcileStaleOwnership handles destroying stale local copies on its next pass when the placement has an active different owner.

Edge cases:

  • Owner unknown to gossip yet (fresh boot, gossip not converged): we still defer. Treating "presumed alive" as the safe default keeps us from racing a peer whose announce just hasn't reached us. The next reconcile pass reclassifies.
  • Owner is dead in gossip: still defer until the leader has committed the orphan transition. Claiming before the FSM records which previous owner lost the row would recreate the overwrite hazard.

func (*Cluster) AttachInternalHandler

func (c *Cluster) AttachInternalHandler(h http.Handler)

AttachInternalHandler wires the public API mux into the cluster-internal mTLS listener so peers can reverse-proxy owner API calls over the cert-pinned channel. No-op only for the single-node/noop construction path where there is no listener to attach to. Called once from cmd/sandboxd after the API server is constructed; the order avoids a service→cluster→api construction cycle.

func (*Cluster) AttachRecreator

func (c *Cluster) AttachRecreator(r SandboxRecreator)

AttachRecreator wires the service-layer recreate hook used by the owner watcher. Called once from cmd/sandboxd/main after both service.New and cluster.New have returned. Safe to call concurrently with the watcher loop.

func (*Cluster) AuditACLForSandbox added in v0.6.1

func (c *Cluster) AuditACLForSandbox(_ context.Context, sandboxID, incarnationID string) (AuditACL, bool, error)

func (*Cluster) AuditCallerWritesAllowed added in v0.6.8

func (c *Cluster) AuditCallerWritesAllowed() bool

AuditCallerWritesAllowed reports whether this node may write the audit caller field.

func (*Cluster) AuditOwnerRef added in v0.6.1

func (c *Cluster) AuditOwnerRef(ctx context.Context, sandboxID string) (string, bool, error)

func (*Cluster) AuthoritativeNodeStorageRetirements added in v0.6.1

func (c *Cluster) AuthoritativeNodeStorageRetirements(ctx context.Context) ([]NodeStorageRetirement, error)

AuthoritativeNodeStorageRetirements answers from the leader, forwarding when this node is a follower.

Discharging a deletion obligation without an ACK is irreversible, so it may not be authorized by a lagging replica or a cached set: an operator who revokes an attestation on one node must not have another node still discharging against it. A follower's own FSM says Authoritative=true about its own state and still cannot order itself against the revoke.

Server and mixed nodes need this as much as workers do — they hold delete outboxes of their own, and without it every one of them fails closed forever on an attestation the operator did make.

func (*Cluster) AuthoritativePlacementsByIDs added in v0.6.1

func (c *Cluster) AuthoritativePlacementsByIDs(ctx context.Context, ids []string) (map[string]Placement, error)

AuthoritativePlacementsByIDs serves destructive reconciliation from the current leader. Followers fetch the bounded result over the same node-pinned mTLS channel used for forwarded Raft operations; they never substitute a potentially stale local FSM read.

func (*Cluster) BeginDeletePlacementExact added in v0.6.1

func (c *Cluster) BeginDeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Cluster) CancelReservation

func (c *Cluster) CancelReservation(ctx context.Context, sandboxID string) error

CancelReservation deletes a pending reservation for sandboxID. Safe to call at any rollback point: opCancelReserve only fires when the row is State == Reserved, so a stale cancel after a successful promote is a no-op. Idempotent; calling on a never-reserved id is also a no-op.

func (*Cluster) ClaimOrphan

func (c *Cluster) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

ClaimOrphan commits sandboxID -> self only if the existing placement is currently orphaned and was orphaned from this node (or predates the previous-owner metadata). This is the false-positive recovery path for a node that was marked dead by gossip but never actually lost its local sandbox.

func (*Cluster) ClaimVolumeReclaim added in v0.6.12

func (c *Cluster) ClaimVolumeReclaim(ctx context.Context, r VolumeReclaim, nodeID string) error

func (*Cluster) ClientForPeer added in v0.6.1

func (c *Cluster) ClientForPeer(nodeID string) *http.Client

ClientForPeer returns a cached mTLS HTTP client that verifies DNS SAN node:<nodeID>. Legacy shared-SAN-only peer certs are rejected.

func (*Cluster) Close

func (c *Cluster) Close() error

Close shuts down gossip + raft. Idempotent.

func (*Cluster) CountAPITokens added in v0.6.8

func (c *Cluster) CountAPITokens(now time.Time) map[controlplane.Role]int

CountAPITokens counts this server's unexpired tokens by role, for the aerolvm_api_tokens gauge. Every server holds the whole FSM, so each reports the fleet's count.

func (*Cluster) CustomDomainsOf added in v0.3.1

func (c *Cluster) CustomDomainsOf(sandboxID string) []string

CustomDomainsOf returns a sorted copy of the hostnames bound to sandboxID, or nil when none. Used by the ingress reconciler so a peer-owned sandbox still gets its TLS matchers installed in the local Caddy.

func (*Cluster) DeletePlacement

func (c *Cluster) DeletePlacement(ctx context.Context, sandboxID string) error

DeletePlacement removes sandboxID from the placement map. Idempotent. The expected owner/incarnation come from the local FSM — the same source RemoveCustomDomain uses — so a follower destroy does not POST to the Raft leader before the write. applyCommand still commits on the leader; a stale local row fails the exact CAS instead of 503ing every delete during an election.

func (*Cluster) DeletePlacementExact added in v0.6.1

func (c *Cluster) DeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Cluster) DeleteSecretOnPeers added in v0.6.1

func (c *Cluster) DeleteSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, generation int64) (acked []string, err error)

DeleteSecretOnPeers DELETEs the sandbox's cluster_secrets rows on peers that may hold a fan-out copy. It returns only authenticated acknowledgements; callers derive pending recipients from requested minus acknowledged so an incomplete transport response can never erase a durable delete obligation.

func (*Cluster) DeleteVolumeAttachmentsForSandbox added in v0.5.4

func (c *Cluster) DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID, incarnationID string) error

func (*Cluster) DestroyAtAllowed added in v0.6.12

func (c *Cluster) DestroyAtAllowed() bool

DestroyAtAllowed reports whether every live member understands destroy_at.

func (*Cluster) DrainedNodes added in v0.6.1

func (c *Cluster) DrainedNodes() map[string]bool

DrainedNodes reads the FSM's drained set with no network hop.

func (*Cluster) ExposedPortsOf

func (c *Cluster) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

ExposedPortsOf returns a copy of the replicated port route map. Returns nil if no placement exists or no ports are recorded.

func (*Cluster) FetchSandboxAuditFromPeer added in v0.6.1

func (c *Cluster) FetchSandboxAuditFromPeer(ctx context.Context, nodeID, sandboxID string, limit int, cursor, kind, incarnationID, caller string) (AuditPeerPage, error)

FetchSandboxAuditFromPeer GETs a peer's local audit slice for sandboxID.

func (*Cluster) ForgetSnapshotNode added in v0.6.7

func (c *Cluster) ForgetSnapshotNode(ctx context.Context, snapshotName string) error

ForgetSnapshotNode drops the named snapshot's home entry (on delete).

func (*Cluster) ForwardHTTP

func (c *Cluster) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

ForwardHTTP reverse-proxies r to target and writes the response into w. The PAT bearer header is preserved (httputil.ReverseProxy passes through headers by default; we only strip hop-by-hop ones).

NodeID and InternalURL are required. The proxy transport is taken from the per-peer client cache, which pins the TLS leaf to node:<NodeID>.

Forwarding loops are detected by the X-Cluster-Forwarded header and returned as 421 Misdirected Request so clients retry against a refreshed placement.

IMPORTANT: a hard network/TLS failure on the internal channel must NOT silently fall back to the public path — that would defeat the cert-pinned security promise B3 is meant to enforce. The reverse proxy's ErrorHandler surfaces such failures as 502 to the original caller, who can then retry against a different peer.

func (*Cluster) HealthyForReads

func (c *Cluster) HealthyForReads() bool

HealthyForReads is true once a leader is known. It does NOT mean the FSM has caught up with the leader; PlacementViewCaughtUp is that property.

func (*Cluster) IdentityAllowed added in v0.6.12

func (c *Cluster) IdentityAllowed() bool

IdentityAllowed reports whether every live member, workers and ingress included, understands cloud identity: the create option (an older worker would drop it and run the sandbox with no identity) and the token push. The identity.token_issued audit record needs no gate: it is a new Kind with only existing fields, which every build hashes the same way.

func (*Cluster) IngressTargets added in v0.3.1

func (c *Cluster) IngressTargets() models.IngressTarget

IngressTargets aggregates live ingress-role members' gossiped PublicHost values. See aggregateIngressTargets for the partition / dedup / ordering rules — those are pinned by ingress_targets_test.go.

func (*Cluster) IsNodeDrained

func (c *Cluster) IsNodeDrained(nodeID string) bool

IsNodeDrained reports the FSM's view of nodeID's drain state. Read from the local FSM with no network hop.

func (*Cluster) Leader

func (c *Cluster) Leader() string

Leader returns the node ID of the current raft leader. Empty if no leader.

func (*Cluster) LeaderAPIURL

func (c *Cluster) LeaderAPIURL() string

LeaderAPIURL returns the API URL of the current leader, or empty if unknown. Used by the API wrapper to forward mutating raft writes to the leader.

func (*Cluster) LeaderReadIndex added in v0.6.9

func (c *Cluster) LeaderReadIndex(ctx context.Context) (uint64, error)

LeaderReadIndex is the leader half of the read index: prove leadership and that every committed entry is applied here, then report the FSM version.

func (*Cluster) ListAPITokens added in v0.6.8

func (c *Cluster) ListAPITokens(context.Context) ([]identity.TokenRecord, error)

ListAPITokens returns every stored token, sorted by team and name.

func (*Cluster) LocalMembers added in v0.6.1

func (c *Cluster) LocalMembers() []Member

LocalMembers returns the gossip membership view without capacity enrichment. Hot paths (list failover_ready) use this to avoid per-row Members() work.

func (*Cluster) LookupMember added in v0.6.1

func (c *Cluster) LookupMember(id string) (Member, bool)

LookupMember returns one gossip member by node ID without scanning the full membership list. Used by placement-page list hydration (O(owners)).

func (*Cluster) LookupToken added in v0.6.8

func (c *Cluster) LookupToken(id string) (identity.TokenRecord, bool, bool)

LookupToken answers the request path on a server node from its own FSM. A server is always current (§5.4: revocation is immediate there), so fresh is always true. It satisfies identity.TokenLookup.

func (*Cluster) Members

func (c *Cluster) Members() []Member

Members returns gossip-known members (self included), enriched with fresh capacity heartbeats where available.

func (*Cluster) NodeStorageRetirements added in v0.6.1

func (c *Cluster) NodeStorageRetirements(context.Context) ([]NodeStorageRetirement, error)

NodeStorageRetirements reads the replicated attestation set from the local FSM. Server-role nodes hold the FSM, so this is a local read.

func (*Cluster) NodeStorageRetirementsForPeer added in v0.6.1

func (c *Cluster) NodeStorageRetirementsForPeer() NodeStorageRetirementsResponse

NodeStorageRetirementsForPeer answers the agent-facing read.

func (*Cluster) NodeStorageRetirementsForPeerAuthoritative added in v0.6.1

func (c *Cluster) NodeStorageRetirementsForPeerAuthoritative(ctx context.Context) (NodeStorageRetirementsResponse, error)

NodeStorageRetirementsForPeerAuthoritative is the peer-facing form of the same authoritative read: the barrier belongs to the read, not to the caller, so a worker asking the leader gets the same guarantee a server does.

func (*Cluster) OwnedRecoveryPlacements added in v0.6.1

func (c *Cluster) OwnedRecoveryPlacements(ownerID string, limit int, pageToken string) OwnedRecoveryResponse

OwnedRecoveryPlacements answers an owner-scoped recovery query for ownerID. Only placements that node owns AND that opted into failover recreate are returned, with the spec and secret handle the owner needs to rebuild them.

Callers must pass the mTLS-authenticated peer identity as ownerID; nothing here derives the owner from a request body.

func (*Cluster) OwnerOf

func (c *Cluster) OwnerOf(sandboxID string) (OwnerInfo, error)

OwnerOf reads the placement map from the local FSM (no network round-trip). Returns ErrUnknownSandbox if no row exists, or ErrOrphaned if the placement exists but has no active owner (auto-orphaned after the owning node died — see voter_autojoin / dead-owner reconciler).

func (*Cluster) OwnerOfName

func (c *Cluster) OwnerOfName(ownerRef, name string) (string, OwnerInfo, error)

OwnerOfName resolves a sandbox name within ownerRef's namespace to its placement owner (see name_key.go). This is intentionally a local FSM read just like OwnerOf; the name index is maintained inside the FSM apply path so it tracks the authoritative placement map exactly.

func (*Cluster) OwnerOfNameKey added in v0.6.5

func (c *Cluster) OwnerOfNameKey(key string) (string, OwnerInfo, error)

OwnerOfNameKey resolves one raw nameIndex key, with no owner filtering. It backs the cluster-internal placement-by-name endpoint, which worker agents call once per key (see Agent.OwnerOfName).

func (*Cluster) PeerDialMember added in v0.6.1

func (c *Cluster) PeerDialMember(m Member) (*http.Client, string, error)

PeerDialMember selects the peer client/URL using the per-node mTLS cache.

func (*Cluster) PeerInternalHTTPClient added in v0.6.1

func (c *Cluster) PeerInternalHTTPClient() *http.Client

PeerInternalHTTPClient exposes the cert-pinned client for cluster list fan-out. End-user Authorization must only ride this internal client.

func (*Cluster) PlacementChanges added in v0.6.1

func (c *Cluster) PlacementChanges(ctx context.Context, since uint64, wait time.Duration) PlacementChangesResponse

PlacementChanges answers a delta-feed poll. With nothing newer than since, it parks until the FSM changes or wait expires. It subscribes BEFORE the first read, so an apply landing between read and park cannot be missed.

A server that is BEHIND the cursor (a lagging follower, or an Agent that switched servers) waits instead of resnapshotting: since is a global Raft index, so "I haven't applied that far yet" is not a gap. Next never moves below since.

func (*Cluster) PlacementOf

func (c *Cluster) PlacementOf(sandboxID string) (Placement, bool)

PlacementOf returns the FSM record for sandboxID. Goes through f.get so the returned Placement is deep-cloned and safe to mutate without aliasing live FSM state (same guarantee as Placements).

func (*Cluster) PlacementPage

func (c *Cluster) PlacementPage(req PlacementPageRequest) PlacementPageResponse

func (*Cluster) PlacementVersion

func (c *Cluster) PlacementVersion() uint64

PlacementVersion returns the FSM's monotonic apply counter — bumps on every committed raft log entry. Exposed for metrics and as a tie-breaker for tests; the ingress reconciler uses SubscribePlacement to wake on apply rather than polling this counter.

func (*Cluster) PlacementViewCaughtUp added in v0.6.9

func (c *Cluster) PlacementViewCaughtUp() bool

PlacementViewCaughtUp reports whether this node's FSM has applied the read index it took after starting. The zero value is caught up, so a Cluster built as a struct literal (tests) behaves as before; only New arms it.

func (*Cluster) PlacementViewStale added in v0.6.12

func (c *Cluster) PlacementViewStale() bool

PlacementViewStale reports whether this server is a follower (or candidate) whose last contact with the leader is older than staleReadBound. The leader, and a Cluster without raft (a single-process FSM, which is the whole truth), are never stale.

func (*Cluster) Placements

func (c *Cluster) Placements() []Placement

func (*Cluster) PlacementsByIDs added in v0.6.1

func (c *Cluster) PlacementsByIDs(ids []string) map[string]Placement

PlacementsByIDs returns hot placement rows for the given IDs (point lookups).

func (*Cluster) PlacementsForShards

func (c *Cluster) PlacementsForShards(filter PlacementShardFilter) []Placement

func (*Cluster) ProbeSecretOnPeers added in v0.6.1

func (c *Cluster) ProbeSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, minGeneration int64) (holding []string, err error)

ProbeSecretOnPeers HEADs peer secret rows and returns nodes that currently hold seal_generation >= minGeneration (authoritative possession, not ACK memory).

func (*Cluster) PruneAuditACL added in v0.6.1

func (c *Cluster) PruneAuditACL(ctx context.Context, cutoff time.Time) error

func (*Cluster) PublishArtifactCatalog added in v0.6.1

func (c *Cluster) PublishArtifactCatalog(ctx context.Context, chunk ArtifactCatalogSnapshot) error

PublishArtifactCatalog applies one chunk of a snapshot.

func (*Cluster) PushSecretBlobToAnyPeer added in v0.6.1

func (c *Cluster) PushSecretBlobToAnyPeer(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

PushSecretBlobToAnyPeer races all eligible recipients and returns as soon as one authenticated peer ACKs. The create path uses this narrow minimum-ACK operation so an unreachable recipient cannot spend the entire ACK window before a healthy backup is attempted. Full convergence continues through PushSecretBlobToPeers and the durable put outbox.

func (*Cluster) PushSecretBlobToPeers added in v0.6.1

func (c *Cluster) PushSecretBlobToPeers(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

PushSecretBlobToPeers POSTs the sealed blob to each non-self recipient that is alive and advertises a reachable internal mTLS URL. Best-effort with bounded backoff; returns the node IDs that ACK'd (callers intersect with live membership for failover_ready). Missing cluster transport is reported for any remote target; self-only single-node calls remain a no-op.

func (*Cluster) PutVolumeAttachments added in v0.5.4

func (c *Cluster) PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error

func (*Cluster) ReadEgressProfiles added in v0.6.6

ReadEgressProfiles answers from this server's FSM.

func (*Cluster) ReadIdentity added in v0.6.12

ReadIdentity answers from this server's FSM.

func (*Cluster) ReadTeamQuota added in v0.6.12

ReadTeamQuota answers from this server's FSM. A follower's answer can trail the leader by an apply; the decisions that must be exact (create and resize) are made on the leader, so a read only informs.

func (*Cluster) ReadWarmPools added in v0.6.12

ReadWarmPools answers from this server's FSM.

func (*Cluster) ReassignPlacement added in v0.5.3

func (c *Cluster) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

ReassignPlacement moves sandboxID to target via opReassign.

func (*Cluster) ReassignStuckPlacement added in v0.6.1

func (c *Cluster) ReassignStuckPlacement(ctx context.Context, requesterID, sandboxID, incarnationID string, capacityRefused bool) error

ReassignStuckPlacement runs the leader's stuck-placement reassignment on behalf of a worker that cannot recreate a sandbox it owns. requesterID is the mTLS-authenticated caller; only the current owner may ask. capacityRefused is the worker's word that its admitter had no room: then a placement no other node can take is handed back (handBackRefusedPlacement).

func (*Cluster) RecordPlacement

func (c *Cluster) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

RecordPlacement commits sandboxID -> self into the FSM via raft along with the (optional) creation spec. Idempotent. Safe to call from any node: applyCommand transparently forwards to the current leader if we're a follower. Passing spec=nil preserves a previously-recorded spec — see fsm.go opPlace handling.

spec MUST be redacted before being passed in; secrets is the provider handle the caller produces via service.SealAndDistribute. Passing an empty handle preserves a previously-recorded handle.

func (*Cluster) RecoveryBlob added in v0.2.2

func (c *Cluster) RecoveryBlob(ctx context.Context, ref string) (RecoveryBlob, bool, error)

func (*Cluster) RegisterSnapshotNode added in v0.6.7

func (c *Cluster) RegisterSnapshotNode(ctx context.Context, snapshotName, nodeID string) error

RegisterSnapshotNode records that nodeID holds the named snapshot's image. Idempotent and move-safe: re-registering a name overwrites the node.

func (*Cluster) RemoveCustomDomain added in v0.3.1

func (c *Cluster) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

RemoveCustomDomain releases the binding. Idempotent — a stale call after a successful DeletePlacement is a no-op.

func (*Cluster) RemoveExposedPort

func (c *Cluster) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

RemoveExposedPort drops a replicated port-exposure intent. Idempotent.

func (*Cluster) RemoveMember added in v0.2.2

func (c *Cluster) RemoveMember(ctx context.Context, nodeID string, force bool) error

RemoveMember explicitly retires nodeID from the raft configuration. It is an operator lifecycle command, not gossip failure detection: the caller should drain the node first, then stop/terminate it, then remove it from raft.

Before RemoveServer, we persist a drain mark and orphan any placements owned by the retiring node so surviving nodes stop routing new work there and clients get a clear 410 for lost sandboxes under the current non-HA policy.

func (*Cluster) ReportStorageObligations added in v0.6.1

func (c *Cluster) ReportStorageObligations(ctx context.Context, report StorageObligationReport) error

ReportStorageObligations submits one owner report. On the leader it is queued for the next batch; elsewhere it is forwarded to the leader, which queues it there.

func (*Cluster) ReserveBatchOnTargets added in v0.2.2

func (c *Cluster) ReserveBatchOnTargets(ctx context.Context, reservations []PlacementReservation) error

ReserveBatchOnTargets commits multiple reservation intents in one Raft entry. It is intended for create-burst frontends that have already selected targets and want the leader to admit the whole chunk against the same capacity view. The public one-create API still uses ReserveOnTarget, but scale harnesses and bulk create callers use this to avoid one Raft log entry per sandbox.

func (*Cluster) ReserveOnTarget

func (c *Cluster) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

ReserveOnTarget commits a capacity-and-name reservation for sandboxID owned by target before any docker side effect runs. The router calls this after SelectPlacement so the cluster has intent recorded the instant the router forwards the body to target. ttl bounds how long the reservation holds the slot before the leader GC sweep cancels it; pick large enough to cover the slowest expected image pull on the target.

redacted MUST be stripped of plaintext credentials (call service.RedactClusterSecrets); secrets is the secret provider handle. Both ride the reservation so a successful promote via opPlace can inherit them without re-shipping the payload — see fsm.go opPlace preserve-on-empty rules.

Returns ErrReservationConflict when the slot is already placed or held by a different owner; ErrNameConflict when redacted.Name is held cluster-wide. Either should map to 4xx at the API surface; transport errors stay 5xx (caller may retry).

func (*Cluster) ResolveCustomDomain added in v0.3.1

func (c *Cluster) ResolveCustomDomain(hostname string) (string, bool)

ResolveCustomDomain answers the TLS-ask probe from the local FSM index in O(1). Returns sandboxID, true when the hostname is bound somewhere in the cluster; false otherwise. hostname is canonicalized inside the FSM.

func (*Cluster) RetireNodeStorage added in v0.6.1

func (c *Cluster) RetireNodeStorage(ctx context.Context, nodeID, actor, reason string, attestedAt time.Time) error

RetireNodeStorage records the attestation in the replicated control plane.

func (*Cluster) RevokeNodeStorageRetirement added in v0.6.1

func (c *Cluster) RevokeNodeStorageRetirement(ctx context.Context, nodeID string) error

RevokeNodeStorageRetirement withdraws the attestation cluster-wide.

func (*Cluster) SSHSessionOwnerRecordAllowed added in v0.6.9

func (c *Cluster) SSHSessionOwnerRecordAllowed() bool

SSHSessionOwnerRecordAllowed reports whether every live member writes the ssh record of a session another node's gateway bridged to it.

func (*Cluster) SecretsOf

func (c *Cluster) SecretsOf(sandboxID string) PlacementSecrets

SecretsOf returns a copy of the provider handle paired with SpecOf's spec.

func (*Cluster) SelectPlacement

func (c *Cluster) SelectPlacement(req capacity.Request) (PlacementTarget, error)

SelectPlacement chooses an owner node for a new sandbox using power-of-two- choices. The algorithm samples up to 2 random alive members (including self) and picks the one with the most CPU+memory headroom relative to the requested resources. This gives near-optimal load balancing without any global coordination — at scale it converges to within a small constant factor of the truly-optimal choice while costing O(1) per placement.

Falls back to self if no alive members are visible yet (just-bootstrapped node) or if no other member can satisfy req. Self always loses ties because "place locally" is the existing single-node behavior; we only forward when a peer is genuinely better.

func (*Cluster) SelectPlacementForCreate added in v0.6.1

func (c *Cluster) SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

SelectPlacementForCreate is SelectPlacement plus the bounded seal recipient set for sandboxID. The server-side member of the pair: no candidate slice crosses a process boundary here, because there is no boundary to cross.

func (*Cluster) SelectPlacementWithCandidates added in v0.6.1

func (c *Cluster) SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

SelectPlacementWithCandidates is SelectPlacement plus the filtered candidate slice the router uses to record the secret recipient set at reserve time (plans/secrets-hardening §3d-1). The target is always one of candidates.

func (*Cluster) SelfAPIURL

func (c *Cluster) SelfAPIURL() string

func (*Cluster) SelfNodeID

func (c *Cluster) SelfNodeID() string

func (*Cluster) ServerKeyRingAllowed added in v0.6.9

func (c *Cluster) ServerKeyRingAllowed() bool

ServerKeyRingAllowed reports whether every live member reads a server-key ring and key-id sealed blobs.

func (*Cluster) SetEgressGatewayReadyProvider added in v0.6.6

func (c *Cluster) SetEgressGatewayReadyProvider(fn func() bool)

SetEgressGatewayReadyProvider registers the egress gateway readiness callback so this node's own lease advertises EgressGatewayReady (plans/egress-domain-filtering.md CEO D20). Peers read the same flag from this node's /v1/capacity.

func (*Cluster) SetLocalSlotPoolsProvider added in v0.6.9

func (c *Cluster) SetLocalSlotPoolsProvider(fn func() []capacity.SlotPool)

SetLocalSlotPoolsProvider registers the network-slot pool callback whose answer is overlaid onto this node's own capacity lease, matching what peers read from its /v1/capacity.

func (*Cluster) SetLocalTemplateCatalogProvider added in v0.6.1

func (c *Cluster) SetLocalTemplateCatalogProvider(fn func() ([]string, bool))

SetLocalTemplateCatalogProvider registers the all-lifecycle template catalogue callback used by administrative item routing.

func (*Cluster) SetLocalTemplateIDsProvider added in v0.4.1

func (c *Cluster) SetLocalTemplateIDsProvider(fn func() ([]string, bool))

SetLocalTemplateIDsProvider registers the Phase 6 PR-D callback used by the capacity lease cache to overlay Firecracker template inventory onto local heartbeats. Idempotent; passing nil unregisters. Wired by main.go after the service is attached. Single-node deployments and nodes without Firecracker enabled never call this and degrade to nil-provider behaviour (no LocalTemplateIDs on the snapshot, peers see the legacy "unknown, allow" placement).

func (*Cluster) SetLocalWasmModuleIDsProvider added in v0.5.3

func (c *Cluster) SetLocalWasmModuleIDsProvider(fn func() ([]string, bool))

SetLocalWasmModuleIDsProvider registers the WASM module inventory callback.

func (*Cluster) SetNodeDrainState

func (c *Cluster) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

SetNodeDrainState marks nodeID as drained (drained=true) or restores it to the SelectPlacement candidate pool (drained=false). Goes through raft so every node — including the drained node itself — sees the new state on its next placement decision. Idempotent.

Drain is a no-op on existing placements: a drained node continues serving the sandboxes it already owns. Combine with the dead-owner reconciler (kill the process) to force ownership transfer, or use a separate evacuate command (not implemented) to move work proactively.

func (*Cluster) SnapshotHomeNode added in v0.6.7

func (c *Cluster) SnapshotHomeNode(_ context.Context, snapshotName string) (string, bool, error)

SnapshotHomeNode resolves the node holding the named snapshot from the local replicated FSM. ok=false means no entry (serve locally).

func (*Cluster) SoleRaftServer added in v0.6.12

func (c *Cluster) SoleRaftServer() bool

SoleRaftServer reports whether this node is the only server, voter or not, in its Raft configuration. Any other server counts, so the answer errs on the side of "not sole".

func (*Cluster) SpecOf

func (c *Cluster) SpecOf(sandboxID string) *models.CreateSandboxRequest

SpecOf returns a deep-copy of the replicated spec for sandboxID, or nil if none is recorded. The returned spec is REDACTED — registry passwords and mount credentials are stripped at write time. Use SecretsOf to retrieve the matching secret ref and service.OpenClusterSecretsForNode to merge credentials back in. Callers may safely mutate the returned struct (it shares no memory with the FSM).

func (*Cluster) StorageObligations added in v0.6.1

func (c *Cluster) StorageObligations(context.Context) ([]StorageObligationView, error)

StorageObligations serves the job view from the local FSM. The expected reporters are the members gossip reports alive that can own sandboxes.

func (*Cluster) SubscribePlacement

func (c *Cluster) SubscribePlacement(ctx context.Context) <-chan struct{}

SubscribePlacement returns a buffered (cap=1) wake channel that fires after every FSM apply on this node. Cancelling ctx removes the subscriber. See the Client.SubscribePlacement contract for semantics.

func (*Cluster) SweepExpiredAPITokens added in v0.6.8

func (c *Cluster) SweepExpiredAPITokens(ctx context.Context, cutoff time.Time) (int, error)

SweepExpiredAPITokens deletes tokens that expired before cutoff. Only the leader sweeps (every delete goes through Raft anyway); on any other node it does nothing. A delete is idempotent, so a sweep cut short by a leader change is finished by the next leader's sweep.

func (*Cluster) TokenPortsAllowed added in v0.6.12

func (c *Cluster) TokenPortsAllowed() bool

TokenPortsAllowed reports whether every live member keeps a port's visibility through failover.

func (*Cluster) UpdatePlacementSecretRecipients added in v0.6.1

func (c *Cluster) UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

UpdatePlacementSecretRecipients commits a replacement seal recipient set (and optional new provider handle after reseal). Preserves IncarnationID. expectedIncarnationID / expectedSealGeneration CAS against the live placement.

func (*Cluster) UpsertSpec

func (c *Cluster) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

UpsertSpec replicates a sandbox spec mutation (resize, lifecycle change) without changing ownership. Idempotent; nil spec + empty secrets is a no-op. Safe to call from any node — applyCommand forwards to the leader as needed.

spec MUST be redacted; empty secrets preserves the previously replicated handle (resize/lifecycle never change credentials).

func (*Cluster) VolumeAttachmentCount added in v0.5.4

func (c *Cluster) VolumeAttachmentCount(_ context.Context, tenant, id string) (int, error)

func (*Cluster) VolumeAttachmentsForSandbox added in v0.6.12

func (c *Cluster) VolumeAttachmentsForSandbox(_ context.Context, sandboxID string) ([]models.VolumeAttachment, error)

func (*Cluster) VolumeByID added in v0.5.4

func (c *Cluster) VolumeByID(_ context.Context, tenant, id string) (models.Volume, error)

func (*Cluster) VolumeByName added in v0.5.4

func (c *Cluster) VolumeByName(_ context.Context, tenant, name string) (models.Volume, error)

func (*Cluster) VolumeDelete added in v0.5.4

func (c *Cluster) VolumeDelete(ctx context.Context, tenant, id string) error

func (*Cluster) VolumeDeleteOwingBytes added in v0.6.12

func (c *Cluster) VolumeDeleteOwingBytes(ctx context.Context, tenant, id, source string) error

func (*Cluster) VolumeExistsForSource added in v0.5.4

func (c *Cluster) VolumeExistsForSource(_ context.Context, source string) (bool, error)

func (*Cluster) VolumeReclaimsFor added in v0.6.12

func (c *Cluster) VolumeReclaimsFor(_ context.Context, nodeID string, limit int) ([]VolumeReclaim, error)

VolumeReclaimsFor answers from this server's replica.

func (*Cluster) VolumeReclaimsReplicated added in v0.6.12

func (c *Cluster) VolumeReclaimsReplicated() bool

VolumeReclaimsReplicated: every Raft server applies the reclaim ops.

func (*Cluster) VolumeUpsert added in v0.5.4

func (c *Cluster) VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)

func (*Cluster) VolumesForTenant added in v0.5.4

func (c *Cluster) VolumesForTenant(_ context.Context, tenant string) ([]models.Volume, error)

func (*Cluster) WatchPlacementChanges added in v0.6.1

func (c *Cluster) WatchPlacementChanges(ctx context.Context, fn PlacementChangeHandler) bool

WatchPlacementChanges streams this server's own change log to fn, in-process (no network): a full view first, then deltas, resyncing on resnapshot. It runs until ctx ends.

func (*Cluster) WriteAPIToken added in v0.6.8

WriteAPIToken creates or revokes a token. The leader checks the upgrade gate and applies; a follower forwards one hop to the leader. The proposer stamps the time into the entry, so every replica applies the same one.

func (*Cluster) WriteEgressProfile added in v0.6.6

WriteEgressProfile puts or deletes a profile. The leader checks the upgrade gate and applies; a follower forwards one hop to the leader.

func (*Cluster) WriteIdentity added in v0.6.12

WriteIdentity applies an identity write on the leader after the upgrade gate; a follower forwards one hop to the leader.

func (*Cluster) WriteTeamQuota added in v0.6.12

WriteTeamQuota puts or deletes a team's quota. The leader checks the upgrade gate and applies; a follower forwards one hop to the leader.

func (*Cluster) WriteWarmPool added in v0.6.12

WriteWarmPool puts or deletes an entry. The leader checks the upgrade gate and applies; a follower forwards one hop to the leader. The proposer stamps the time into the entry, so every replica applies the same one.

type ClusterTLS

type ClusterTLS struct {
	// contains filtered or unexported fields
}

ClusterTLS is the loaded TLS material the daemon uses for cluster-internal channels (raft transport + leader-forwarded HTTP applies). Built once at boot from the on-disk files and shared by the raft StreamLayer and the internal HTTPS listener / client.

func (*ClusterTLS) NodeID added in v0.6.1

func (t *ClusterTLS) NodeID() string

NodeID returns the node-bound identity extracted from the local node cert.

type DestroyAtGate added in v0.6.12

type DestroyAtGate interface {
	DestroyAtAllowed() bool
}

DestroyAtGate is implemented by the cluster clients that can answer; the single-node Noop does not (a single node forwards to no older owner).

type DrainStateResponse

type DrainStateResponse struct {
	Drained bool `json:"drained"`
}

type DrainedNodesReader added in v0.6.1

type DrainedNodesReader interface {
	DrainedNodes() map[string]bool
}

DrainedNodesReader is implemented by clients that can return the drained set in one read. It is deliberately NOT part of Client: callers type-assert and fall back to per-node IsNodeDrained, so no test double has to grow a method it never exercises.

type DrainedNodesResponse added in v0.6.1

type DrainedNodesResponse struct {
	Drained []string `json:"drained"`
}

DrainedNodesResponse is the wire shape of PublicInternalDrainedNodesPath.

type EgressProfileKey added in v0.6.6

type EgressProfileKey struct {
	Owner string `json:"owner"`
	Name  string `json:"name"`
}

EgressProfileKey names one owner's profile.

type EgressProfileReadRequest added in v0.6.6

type EgressProfileReadRequest struct {
	Owner string   `json:"owner"`
	Names []string `json:"names,omitempty"`
	After string   `json:"after,omitempty"`
	Limit int      `json:"limit,omitempty"`
}

EgressProfileReadRequest reads an owner's profiles: the named ones, or a page by name when Names is empty.

type EgressProfileReadResponse added in v0.6.6

type EgressProfileReadResponse struct {
	Profiles      []EgressProfileRecord `json:"profiles"`
	Authoritative bool                  `json:"authoritative"`
}

EgressProfileReadResponse answers a read from a server's FSM. A name with no profile is simply absent.

type EgressProfileRecord added in v0.6.6

type EgressProfileRecord struct {
	Owner           string   `json:"owner"`
	Name            string   `json:"name"`
	AllowOut        []string `json:"allow_out"`
	Description     string   `json:"description,omitempty"`
	HostnameCount   int      `json:"hostname_count"`
	Generation      int64    `json:"generation"`
	CreatedUnixNano int64    `json:"created_unix_nano"`
	UpdatedUnixNano int64    `json:"updated_unix_nano"`
}

EgressProfileRecord is one replicated profile. HostnameCount is computed by the proposer (the entries are canonical) so the apply-time cap check needs only counts.

type EgressProfileWriteRequest added in v0.6.6

type EgressProfileWriteRequest struct {
	Delete  bool                `json:"delete,omitempty"`
	Profile EgressProfileRecord `json:"profile"`
}

EgressProfileWriteRequest puts (Delete false) or deletes one profile. A put carries the canonical entries and their hostname count.

type EgressProfileWriteResponse added in v0.6.6

type EgressProfileWriteResponse struct {
	Profile EgressProfileRecord `json:"profile"`
	Changed bool                `json:"changed"`
	Code    string              `json:"code,omitempty"`
	Message string              `json:"message,omitempty"`
}

EgressProfileWriteResponse is the outcome of a write. Code is set, and Profile empty, when the FSM refused it.

func EgressProfileWriteOutcome added in v0.6.6

func EgressProfileWriteOutcome(err error) EgressProfileWriteResponse

EgressProfileWriteOutcome turns a write error into the body a peer gets, and a body back into the same error, so sentinels survive forwarding.

type Endpoint

type Endpoint struct {
	NodeID      string
	InternalURL string
	APIURL      string
}

Endpoint binds a peer identity to its cluster-internal mTLS URL. APIURL is response metadata; ForwardHTTP never uses it as a downgrade.

type ExposedPortRoute

type ExposedPortRoute struct {
	Protocol  string `json:"protocol"`
	HostPort  int    `json:"host_port,omitempty"`
	PublicURL string `json:"public_url,omitempty"`
	// Visibility is "token" for a port whose public URL needs a sandbox
	// token (A2), so a failover recreate gates it again; omitted for a
	// public port, so every placement written before it decodes the same.
	Visibility string `json:"visibility,omitempty"`
}

Placement is one row of the FSM's placement map. Spec is the replicated sandbox creation request used by the new owner to re-materialize a sandbox after its previous owner died (see owner_watcher.go). Spec is a pointer so older snapshots written before spec replication still decode cleanly.

SecretRef/SecretVersion identify the secret provider record that can rehydrate the redacted Spec on the owner or an authorized recovery target. They are a handle, not secret material — raw sealed bytes have no field anywhere in cluster state, so Raft can never fan secret material out to every participant.

ExposedPortRoute is the replicated routing intent for one exposed sandbox port. Protocol drives which Caddy surface is used. HostPort is populated for raw TCP so every node can bind/proxy the same cluster-wide port.

type FailoverRefusal added in v0.6.12

type FailoverRefusal struct {
	SandboxID     string
	IncarnationID string
	OwnerRef      string
	// FromNodeID is the dead or stuck owner the sandbox could not leave.
	FromNodeID string
	Reason     string
	// Rule is the rule as text ("gpu=a10, zone=eu-west-1a").
	Rule string
}

FailoverRefusal is one failover the cluster declined to perform because moving the sandbox would break a rule it was created with.

type FailoverRefusalRecorder added in v0.6.12

type FailoverRefusalRecorder interface {
	RecordFailoverRefused(FailoverRefusal)
}

FailoverRefusalRecorder is the optional half of SandboxRecreator that writes the audit record for a refused failover. The service implements it; a recreator that does not just leaves the counter and the log line.

type IdentityGate added in v0.6.12

type IdentityGate interface {
	IdentityAllowed() bool
}

IdentityGate is implemented by cluster clients that can tell whether every live member runs a build with cloud identity. The service asserts it, so the single-node Noop (which does not implement it) and a cluster service that has not attached its client yet both stay closed.

type IdentityKeysWrite added in v0.6.12

type IdentityKeysWrite struct {
	ExpectedGeneration int64                     `json:"expected_generation"`
	Keys               []cloudidentity.KeyRecord `json:"keys"`
}

IdentityKeysWrite is opPutIdentityKeys' payload.

type IdentityReadRequest added in v0.6.12

type IdentityReadRequest struct {
	Keys     bool     `json:"keys,omitempty"`
	Owner    string   `json:"owner,omitempty"`
	Tags     []string `json:"tags,omitempty"`
	RolePage bool     `json:"role_page,omitempty"`
	After    string   `json:"after,omitempty"`
	Limit    int      `json:"limit,omitempty"`
}

IdentityReadRequest reads the key set (Keys) and/or an owner's roles: the named tags, or a page by tag when Tags is empty and RolePage is set.

type IdentityReadResponse added in v0.6.12

type IdentityReadResponse struct {
	Keys          *cloudidentity.KeySet `json:"keys,omitempty"`
	Roles         []IdentityRoleRecord  `json:"roles"`
	Authoritative bool                  `json:"authoritative"`
}

IdentityReadResponse answers a read from a server's FSM. A tag with no role is simply absent.

type IdentityRoleKey added in v0.6.12

type IdentityRoleKey struct {
	Owner string `json:"owner"`
	Tag   string `json:"tag"`
}

IdentityRoleKey names one owner's role tag.

type IdentityRoleRecord added in v0.6.12

type IdentityRoleRecord struct {
	Owner           string `json:"owner"`
	Tag             string `json:"tag"`
	MinRole         string `json:"min_role"`
	CreatedUnixNano int64  `json:"created_unix_nano"`
	UpdatedUnixNano int64  `json:"updated_unix_nano"`
}

IdentityRoleRecord is one replicated role tag.

type IdentityWriteRequest added in v0.6.12

type IdentityWriteRequest struct {
	Keys       *IdentityKeysWrite  `json:"keys,omitempty"`
	Role       *IdentityRoleRecord `json:"role,omitempty"`
	DeleteRole bool                `json:"delete_role,omitempty"`
}

IdentityWriteRequest is one identity write: a key-set replace, or a role put (DeleteRole false) or delete.

type IdentityWriteResponse added in v0.6.12

type IdentityWriteResponse struct {
	Keys    *cloudidentity.KeySet `json:"keys,omitempty"`
	Role    *IdentityRoleRecord   `json:"role,omitempty"`
	Changed bool                  `json:"changed,omitempty"`
	Code    string                `json:"code,omitempty"`
	Message string                `json:"message,omitempty"`
}

IdentityWriteResponse is the outcome. Code is set when the FSM refused.

func IdentityWriteOutcome added in v0.6.12

func IdentityWriteOutcome(err error) IdentityWriteResponse

IdentityWriteOutcome turns a write error into the body a peer gets.

type IngressRouteOwner

type IngressRouteOwner struct {
	NodeID        string `json:"node_id"`
	APIURL        string `json:"api_url,omitempty"`
	InternalURL   string `json:"internal_url,omitempty"`
	DataPlaneHost string `json:"data_plane_host,omitempty"`
}

type IngressShardFilterCache added in v0.6.1

type IngressShardFilterCache struct {
	// contains filtered or unexported fields
}

IngressShardFilterCache retains only the last topology for one reconciler. Capacity heartbeats and endpoint changes do not change HRW ownership. Compare the sorted live ingress IDs, not the entire membership (or a lossy hash).

func (*IngressShardFilterCache) ForNode added in v0.6.1

func (c *IngressShardFilterCache) ForNode(members []Member, nodeID, selfRole string) PlacementShardFilter

type IngressShardRoute

type IngressShardRoute struct {
	SandboxID  string              `json:"sandbox_id"`
	Shard      int                 `json:"shard"`
	ShardCount int                 `json:"shard_count"`
	Owners     []IngressRouteOwner `json:"owners"`
	// RingVersion identifies the membership view this answer was computed
	// from. Route installation and route lookup hash the same ordered set of
	// ingress node IDs, so two nodes that disagree about membership hand out
	// routes for different rings — the upstream can be pointed at a node that
	// never installed the shard. Publishing the version makes that divergence
	// observable (compare across ingress nodes) instead of silent.
	RingVersion string `json:"ring_version,omitempty"`
}

func IngressRouteForSandbox

func IngressRouteForSandbox(members []Member, sandboxID string) IngressShardRoute

type LeaderContactReporter added in v0.6.12

type LeaderContactReporter interface {
	PlacementViewStale() bool
}

LeaderContactReporter is implemented by clients that answer placement reads from a local replica fed by a raft leader (*Cluster).

type LocalSandboxState

type LocalSandboxState struct {
	ID           string
	Spec         *models.CreateSandboxRequest
	Secrets      PlacementSecrets
	ExposedPorts map[int]ExposedPortRoute
	// CustomHostnames is the per-sandbox bound hostname set known to the
	// local store. Carried through AssertOwnership so a sandbox created before
	// the FSM learned about its hostnames (e.g. cluster mode enabled after the
	// sandbox already had custom domains) backfills the replicated set on
	// boot — without this a failover-recreate would lose the user's TLS
	// matchers.
	CustomHostnames []string
}

LocalSandboxState is one entry in the boot-time AssertOwnership payload. Carrying Spec + ExposedPorts (rather than just an ID) lets the boot replay backfill the FSM with everything a future failover-recreate needs — without this, sandboxes that pre-date cluster mode (or pre-date the spec/ports replication features) would never gain a replicated spec until their next mutating call.

Spec MUST be redacted (no plaintext registry password / mount credentials) before being handed to AssertOwnership; Secrets carries the provider ref that the new owner re-merges on recreate. cmd/sandboxd takes care of this via service.SealAndDistribute + RedactClusterSecrets so the cluster layer never sees plaintext. Secrets may be empty when the sandbox has no secrets to ship.

type Member

type Member struct {
	NodeID string `json:"node_id"`
	// NodeName is the operator-friendly display label (SB_NODE_NAME). Empty
	// for peers running pre-NodeName builds; dashboards fall back to NodeID.
	NodeName      string `json:"node_name,omitempty"`
	APIURL        string `json:"api_url"`
	DataPlaneHost string `json:"data_plane_host,omitempty"`
	RaftAddr      string `json:"raft_addr,omitempty"`
	// InternalURL is the required cluster-internal mTLS endpoint used for peer
	// RPC and leader-forwarded applies.
	InternalURL string `json:"internal_url,omitempty"`
	// Role is the peer's gossiped SB_NODE_ROLE. Empty for older builds that
	// pre-date the field; callers treat empty as the legacy "mixed" default.
	Role string `json:"role,omitempty"`
	// PublicHost is the peer's gossiped public ingress address
	// (config.EffectivePublicHost). Aggregated by Cluster.IngressTargets to
	// answer the DNS-helper API. Empty for peers without a public host set
	// or running pre-PublicHost builds.
	PublicHost string `json:"public_host,omitempty"`
	// FSMOpsVersion is the peer's gossiped FSM op set (0 for older builds).
	FSMOpsVersion int               `json:"fsm_ops_version,omitempty"`
	Alive         bool              `json:"alive"`
	Capacity      capacity.Snapshot `json:"capacity"`
	// CapacityUpdatedUnix is when this node's last capacity heartbeat was
	// observed by the scheduler. CapacityStale means the last heartbeat is
	// missing or too old for placement admission.
	CapacityUpdatedUnix int64 `json:"capacity_updated_unix,omitempty"`
	CapacityStale       bool  `json:"capacity_stale,omitempty"`
	// PATScope is the peer's gossiped SB_PAT_SCOPE: "host", or empty for
	// the default "any" (and for older builds). It feeds the
	// aerolvm_pat_scope_drift gauge and grants nothing.
	PATScope string `json:"pat_scope,omitempty"`
}

Member is a snapshot of a peer's gossiped state.

func IdentityMembers added in v0.6.1

func IdentityMembers(c Client) []Member

IdentityMembers is the membership view for callers that need only identity, role, liveness or endpoints — never capacity or artifact inventories.

Identity/liveness gossip is already fleet-wide and local, so these callers have the answer in-process. Members() on an agent is a control-plane round trip that redistributes the whole membership record (~850 B per peer, ~1.7 MB at 2,000 nodes) including capacity snapshots and template/module inventories the caller is about to throw away. Local gossip first; the control plane is the fallback for a node whose gossip is not up yet.

func IngressRingMembers added in v0.6.1

func IngressRingMembers(c Client) []Member

IngressRouteForSandbox returns the ingress owner set an upstream router should target for sandboxID. Small ingress tiers all own every sandbox route; very large ingress tiers return the primary shard owner plus one failover replica so a single ingress death does not create a traffic vacuum. IngressRingMembers is the single membership view the shard-aware ingress path may hash. Reconciliation (which shards this node installs) and the route lookup (which node the upstream is sent to) must agree: an Agent's Members() is a control-plane snapshot while LocalMembers() is local gossip, and hashing one on the install side and the other on the lookup side routes traffic to nodes that never installed the shard. Local gossip is the preferred source — it is what the installer can act on — with the control-plane view as the fallback for a node whose gossip is not up yet.

type NodeStorageRetirement added in v0.6.1

type NodeStorageRetirement struct {
	NodeID string `json:"node_id"`
	Actor  string `json:"actor,omitempty"`
	Reason string `json:"reason,omitempty"`
	// AttestedUnixNano is the fence discharge compares copy provenance
	// against, in NANOSECONDS. Second granularity would round the attestation
	// backwards, so a copy distributed a few hundred milliseconds before it
	// would read as newer than the attestation and stay pinned forever.
	AttestedUnixNano int64 `json:"attested_unix_nano"`
}

NodeStorageRetirement is an operator's attestation that a node's storage was destroyed, replicated through the placement FSM.

It has to be cluster state, not a row on the node that served the API call: the deletion obligations an attestation discharges live in the delete outbox of whichever node owns the secret, and an operator's request reaches an arbitrary ingress or server. A node-local table meant the owners never learned, and list/revoke through a different entry node saw a different set.

The payload is tiny administrative metadata — one row per decommissioned node — so replicating it costs nothing like the placement state does.

func (NodeStorageRetirement) AttestedAt added in v0.6.1

func (r NodeStorageRetirement) AttestedAt() time.Time

AttestedAt is the fence discharge decisions compare obligation provenance against.

type NodeStorageRetirementsResponse added in v0.6.1

type NodeStorageRetirementsResponse struct {
	Retirements []NodeStorageRetirement `json:"retirements"`
	// Authoritative distinguishes "no attestations" from "could not ask".
	// A discharge is irreversible evidence-wise, so a non-authoritative
	// answer must never be read as an empty set.
	Authoritative bool `json:"authoritative,omitempty"`
}

NodeStorageRetirementsResponse is the agent-facing read.

type Noop

type Noop struct {
	// contains filtered or unexported fields
}

Noop is the single-node-mode Client implementation. Every method behaves as if this node owns every sandbox and is the only cluster member. Used when cfg.EnableCluster is false so callsites can be unconditional.

func NewNoop

func NewNoop(nodeID, apiURL, publicHost string) *Noop

NewNoop returns a single-node Client. nodeID and apiURL are reported back for observability but never actually used for routing. publicHost is the operator-configured public ingress address (config.EffectivePublicHost) that IngressTargets reports as the single-member target. Empty when the daemon runs in IP-only mode (no SB_PUBLIC_HOST / SB_DOMAIN), which also means custom domains are disabled.

func (*Noop) AddCustomDomain added in v0.3.1

func (n *Noop) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

func (*Noop) AddExposedPort

func (n *Noop) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

func (*Noop) ApplyEncoded

func (n *Noop) ApplyEncoded(ctx context.Context, payload []byte) error

func (*Noop) AssertOwnership

func (n *Noop) AssertOwnership(ctx context.Context, local []LocalSandboxState) error

func (*Noop) AttachInternalHandler

func (n *Noop) AttachInternalHandler(h http.Handler)

AttachInternalHandler is a no-op for single-node mode — there's no mTLS listener to wire into, so nothing to do.

func (*Noop) AuditACLForSandbox added in v0.6.1

func (n *Noop) AuditACLForSandbox(context.Context, string, string) (AuditACL, bool, error)

func (*Noop) AuditOwnerRef added in v0.6.1

func (n *Noop) AuditOwnerRef(context.Context, string) (string, bool, error)

func (*Noop) AuthoritativePlacementsByIDs added in v0.6.1

func (n *Noop) AuthoritativePlacementsByIDs(context.Context, []string) (map[string]Placement, error)

func (*Noop) BeginDeletePlacementExact added in v0.6.1

func (n *Noop) BeginDeletePlacementExact(context.Context, string, string, string) error

func (*Noop) CancelReservation

func (n *Noop) CancelReservation(ctx context.Context, sandboxID string) error

func (*Noop) ClaimOrphan

func (n *Noop) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Noop) Close

func (n *Noop) Close() error

func (*Noop) CustomDomainsOf added in v0.3.1

func (n *Noop) CustomDomainsOf(sandboxID string) []string

func (*Noop) DeletePlacement

func (n *Noop) DeletePlacement(_ context.Context, sandboxID string) error

func (*Noop) DeletePlacementExact added in v0.6.1

func (n *Noop) DeletePlacementExact(_ context.Context, sandboxID, _, incarnationID string) error

func (*Noop) DeleteVolumeAttachmentsForSandbox added in v0.5.4

func (n *Noop) DeleteVolumeAttachmentsForSandbox(_ context.Context, sandboxID, incarnationID string) error

func (*Noop) ExposedPortsOf

func (n *Noop) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

func (*Noop) FetchSandboxAuditFromPeer added in v0.6.1

func (n *Noop) FetchSandboxAuditFromPeer(context.Context, string, string, int, string, string, string, string) (AuditPeerPage, error)

FetchSandboxAuditFromPeer on Noop always fails — single-node mode has no peers.

func (*Noop) ForgetSnapshotNode added in v0.6.7

func (n *Noop) ForgetSnapshotNode(_ context.Context, snapshotName string) error

func (*Noop) ForwardHTTP

func (n *Noop) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

func (*Noop) IngressTargets added in v0.3.1

func (n *Noop) IngressTargets() models.IngressTarget

IngressTargets reports the single-node deployment's public address as the DNS target. Empty publicHost (IP-only mode) returns the Unknown source so the service layer can surface a clean 412 rather than fake records.

func (*Noop) IsNodeDrained

func (n *Noop) IsNodeDrained(nodeID string) bool

func (*Noop) Leader

func (n *Noop) Leader() string

func (*Noop) LocalMembers added in v0.6.1

func (n *Noop) LocalMembers() []Member

LocalMembers mirrors Members in single-node mode.

func (*Noop) LookupMember added in v0.6.1

func (n *Noop) LookupMember(id string) (Member, bool)

LookupMember resolves the single Noop member when id matches.

func (*Noop) Members

func (n *Noop) Members() []Member

func (*Noop) OwnerOf

func (n *Noop) OwnerOf(sandboxID string) (OwnerInfo, error)

func (*Noop) OwnerOfName

func (n *Noop) OwnerOfName(ownerRef, name string) (string, OwnerInfo, error)

func (*Noop) OwnerOfNameKey added in v0.6.5

func (n *Noop) OwnerOfNameKey(key string) (string, OwnerInfo, error)

func (*Noop) PeerInternalHTTPClient added in v0.6.1

func (n *Noop) PeerInternalHTTPClient() *http.Client

func (*Noop) PlacementOf

func (n *Noop) PlacementOf(sandboxID string) (Placement, bool)

PlacementOf has no record in single-node mode — there's no FSM. Returns the zero Placement and false so callers fall back to the local sandbox row.

func (*Noop) PlacementPage

func (*Noop) PlacementVersion

func (n *Noop) PlacementVersion() uint64

PlacementVersion always returns 0 in single-node mode — there's no FSM and no need to wake an ingress reconciler that isn't running.

func (*Noop) Placements

func (n *Noop) Placements() []Placement

func (*Noop) PlacementsByIDs added in v0.6.1

func (n *Noop) PlacementsByIDs(ids []string) map[string]Placement

PlacementsByIDs is empty in single-node mode (no FSM).

func (*Noop) PlacementsForShards

func (n *Noop) PlacementsForShards(PlacementShardFilter) []Placement

func (*Noop) PruneAuditACL added in v0.6.1

func (n *Noop) PruneAuditACL(context.Context, time.Time) error

func (*Noop) PutVolumeAttachments added in v0.5.4

func (n *Noop) PutVolumeAttachments(_ context.Context, attachments []models.VolumeAttachment) error

func (*Noop) ReassignPlacement added in v0.5.3

func (n *Noop) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

func (*Noop) RecordPlacement

func (n *Noop) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Noop) RegisterSnapshotNode added in v0.6.7

func (n *Noop) RegisterSnapshotNode(_ context.Context, snapshotName, nodeID string) error

Snapshot-home routing is a cluster concern; single-node serves every snapshot locally, so the index writes are no-ops and the lookup always says "not found" (serve locally — the local SQLite row is the sole source of truth).

func (*Noop) RemoveCustomDomain added in v0.3.1

func (n *Noop) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

func (*Noop) RemoveExposedPort

func (n *Noop) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

func (*Noop) RemoveMember added in v0.2.2

func (n *Noop) RemoveMember(ctx context.Context, nodeID string, force bool) error

func (*Noop) ReserveOnTarget

func (n *Noop) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

func (*Noop) ResolveCustomDomain added in v0.3.1

func (n *Noop) ResolveCustomDomain(hostname string) (string, bool)

func (*Noop) SecretsOf

func (n *Noop) SecretsOf(sandboxID string) PlacementSecrets

func (*Noop) SelectPlacement

func (n *Noop) SelectPlacement(req capacity.Request) (PlacementTarget, error)

func (*Noop) SelectPlacementForCreate added in v0.6.1

func (n *Noop) SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

func (*Noop) SelectPlacementWithCandidates added in v0.6.1

func (n *Noop) SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

func (*Noop) SelfAPIURL

func (n *Noop) SelfAPIURL() string

func (*Noop) SelfNodeID

func (n *Noop) SelfNodeID() string

func (*Noop) SetNodeDrainState

func (n *Noop) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

func (*Noop) SnapshotHomeNode added in v0.6.7

func (n *Noop) SnapshotHomeNode(_ context.Context, snapshotName string) (string, bool, error)

func (*Noop) SpecOf

func (n *Noop) SpecOf(sandboxID string) *models.CreateSandboxRequest

func (*Noop) SubscribePlacement

func (n *Noop) SubscribePlacement(ctx context.Context) <-chan struct{}

SubscribePlacement returns nil in single-node mode. Selecting on a nil channel never proceeds, so an ingress reconciler that select{}s on this channel + a slow ticker just behaves as if the cluster never has placement events — which is exactly the truth.

func (*Noop) UpdatePlacementSecretRecipients added in v0.6.1

func (n *Noop) UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

func (*Noop) UpsertSpec

func (n *Noop) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Noop) VolumeAttachmentCount added in v0.5.4

func (n *Noop) VolumeAttachmentCount(_ context.Context, tenant, id string) (int, error)

func (*Noop) VolumeAttachmentsForSandbox added in v0.6.12

func (n *Noop) VolumeAttachmentsForSandbox(_ context.Context, sandboxID string) ([]models.VolumeAttachment, error)

func (*Noop) VolumeByID added in v0.5.4

func (n *Noop) VolumeByID(_ context.Context, tenant, id string) (models.Volume, error)

func (*Noop) VolumeByName added in v0.5.4

func (n *Noop) VolumeByName(_ context.Context, tenant, name string) (models.Volume, error)

func (*Noop) VolumeDelete added in v0.5.4

func (n *Noop) VolumeDelete(_ context.Context, tenant, id string) error

func (*Noop) VolumeExistsForSource added in v0.5.4

func (n *Noop) VolumeExistsForSource(_ context.Context, source string) (bool, error)

func (*Noop) VolumeUpsert added in v0.5.4

func (n *Noop) VolumeUpsert(_ context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)

func (*Noop) VolumesForTenant added in v0.5.4

func (n *Noop) VolumesForTenant(_ context.Context, tenant string) ([]models.Volume, error)

type OwnedRecoveryRequest added in v0.6.1

type OwnedRecoveryRequest struct {
	Limit     int    `json:"limit,omitempty"`
	PageToken string `json:"page_token,omitempty"`
}

OwnedRecoveryRequest asks the control plane for the recovery work assigned to the CALLING node. The owner is never taken from the body: the handler uses the mTLS-authenticated peer identity, so a node can only ever ask for its own placements.

type OwnedRecoveryResponse added in v0.6.1

type OwnedRecoveryResponse struct {
	Placements    []Placement `json:"placements"`
	NextPageToken string      `json:"next_page_token,omitempty"`
	// Authoritative distinguishes "this node owns no recovery work" from
	// "the control plane could not answer". The watcher must not treat the
	// second as the first.
	Authoritative bool `json:"authoritative,omitempty"`
}

OwnedRecoveryResponse carries full placements — spec and secret handle included — for the caller's own failover-recreate sandboxes. It is the worker-side equivalent of the FSM read a server-role owner watcher does locally; without it a dedicated worker has no way to learn that a placement was reassigned to it.

type OwnerCache added in v0.6.12

type OwnerCache struct {
	// contains filtered or unexported fields
}

OwnerCache is a bounded, least-recently-used map from sandbox id to owner node. Safe for concurrent use; every method is a no-op on a nil cache.

func NewOwnerCache added in v0.6.12

func NewOwnerCache(self string, maxEntries int, ttl time.Duration, member func(nodeID string) (Member, bool)) *OwnerCache

NewOwnerCache returns a cache for node self holding at most maxEntries entries for ttl each. member resolves a node id to its gossip record: an entry is used only while its node is alive there with an internal URL.

func OwnerCacheOf added in v0.6.12

func OwnerCacheOf(c Client) *OwnerCache

OwnerCacheOf returns c's owner cache, or nil: a server-role node, a single node, or SB_CLUSTER_OWNER_CACHE off.

func (*OwnerCache) Forget added in v0.6.12

func (oc *OwnerCache) Forget(sandboxID string)

Forget drops sandboxID's entry, if any.

func (*OwnerCache) Get added in v0.6.12

func (oc *OwnerCache) Get(sandboxID string) (OwnerRoute, bool)

Get returns where to forward sandboxID, or false when there is no fresh entry or its node is not a live gossip member with an internal URL.

func (*OwnerCache) Len added in v0.6.12

func (oc *OwnerCache) Len() int

Len is the number of entries held, expired ones included until touched.

func (*OwnerCache) Put added in v0.6.12

func (oc *OwnerCache) Put(sandboxID string, owner OwnerInfo, ownerRef string, hasOwnerRef bool)

Put records that a fresh lookup found sandboxID on owner. An owner that is this node is never stored: a local serve must always come from a lookup. An unscoped caller's lookup does not read the tenant, so it keeps a tenant an earlier lookup learned for the same owner.

type OwnerCacheHolder added in v0.6.12

type OwnerCacheHolder interface {
	OwnerCache() *OwnerCache
}

OwnerCacheHolder is a Client that may keep an owner cache. Only the Agent implements it.

type OwnerInfo

type OwnerInfo struct {
	NodeID string
	APIURL string
	// InternalURL is the owner's required cluster-internal mTLS URL.
	InternalURL string
	IsSelf      bool
	// Epoch is the placement's OwnerEpoch when it was read (0: unknown).
	Epoch uint64
}

OwnerInfo is returned by OwnerOf.

type OwnerRoute added in v0.6.12

type OwnerRoute struct {
	// Endpoint is the owner's current gossip address, read when the entry is
	// used, so a node that restarted on a new URL is reached at the new one.
	Endpoint Endpoint
	// OwnerRef is the sandbox's tenant, known (HasOwnerRef) only when the
	// lookup that filled the entry read the placement for a scoped caller.
	OwnerRef    string
	HasOwnerRef bool
	// Epoch is the placement's OwnerEpoch the lookup that filled the entry
	// read (0: unknown). The forward carries it, and the owner turns down an
	// entry older than its own reading (HeaderOwnerEpoch).
	Epoch uint64
}

OwnerRoute is where a cached entry sends a sandbox's requests.

type Placement

type Placement struct {
	SandboxID           string              `json:"sandbox_id"`
	OwnerNodeID         string              `json:"owner_node_id"`
	OwnerAPIURL         string              `json:"owner_api_url"`
	OwnerDataPlaneHost  string              `json:"owner_data_plane_host,omitempty"`
	OwnerState          PlacementOwnerState `json:"owner_state,omitempty"`
	OrphanedOwnerNodeID string              `json:"orphaned_owner_node_id,omitempty"`
	OrphanedUnix        int64               `json:"orphaned_unix,omitempty"`
	// OwnerSinceUnixNano is when OwnerNodeID took the placement over from
	// another node: the leader's append time (raft.Log.AppendedAt) of the
	// entry that moved it, never a clock read in apply, so every replica and
	// every replay computes the same value. 0 means no takeover is known: the
	// placement still has the owner it was created (reserved or placed)
	// with, the entry carried no append time, or an older build applied it.
	// Orphaning, and a claim back by the node it was orphaned from, keep the
	// value, as no other node owned the sandbox in between. The owner reads
	// it when it reuses a local row from an earlier ownership, to refuse the
	// sandbox tokens issued before the takeover (another node may have
	// revoked them): Service.RecreateSandboxReport.
	OwnerSinceUnixNano int64 `json:"owner_since_unix_nano,omitempty"`
	// OwnerEpoch is the Raft log index of the entry that last changed
	// OwnerNodeID: the entry that wrote the row, and every reassign, orphaning
	// and claim since (invariant I2, plan C4 part 4). Unlike
	// OwnerSinceUnixNano it is a log index, not a leader's clock, so a later
	// owner change always has a larger epoch, even across leader changes. A
	// forward carries the epoch the edge saw (HeaderOwnerEpoch), and the owner
	// compares it with its own reading. 0 means unknown (a row written by an
	// older build, or restored from its snapshot): readers then fall back to
	// looking the owner up.
	OwnerEpoch  uint64 `json:"owner_epoch,omitempty"`
	Version     uint64 `json:"version"`
	CreatedUnix int64  `json:"created_unix"`
	UpdatedUnix int64  `json:"updated_unix"`
	// Name is the small, hot copy of Spec.Name used to rebuild the cluster-wide
	// uniqueness index without loading the larger recovery spec payload.
	Name string `json:"name,omitempty"`
	// RecoveryRef points at the out-of-snapshot recovery payload for this row.
	// Spec/secret fields are hydrated from that store only for point lookups and
	// recreate flows.
	RecoveryRef string                       `json:"-"`
	Spec        *models.CreateSandboxRequest `json:"spec,omitempty"`
	// PublicTraffic mirrors Spec.AllowPublicTraffic on the HOT row.
	//
	// Spec is split into the recovery store on write and redacted from every
	// paged/point read, so a dedicated ingress node — which has no FSM and
	// reads only those pages — never saw it. The ingress route builder skips
	// any placement whose public-traffic flag it cannot see, so on a hetero
	// cluster it installed no L4 route for any remote sandbox: raw TCP
	// exposures were unreachable (T18, UC-34). HTTP kept working only through
	// the per-request ingress proxy fallback. One bool rides the row instead.
	//
	// false means "not public, or recorded by a build that predates this
	// field"; placementAllowsPublicTraffic still consults Spec when present,
	// so a legacy row behaves exactly as before until its next write.
	PublicTraffic bool `json:"public_traffic,omitempty"`
	// FailoverRecreate mirrors Spec.Failover.ShouldRecreate() on the HOT row,
	// for the same reason as PublicTraffic: an ingress reads only redacted
	// pages and feed rows. When it loses the servers past the ownership
	// lease it puts only these sandboxes in flux, since only they can be
	// handed to another worker behind its back (placementMovableOnFailover).
	// false means "no recreate policy, or recorded by a build that predates
	// this field"; a row read with its Spec still consults the Spec.
	FailoverRecreate bool `json:"failover_recreate,omitempty"`
	// EgressProfiles / EgressInlineHostnames mirror Spec.EgressProfiles and
	// the hostname count of Spec.NetworkAllowOut on the HOT row, for the
	// FSM's profile reference index and union-cap check (D21). Empty for a
	// sandbox without profiles.
	EgressProfiles        []string `json:"egress_profiles,omitempty"`
	EgressInlineHostnames int      `json:"egress_inline_hostnames,omitempty"`
	// QuotaCPU / QuotaMemoryMB mirror the size a team's sandbox asked for
	// (Spec.CPU / Spec.MemoryMB with server defaults) on the HOT row, so the
	// FSM's team-usage index is rebuilt on Restore without hydrating specs
	// (fsm_team_quotas.go). Zero for a sandbox no team owns.
	QuotaCPU      float64 `json:"quota_cpu,omitempty"`
	QuotaMemoryMB int     `json:"quota_memory_mb,omitempty"`
	SecretRef     string  `json:"secret_ref,omitempty"`
	SecretVersion int     `json:"secret_version,omitempty"`
	// SecretRecipients is the seal recipient set recorded at reserve time
	// (owner + N backups). The create target seals to this set and must not
	// recompute it. It is empty only when the placement has no replicated
	// credential payload.
	SecretRecipients []string `json:"secret_recipients,omitempty"`
	// IncarnationID uniquely tags this placement lifetime for secret refs /
	// seal bindings. Minted at reserve and preserved on reassign/promote.
	IncarnationID string `json:"incarnation_id,omitempty"`
	// SecretSealGeneration is the last Raft-coordinated seal generation for
	// this placement (reseal CAS). It is 0 only when no secret was sealed.
	SecretSealGeneration int64 `json:"secret_seal_generation,omitempty"`
	// OwnerRef is the control-plane tenant account (tenancy). Distinct from
	// OwnerNodeID (which cluster node hosts the sandbox).
	OwnerRef string `json:"owner_ref,omitempty"`
	// AuditNodeIDs is the bounded history of nodes that have owned this
	// sandbox and may therefore hold local audit evidence. It lets audit reads
	// remain O(owner changes), rather than fanning out to every worker after a
	// failover or delete. AuditNodesTruncated forces the reader to fall back to
	// full worker fan-out so the bound can never create a silent evidence gap.
	AuditNodeIDs        []string `json:"audit_node_ids,omitempty"`
	AuditNodesTruncated bool     `json:"audit_nodes_truncated,omitempty"`

	ExposedPorts      map[int]string           `json:"exposed_ports,omitempty"`
	ExposedPortRoutes map[int]ExposedPortRoute `json:"exposed_port_routes,omitempty"`
	// CustomHostnames is the replicated set of user-bound hostnames pointing at
	// this sandbox. Kept sorted and lower-cased so snapshot bytes are stable
	// across rebuilds and every node derives the same Caddy matcher order. The
	// FSM additionally maintains a hostname→sandboxID index so the TLS-ask
	// resolver and ingress reconciler can look up ownership in O(1) from any
	// node, including the ingress-only ones that don't run the local SQLite
	// custom-domains table. Hostnames are added/removed via the
	// opAddCustomDomain/opRemoveCustomDomain raft commands; opPlace preserves
	// the existing slice the same way ExposedPorts is preserved so a
	// re-place/reassign cannot erase domains a prior raft entry installed.
	CustomHostnames []string `json:"custom_hostnames,omitempty"`
	// State is empty for materialized placements, PlacementStateReserved for
	// capacity-only intents, and PlacementStateDeleting while lifecycle-wide
	// finalization owns the row. Reserved and deleting states are eligible for
	// TTL-driven reconciliation; opPlace promotes a reservation to empty.
	State PlacementState `json:"state,omitempty"`
	// ExpiresUnix bounds reserved and deleting states. The leader reconciler
	// cancels expired reservations and completes abandoned exact-lifecycle
	// deletes after their finalization lease expires.
	ExpiresUnix int64 `json:"expires_unix,omitempty"`
}

ExposedPorts is the legacy replicated set of port→protocol intents. Newer code also fills ExposedPortRoutes with HostPort/PublicURL metadata so remote ingress nodes can install owner-aware data-plane routes. Both fields are kept so older snapshots still restore cleanly.

func (Placement) IsDeleting added in v0.6.1

func (p Placement) IsDeleting() bool

IsDeleting reports whether lifecycle-wide finalization owns the placement.

func (Placement) IsOrphaned

func (p Placement) IsOrphaned() bool

IsOrphaned reports whether the placement has no active owner.

func (Placement) IsReserved

func (p Placement) IsReserved() bool

IsReserved reports whether p is a reservation awaiting promotion.

type PlacementChange added in v0.6.1

type PlacementChange struct {
	// Index is the Raft index of the newest change seen for this id in the
	// served range. For a delete, a client drops its row only if the row's
	// Version is not above Index.
	Index     uint64     `json:"index"`
	SandboxID string     `json:"sandbox_id"`
	Deleted   bool       `json:"deleted,omitempty"`
	Placement *Placement `json:"placement,omitempty"`
}

PlacementChange is one sandbox's latest state since a cursor.

type PlacementChangeHandler added in v0.6.1

type PlacementChangeHandler func(full []Placement, changes []PlacementChange)

PlacementChangeHandler receives placement changes in order. full is non-nil after a (re)sync and is then the COMPLETE view; changes follow incrementally. Handlers must not block for long.

type PlacementChangeWatcher added in v0.6.1

type PlacementChangeWatcher interface {
	WatchPlacementChanges(ctx context.Context, fn PlacementChangeHandler) bool
}

PlacementChangeWatcher streams placement changes to a consumer that keeps its own index (the ingress route responder). It is implemented by the Agent (from its delta feed) and by a server Cluster (from its own change log, in-process).

type PlacementChangesResponse added in v0.6.1

type PlacementChangesResponse struct {
	Changes []PlacementChange `json:"changes"`
	// Next is the cursor to send as since on the next call.
	Next uint64 `json:"next"`
	// Resnapshot means the log no longer covers since. The client must
	// page-walk /internal/placements/page and then resume from Next.
	Resnapshot bool   `json:"resnapshot,omitempty"`
	Floor      uint64 `json:"floor"`
}

PlacementChangesResponse answers GET /v1/cluster/internal/placements/changes.

type PlacementLookupResponse

type PlacementLookupResponse struct {
	SandboxID string    `json:"sandbox_id"`
	Placement Placement `json:"placement"`
	Owner     OwnerInfo `json:"owner"`
	Orphaned  bool      `json:"orphaned"`
}

type PlacementOwnerState

type PlacementOwnerState string

PlacementOwnerState records the ownership lifecycle independently from the reservation lifecycle. Empty means the placement has an active owner.

const (
	PlacementOwnerStateActive   PlacementOwnerState = ""
	PlacementOwnerStateOrphaned PlacementOwnerState = "orphaned"
)

type PlacementPageRequest

type PlacementPageRequest struct {
	Limit       int                  `json:"limit,omitempty"`
	PageToken   string               `json:"page_token,omitempty"`
	ShardFilter PlacementShardFilter `json:"shard_filter,omitempty"`
	// OwnerRef, when set, returns only placements for that tenant account.
	OwnerRef string `json:"owner_ref,omitempty"`
	// OwnerNodeID, when set, returns only placements this cluster node owns.
	// Distinct from OwnerRef (tenant account). It is the read path a worker's
	// reconcile sweep uses to enumerate its own rows: at 100k sandboxes the
	// unfiltered view is ~72 MB, which every worker would otherwise pull on
	// every sweep. Reserved and orphaned rows are excluded — the FSM's owner
	// index only tracks materialized placements.
	OwnerNodeID string `json:"owner_node_id,omitempty"`
}

func (PlacementPageRequest) Normalize

type PlacementPageResponse

type PlacementPageResponse struct {
	Placements    []Placement `json:"placements"`
	NextPageToken string      `json:"next_page_token,omitempty"`
	// Authoritative is true when the page was produced by a ready FSM / control
	// plane. Empty Authoritative pages mean "tenant (or fleet) has zero rows"
	// — not "placement view unavailable". Agents set this false on CP errors.
	Authoritative bool `json:"authoritative,omitempty"`
	// SkippedSandboxIDs names rows that could not be delivered because one row
	// alone exceeds the response byte budget. They are neither present nor
	// absent, and a caller that makes cleanup decisions from this view (route
	// GC deletes routes for placements it cannot see) must treat them as
	// unavailable rather than deleted.
	SkippedSandboxIDs []string `json:"skipped_sandbox_ids,omitempty"`
}

type PlacementReservation added in v0.2.2

type PlacementReservation struct {
	SandboxID string
	Target    PlacementTarget
	Redacted  *models.CreateSandboxRequest
	Secrets   PlacementSecrets
	TTL       time.Duration
}

PlacementReservation is one item in a leader-side batch reservation request. Redacted MUST have plaintext credentials stripped before the caller passes it in; Secrets carries only a provider handle (and optional Recipients for the reserve-time seal set).

type PlacementSecrets

type PlacementSecrets struct {
	Ref     string `json:"ref,omitempty"`
	Version int    `json:"version,omitempty"`
	// Recipients is the seal recipient set chosen at reserve time for
	// failover.policy=recreate. It rides opReserve onto Placement.SecretRecipients
	// (node IDs only — never ciphertext). Reserved promotes normally repeat the
	// same set; boot ownership replay carries the durable local set so rebuilding
	// a missing placement cannot silently downgrade HA to one holder.
	Recipients []string `json:"recipients,omitempty"`
	// OwnerRef is the control-plane tenant account for this sandbox. Not
	// secret material — rides Place/Reserve so failover recreate preserves
	// tenancy when the owner-watcher uses an unscoped internal context.
	OwnerRef string `json:"owner_ref,omitempty"`
	// IncarnationID tags this placement lifetime for secret binding. Minted at
	// reserve and preserved across reassign.
	IncarnationID string `json:"incarnation_id,omitempty"`
	// SealGeneration is the seal generation last coordinated via Raft
	// (reseal / recipient expansion).
	SealGeneration int64 `json:"seal_generation,omitempty"`
}

PlacementSecrets is the recoverability handle paired with a redacted placement spec. It carries a secret provider reference only — the encrypted secret payload lives behind the service secret provider, never in Raft state. There is deliberately no field for raw sealed bytes: the raft log and FSM snapshots are not erasable per-sandbox, so making the field unrepresentable is what enforces the "no secret material in the log" rule.

type PlacementShardFilter

type PlacementShardFilter struct {
	ShardCount int   `json:"shard_count,omitempty"`
	Shards     []int `json:"shards,omitempty"`
	// None selects NO shards at all. It exists because the zero value already
	// means "all shards": a node with no public-route work of its own had no
	// way to say so, and the difference between "nothing" and "the whole
	// fleet" is a full placement download per node on every reconcile tick.
	None bool `json:"none,omitempty"`
}

PlacementShardFilter asks the control plane for only a subset of placement shards. Empty Shards means "all shards" for compatibility with existing callers; ShardCount <= 0 uses DefaultPlacementShardCount.

func IngressShardFilterForNode

func IngressShardFilterForNode(members []Member, nodeID, selfRole string) PlacementShardFilter

IngressShardFilterForNode returns the placement shard subset a single ingress-capable node should reconcile. Small ingress tiers replicate the full public route table to each ingress node so ordinary DNS round-robin / TCP load balancers work for every sandbox. Very large ingress tiers shard the table and require a shard-aware upstream router.

selfRole is this node's own SB_NODE_ROLE. A node that does not serve ingress gets NoPlacementShards: it has no public-route work, and synthesizing a ring membership for it (the old behavior when it was absent from the ingress ids) handed a dedicated worker either a slice of unrelated shards or, at small ingress counts, the entire placement map.

func NoPlacementShards added in v0.6.1

func NoPlacementShards() PlacementShardFilter

NoPlacementShards is the filter a node with no public-route responsibility asks with.

func (PlacementShardFilter) Normalize

Normalize returns a deterministic, de-duplicated copy of f.

type PlacementState

type PlacementState string

PlacementState distinguishes a reservation (capacity held, no docker yet) from a placement (sandbox materialized). Empty defaults to Placed so pre-reservation snapshots restore correctly: every old row is a real placement, never a pending reservation.

const (
	PlacementStatePlaced   PlacementState = "" // empty = legacy/placed
	PlacementStateReserved PlacementState = "reserved"
	// PlacementStateDeleting is a distributed lifecycle fence. The current
	// owner has committed delete intent, so failover/recreate and mutations
	// must stop while durable secret/artifact cleanup completes.
	PlacementStateDeleting PlacementState = "deleting"
)

type PlacementTarget

type PlacementTarget struct {
	NodeID        string
	APIURL        string
	DataPlaneHost string
	// InternalURL is the peer's required cluster-internal mTLS URL.
	InternalURL string
	IsSelf      bool
}

PlacementTarget is returned by SelectPlacement.

type ReadIndexResponse added in v0.6.9

type ReadIndexResponse struct {
	ReadIndex uint64 `json:"read_index"`
}

ReadIndexResponse is the body of InternalReadIndexPath.

type ReassignStuckRequest added in v0.6.1

type ReassignStuckRequest struct {
	SandboxID string `json:"sandbox_id"`
	// IncarnationID fences the request against a placement that has already
	// moved on since this node last read it.
	IncarnationID string `json:"incarnation_id,omitempty"`
	// CapacityRefused says this node's admitter refused the recreate for
	// room. With no other node to take it, the leader then orphans the
	// placement rather than leave it here, where it cannot run. A server that
	// predates the field ignores it and keeps the placement here, as before.
	CapacityRefused bool `json:"capacity_refused,omitempty"`
}

ReassignStuckRequest asks the leader to move a placement this node has failed to recreate. Target selection stays on the FSM side: it needs drain state, pending reservations and capacity leases that an Agent does not hold.

type RecoveryBlob added in v0.2.2

type RecoveryBlob struct {
	Ref           string                       `json:"ref"`
	SandboxID     string                       `json:"sandbox_id"`
	Spec          *models.CreateSandboxRequest `json:"spec,omitempty"`
	SecretRef     string                       `json:"secret_ref,omitempty"`
	SecretVersion int                          `json:"secret_version,omitempty"`
}

type SSHSessionRecordGate added in v0.6.9

type SSHSessionRecordGate interface {
	SSHSessionOwnerRecordAllowed() bool
}

SSHSessionRecordGate is implemented by the cluster clients that can answer; the single-node Noop does not (a single node bridges nothing).

type SandboxRecreateReporter added in v0.5.6

type SandboxRecreateReporter interface {
	RecreateSandboxReport(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets PlacementSecrets, exposedPorts map[int]ExposedPortRoute, ownerSinceUnixNano int64) (attempted bool, err error)
}

SandboxRecreateReporter is the optional outcome-reporting form of SandboxRecreator. attempted is false only for a successful steady-state no-op where the sandbox and its replicated routes were already present. Failures always report attempted=true so retry/error metrics retain the signal that drives stuck-placement reassignment.

ownerSinceUnixNano is the placement's OwnerSinceUnixNano, handed over from the row the watcher already read: an owner that finds a local row from an earlier ownership refuses the sandbox tokens issued before it took the placement back. Reading the placement again in the service instead would cost a worker one control-plane round trip per owned sandbox per tick.

type SandboxRecreator

type SandboxRecreator interface {
	RecreateSandbox(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets PlacementSecrets, exposedPorts map[int]ExposedPortRoute) error
}

SandboxRecreator is the cluster's escape hatch back into the service layer for the failover-recreate path. The owner watcher (see owner_watcher.go) invokes RecreateSandbox for any FSM placement that points to self but has no corresponding local sandbox — typically because the previous owner died and the dead-owner reconciler reassigned the placement here.

exposedPorts carries the replicated port routing intents the previous owner had recorded; the implementation is expected to re-issue ExposePort for each entry after the create succeeds. Raw TCP entries include the original HostPort so the recreated owner can preserve the public endpoint when the port is available on the new node.

Implementations MUST be idempotent: the watcher polls and may invoke RecreateSandbox multiple times for the same id while a previous attempt is still in flight or after an unexpected restart.

secrets is the provider ref that can rehydrate the redacted spec. Legacy sealed bytes may be present only for old placement rows. The implementation is expected to resolve and re-merge it before instantiating the container — without this step, recreated sandboxes would lose access to the user's private registry / mount credentials.

type SecretPeerMinACKPusher added in v0.6.1

type SecretPeerMinACKPusher interface {
	PushSecretBlobToAnyPeer(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)
}

SecretPeerMinACKPusher is implemented by the production cluster clients. Keeping it separate preserves the small full-fanout test seam while letting the synchronous HA-create path stop immediately after its first backup ACK.

type SecretPeerPusher added in v0.6.1

type SecretPeerPusher interface {
	PushSecretBlobToPeers(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)
	DeleteSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, generation int64) (acked []string, err error)
	ProbeSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, minGeneration int64) (holding []string, err error)
}

SecretPeerPusher is the narrow seam Service uses for async fan-out so tests can inject a fake without a full Cluster.

type SelectPlacementRequest

type SelectPlacementRequest struct {
	Request capacity.Request `json:"request"`
	// SandboxID and RecipientBackups ask the control plane to pick the seal
	// recipients itself and return only those. Without them the server falls
	// back to returning the full candidate slice, which is what an agent
	// running the previous build still expects — keep both paths until every
	// node in a rolling upgrade sends the id.
	SandboxID        string `json:"sandbox_id,omitempty"`
	RecipientBackups int    `json:"recipient_backups,omitempty"`
	// TargetOnly asks for the chosen node and nothing else. Build, template,
	// JS-bundle and local-image routing all want a single target, but the
	// only shape that existed without a SandboxID also serialized every
	// eligible worker back to the caller — O(fleet) bytes for a one-field
	// answer. An older server ignores the flag and simply returns the
	// candidates the caller then discards, so this is rolling-upgrade safe.
	TargetOnly bool `json:"target_only,omitempty"`
}

type SelectPlacementResponse

type SelectPlacementResponse struct {
	Target PlacementTarget `json:"target"`
	// Candidates is the legacy shape: every eligible worker, serialized to the
	// caller so IT could pick secret recipients. At 2,000 nodes that made each
	// create's response O(fleet) in bytes, allocations and control-plane CPU,
	// for an answer of at most a handful of node ids. Populated only when the
	// request did not carry a SandboxID.
	Candidates []Member `json:"candidates,omitempty"`
	// Recipients is the bounded answer: owner first, then the chosen backups.
	Recipients []string `json:"recipients,omitempty"`
	Error      string   `json:"error,omitempty"`
	// AtCapacity marks a placement refused because every worker that can run
	// the request is full (ErrClusterAtCapacity). Error then carries exactly
	// ErrNoPlacementTarget's text, which is all an agent on the previous
	// build matches — it keeps answering a 503 during a rolling upgrade —
	// while a current agent restores ErrClusterAtCapacity from this flag.
	AtCapacity bool `json:"at_capacity,omitempty"`
	// PlacementUnsatisfiable marks a refusal because no worker carries the
	// request's labels (ErrPlacementUnsatisfiable); Error names the rule.
	PlacementUnsatisfiable bool `json:"placement_unsatisfiable,omitempty"`
	// LabelsChecked says this control plane filtered on Request.Labels. A
	// server on an older build drops the field it cannot read and places as
	// if there were no rule; an agent that sent labels and does not get this
	// back fails closed (ErrPlacementRulesUnsupported) instead of creating
	// the sandbox wherever that server chose.
	LabelsChecked bool `json:"labels_checked,omitempty"`
}

func SelectPlacementErrorResponse added in v0.6.9

func SelectPlacementErrorResponse(err error) SelectPlacementResponse

SelectPlacementErrorResponse renders a placement failure for an agent.

type ServerKeyRingGate added in v0.6.9

type ServerKeyRingGate interface {
	ServerKeyRingAllowed() bool
}

ServerKeyRingGate is implemented by the cluster clients that can answer; the single-node Noop does not (a single node has no one to wait for).

type SnapshotNodeQueryResponse added in v0.6.7

type SnapshotNodeQueryResponse struct {
	NodeID string `json:"node_id,omitempty"`
	Found  bool   `json:"found,omitempty"`
}

SnapshotNodeQueryResponse is the wire shape of the internal snapshot-node read.

type SoleServerReporter added in v0.6.12

type SoleServerReporter interface {
	SoleRaftServer() bool
}

SoleServerReporter is implemented by the clients that run Raft themselves (*Cluster). A worker Agent and the single-node Noop do not.

type StorageObligation added in v0.6.1

type StorageObligation struct {
	NodeID         string `json:"node_id"`
	OpenedUnixNano int64  `json:"opened_unix_nano,omitempty"`
}

StorageObligation is the open job for one draining node.

type StorageObligationReport added in v0.6.1

type StorageObligationReport struct {
	Reporter string `json:"reporter"`
	// Seq increases on every report from this owner. The FSM ignores a report
	// whose Seq is not newer than the one it holds, so a retried or
	// out-of-order delivery can never resurrect an outbox row the owner has
	// already cleared.
	Seq uint64 `json:"seq"`
	// ObservedUnixNano is the owner's own clock when it took the snapshot.
	// Informational only: staleness is judged on ReceivedUnixNano.
	ObservedUnixNano int64 `json:"observed_unix_nano,omitempty"`
	// Owed maps node id -> number of outbox rows still owing that node a
	// delete. Empty is a real report ("I owe nothing"), not an absent one.
	Owed map[string]int `json:"owed,omitempty"`
	// ReceivedUnixNano is stamped by the leader when the batch is proposed,
	// never by Apply: every replica must reach the same state from the log.
	ReceivedUnixNano int64 `json:"received_unix_nano,omitempty"`
}

StorageObligationReport is one owner's current snapshot of the deletes it still owes. It REPLACES that owner's previous report.

type StorageObligationReporter added in v0.6.1

type StorageObligationReporter struct {
	NodeID     string    `json:"node_id"`
	Owed       int       `json:"owed"`
	ReportedAt time.Time `json:"reported_at,omitempty"`
	Stale      bool      `json:"stale"`
}

StorageObligationReporter is one owner's contribution to a job, as the operator sees it.

type StorageObligationReporterClient added in v0.6.1

type StorageObligationReporterClient interface {
	ReportStorageObligations(ctx context.Context, report StorageObligationReport) error
}

StorageObligationReporterClient is implemented by clients that can submit an owner report. Optional (type-asserted) so no test double must grow it.

type StorageObligationView added in v0.6.1

type StorageObligationView struct {
	NodeID   string    `json:"node_id"`
	OpenedAt time.Time `json:"opened_at,omitempty"`
	// Holders is how many placements still list the node as a secret
	// recipient — copies the reseal has not moved yet.
	Holders int `json:"holders"`
	// PendingDeletes sums the owners' reported outbox rows owed to the node.
	PendingDeletes int `json:"pending_deletes"`
	// Reporters are the owners that report owing this node something.
	Reporters []StorageObligationReporter `json:"reporters,omitempty"`
	// StaleReporters are owners expected to report whose last report is too
	// old (or absent). Their outbox is unknown, so PendingDeletes may be low.
	StaleReporters []StorageObligationReporter `json:"stale_reporters,omitempty"`
	// Complete is false whenever a reporter is stale: the counts are then a
	// floor, never an all-clear.
	Complete bool `json:"complete"`
	// ReadyForAttestation: no copies left, no deletes owed, every reporter
	// fresh. The job still stays open until an operator attests.
	ReadyForAttestation bool `json:"ready_for_attestation"`
}

StorageObligationView is what GET /v1/cluster/storage-retirements shows for an open job.

type StorageObligationsReader added in v0.6.1

type StorageObligationsReader interface {
	StorageObligations(ctx context.Context) ([]StorageObligationView, error)
}

StorageObligationsReader is implemented by clients that can serve the job view.

type StorageObligationsResponse added in v0.6.1

type StorageObligationsResponse struct {
	Obligations []StorageObligationView `json:"obligations"`
}

StorageObligationsResponse is the internal wire shape agents read.

type TeamQuotaReadRequest added in v0.6.12

type TeamQuotaReadRequest struct {
	Team string `json:"team"`
}

TeamQuotaReadRequest reads one team's quota and usage.

type TeamQuotaReadResponse added in v0.6.12

type TeamQuotaReadResponse struct {
	Quota         TeamQuotaRecord       `json:"quota"`
	Found         bool                  `json:"found"`
	Usage         models.TeamQuotaUsage `json:"usage"`
	Authoritative bool                  `json:"authoritative"`
}

TeamQuotaReadResponse answers from a server's FSM. Found is false when the team has no quota; Usage is filled either way.

type TeamQuotaRecord added in v0.6.12

type TeamQuotaRecord struct {
	Team            string  `json:"team"`
	MaxSandboxes    int     `json:"max_sandboxes,omitempty"`
	MaxCPU          float64 `json:"max_cpu,omitempty"`
	MaxMemoryMB     int     `json:"max_memory_mb,omitempty"`
	Generation      int64   `json:"generation"`
	CreatedUnixNano int64   `json:"created_unix_nano"`
	UpdatedUnixNano int64   `json:"updated_unix_nano"`
}

TeamQuotaRecord is one replicated team quota.

type TeamQuotaWriteRequest added in v0.6.12

type TeamQuotaWriteRequest struct {
	Delete bool            `json:"delete,omitempty"`
	Quota  TeamQuotaRecord `json:"quota"`
}

TeamQuotaWriteRequest puts (Delete false) or deletes one team's quota.

type TeamQuotaWriteResponse added in v0.6.12

type TeamQuotaWriteResponse struct {
	Quota   TeamQuotaRecord `json:"quota"`
	Changed bool            `json:"changed"`
	Code    string          `json:"code,omitempty"`
	Message string          `json:"message,omitempty"`
}

TeamQuotaWriteResponse is the outcome of a write.

func TeamQuotaWriteOutcome added in v0.6.12

func TeamQuotaWriteOutcome(err error) TeamQuotaWriteResponse

TeamQuotaWriteOutcome turns a write error into the body a peer gets.

type TokenPortGate added in v0.6.12

type TokenPortGate interface {
	TokenPortsAllowed() bool
}

TokenPortGate is implemented by the cluster clients that can answer; the single-node Noop does not (a single node has no older peer to fail over through).

type VolumeQueryResponse added in v0.5.4

type VolumeQueryResponse struct {
	Volume  *models.Volume  `json:"volume,omitempty"`
	Volumes []models.Volume `json:"volumes,omitempty"`
	Exists  bool            `json:"exists,omitempty"`
	Count   int             `json:"count,omitempty"`
	// Attachments answers kind=sandbox. A server that predates it answers
	// 400 (unknown kind), which the agent reports as an error: the caller
	// then keeps the volumes rather than guess.
	Attachments []models.VolumeAttachment `json:"attachments,omitempty"`
	// Reclaims answers kind=reclaims (fsm_volume_reclaims.go). A server
	// that predates it answers 400; the agent's sweep skips that pass.
	Reclaims []VolumeReclaim `json:"reclaims,omitempty"`
}

VolumeQueryResponse is the wire shape of the internal volume read endpoint. Exactly one of Volume / Volumes / Exists is meaningful per request kind.

type VolumeReclaim added in v0.6.12

type VolumeReclaim struct {
	Tenant   string `json:"tenant"`
	VolumeID string `json:"volume_id"`
	Name     string `json:"name,omitempty"`
	Backend  string `json:"backend"`
	Source   string `json:"source"`
	// SandboxID is the destroyed sandbox whose release made it owed.
	SandboxID string `json:"sandbox_id,omitempty"`
	// NodeID is the node whose volume-reclaim sweep drains it.
	NodeID string `json:"node_id"`
	// RecordedUnixNano versions the entry: an ack or claim names it, so a
	// late ack of an older release never drops a newer one.
	RecordedUnixNano int64 `json:"recorded_unix_nano"`
}

VolumeReclaim is one delete_with_sandbox volume whose last attachment a placement delete released: its bytes (and its row, if still there) are owed a delete. Times come from the proposer, never the applier's clock.

type VolumeReclaimClient added in v0.6.12

type VolumeReclaimClient interface {
	// VolumeReclaimsReplicated reports whether a placement delete records
	// the reclaims: every server applies the ops.
	VolumeReclaimsReplicated() bool
	// VolumeReclaimsFor lists up to limit entries nodeID should drain: its
	// own, then ones whose node is gone or never drains.
	VolumeReclaimsFor(ctx context.Context, nodeID string, limit int) ([]VolumeReclaim, error)
	// ClaimVolumeReclaim moves r to nodeID if it is still r, held by
	// r.NodeID; ErrVolumeReclaimMoved otherwise.
	ClaimVolumeReclaim(ctx context.Context, r VolumeReclaim, nodeID string) error
	// AckVolumeReclaim drops r once its bytes are deleted. Idempotent.
	AckVolumeReclaim(ctx context.Context, r VolumeReclaim) error
	// VolumeDeleteOwingBytes deletes an unattached volume row and records
	// its reclaim (source when the row has none) in the same apply, drained
	// by a worker. The caller checks VolumeReclaimsReplicated first.
	// ErrUnknownVolume / ErrVolumeInUse as VolumeDelete.
	VolumeDeleteOwingBytes(ctx context.Context, tenant, id, source string) error
}

VolumeReclaimClient is implemented by the cluster clients that replicate delete_with_sandbox volume reclaims. Optional (type-asserted) so the single-node Noop and test doubles need not grow it.

type WarmPoolReadRequest added in v0.6.12

type WarmPoolReadRequest struct {
	Since uint64 `json:"since,omitempty"`
}

WarmPoolReadRequest reads every entry. Since is the Version of the last answer the reader applied; 0 asks for the list whatever it is.

type WarmPoolReadResponse added in v0.6.12

type WarmPoolReadResponse struct {
	Version       uint64           `json:"version"`
	Unchanged     bool             `json:"unchanged,omitempty"`
	Pools         []WarmPoolRecord `json:"pools,omitempty"`
	Authoritative bool             `json:"authoritative"`
}

WarmPoolReadResponse answers a read from a server's FSM. Unchanged means Since is current and Pools is left out.

type WarmPoolRecord added in v0.6.12

type WarmPoolRecord struct {
	Image string `json:"image"`
	Depth int    `json:"depth"`
	// MinDepth/MaxDepth: the depth follows demand between them (W2). They
	// ship in the same release as the ops, so the fv 6 gate covers them: no
	// server can drop them as unknown fields.
	MinDepth int `json:"min_depth,omitempty"`
	MaxDepth int `json:"max_depth,omitempty"`
	// OnImageChange is replace_now or replace_as_used (W3); same release,
	// same gate. "" reads as replace_now.
	OnImageChange   string `json:"on_image_change,omitempty"`
	Generation      int64  `json:"generation"`
	CreatedUnixNano int64  `json:"created_unix_nano"`
	UpdatedUnixNano int64  `json:"updated_unix_nano"`
}

WarmPoolRecord is one replicated entry. Times come from the proposer's stamp, never the applier's clock, so replicas agree.

type WarmPoolWriteRequest added in v0.6.12

type WarmPoolWriteRequest struct {
	Delete bool           `json:"delete,omitempty"`
	Pool   WarmPoolRecord `json:"pool"`
}

WarmPoolWriteRequest puts (Delete false) or deletes one entry. A delete needs only the image.

type WarmPoolWriteResponse added in v0.6.12

type WarmPoolWriteResponse struct {
	Pool    WarmPoolRecord `json:"pool"`
	Changed bool           `json:"changed"`
	Code    string         `json:"code,omitempty"`
	Message string         `json:"message,omitempty"`
}

WarmPoolWriteResponse is the outcome. Code is set, and Pool empty, when the FSM or the upgrade gate refused the write.

func WarmPoolWriteOutcome added in v0.6.12

func WarmPoolWriteOutcome(err error) WarmPoolWriteResponse

WarmPoolWriteOutcome turns a write error into the body a peer gets, and err back into the same error, so sentinels survive forwarding.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL