cluster

package
v0.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2026 License: MIT Imports: 48 Imported by: 0

Documentation

Overview

Package cluster implements Phase-1 cluster mode for sandboxd.

Architecture (Phase 1):

  • Membership: SWIM gossip via hashicorp/memberlist. Gossip carries identity and role metadata only so memberlist's 512-byte NodeMeta limit cannot strip Raft addresses.

  • Capacity heartbeats: server-role nodes fetch authenticated capacity.Snapshot payloads from worker-capable peers and require a fresh heartbeat before placement can target that worker.

  • Placement map: server-role nodes run a small Raft FSM (hashicorp/raft) holding sandbox_id -> owner_node_id plus replicated recovery metadata. Mutations happen on the leader; server reads are local from the FSM. Worker/ingress-only nodes run Agent instead: they gossip identity and receive owner API forwards, but all placement reads/writes go to the server quorum over authenticated RPC. They do not store the FSM and do not join Raft as non-voters.

  • Owner-sharded execution: once a sandbox is placed on node N, all of its state and lifecycle stays on N. The local SQLite store is unchanged. Cross-node API calls (toolbox, sessions, port forwards) are transparently reverse-proxied to the owner via internal/cluster.ForwardHTTP.

  • No central control plane: any node can accept any request. Mutating requests for sandbox X are forwarded to X's owner; CreateSandbox forwards to the placement target chosen by power-of-two-choices.

Single-node mode (cfg.EnableCluster = false) uses Noop so that callsites can stay unconditional.

Index

Constants

View Source
const (
	PublicInternalApplyPath           = "/v1/cluster/internal/apply"
	PublicInternalPlacementPath       = "/v1/cluster/internal/placement/"
	PublicInternalPlacementByNamePath = "/v1/cluster/internal/placement-by-name/"
	PublicInternalPlacementsPath      = "/v1/cluster/internal/placements"
	PublicInternalPlacementsQueryPath = "/v1/cluster/internal/placements/query"
	PublicInternalPlacementsPagePath  = "/v1/cluster/internal/placements/page"
	PublicInternalPlacementsByIDsPath = "/v1/cluster/internal/placements-by-ids"
	// PublicInternalOwnedRecoveryPath serves a node its OWN failover-recreate
	// placements (spec + secret handle). It is how a dedicated worker — which
	// has no FSM — learns that a placement was reassigned to it.
	PublicInternalOwnedRecoveryPath = "/v1/cluster/internal/placements/owned-recovery"
	// PublicInternalReassignStuckPath lets an owner that keeps failing to
	// recreate a sandbox ask the leader to move it. Target selection stays on
	// the FSM side.
	PublicInternalReassignStuckPath   = "/v1/cluster/internal/placements/reassign-stuck"
	PublicInternalRecoveryPath        = "/v1/cluster/internal/recovery/"
	PublicInternalSelectPlacementPath = "/v1/cluster/internal/select-placement"
	PublicInternalVolumePath          = "/v1/cluster/internal/volume"
	PublicInternalDrainStatePath      = "/v1/cluster/internal/drain/"
	PublicInternalAuditACLPath        = "/v1/cluster/internal/audit-acl/"
	PublicInternalClusterLeaderPath   = "/v1/cluster/leader"
	// PublicInternalSecretPath receives peer fan-out of sealed secret blobs
	// (POST upsert) and delete-fanout (DELETE .../{sandboxID}). Auth is PAT +
	// d.Auth like every other /v1/cluster/internal/... route.
	PublicInternalSecretPath = "/v1/cluster/internal/secrets"
	// PublicInternalSandboxAuditPath is the prefix for peer-local secret audit
	// reads. Full path: .../sandboxes/{id}/audit
	PublicInternalSandboxAuditPath = "/v1/cluster/internal/sandboxes/"
	// PublicInternalNodeStorageRetirementsPath serves the replicated operator
	// attestations that a node's storage was destroyed. Obligation owners are
	// workers, which hold no FSM, so they read the set from the server tier
	// rather than from a table on whichever node the operator called.
	PublicInternalNodeStorageRetirementsPath = "/v1/cluster/internal/node-storage-retirements"
	// PublicInternalArtifactCatalogPath serves and accepts the replicated
	// template / JS-bundle metadata catalogue.
	PublicInternalArtifactCatalogPath = "/v1/cluster/internal/artifact-catalog"
	// PublicInternalArtifactCatalogEpochPath allocates ONE publisher its
	// fencing token. It is deliberately separate from the catalogue read: a
	// publisher needs a single number, and serving it from the general page
	// would hand every publisher the fleet's whole coverage list and scan
	// every node's inventory under the FSM lock to produce it.
	PublicInternalArtifactCatalogEpochPath = "/v1/cluster/internal/artifact-catalog/epoch"
)
View Source
const (
	// ArtifactKindTemplate / ArtifactKindJSBundle name the two catalogues.
	// Templates are not tenant-scoped, so they publish under the empty
	// tenant; JS bundles carry their owner.
	ArtifactKindTemplate = "template"
	ArtifactKindJSBundle = "js-bundle"

	// MaxArtifactCatalogChunkRows / MaxArtifactCatalogChunkBytes bound ONE
	// raft command. The apply transport caps a request at 1 MiB and the
	// payload is base64-encoded inside the command, so a publisher that sent
	// its whole inventory at once would be refused for an ordinary inventory
	// of small rows — 4,096 of them encode to ~1.6 MB.
	MaxArtifactCatalogChunkRows  = 256
	MaxArtifactCatalogChunkBytes = 512 << 10

	// MaxArtifactCatalogPageRows / artifactCatalogPageByteBudget bound one
	// READ. The cursor carries the rest, so a large catalogue is walked
	// rather than truncated.
	MaxArtifactCatalogPageRows = 2048
)
View Source
const (
	// CreateBackpressureRetryAfterSeconds is the public Retry-After hint for
	// queue/concurrency backpressure. Keep it short: these rejects are caused by
	// transient in-flight create fan-in, not by a long operator action.
	CreateBackpressureRetryAfterSeconds = 5
	CapacityRetryAfterSeconds           = 30
	// InvalidTopologyRetryAfterSeconds is the hint for ErrInvalidTopology
	// rejects, which only clear when an operator reshapes the cluster
	// (promoting servers or adding workers). A long back-off keeps clients
	// from busy-retrying while the human change is in flight.
	InvalidTopologyRetryAfterSeconds = 300
)
View Source
const (
	DefaultPlacementPageLimit = 1000
	MaxPlacementPageLimit     = 5000
	// MaxReplicatedIngressRouteNodes is the largest ingress tier where every
	// ingress-capable node keeps the full public route table. Ordinary DNS
	// round-robin / TCP load balancers can only work if any advertised ingress
	// node can answer for any sandbox.
	MaxReplicatedIngressRouteNodes = 10
)
View Source
const (
	// PublicWasmMigratePath is the operator-facing live-migration endpoint (§4.4).
	PublicWasmMigratePath = "/v1/cluster/wasm-migrate"
	// PublicInternalWasmMigratePath prefixes per-sandbox internal export/import routes.
	PublicInternalWasmMigratePath = "/v1/cluster/internal/wasm-migrate/"

	// WasmMigrateCloneGenHeader carries the §4.8 clone-generation token on export.
	WasmMigrateCloneGenHeader = "X-Aerol-Clone-Generation"
	// WasmMigrateTarMediaType is the streamed mem.snap tarball content type.
	WasmMigrateTarMediaType = "application/vnd.aerolvm.wasm-migrate.v1+tar"
)
View Source
const DefaultPlacementShardCount = 16384

DefaultPlacementShardCount is the stable shard space for placement-index reads. The value intentionally stays independent from the number of ingress nodes: ingress ownership changes remap shard IDs to nodes, but a sandbox's shard ID remains stable as the fleet scales up and down.

View Source
const InternalAPIPath = "/internal/apply"

InternalAPIPath is the URL path served by the cluster-internal mTLS listener. Distinct from the public /v1/cluster/internal/apply route so a misconfigured load balancer can't accidentally tunnel public traffic into the cluster-internal channel.

View Source
const MaxMixedClusterNodes = 10

MaxMixedClusterNodes is the largest live cluster size allowed to keep the legacy mixed/hybrid convenience topology. Above this, production clusters must run dedicated server, worker, and ingress tiers.

View Source
const MaxPlacementChangesWait = 25 * time.Second

MaxPlacementChangesWait bounds one long-poll. It stays under the Agent's control-plane request timeout plus slack, so a parked poll is never cut off by the caller's own deadline.

View Source
const MaxServerTierNodes = 7

MaxServerTierNodes caps the live server-role tier. The voter cap (SB_CLUSTER_MAX_AUTO_VOTERS, default 5) bounds how many nodes *vote*, not how many hold state: a surplus server-role node is added as a Raft non-voter and still receives the full log and FSM — at 100k sandboxes that is ~72 MB of placement state plus a leader replication stream each. Scaling out belongs in the worker and ingress tiers, which run Agent and hold no FSM at all, so the control plane stays a small fixed set however large the fleet grows.

7 = the default 5 voters plus two slots, so a rolling replacement can bring a new server up before retiring the old one without tripping the gate. Deliberately a constant, like MaxMixedClusterNodes: an env override here would just re-open the foot-gun it exists to close.

View Source
const PeerNodeIDHeader = "X-Cluster-Peer-Node-ID"

PeerNodeIDHeader is set by production peer callers so the receiver can bind the mTLS leaf to the claimed gossip node id.

View Source
const PublicInternalDrainedNodesPath = "/v1/cluster/internal/drained-nodes"

PublicInternalDrainedNodesPath returns the whole drained set in one call.

The per-node route (PublicInternalDrainStatePath) is one control-plane round trip per question. The secret reseal asks about every current holder of every HA sandbox an owner has, on a 30s tick, on every node — at 2,000 workers that is thousands of round trips a second for a set that is almost always empty and changes only when an operator drains something.

View Source
const PublicInternalPlacementChangesPath = "/v1/cluster/internal/placements/changes"

PublicInternalPlacementChangesPath is the long-poll delta feed endpoint.

View Source
const PublicInternalStorageObligationsPath = "/v1/cluster/internal/storage-obligations"

PublicInternalStorageObligationsPath serves the job view to agents (an ingress or worker has no FSM). One call to a server — never a fan-out.

Variables

View Source
var (
	// ErrVolumeQuotaExceeded is returned by opUpsertVolume when inserting a new
	// row would exceed the tenant's configured volume-count cap.
	ErrVolumeQuotaExceeded = errors.New("cluster: tenant volume quota exceeded")
	// ErrUnknownVolume is returned by volume reads/deletes when no replicated row
	// exists for the (tenant, id).
	ErrUnknownVolume = errors.New("cluster: unknown volume")
	// ErrVolumeInUse is returned when a delete races with or follows a sandbox
	// attachment that still points at the volume.
	ErrVolumeInUse = errors.New("cluster: volume is still attached")
)
View Source
var ErrArtifactCatalogSuperseded = errors.New("cluster: artifact catalogue publication is superseded")

ErrArtifactCatalogSuperseded reports a publication from a process the catalogue has already moved past. The publisher re-seeds its epoch from the authority and republishes rather than believing it is clean.

View Source
var ErrArtifactNodeUnavailable = fmt.Errorf("cluster: the node holding this artifact is unavailable: %w", ErrNoPlacementTarget)

ErrArtifactNodeUnavailable is ErrNoPlacementTarget's specific form for a request pinned to one node by an artifact it holds (a node-bound js-bundle or built image) when that node is not a live, capacity-reporting member. errors.Is(err, ErrNoPlacementTarget) stays true; the API adds a code so a client can tell "re-create the artifact" from "wait for capacity".

View Source
var ErrCapacityExceeded = errors.New("cluster: target capacity exceeded after pending reservations")

ErrCapacityExceeded is returned when an opReserve apply finds that the chosen target no longer has headroom for the request once concurrent in-flight reservations are added to its gossiped reserved totals. Routers translate this to 503 so the client retries; the next SelectPlacement will see the new reservation and pick a different node.

View Source
var ErrCreateBackpressure = errors.New("cluster: create backpressure")

ErrCreateBackpressure is returned when the leader-side create queue for a worker is full. This is intentionally distinct from ErrCapacityExceeded: capacity means "pick another worker or wait for resources"; backpressure means "too many creates are already in-flight to this worker, retry shortly."

View Source
var ErrCustomHostnameConflict = errors.New("cluster: custom hostname already in use")

ErrCustomHostnameConflict is returned when an opAddCustomDomain entry asks the FSM to claim a hostname already held by a different sandbox. Maps to the same 409 the local SQLite custom-domains insert returns — the FSM is the cluster-wide tiebreaker that catches the race where two sandboxes on different owners try to claim the same hostname concurrently. The API/service layer surfaces this as models.ErrCustomDomainConflict after reading the raft Apply result.

View Source
var ErrHostPortReserved = errors.New("cluster: tcp host port already reserved")

ErrHostPortReserved is returned when the FSM rejects a raw-TCP exposure because another placement already owns the requested cluster-wide host port. Callers can retry with another candidate instead of surfacing a random expose failure to the user.

View Source
var ErrIncarnationConflict = errors.New("cluster: sandbox incarnation conflict")

ErrIncarnationConflict fences a placement update carrying secret state for a different lifetime of the same sandbox ID.

View Source
var ErrInvalidSecretHandle = errors.New("cluster: invalid secret handle")

ErrInvalidSecretHandle rejects partial, generation-less, or cross-lifecycle provider handles before they can enter replicated placement state.

View Source
var ErrInvalidTopology = errors.New("cluster: invalid topology")

ErrInvalidTopology is returned when the live cluster shape violates a production topology invariant. API layers translate this to 503 so clients retry after the operator fixes membership instead of treating it as a malformed request.

View Source
var ErrLastVoter = errors.New("cluster: cannot remove last raft voter")

ErrLastVoter prevents an explicit removal from deleting the last voting raft server and leaving the cluster with no quorum path.

View Source
var ErrMemberStillAlive = errors.New("cluster: raft member is still alive")

ErrMemberStillAlive protects operators from accidentally removing a live control-plane node. Stop the node first, or pass force through the public lifecycle API when intentionally retiring a live member.

View Source
var ErrMembershipPending = errors.New("cluster: peer membership not yet recognised by the control plane")

ErrMembershipPending means a control-plane server refused this node only because its gossip view does not (yet) list the node as alive.

A restarting node hits this on every boot: its clean shutdown broadcast a gossip Leave, and its first control-plane call lands ~100ms after start — before the server has processed the rejoin. The live hetero run (T18, 2026-09-27) lost a worker permanently to it: enterprise treated the 403 as fatal, each restart re-broadcast the Leave and re-lost the same race, and systemd's restart limit made it final.

View Source
var ErrNameConflict = errors.New("cluster: sandbox name already in use")

ErrNameConflict is returned when the placement FSM rejects an opPlace / opUpsertSpec because a different sandbox already owns the requested Name. Sandbox names are unique cluster-wide; without this check, two concurrent creates landing on different owners could each succeed locally and present ambiguous name-based lookups to facades like Daytona that resolve sandboxes by name. Callers handle this by rolling back the local create and surfacing 409 Conflict to the user.

View Source
var ErrNoLeader = errors.New("cluster: timed out waiting for leader")

ErrNoLeader is returned when no leader is seated yet — an election is in flight, or this node has just started and has not learned who the leader is. Distinct from ErrNotLeader ("someone else is the leader"), but in the same class for callers: both mean "ask again shortly", neither means the data is bad.

View Source
var ErrNoPlacementTarget = errors.New("cluster: no worker placement target available")

ErrNoPlacementTarget is returned when no alive worker-capable node can accept a new sandbox. This is distinct from "self wins": a pure server or ingress node must not silently fall back to local Docker ownership.

View Source
var ErrNoReassignTarget = errors.New("cluster: no alternate node available for stuck placement")

ErrNoReassignTarget reports that no live node other than the failing owner can host the placement. The caller keeps retrying where it is rather than orphaning a recoverable sandbox — but it must not be told the placement moved, or it resets the failure counter that drives the escalation.

View Source
var ErrNotLeader = errors.New("cluster: not raft leader")

ErrNotLeader is returned by mutating Cluster operations when this node is not the Raft leader. Callers should retry against the leader (which Client resolves automatically by following raft leadership).

View Source
var ErrOrphanClaimConflict = errors.New("cluster: orphaned placement belongs to a different previous owner")

ErrOrphanClaimConflict is returned when a node tries to reclaim an orphaned placement that was explicitly orphaned from a different previous owner. This prevents a returning node from stealing another dead node's sandbox just because it has a stale local row with the same ID.

View Source
var ErrOrphaned = errors.New("cluster: sandbox owner is dead, placement orphaned")

ErrOrphaned is returned by OwnerOf when a placement exists but its owner has been auto-evicted and the dead-owner reconciler has cleared the pointer. This is the terminal state for default non-HA sandboxes; callers should surface 410 Gone. Sandboxes that opted into failover.policy=recreate are reassigned before the remaining dead-owner placements are orphaned.

View Source
var ErrPeerInternalURLInvalid = errors.New("cluster: peer InternalURL must be an absolute https URL")

ErrPeerInternalURLInvalid is returned before any credentials are attached when gossip advertises a non-HTTPS or malformed internal endpoint.

View Source
var ErrPeerInternalURLRequired = errors.New("cluster: peer InternalURL required (mTLS fail-closed)")

ErrPeerInternalURLRequired is returned when local TLS is configured but the peer has no InternalURL. Callers must not fall back to public APIURL+PAT.

View Source
var ErrRecoveryPayloadTooLarge = errors.New("cluster: recovery payload exceeds inline size limit")

ErrRecoveryPayloadTooLarge is returned when a redacted spec + secret handle encodes past inlineRecoveryMaxBytes. It surfaces at create validation (the user-facing 400) and again defensively at command encode time — hitting the latter without the former means a service-side mutation grew the spec after validation, which is a bug, not a routing case.

View Source
var ErrReservationConflict = errors.New("cluster: sandbox already placed or reserved")

ErrReservationConflict is returned when opReserve tries to reserve a sandbox ID that already has a non-expired placement (placed or actively reserved by a different owner). Indicates either a router racing a completed sandbox or a router with a stale view.

View Source
var ErrSecretRecipientsCASMismatch = errors.New("cluster: secret recipients CAS mismatch")

ErrSecretRecipientsCASMismatch is returned when opUpdateSecretRecipients carries expected incarnation / seal-generation pretenses that no longer match the live placement (stale reseal loser).

View Source
var ErrStuckReassignNotOwner = errors.New("cluster: reassign requester is not the current placement owner")

ErrStuckReassignNotOwner rejects a reassignment request from a node that is not the placement's current owner (or names a superseded incarnation). A stale request from a previous owner must not bounce a live sandbox off the node that has just taken it over.

View Source
var ErrUnknownMember = errors.New("cluster: unknown raft member")

ErrUnknownMember is returned when an operator asks to remove a node that is not present in the current raft configuration.

View Source
var ErrUnknownSandbox = errors.New("cluster: unknown sandbox placement")

ErrUnknownSandbox is returned by OwnerOf when no placement record exists for the given sandbox ID. Callers should treat this as "owned locally" only when they have just-created the sandbox and not yet committed its placement.

Functions

func ApplyErrorRetryAfterSeconds added in v0.6.1

func ApplyErrorRetryAfterSeconds(err error) int

ApplyErrorRetryAfterSeconds is the Retry-After an apply listener sets for a failure that is worth retrying on a delay; 0 means none.

func ApplyErrorStatus added in v0.6.1

func ApplyErrorStatus(err error) int

ApplyErrorStatus is the HTTP status an apply listener answers a failed apply with. It is exported because the v1 apply handler is the other listener and has to answer a given verdict identically.

func AuthenticatedPeerNodeID added in v0.6.1

func AuthenticatedPeerNodeID(r *http.Request) (string, error)

AuthenticatedPeerNodeID returns the node identity authenticated by the request's verified mTLS leaf. Cluster-control headers are not credentials: callers must first prove possession of a CA-signed node certificate, and an optional explicit node claim must match that certificate.

func CanOwnSandboxRole

func CanOwnSandboxRole(role string) bool

CanOwnSandboxRole reports whether a gossiped node role may own sandboxes. Empty is treated as worker-capable for rolling upgrades from builds that did not advertise role metadata.

func CanServeControlPlaneRole

func CanServeControlPlaneRole(role string) bool

CanServeControlPlaneRole reports whether a gossiped node role may host the authoritative Raft/FSM control plane. Empty is treated as server-capable for rolling upgrades from builds that did not advertise role metadata.

func CanServeIngressRole

func CanServeIngressRole(role string) bool

CanServeIngressRole reports whether a gossiped node role may host public ingress routes. Empty is treated as ingress-capable for rolling upgrades from builds that did not advertise role metadata.

func ClientForPeer added in v0.6.1

func ClientForPeer(base *http.Client, expectedNodeID string) *http.Client

ClientForPeer returns an HTTP client that verifies the dialed peer presents DNS SAN node:<expectedNodeID> (in addition to the shared clusterServerName hostname check). Empty expected or a non-TLS base is a no-op.

func ExportWasmMigrateLocal added in v0.5.3

func ExportWasmMigrateLocal(export func(w io.Writer) (string, error)) ([]byte, string, error)

ExportWasmMigrateLocal buffers export output for tests.

func ExposedPortRoutesForPlacement

func ExposedPortRoutesForPlacement(p Placement) map[int]ExposedPortRoute

func ExtractPeerNodeID added in v0.6.1

func ExtractPeerNodeID(cert *x509.Certificate) string

ExtractPeerNodeID returns the node-bound identity from a peer certificate. Only DNS/URI SAN `node:<id>` is authoritative; common-name and shared-SAN fallbacks are intentionally unsupported.

func IngressRingVersion added in v0.6.1

func IngressRingVersion(members []Member) string

IngressRingVersion fingerprints the ordered set of ingress-eligible node IDs a ring was computed from. Equal versions mean two nodes will compute the same owners for every sandbox; different versions mean they are mid- convergence and their answers may disagree.

func IsControlPlaneUnavailable added in v0.6.1

func IsControlPlaneUnavailable(err error) bool

IsControlPlaneUnavailable reports whether err means only that the control plane could not serve this node at this instant — no leader seated, or the server not yet seeing this node in gossip. Neither says anything about the data being validated, so neither may be treated as fatal at boot.

func IsLeaderUnavailable added in v0.6.1

func IsLeaderUnavailable(err error) bool

IsLeaderUnavailable reports whether an error means only that leadership was not reachable at this instant.

This is the distinction that decides whether a caller may treat a failure as fatal. A boot-time validation that exits(1) on ErrNoLeader turns a routine election into a node that never starts — and with systemd's restart limit, never starts AGAIN. Leadership being momentarily unsettled is the normal state of a cluster that is starting up; it is not evidence that anything is wrong with what was being validated.

func IsLivePeer added in v0.6.1

func IsLivePeer(c Client, nodeID string) bool

IsLivePeer reports whether nodeID is present and alive in the caller's current gossip view. It deliberately uses the local view: authenticating a forwarded request must not recurse through another cluster HTTP request.

func IsMixedArchitectureRole added in v0.2.2

func IsMixedArchitectureRole(role string) bool

IsMixedArchitectureRole reports whether a role value runs more than one production tier. Empty legacy metadata is treated as mixed because old nodes behaved as server+worker+ingress.

func LargeClusterTopologyError added in v0.2.2

func LargeClusterTopologyError(members []Member) error

LargeClusterTopologyError enforces the production topology contract: clusters above MaxMixedClusterNodes must have dedicated server, worker, and ingress tiers. Small clusters keep the legacy convenience behavior.

func LiveMemberCount added in v0.2.2

func LiveMemberCount(members []Member) int

LiveMemberCount counts live members with a usable node ID.

func MaxArtifactCatalogRowsPerNode added in v0.6.1

func MaxArtifactCatalogRowsPerNode() int

MaxArtifactCatalogRowsPerNode is the publisher-side cap on one node's whole inventory of a kind. A node over it does not publish at all: it stays uncovered and the aggregator keeps asking it, which is slower but never wrong.

func MintIncarnationID added in v0.6.1

func MintIncarnationID() (string, error)

MintIncarnationID returns a 16-byte random hex string that uniquely tags a sandbox placement lifetime. Reassign preserves the value; every fresh reserve or unreserved placement mints a new one.

func PeerDial added in v0.6.1

func PeerDial(m Member, internalClient *http.Client) (client *http.Client, base string, err error)

PeerDial selects the peer HTTP client and internal URL. Cluster peer RPCs are always mTLS and fail closed when either side is missing.

Prefer PeerDialCached (or Cluster/Agent.PeerDialMember) so mTLS VerifyPeer transport clones are reused per node.

func PeerDialCached added in v0.6.1

func PeerDialCached(m Member, internalClient, cached *http.Client) (client *http.Client, base string, err error)

PeerDialCached is PeerDial with an optional prebuilt per-peer mTLS client. When cached is non-nil it is returned as-is (no transport clone). Otherwise a VerifyPeerCertificate client is built via ClientForPeer.

func PeerDialPath added in v0.6.1

func PeerDialPath(m Member, internalClient *http.Client, path string) (*http.Client, string, error)

PeerDialPath is PeerDial plus a path suffix (e.g. /v1/cluster/internal/secrets).

func PlacementShardForSandbox

func PlacementShardForSandbox(sandboxID string, count int) int

PlacementShardForSandbox maps sandboxID to a stable shard ID in [0, count).

func PostWasmMigrateImport added in v0.5.3

func PostWasmMigrateImport(ctx context.Context, c Client, target Member, sandboxID, cloneGen string, body io.Reader) error

PostWasmMigrateImport pushes a mem.snap tarball to target.

func RecordMTLSUnknownPeer added in v0.6.1

func RecordMTLSUnknownPeer()

RecordMTLSUnknownPeer increments when an inbound mTLS peer id is not an Alive gossip member. Soft mode warns via this metric; enterprise rejects.

func RecordOwnerForwardStale

func RecordOwnerForwardStale()

func SelectSecretRecipients added in v0.6.1

func SelectSecretRecipients(sandboxID string, candidates []Member, ownerID string, maxBackups int) []string

SelectSecretRecipients builds the seal recipient set: owner first, then up to maxBackups other candidates chosen by deterministic rendezvous hashing over (sandboxID, nodeID). Sorting by NodeID alone hotspot-loads the lexicographically first workers; HRW spreads backup rows evenly while remaining stable for a given sandbox. When the cluster is smaller than owner+maxBackups, every eligible candidate is included. Empty ownerID or candidates yields nil.

func SetPeerNodeIDHeader added in v0.6.1

func SetPeerNodeIDHeader(req *http.Request, selfNodeID string)

SetPeerNodeIDHeader attaches this node's id for inbound withInternalMTLS checks.

func StreamWasmMigrateExport added in v0.5.3

func StreamWasmMigrateExport(ctx context.Context, c Client, owner OwnerInfo, sandboxID string, w io.Writer) (cloneGen string, err error)

StreamWasmMigrateExport pulls a mem.snap tarball from owner into w.

func ValidateRecoveryPayloadSize added in v0.5.6

func ValidateRecoveryPayloadSize(sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

ValidateRecoveryPayloadSize reports whether the recovery record for (sandboxID, spec, secrets) fits in a raft log entry. Exported so the service layer can reject oversized specs with a clean validation error before any admission/container work happens. Encodes the exact bytes the FSM's recovery store persists, so validation and apply cannot disagree.

func VerifyPeerNodeID added in v0.6.1

func VerifyPeerNodeID(cert *x509.Certificate, expected string) bool

VerifyPeerNodeID reports whether expected matches the peer cert's node-bound identity. Empty expected is a no-op (returns true). Used by withInternalMTLS when the caller supplies an expected peer id.

Types

type Agent

type Agent struct {
	// contains filtered or unexported fields
}

Agent is the worker/ingress-side cluster client. It deliberately does not start Raft, does not create a placement FSM, and never joins the raft configuration as a non-voter. It gossips identity/addresses, serves local capacity heartbeats, and delegates every authoritative placement read/write to server-role nodes.

func NewAgent

func NewAgent(cfg config.Config, logger *slog.Logger, admitter *capacity.Admitter) (*Agent, error)

func (*Agent) AddCustomDomain added in v0.3.1

func (a *Agent) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

AddCustomDomain forwards to the cluster Apply pipe. hostname is canonicalized here (the public Cluster wrapper does the same on the other path); the FSM then enforces cluster-wide uniqueness.

func (*Agent) AddExposedPort

func (a *Agent) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

func (*Agent) AllocateArtifactCatalogEpoch added in v0.6.1

func (a *Agent) AllocateArtifactCatalogEpoch(ctx context.Context, kind, _ string, holder string) (int64, error)

AllocateArtifactCatalogEpoch asks the control plane to issue THIS node's fencing token. The server fills the node from the authenticated peer identity, so a node can only ever be issued its own.

It deliberately does not go through the catalogue page: a publisher needs one number, and the page answers with every publisher's coverage (tens of kilobytes at fleet scale) after scanning every node's inventory under the FSM read lock.

func (*Agent) ApplyEncoded

func (a *Agent) ApplyEncoded(ctx context.Context, payload []byte) error

func (*Agent) ArtifactCatalog added in v0.6.1

func (a *Agent) ArtifactCatalog(ctx context.Context, req ArtifactCatalogRequest) (ArtifactCatalogPage, error)

ArtifactCatalog reads one page from the server tier. An unreachable control plane is an error: the caller falls back to the fan-out rather than reporting a tenant's artifacts as absent.

func (*Agent) AssertOwnership

func (a *Agent) AssertOwnership(ctx context.Context, local []LocalSandboxState) error

func (*Agent) AttachInternalHandler

func (a *Agent) AttachInternalHandler(h http.Handler)

func (*Agent) AttachRecreator added in v0.6.1

func (a *Agent) AttachRecreator(r SandboxRecreator)

AttachRecreator wires the service-layer recreate hook used by the worker owner watcher. It mirrors Cluster.AttachRecreator so pkg/daemon attaches the same hook whatever role the node runs — a dedicated worker previously had no such method, so the daemon's interface probe silently skipped it and the only automatic recreation loop in the product belonged to *Cluster.

func (*Agent) AuditACLForSandbox added in v0.6.1

func (a *Agent) AuditACLForSandbox(ctx context.Context, sandboxID, incarnationID string) (AuditACL, bool, error)

func (*Agent) AuditOwnerRef added in v0.6.1

func (a *Agent) AuditOwnerRef(ctx context.Context, sandboxID string) (string, bool, error)

func (*Agent) AuthoritativeNodeStorageRetirements added in v0.6.1

func (a *Agent) AuthoritativeNodeStorageRetirements(ctx context.Context) ([]NodeStorageRetirement, error)

AuthoritativeNodeStorageRetirements asks the control plane for the LEADER's answer. A worker discharging an obligation without an ACK is doing something irreversible, so it must not be authorized by a server whose FSM has not yet applied the operator's revoke.

func (*Agent) AuthoritativePlacementsByIDs added in v0.6.1

func (a *Agent) AuthoritativePlacementsByIDs(ctx context.Context, ids []string) (map[string]Placement, error)

func (*Agent) BeginDeletePlacementExact added in v0.6.1

func (a *Agent) BeginDeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Agent) CancelReservation

func (a *Agent) CancelReservation(ctx context.Context, sandboxID string) error

func (*Agent) ClaimOrphan

func (a *Agent) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Agent) ClientForPeer added in v0.6.1

func (a *Agent) ClientForPeer(nodeID string) *http.Client

ClientForPeer returns a cached mTLS HTTP client that verifies DNS SAN node:<nodeID>. Legacy shared-SAN-only peer certs are rejected.

func (*Agent) Close

func (a *Agent) Close() error

func (*Agent) CustomDomainsOf added in v0.3.1

func (a *Agent) CustomDomainsOf(sandboxID string) []string

CustomDomainsOf returns the hostnames bound to sandboxID. Agent doesn't run the placement FSM locally, so this rides the same remote placement lookup the existing ExposedPortsOf path uses — failure modes match: a transient network blip yields nil, which the ingress reconciler treats as "no custom matchers right now" until the placement subscription wakes a re-read.

func (*Agent) DeletePlacement

func (a *Agent) DeletePlacement(ctx context.Context, sandboxID string) error

func (*Agent) DeletePlacementExact added in v0.6.1

func (a *Agent) DeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Agent) DeleteSecretOnPeers added in v0.6.1

func (a *Agent) DeleteSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, generation int64) (acked []string, err error)

func (*Agent) DeleteVolumeAttachmentsForSandbox added in v0.5.4

func (a *Agent) DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID, incarnationID string) error

func (*Agent) DrainedNodes added in v0.6.1

func (a *Agent) DrainedNodes() map[string]bool

DrainedNodes returns the control plane's drained set, cached for drainedNodesTTL. On a failed refresh it keeps serving the last good set: forgetting a drain because one request failed would put a secret copy straight back onto the node being evacuated.

func (*Agent) ExposedPortsOf

func (a *Agent) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

func (*Agent) FetchSandboxAuditFromPeer added in v0.6.1

func (a *Agent) FetchSandboxAuditFromPeer(ctx context.Context, nodeID, sandboxID string, limit int, cursor, kind, incarnationID string) (AuditPeerPage, error)

FetchSandboxAuditFromPeer GETs a peer's local audit slice (worker agent).

func (*Agent) ForwardHTTP

func (a *Agent) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

func (*Agent) IngressTargets added in v0.3.1

func (a *Agent) IngressTargets() models.IngressTarget

IngressTargets aggregates live ingress-role members' PublicHost values. Identity and role are gossip facts, so this reads the local view and only falls back to the control plane when gossip has nothing yet — the aggregate is a handful of hostnames, not a reason to redistribute capacity snapshots for the whole fleet.

func (*Agent) IsNodeDrained

func (a *Agent) IsNodeDrained(nodeID string) bool

func (*Agent) Leader

func (a *Agent) Leader() string

func (*Agent) LocalMembers added in v0.6.1

func (a *Agent) LocalMembers() []Member

LocalMembers returns the gossip view only — never the control-plane HTTP members call. Used by list failover_ready so a page of sandboxes does not trigger N membership RPCs.

func (*Agent) LookupMember added in v0.6.1

func (a *Agent) LookupMember(id string) (Member, bool)

LookupMember resolves one peer by ID via the local gossip index (O(1)), falling back to a membership scan only when the index misses.

func (*Agent) Members

func (a *Agent) Members() []Member

func (*Agent) NodeStorageRetirements added in v0.6.1

func (a *Agent) NodeStorageRetirements(ctx context.Context) ([]NodeStorageRetirement, error)

NodeStorageRetirements reads the replicated set from the server tier. An unavailable control plane is an ERROR, never an empty set: discharging an obligation is irreversible, and so is failing to discharge one that the operator attested.

This is the DISCOVERY read, answerable by any server. A caller about to act irreversibly uses AuthoritativeNodeStorageRetirements.

func (*Agent) OwnerOf

func (a *Agent) OwnerOf(sandboxID string) (OwnerInfo, error)

func (*Agent) OwnerOfName

func (a *Agent) OwnerOfName(name string) (string, OwnerInfo, error)

func (*Agent) PeerDialMember added in v0.6.1

func (a *Agent) PeerDialMember(m Member) (*http.Client, string, error)

PeerDialMember selects the peer client/URL using the per-node mTLS cache.

func (*Agent) PeerInternalHTTPClient added in v0.6.1

func (a *Agent) PeerInternalHTTPClient() *http.Client

PeerInternalHTTPClient exposes the cert-pinned client for cluster list fan-out.

func (*Agent) PlacementOf

func (a *Agent) PlacementOf(sandboxID string) (Placement, bool)

func (*Agent) PlacementPage

func (a *Agent) PlacementPage(req PlacementPageRequest) PlacementPageResponse

func (*Agent) PlacementVersion

func (a *Agent) PlacementVersion() uint64

func (*Agent) Placements

func (a *Agent) Placements() []Placement

func (*Agent) PlacementsByIDs added in v0.6.1

func (a *Agent) PlacementsByIDs(ids []string) map[string]Placement

PlacementsByIDs batch-looks up IDs via a single control-plane POST. Prefer this over Placements() when only a page of IDs is needed.

func (*Agent) PlacementsForShards

func (a *Agent) PlacementsForShards(filter PlacementShardFilter) []Placement

PlacementsForShards reads this node's slice of the placement view, PAGED.

It used to be one unbounded request. A minimal 100k-placement answer encodes to ~28 MB, well past the agent's 16 MB JSON response ceiling, so the read failed and a cold agent fell back to an empty view — and the unfiltered endpoint does have a production caller: an ingress tier at or below MaxReplicatedIngressRouteNodes reaches it through an all-shards filter. Paging the page endpoint keeps every response bounded by MaxPlacementPageLimit rows instead of raising the ceiling.

func (*Agent) ProbeSecretOnPeers added in v0.6.1

func (a *Agent) ProbeSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, minGeneration int64) (holding []string, err error)

func (*Agent) PruneAuditACL added in v0.6.1

func (a *Agent) PruneAuditACL(context.Context, time.Time) error

func (*Agent) PublishArtifactCatalog added in v0.6.1

func (a *Agent) PublishArtifactCatalog(ctx context.Context, chunk ArtifactCatalogSnapshot) error

PublishArtifactCatalog forwards one chunk to the control plane.

func (*Agent) PushSecretBlobToAnyPeer added in v0.6.1

func (a *Agent) PushSecretBlobToAnyPeer(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

func (*Agent) PushSecretBlobToPeers added in v0.6.1

func (a *Agent) PushSecretBlobToPeers(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

Agent mirrors for worker nodes that seal locally and need to fan out.

func (*Agent) PutVolumeAttachments added in v0.5.4

func (a *Agent) PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error

func (*Agent) ReassignPlacement added in v0.5.3

func (a *Agent) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

func (*Agent) RecordPlacement

func (a *Agent) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Agent) RemoveCustomDomain added in v0.3.1

func (a *Agent) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

func (*Agent) RemoveExposedPort

func (a *Agent) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

func (*Agent) RemoveMember added in v0.2.2

func (a *Agent) RemoveMember(ctx context.Context, nodeID string, force bool) error

func (*Agent) ReportStorageObligations added in v0.6.1

func (a *Agent) ReportStorageObligations(ctx context.Context, report StorageObligationReport) error

ReportStorageObligations submits one owner report through the control plane.

func (*Agent) ReserveOnTarget

func (a *Agent) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

func (*Agent) ResolveCustomDomain added in v0.3.1

func (a *Agent) ResolveCustomDomain(hostname string) (string, bool)

ResolveCustomDomain returns ("", false) on agent nodes. A reverse-by-hostname lookup endpoint isn't wired through yet (the TLS-ask handler is intended to live alongside a node that runs raft locally — server/ingress roles). See task #17 for adding a remote lookup if agent-role ingress becomes a supported topology.

func (*Agent) RetireNodeStorage added in v0.6.1

func (a *Agent) RetireNodeStorage(ctx context.Context, nodeID, actor, reason string, attestedAt time.Time) error

RetireNodeStorage forwards the attestation to the control plane. Agents hold no FSM, so the command rides the same leader-forwarding path as every other write an agent issues.

func (*Agent) RevokeNodeStorageRetirement added in v0.6.1

func (a *Agent) RevokeNodeStorageRetirement(ctx context.Context, nodeID string) error

RevokeNodeStorageRetirement withdraws the attestation cluster-wide.

func (*Agent) SecretsOf

func (a *Agent) SecretsOf(sandboxID string) PlacementSecrets

func (*Agent) SelectPlacement

func (a *Agent) SelectPlacement(req capacity.Request) (PlacementTarget, error)

SelectPlacement asks the control plane for a target and nothing else.

func (*Agent) SelectPlacementForCreate added in v0.6.1

func (a *Agent) SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

SelectPlacementForCreate picks a target and the sandbox's seal recipients in one control-plane round trip, returning the bounded recipient ids instead of the candidate fleet.

func (*Agent) SelectPlacementWithCandidates added in v0.6.1

func (a *Agent) SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

func (*Agent) SelfAPIURL

func (a *Agent) SelfAPIURL() string

func (*Agent) SelfNodeID

func (a *Agent) SelfNodeID() string

func (*Agent) SetNodeDrainState

func (a *Agent) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

func (*Agent) SpecOf

func (a *Agent) SpecOf(sandboxID string) *models.CreateSandboxRequest

func (*Agent) StorageObligations added in v0.6.1

func (a *Agent) StorageObligations(ctx context.Context) ([]StorageObligationView, error)

StorageObligations asks one control-plane server for the view.

func (*Agent) SubscribePlacement

func (a *Agent) SubscribePlacement(ctx context.Context) <-chan struct{}

SubscribePlacement wakes the ingress reconciler on placement changes. Only the delta feed can provide that on an Agent; without it (flag off), this is nil and the reconciler runs on its timer, as before.

func (*Agent) UpdatePlacementSecretRecipients added in v0.6.1

func (a *Agent) UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

func (*Agent) UpsertSpec

func (a *Agent) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Agent) VolumeAttachmentCount added in v0.5.4

func (a *Agent) VolumeAttachmentCount(ctx context.Context, tenant, id string) (int, error)

func (*Agent) VolumeByID added in v0.5.4

func (a *Agent) VolumeByID(ctx context.Context, tenant, id string) (models.Volume, error)

func (*Agent) VolumeByName added in v0.5.4

func (a *Agent) VolumeByName(ctx context.Context, tenant, name string) (models.Volume, error)

func (*Agent) VolumeDelete added in v0.5.4

func (a *Agent) VolumeDelete(ctx context.Context, tenant, id string) error

func (*Agent) VolumeExistsForSource added in v0.5.4

func (a *Agent) VolumeExistsForSource(ctx context.Context, source string) (bool, error)

func (*Agent) VolumeUpsert added in v0.5.4

func (a *Agent) VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)

func (*Agent) VolumesForTenant added in v0.5.4

func (a *Agent) VolumesForTenant(ctx context.Context, tenant string) ([]models.Volume, error)

func (*Agent) WatchPlacementChanges added in v0.6.1

func (a *Agent) WatchPlacementChanges(ctx context.Context, fn PlacementChangeHandler) bool

WatchPlacementChanges registers fn with the delta feed. It returns false when the feed is off (flag off). fn first gets the current full view, if already synced.

type ArtifactCatalogEpochRequest added in v0.6.1

type ArtifactCatalogEpochRequest struct {
	Kind string `json:"kind"`
	// Holder identifies the requesting process. A retry from the same holder
	// is answered with the token it already has rather than a new one.
	Holder string `json:"holder"`
}

ArtifactCatalogEpochRequest asks the authority for one publisher token. The node is NOT a body field: the server fills it from the authenticated peer identity, so a node can only ever be issued its own.

type ArtifactCatalogEpochResponse added in v0.6.1

type ArtifactCatalogEpochResponse struct {
	Epoch int64 `json:"epoch"`
}

ArtifactCatalogEpochResponse carries exactly the issued token.

type ArtifactCatalogPage added in v0.6.1

type ArtifactCatalogPage struct {
	Rows []ArtifactCatalogRow `json:"rows"`
	// Publishers are the nodes whose inventory of this KIND the catalogue
	// holds in full. The aggregator asks everyone else. It does not depend on
	// the tenant being read: a node that published an inventory containing
	// nothing for this tenant has still answered for it.
	Publishers []string `json:"publishers"`
	// NextPageToken continues the walk. Publishers are complete on the first
	// page; rows are not, so a caller that needs every row pages to the end.
	NextPageToken string `json:"next_page_token,omitempty"`
	// Authoritative distinguishes "nothing published yet" from "could not
	// ask". A non-authoritative answer must fall back to the fan-out.
	Authoritative bool `json:"authoritative,omitempty"`
	// PublisherEpoch is the epoch the catalogue holds for the node that
	// ASKED. A publisher takes the next one, which is how a restarted process
	// outranks requests its predecessor may still have in flight. It is
	// filled from the authenticated peer identity on the peer path, never
	// from the request body.
	PublisherEpoch int64 `json:"publisher_epoch,omitempty"`
}

ArtifactCatalogPage is what a reader gets back.

type ArtifactCatalogRequest added in v0.6.1

type ArtifactCatalogRequest struct {
	Kind      string `json:"kind"`
	Tenant    string `json:"tenant,omitempty"`
	Limit     int    `json:"limit,omitempty"`
	PageToken string `json:"page_token,omitempty"`
	// ForNodeID asks for that node's publisher epoch alongside the page. The
	// peer handler overrides it with the authenticated identity; a node may
	// only ever ask for its own.
	ForNodeID string `json:"for_node_id,omitempty"`
}

ArtifactCatalogRequest is the agent-facing read.

type ArtifactCatalogRow added in v0.6.1

type ArtifactCatalogRow struct {
	ID string `json:"id"`
	// Tenant scopes the row inside its kind. Empty means "not tenant-scoped"
	// (templates). It lives on the row, not on the publication, because
	// coverage has to be answerable for a tenant the publisher holds nothing
	// for.
	Tenant  string `json:"tenant,omitempty"`
	Payload []byte `json:"payload"`
}

ArtifactCatalogRow is one artifact's metadata as its holder serialized it. The FSM does not interpret Payload — the API layer owns the wire type — so adding a field to models.Template cannot require an FSM change.

type ArtifactCatalogSnapshot added in v0.6.1

type ArtifactCatalogSnapshot struct {
	Kind string `json:"kind"`
	// NodeID is the publisher. Callers never supply another node's id: the
	// peer-facing path takes it from the authenticated identity.
	NodeID string `json:"node_id"`
	// Epoch is the AUTHORITY-ISSUED fencing token for the publishing process:
	// a node reads the catalogue's committed epoch at boot and takes the next
	// one. A process identifier alone (a UUID) identifies a process without
	// ordering processes, so a request delayed in transport from a replaced
	// process could take ownership back and republish an obsolete inventory.
	Epoch int64 `json:"epoch"`
	// Revision orders publications WITHIN one epoch.
	Revision int64                `json:"revision"`
	Rows     []ArtifactCatalogRow `json:"rows"`
	// First starts a new snapshot (discarding any half-delivered one) and
	// Final commits it. A single-chunk snapshot sets both.
	First bool `json:"first,omitempty"`
	Final bool `json:"final,omitempty"`
	// Withdraw removes this node's coverage instead of replacing it. A node
	// whose inventory cannot be represented — over the per-node cap — must
	// say so: the aggregator skips a node it believes it covers, so leaving
	// the old coverage in place advertises an inventory the node no longer
	// has and nothing ever asks it again.
	Withdraw bool `json:"withdraw,omitempty"`
}

ArtifactCatalogSnapshot is one node's whole inventory of one kind, as the publisher wants it to become. It is delivered as a sequence of chunks and applied all-or-nothing.

func ChunkArtifactCatalogSnapshot added in v0.6.1

func ChunkArtifactCatalogSnapshot(kind, nodeID string, epoch, revision int64, rows []ArtifactCatalogRow) []ArtifactCatalogSnapshot

ChunkArtifactCatalogSnapshot splits an inventory into commands that fit the apply transport. The sequence is always non-empty: an EMPTY inventory is a real statement ("this node holds nothing of this kind"), and losing it is what keeps a tenant's list asking every worker forever.

func WithdrawArtifactCatalogCoverage added in v0.6.1

func WithdrawArtifactCatalogCoverage(kind, nodeID string, epoch, revision int64) ArtifactCatalogSnapshot

WithdrawArtifactCatalogCoverage builds the publication that removes this node's coverage of a kind.

type AuditACL added in v0.6.1

type AuditACL struct {
	SandboxID           string   `json:"sandbox_id"`
	IncarnationID       string   `json:"incarnation_id,omitempty"`
	OwnerRef            string   `json:"owner_ref"`
	AuditNodeIDs        []string `json:"audit_node_ids,omitempty"`
	AuditNodesTruncated bool     `json:"audit_nodes_truncated,omitempty"`
	ExpiresUnix         int64    `json:"expires_unix,omitempty"`
	// RetainedVersion is the Raft log index of the delete that created this
	// row. It provides a deterministic latest-lifecycle ordering without wall
	// clocks or a second replicated table.
	RetainedVersion uint64 `json:"retained_version"`
}

AuditACL is the minimal post-delete existence/authorization and evidence-node record retained in Raft. OwnerRef may be empty for operator-owned local workflows. It deliberately carries no spec, routes, secret refs, or events.

type AuditACLResponse added in v0.6.1

type AuditACLResponse struct {
	ACL    AuditACL `json:"acl"`
	Exists bool     `json:"exists"`
}

type AuditEventDTO added in v0.6.1

type AuditEventDTO = auditlog.Event

AuditEventDTO is the wire shape of one secret-audit event on the peer internal endpoint. Kept in cluster to avoid an import cycle with service.

type AuditPeerFetcher added in v0.6.1

type AuditPeerFetcher interface {
	FetchSandboxAuditFromPeer(ctx context.Context, nodeID, sandboxID string, limit int, cursor, kind, incarnationID string) (AuditPeerPage, error)
}

AuditPeerFetcher is the narrow seam Service uses for audit fan-out so tests can inject a fake without a full Cluster.

type AuditPeerPage added in v0.6.1

type AuditPeerPage struct {
	Events     []AuditEventDTO `json:"events"`
	NextCursor string          `json:"next_cursor,omitempty"`
}

AuditPeerPage is the JSON body returned by the internal audit endpoint.

type Client

type Client interface {
	// SelfNodeID returns this node's stable cluster identifier.
	SelfNodeID() string
	// SelfAPIURL returns this node's externally-reachable API base URL.
	SelfAPIURL() string

	// OwnerOf returns the node currently owning sandboxID, or
	// ErrUnknownSandbox if no placement record exists.
	OwnerOf(sandboxID string) (OwnerInfo, error)

	// OwnerOfName resolves a cluster-wide sandbox name to its sandbox ID and
	// current owner. Names are indexed from replicated specs and exist for
	// facade APIs that accept either ID or name. Returns ErrUnknownSandbox if
	// no placement record claims name.
	OwnerOfName(name string) (string, OwnerInfo, error)

	// SelectPlacement chooses a node to host a new sandbox with the given
	// resource request. In single-node mode it always returns self.
	SelectPlacement(req capacity.Request) (PlacementTarget, error)

	// SelectPlacementWithCandidates is SelectPlacement plus the filtered
	// candidate slice used to build the secret recipient set at reserve time.
	// Callers that only need the winner keep using SelectPlacement.
	SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

	// RecordPlacement commits sandboxID -> self into the FSM along with the
	// (optional) creation spec used to re-materialize the sandbox after a
	// failover. Idempotent — re-recording with the same owner is a no-op, and
	// passing spec=nil preserves any spec that was previously replicated for
	// this id (so a boot-time replay can't erase a richer record written by
	// the original CreateSandbox call).
	//
	// spec MUST be redacted (no plaintext credentials) before being passed in;
	// secrets is the provider handle produced by the service layer. Passing an
	// empty handle preserves any previously replicated ref (mirrors the
	// spec-preservation rule — a boot-time replay that has the spec but not
	// the secrets can't erase the original create's recoverability handle).
	RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

	// ClaimOrphan promotes an orphaned placement back to self. It succeeds
	// only when the placement is currently orphaned and either has no recorded
	// previous owner (legacy row) or was orphaned from this node. It preserves
	// any replicated spec/secrets when the caller passes nil.
	ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

	// UpsertSpec replaces the replicated spec for sandboxID without touching
	// ownership. Mutating handlers (resize, lifecycle) call this after a
	// successful local mutation so the FSM stays current — otherwise a
	// failover-recreated sandbox would revert to its create-time shape.
	//
	// spec MUST be redacted. An empty secrets handle preserves the previously
	// replicated ref — resize/lifecycle never touch credentials, so passing an
	// empty handle is the right choice for those callers; the "real" secrets
	// are only re-shipped when the user re-runs create or rotates them
	// explicitly.
	UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

	// SelectPlacementForCreate is SelectPlacement plus the seal recipients for
	// sandboxID, chosen where the membership already lives. It exists so a
	// create never has to ship the candidate fleet to the caller: at 2,000
	// nodes SelectPlacementWithCandidates makes every create's answer O(fleet)
	// in bytes and allocations to produce at most a handful of node ids.
	// recipientBackups <= 0 means the create wants no fan-out and skips the
	// selection entirely.
	SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

	// UpdatePlacementSecretRecipients replaces Placement.SecretRecipients
	// (and optionally the seal handle after a reseal) without touching
	// ownership or IncarnationID. expectedIncarnationID / expectedOwnerNodeID /
	// expectedSealGeneration are CAS pretenses: when the live placement
	// differs, the FSM rejects with ErrSecretRecipientsCASMismatch.
	//
	// expectedOwnerNodeID is not redundant with the other two. opReassign
	// changes the owner while preserving both the incarnation and the seal
	// generation, so without it a node that has just lost the lifecycle can
	// still land a reseal it started as owner. Pass the node this caller
	// believes coordinates the secret: itself when it owns the placement, or
	// "" for the leader-coordinated ownerless case (the empty expectation is
	// still an expectation — the FSM compares it).
	UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

	// SpecOf returns the most-recently-replicated CreateSandboxRequest for
	// sandboxID, or nil if no spec is recorded (pre-cluster sandbox, or no
	// placement). The returned spec is REDACTED — no plaintext secrets. Use
	// SecretsOf to retrieve the matching provider handle if you need to
	// reconstruct the full spec for a recreate.
	SpecOf(sandboxID string) *models.CreateSandboxRequest

	// SecretsOf returns the secret provider handle paired with SpecOf's spec.
	// Empty when the sandbox was created without any private-registry / mount
	// credentials, or when the placement predates secret replication.
	SecretsOf(sandboxID string) PlacementSecrets

	// AddExposedPort records intent that sandboxID has port exposed. Raw TCP
	// routes include HostPort so every ingress node can bind/proxy the same
	// cluster-wide endpoint. Idempotent when the route metadata is unchanged.
	AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

	// RemoveExposedPort drops a port intent from the placement. Idempotent.
	RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

	// ExposedPortsOf returns a copy of the replicated port route map for
	// sandboxID, or nil if none. Used by the recreator and ingress reconciler.
	ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

	// AddCustomDomain records intent that sandboxID owns hostname (already
	// canonicalized lower-case) cluster-wide. Returns ErrCustomHostnameConflict
	// when a different sandbox already holds it. Idempotent for the same
	// (sandbox, hostname) pair so retries are safe.
	AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

	// RemoveCustomDomain drops hostname from sandboxID's set. No-op when the
	// hostname isn't claimed by this sandbox (the local row may already be
	// gone after a Delete) so callers can retry without special-casing.
	RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

	// CustomDomainsOf returns a copy of the replicated hostname set for
	// sandboxID, or nil when no placement exists or the set is empty. The
	// ingress reconciler uses it to populate Caddy matchers on every node.
	CustomDomainsOf(sandboxID string) []string

	// ResolveCustomDomain answers "which sandbox owns this hostname?" from
	// the local FSM in O(1). Used by the TLS-ask handler on ingress nodes
	// that may not own the sandbox row themselves. hostname is matched
	// case-insensitively. Returns sandboxID, true when known.
	ResolveCustomDomain(hostname string) (string, bool)

	// DeletePlacement removes sandboxID from the FSM. Idempotent.
	DeletePlacement(ctx context.Context, sandboxID string) error
	// DeletePlacementExact removes only the placement lifecycle still owned by
	// expectedOwnerNodeID. It is the destroy/finalizer primitive: a delayed
	// cleanup must not erase an ID-reused or concurrently reassigned placement.
	DeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error
	// BeginDeletePlacementExact atomically freezes an exact owner/lifecycle
	// before irreversible secret and external-artifact cleanup. It is
	// idempotent for the same lifecycle and rejects reassignment races.
	BeginDeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error
	// AuditOwnerRef resolves the bounded, short-grace routing stub Raft keeps
	// for a deleted sandbox (owner_ref + evidence nodes). The stub exists only
	// for SB_AUDIT_DELETED_GRACE and under SB_AUDIT_DELETED_INDEX_MAX; the
	// durable authorization record is the evidence node's sandbox_audit_acl
	// row, and long-term history is the export backend's. PruneAuditACL
	// removes expired stubs through a deterministic Raft command.
	AuditOwnerRef(ctx context.Context, sandboxID string) (string, bool, error)
	// AuditACLForSandbox resolves an exact retained lifecycle when
	// incarnationID is non-empty, or the newest retained lifecycle otherwise.
	AuditACLForSandbox(ctx context.Context, sandboxID, incarnationID string) (AuditACL, bool, error)
	PruneAuditACL(ctx context.Context, cutoff time.Time) error

	// ReserveOnTarget writes opReserve into the FSM holding capacity + name
	// for target before the body is forwarded to it. The reservation carries
	// the redacted spec + secret ref so the target's RecordPlacement promote
	// step can run with nil Spec/empty Secrets and inherit them
	// atomically. ttl bounds how long the reservation can hold capacity if
	// the target never promotes (the leader's GC sweep cancels expired rows
	// at ~5s tick cadence). Spec MUST already be redacted — the raft log
	// must NOT carry plaintext credentials.
	ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

	// CancelReservation drops a pending reservation from the FSM. No-op on
	// missing or already-promoted (Placed) rows, so router rollback / TTL GC
	// / late successful promote can race harmlessly.
	CancelReservation(ctx context.Context, sandboxID string) error

	// SetNodeDrainState marks nodeID as drained (excluded from
	// SelectPlacement) or restores it to the candidate pool. Idempotent on
	// both edges. The mark lives in the FSM and survives the drained node
	// going away — the operator's intent must outlast the process they're
	// about to stop.
	SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

	// ReassignPlacement moves sandboxID to target without touching the
	// replicated spec or port intents. Used by WASM live migration once the
	// receiving node has imported a §4.8.1 checkpoint.
	ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

	// Platform-volume metadata replication. These keep a Daytona volume's
	// id/name/source consistent cluster-wide so get/list/delete-by-id survive the
	// tenant's API ownership moving between nodes. The data itself is
	// deterministic in S3/NFS and is never replicated.
	//
	// VolumeUpsert is an idempotent get-or-create: a duplicate (tenant, name)
	// converges on the existing row. It returns the canonical row and whether
	// this call created it, or ErrVolumeQuotaExceeded.
	VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)
	// VolumeDelete removes the (tenant, id) row, or returns ErrUnknownVolume.
	VolumeDelete(ctx context.Context, tenant, id string) error
	// VolumeByID / VolumeByName resolve a single row or return ErrUnknownVolume.
	VolumeByID(ctx context.Context, tenant, id string) (models.Volume, error)
	VolumeByName(ctx context.Context, tenant, name string) (models.Volume, error)
	// VolumesForTenant lists the tenant's rows, newest first.
	VolumesForTenant(ctx context.Context, tenant string) ([]models.Volume, error)
	// VolumeExistsForSource reports whether any replicated row points at source
	// (the reclaim worker's cluster-wide live-data guard).
	VolumeExistsForSource(ctx context.Context, source string) (bool, error)
	// VolumeAttachmentCount returns the cluster-wide live attachment count for a
	// volume. Put/Delete keep this index in the same raft log as volume metadata
	// so Daytona delete cannot miss a sandbox that lives on another worker.
	VolumeAttachmentCount(ctx context.Context, tenant, id string) (int, error)
	PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error
	DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID, incarnationID string) error

	// RemoveMember explicitly removes nodeID from the raft configuration after
	// marking it drained and orphaning any placements it owned. Unknown raft
	// members return ErrUnknownMember. Live members require force=true so an
	// operator cannot accidentally cut out a healthy server by typo.
	RemoveMember(ctx context.Context, nodeID string, force bool) error

	// IsNodeDrained reports whether nodeID is currently marked drained.
	// Reads the local FSM (no network hop). Used by observability endpoints
	// — placement scoring uses an internal accessor that takes one lock for
	// the whole sweep.
	IsNodeDrained(nodeID string) bool

	// ApplyEncoded is the receiving end of leader-forwarded raft writes. The
	// internal API endpoint pipes the request body through here on the leader
	// so any owner-side mutating call (Record/Upsert/Add/Remove/Delete) made on
	// a follower can transparently land on the leader's raft. Returns
	// ErrNotLeader if leadership has shifted; the forwarder retries.
	ApplyEncoded(ctx context.Context, payload []byte) error

	// AssertOwnership ensures the FSM lists self as owner for every entry in
	// local, and backfills any missing Spec / ExposedPorts so failover-recreate
	// works for sandboxes that pre-date the spec-replication features. Used at
	// boot. Idempotent.
	AssertOwnership(ctx context.Context, local []LocalSandboxState) error

	// ForwardHTTP reverse-proxies r to the given peer. Target NodeID and
	// InternalURL are required and the TLS leaf is pinned to node:<NodeID>.
	ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

	// AttachInternalHandler wires the API server's HTTP handler into the
	// cluster-internal mTLS listener so peers can reverse-proxy owner API calls
	// over the cert-pinned channel (not just leader-forwarded raft applies).
	// No-op for Noop and for Cluster instances with SB_CLUSTER_TLS_DIR unset.
	// Safe to call exactly once after construction; subsequent calls overwrite.
	AttachInternalHandler(h http.Handler)

	// Members returns a snapshot of all known cluster members.
	Members() []Member

	// LocalMembers returns the local gossip membership view without a
	// control-plane HTTP round-trip. Hot paths (list failover_ready) prefer
	// this over Members() on agent nodes so a page of sandboxes does not
	// fan out N membership RPCs.
	LocalMembers() []Member

	// IngressTargets aggregates live ingress-role nodes' gossiped PublicHost
	// values into the set of public addresses users must point DNS at for
	// custom domains. Hostnames and raw IPs are partitioned via net.ParseIP;
	// duplicates are removed; output ordering is stable so the API response
	// is byte-identical across calls when membership is unchanged. Source
	// reflects whether the cluster published a hostname, IPs, both, or
	// nothing usable (see models.IngressTargetSource*). Service-layer
	// IngressDNSTarget wraps this; clients hit GET /v1/ingress/dns.
	IngressTargets() models.IngressTarget

	// Placements returns the local FSM's hot placement snapshot. Recovery
	// payloads (Spec/secrets) are omitted; use PlacementOf/SpecOf for point
	// lookups that need them.
	Placements() []Placement

	// PlacementsForShards returns only placements whose sandbox ID belongs to
	// one of the requested placement shards. Ingress nodes use this instead of
	// pulling the full global placement map; server nodes serve it from the
	// FSM shard index and agent nodes delegate it to a server-role control
	// plane peer. Empty Shards means "all placements." Returned rows are hot
	// rows without Spec/secrets.
	PlacementsForShards(filter PlacementShardFilter) []Placement

	// PlacementPage returns a bounded global placement-index page sorted by
	// sandbox ID. It is the scalable read path for operators/control planes
	// that need to enumerate large clusters without asking every worker for
	// its local DB. Returned rows are hot rows without Spec/secrets.
	PlacementPage(req PlacementPageRequest) PlacementPageResponse

	// PlacementOf returns the full Placement record for sandboxID and true,
	// or a zero Placement and false if no record exists. Operator/debug
	// endpoints use this for convergence-status reads where the per-aspect
	// getters (OwnerOf/ExposedPortsOf) would lose the placement.Version
	// needed to compute "is this sandbox's route installed yet on this node".
	PlacementOf(sandboxID string) (Placement, bool)

	// PlacementsByIDs returns hot placement rows for the requested sandbox IDs
	// without dumping the full Placements() view. Missing IDs are omitted.
	// Used by list/get failover_ready batching so a page of N sandboxes does
	// not pay for a full FSM scan.
	PlacementsByIDs(ids []string) map[string]Placement

	// AuthoritativePlacementsByIDs returns the same bounded point set from the
	// current Raft leader. Unlike PlacementsByIDs, an unavailable or changing
	// leader is an error rather than an empty result. Destructive reconcilers
	// use this so a stale follower/control-plane outage cannot masquerade as
	// placement absence and retire live lifecycle data.
	AuthoritativePlacementsByIDs(ctx context.Context, ids []string) (map[string]Placement, error)

	// PlacementVersion is the FSM's monotonic apply counter — bumps on every
	// raft log entry the FSM applied. Exposed for metrics/observability and
	// as a tie-breaker for tests; the ingress reconciler now uses
	// SubscribePlacement to wake on apply rather than polling this counter.
	// Zero means "no version data" (Noop or fresh cluster).
	PlacementVersion() uint64

	// SubscribePlacement returns a buffered (cap=1) channel that receives a
	// signal after every FSM apply on this node. The channel is fed directly
	// from FSM.Apply, so a leader-side commit reaches every node's
	// subscribers as soon as raft delivers the log entry — no poll interval.
	//
	// Multiple applies between reads collapse into one wake (cap=1, drops on
	// full). Cancel ctx (or the returned cancel func) to deregister; both
	// are safe to call multiple times. Callers MUST tolerate spurious wakes —
	// the channel says "something changed in the FSM," not "the placement
	// you care about changed."
	//
	// In single-node mode (Noop) the returned channel never fires; this is
	// safe to use in a select{} alongside a ticker because Go's select treats
	// a never-firing nil channel as permanently un-ready, not an error.
	SubscribePlacement(ctx context.Context) <-chan struct{}

	// Leader returns the node ID of the current Raft leader, empty if none.
	Leader() string

	// Close shuts down the cluster cleanly.
	Close() error
}

Client is the surface the rest of the daemon (Service, API handlers) interacts with. Both *Cluster and *Noop satisfy it so callsites stay unconditional.

type Cluster

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster is the multi-node Client implementation. Construct with New.

func New

func New(cfg config.Config, logger *slog.Logger, admitter *capacity.Admitter) (*Cluster, error)

New constructs the server-role Cluster for cfg.EnableCluster=true. Caller takes ownership of Close. For worker/ingress-only roles call NewAgent; for cfg.EnableCluster=false call NewNoop.

func (*Cluster) AddCustomDomain added in v0.3.1

func (c *Cluster) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

AddCustomDomain replicates a sandbox→hostname binding through raft so every node can answer the TLS-ask probe and install the matching Caddy matcher. hostname is canonicalized (trim + lower) here so callers don't have to remember to do it themselves — the local SQLite layer already trims. Idempotent for the same (sandbox, hostname) pair; returns ErrCustomHostnameConflict when the hostname is held by a different sandbox.

func (*Cluster) AddExposedPort

func (c *Cluster) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

AddExposedPort replicates a port-exposure intent. Idempotent when the route metadata is unchanged. Safe to call from any node.

func (*Cluster) AllocateArtifactCatalogEpoch added in v0.6.1

func (c *Cluster) AllocateArtifactCatalogEpoch(ctx context.Context, kind, nodeID, holder string) (int64, error)

AllocateArtifactCatalogEpoch issues this process its fencing token through the replicated log.

Why an allocation and not a read: reading the committed epoch and locally choosing "one more" claims nothing. Two processes for the same node — a restart overlapping its predecessor, or a node rejoining while its old process drains — both read the same value, both pick the same successor, and the catalogue can no longer order them: whichever has the higher REVISION wins, which is the opposite of what the token is for. Allocating through the log makes every token distinct and monotonic.

func (*Cluster) ApplyEncoded

func (c *Cluster) ApplyEncoded(ctx context.Context, payload []byte) error

ApplyEncoded is the receiving side of leader-forwarded raft writes. It decodes-validates the payload (so a malformed body never reaches the FSM) and applies it locally. Returns ErrNotLeader if leadership has changed since the forwarder picked us — the caller is expected to retry.

func (*Cluster) ArtifactCatalog added in v0.6.1

ArtifactCatalog reads one page of one catalogue from the local FSM.

func (*Cluster) ArtifactCatalogForPeer added in v0.6.1

func (c *Cluster) ArtifactCatalogForPeer(req ArtifactCatalogRequest) ArtifactCatalogPage

ArtifactCatalogForPeer answers the agent-facing read.

func (*Cluster) ArtifactCatalogHolders added in v0.6.1

func (c *Cluster) ArtifactCatalogHolders(kind, tenant, id string) []string

ArtifactCatalogHolders returns the nodes whose committed inventory of kind holds (tenant, id), sorted. A point read for routing one artifact request: the gossip inventory that item routes use first trails a create by a capacity heartbeat, while a creator publishes here before it answers, so "created, then read at once" must fall back to this or 404.

func (*Cluster) AssertOwnership

func (c *Cluster) AssertOwnership(ctx context.Context, local []LocalSandboxState) error

AssertOwnership reconciles local sandbox state against the cluster FSM at boot. Used at boot. Idempotent. Best-effort callers should log errors and retry later.

Three-way decision per local row, with the FSM as the source of truth for ownership (never overwrite an existing non-self owner):

  • **No FSM placement.** Write opPlace claiming self as owner, with spec + secret ref so the new placement is born recoverable. This is the normal case for sandboxes created before AssertOwnership last ran.

  • **FSM owner == self.** No ownership change. Backfill missing Spec / secret ref via opUpsertSpec (closes the pre-spec-replication gap) and replay ExposedPort intents via opAddExposedPort (no-ops when already recorded).

  • **FSM orphaned from self.** Reclaim via opClaimOrphan, then replay the same spec/port backfill. This is the false-positive dead-owner recovery path: gossip/Raft marked us dead, but the local sandbox never left.

  • **FSM owner != self, or orphaned from another owner.** **Do NOT reclaim.** This is the failover-recovery/stale-local-row case. Calling RecordPlacement here would overwrite another owner or steal a different dead node's orphan. Instead: log loudly and leave the FSM alone. service.reconcileStaleOwnership handles destroying stale local copies on its next pass when the placement has an active different owner.

Edge cases:

  • Owner unknown to gossip yet (fresh boot, gossip not converged): we still defer. Treating "presumed alive" as the safe default keeps us from racing a peer whose announce just hasn't reached us. The next reconcile pass reclassifies.
  • Owner is dead in gossip: still defer until the leader has committed the orphan transition. Claiming before the FSM records which previous owner lost the row would recreate the overwrite hazard.

func (*Cluster) AttachInternalHandler

func (c *Cluster) AttachInternalHandler(h http.Handler)

AttachInternalHandler wires the public API mux into the cluster-internal mTLS listener so peers can reverse-proxy owner API calls over the cert-pinned channel. No-op only for the single-node/noop construction path where there is no listener to attach to. Called once from cmd/sandboxd after the API server is constructed; the order avoids a service→cluster→api construction cycle.

func (*Cluster) AttachRecreator

func (c *Cluster) AttachRecreator(r SandboxRecreator)

AttachRecreator wires the service-layer recreate hook used by the owner watcher. Called once from cmd/sandboxd/main after both service.New and cluster.New have returned. Safe to call concurrently with the watcher loop.

func (*Cluster) AuditACLForSandbox added in v0.6.1

func (c *Cluster) AuditACLForSandbox(_ context.Context, sandboxID, incarnationID string) (AuditACL, bool, error)

func (*Cluster) AuditOwnerRef added in v0.6.1

func (c *Cluster) AuditOwnerRef(ctx context.Context, sandboxID string) (string, bool, error)

func (*Cluster) AuthoritativeNodeStorageRetirements added in v0.6.1

func (c *Cluster) AuthoritativeNodeStorageRetirements(ctx context.Context) ([]NodeStorageRetirement, error)

AuthoritativeNodeStorageRetirements answers from the leader, forwarding when this node is a follower.

Discharging a deletion obligation without an ACK is irreversible, so it may not be authorized by a lagging replica or a cached set: an operator who revokes an attestation on one node must not have another node still discharging against it. A follower's own FSM says Authoritative=true about its own state and still cannot order itself against the revoke.

Server and mixed nodes need this as much as workers do — they hold delete outboxes of their own, and without it every one of them fails closed forever on an attestation the operator did make.

func (*Cluster) AuthoritativePlacementsByIDs added in v0.6.1

func (c *Cluster) AuthoritativePlacementsByIDs(ctx context.Context, ids []string) (map[string]Placement, error)

AuthoritativePlacementsByIDs serves destructive reconciliation from the current leader. Followers fetch the bounded result over the same node-pinned mTLS channel used for forwarded Raft operations; they never substitute a potentially stale local FSM read.

func (*Cluster) BeginDeletePlacementExact added in v0.6.1

func (c *Cluster) BeginDeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Cluster) CancelReservation

func (c *Cluster) CancelReservation(ctx context.Context, sandboxID string) error

CancelReservation deletes a pending reservation for sandboxID. Safe to call at any rollback point: opCancelReserve only fires when the row is State == Reserved, so a stale cancel after a successful promote is a no-op. Idempotent; calling on a never-reserved id is also a no-op.

func (*Cluster) ClaimOrphan

func (c *Cluster) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

ClaimOrphan commits sandboxID -> self only if the existing placement is currently orphaned and was orphaned from this node (or predates the previous-owner metadata). This is the false-positive recovery path for a node that was marked dead by gossip but never actually lost its local sandbox.

func (*Cluster) ClientForPeer added in v0.6.1

func (c *Cluster) ClientForPeer(nodeID string) *http.Client

ClientForPeer returns a cached mTLS HTTP client that verifies DNS SAN node:<nodeID>. Legacy shared-SAN-only peer certs are rejected.

func (*Cluster) Close

func (c *Cluster) Close() error

Close shuts down gossip + raft. Idempotent.

func (*Cluster) CustomDomainsOf added in v0.3.1

func (c *Cluster) CustomDomainsOf(sandboxID string) []string

CustomDomainsOf returns a sorted copy of the hostnames bound to sandboxID, or nil when none. Used by the ingress reconciler so a peer-owned sandbox still gets its TLS matchers installed in the local Caddy.

func (*Cluster) DeletePlacement

func (c *Cluster) DeletePlacement(ctx context.Context, sandboxID string) error

DeletePlacement removes sandboxID from the placement map. Idempotent. The expected owner/incarnation come from the local FSM — the same source RemoveCustomDomain uses — so a follower destroy does not POST to the Raft leader before the write. applyCommand still commits on the leader; a stale local row fails the exact CAS instead of 503ing every delete during an election.

func (*Cluster) DeletePlacementExact added in v0.6.1

func (c *Cluster) DeletePlacementExact(ctx context.Context, sandboxID, expectedOwnerNodeID, expectedIncarnationID string) error

func (*Cluster) DeleteSecretOnPeers added in v0.6.1

func (c *Cluster) DeleteSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, generation int64) (acked []string, err error)

DeleteSecretOnPeers DELETEs the sandbox's cluster_secrets rows on peers that may hold a fan-out copy. It returns only authenticated acknowledgements; callers derive pending recipients from requested minus acknowledged so an incomplete transport response can never erase a durable delete obligation.

func (*Cluster) DeleteVolumeAttachmentsForSandbox added in v0.5.4

func (c *Cluster) DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID, incarnationID string) error

func (*Cluster) DrainedNodes added in v0.6.1

func (c *Cluster) DrainedNodes() map[string]bool

DrainedNodes reads the FSM's drained set with no network hop.

func (*Cluster) ExposedPortsOf

func (c *Cluster) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

ExposedPortsOf returns a copy of the replicated port route map. Returns nil if no placement exists or no ports are recorded.

func (*Cluster) FetchSandboxAuditFromPeer added in v0.6.1

func (c *Cluster) FetchSandboxAuditFromPeer(ctx context.Context, nodeID, sandboxID string, limit int, cursor, kind, incarnationID string) (AuditPeerPage, error)

FetchSandboxAuditFromPeer GETs a peer's local audit slice for sandboxID.

func (*Cluster) ForwardHTTP

func (c *Cluster) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

ForwardHTTP reverse-proxies r to target and writes the response into w. The PAT bearer header is preserved (httputil.ReverseProxy passes through headers by default; we only strip hop-by-hop ones).

NodeID and InternalURL are required. The proxy transport is taken from the per-peer client cache, which pins the TLS leaf to node:<NodeID>.

Forwarding loops are detected by the X-Cluster-Forwarded header and returned as 421 Misdirected Request so clients retry against a refreshed placement.

IMPORTANT: a hard network/TLS failure on the internal channel must NOT silently fall back to the public path — that would defeat the cert-pinned security promise B3 is meant to enforce. The reverse proxy's ErrorHandler surfaces such failures as 502 to the original caller, who can then retry against a different peer.

func (*Cluster) HealthyForReads

func (c *Cluster) HealthyForReads() bool

HealthyForReads is true once the FSM has caught up to the leader's last log index — i.e. our local OwnerOf reads are not stale by more than a single round trip. Used by EnsureClusterReady.

func (*Cluster) IngressTargets added in v0.3.1

func (c *Cluster) IngressTargets() models.IngressTarget

IngressTargets aggregates live ingress-role members' gossiped PublicHost values. See aggregateIngressTargets for the partition / dedup / ordering rules — those are pinned by ingress_targets_test.go.

func (*Cluster) IsNodeDrained

func (c *Cluster) IsNodeDrained(nodeID string) bool

IsNodeDrained reports the FSM's view of nodeID's drain state. Read from the local FSM with no network hop.

func (*Cluster) Leader

func (c *Cluster) Leader() string

Leader returns the node ID of the current raft leader. Empty if no leader.

func (*Cluster) LeaderAPIURL

func (c *Cluster) LeaderAPIURL() string

LeaderAPIURL returns the API URL of the current leader, or empty if unknown. Used by the API wrapper to forward mutating raft writes to the leader.

func (*Cluster) LocalMembers added in v0.6.1

func (c *Cluster) LocalMembers() []Member

LocalMembers returns the gossip membership view without capacity enrichment. Hot paths (list failover_ready) use this to avoid per-row Members() work.

func (*Cluster) LookupMember added in v0.6.1

func (c *Cluster) LookupMember(id string) (Member, bool)

LookupMember returns one gossip member by node ID without scanning the full membership list. Used by placement-page list hydration (O(owners)).

func (*Cluster) Members

func (c *Cluster) Members() []Member

Members returns gossip-known members (self included), enriched with fresh capacity heartbeats where available.

func (*Cluster) NodeStorageRetirements added in v0.6.1

func (c *Cluster) NodeStorageRetirements(context.Context) ([]NodeStorageRetirement, error)

NodeStorageRetirements reads the replicated attestation set from the local FSM. Server-role nodes hold the FSM, so this is a local read.

func (*Cluster) NodeStorageRetirementsForPeer added in v0.6.1

func (c *Cluster) NodeStorageRetirementsForPeer() NodeStorageRetirementsResponse

NodeStorageRetirementsForPeer answers the agent-facing read.

func (*Cluster) NodeStorageRetirementsForPeerAuthoritative added in v0.6.1

func (c *Cluster) NodeStorageRetirementsForPeerAuthoritative(ctx context.Context) (NodeStorageRetirementsResponse, error)

NodeStorageRetirementsForPeerAuthoritative is the peer-facing form of the same authoritative read: the barrier belongs to the read, not to the caller, so a worker asking the leader gets the same guarantee a server does.

func (*Cluster) OwnedRecoveryPlacements added in v0.6.1

func (c *Cluster) OwnedRecoveryPlacements(ownerID string, limit int, pageToken string) OwnedRecoveryResponse

OwnedRecoveryPlacements answers an owner-scoped recovery query for ownerID. Only placements that node owns AND that opted into failover recreate are returned, with the spec and secret handle the owner needs to rebuild them.

Callers must pass the mTLS-authenticated peer identity as ownerID; nothing here derives the owner from a request body.

func (*Cluster) OwnerOf

func (c *Cluster) OwnerOf(sandboxID string) (OwnerInfo, error)

OwnerOf reads the placement map from the local FSM (no network round-trip). Returns ErrUnknownSandbox if no row exists, or ErrOrphaned if the placement exists but has no active owner (auto-orphaned after the owning node died — see voter_autojoin / dead-owner reconciler).

func (*Cluster) OwnerOfName

func (c *Cluster) OwnerOfName(name string) (string, OwnerInfo, error)

OwnerOfName resolves a replicated sandbox Name to its placement owner. This is intentionally a local FSM read just like OwnerOf; the name index is maintained inside the FSM apply path so it tracks the authoritative placement map exactly.

func (*Cluster) PeerDialMember added in v0.6.1

func (c *Cluster) PeerDialMember(m Member) (*http.Client, string, error)

PeerDialMember selects the peer client/URL using the per-node mTLS cache.

func (*Cluster) PeerInternalHTTPClient added in v0.6.1

func (c *Cluster) PeerInternalHTTPClient() *http.Client

PeerInternalHTTPClient exposes the cert-pinned client for cluster list fan-out. End-user Authorization must only ride this internal client.

func (*Cluster) PlacementChanges added in v0.6.1

func (c *Cluster) PlacementChanges(ctx context.Context, since uint64, wait time.Duration) PlacementChangesResponse

PlacementChanges answers a delta-feed poll. With nothing newer than since, it parks until the FSM changes or wait expires. It subscribes BEFORE the first read, so an apply landing between read and park cannot be missed.

A server that is BEHIND the cursor (a lagging follower, or an Agent that switched servers) waits instead of resnapshotting: since is a global Raft index, so "I haven't applied that far yet" is not a gap. Next never moves below since.

func (*Cluster) PlacementOf

func (c *Cluster) PlacementOf(sandboxID string) (Placement, bool)

PlacementOf returns the FSM record for sandboxID. Goes through f.get so the returned Placement is deep-cloned and safe to mutate without aliasing live FSM state (same guarantee as Placements).

func (*Cluster) PlacementPage

func (c *Cluster) PlacementPage(req PlacementPageRequest) PlacementPageResponse

func (*Cluster) PlacementVersion

func (c *Cluster) PlacementVersion() uint64

PlacementVersion returns the FSM's monotonic apply counter — bumps on every committed raft log entry. Exposed for metrics and as a tie-breaker for tests; the ingress reconciler uses SubscribePlacement to wake on apply rather than polling this counter.

func (*Cluster) Placements

func (c *Cluster) Placements() []Placement

func (*Cluster) PlacementsByIDs added in v0.6.1

func (c *Cluster) PlacementsByIDs(ids []string) map[string]Placement

PlacementsByIDs returns hot placement rows for the given IDs (point lookups).

func (*Cluster) PlacementsForShards

func (c *Cluster) PlacementsForShards(filter PlacementShardFilter) []Placement

func (*Cluster) ProbeSecretOnPeers added in v0.6.1

func (c *Cluster) ProbeSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, minGeneration int64) (holding []string, err error)

ProbeSecretOnPeers HEADs peer secret rows and returns nodes that currently hold seal_generation >= minGeneration (authoritative possession, not ACK memory).

func (*Cluster) PruneAuditACL added in v0.6.1

func (c *Cluster) PruneAuditACL(ctx context.Context, cutoff time.Time) error

func (*Cluster) PublishArtifactCatalog added in v0.6.1

func (c *Cluster) PublishArtifactCatalog(ctx context.Context, chunk ArtifactCatalogSnapshot) error

PublishArtifactCatalog applies one chunk of a snapshot.

func (*Cluster) PushSecretBlobToAnyPeer added in v0.6.1

func (c *Cluster) PushSecretBlobToAnyPeer(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

PushSecretBlobToAnyPeer races all eligible recipients and returns as soon as one authenticated peer ACKs. The create path uses this narrow minimum-ACK operation so an unreachable recipient cannot spend the entire ACK window before a healthy backup is attempted. Full convergence continues through PushSecretBlobToPeers and the durable put outbox.

func (*Cluster) PushSecretBlobToPeers added in v0.6.1

func (c *Cluster) PushSecretBlobToPeers(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)

PushSecretBlobToPeers POSTs the sealed blob to each non-self recipient that is alive and advertises a reachable internal mTLS URL. Best-effort with bounded backoff; returns the node IDs that ACK'd (callers intersect with live membership for failover_ready). Missing cluster transport is reported for any remote target; self-only single-node calls remain a no-op.

func (*Cluster) PutVolumeAttachments added in v0.5.4

func (c *Cluster) PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error

func (*Cluster) ReassignPlacement added in v0.5.3

func (c *Cluster) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

ReassignPlacement moves sandboxID to target via opReassign.

func (*Cluster) ReassignStuckPlacement added in v0.6.1

func (c *Cluster) ReassignStuckPlacement(ctx context.Context, requesterID, sandboxID, incarnationID string) error

ReassignStuckPlacement runs the leader's stuck-placement reassignment on behalf of a worker that cannot recreate a sandbox it owns. requesterID is the mTLS-authenticated caller; only the current owner may ask.

func (*Cluster) RecordPlacement

func (c *Cluster) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

RecordPlacement commits sandboxID -> self into the FSM via raft along with the (optional) creation spec. Idempotent. Safe to call from any node: applyCommand transparently forwards to the current leader if we're a follower. Passing spec=nil preserves a previously-recorded spec — see fsm.go opPlace handling.

spec MUST be redacted before being passed in; secrets is the provider handle the caller produces via service.SealAndDistribute. Passing an empty handle preserves a previously-recorded handle.

func (*Cluster) RecoveryBlob added in v0.2.2

func (c *Cluster) RecoveryBlob(ctx context.Context, ref string) (RecoveryBlob, bool, error)

func (*Cluster) RemoveCustomDomain added in v0.3.1

func (c *Cluster) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

RemoveCustomDomain releases the binding. Idempotent — a stale call after a successful DeletePlacement is a no-op.

func (*Cluster) RemoveExposedPort

func (c *Cluster) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

RemoveExposedPort drops a replicated port-exposure intent. Idempotent.

func (*Cluster) RemoveMember added in v0.2.2

func (c *Cluster) RemoveMember(ctx context.Context, nodeID string, force bool) error

RemoveMember explicitly retires nodeID from the raft configuration. It is an operator lifecycle command, not gossip failure detection: the caller should drain the node first, then stop/terminate it, then remove it from raft.

Before RemoveServer, we persist a drain mark and orphan any placements owned by the retiring node so surviving nodes stop routing new work there and clients get a clear 410 for lost sandboxes under the current non-HA policy.

func (*Cluster) ReportStorageObligations added in v0.6.1

func (c *Cluster) ReportStorageObligations(ctx context.Context, report StorageObligationReport) error

ReportStorageObligations submits one owner report. On the leader it is queued for the next batch; elsewhere it is forwarded to the leader, which queues it there.

func (*Cluster) ReserveBatchOnTargets added in v0.2.2

func (c *Cluster) ReserveBatchOnTargets(ctx context.Context, reservations []PlacementReservation) error

ReserveBatchOnTargets commits multiple reservation intents in one Raft entry. It is intended for create-burst frontends that have already selected targets and want the leader to admit the whole chunk against the same capacity view. The public one-create API still uses ReserveOnTarget, but scale harnesses and bulk create callers use this to avoid one Raft log entry per sandbox.

func (*Cluster) ReserveOnTarget

func (c *Cluster) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

ReserveOnTarget commits a capacity-and-name reservation for sandboxID owned by target before any docker side effect runs. The router calls this after SelectPlacement so the cluster has intent recorded the instant the router forwards the body to target. ttl bounds how long the reservation holds the slot before the leader GC sweep cancels it; pick large enough to cover the slowest expected image pull on the target.

redacted MUST be stripped of plaintext credentials (call service.RedactClusterSecrets); secrets is the secret provider handle. Both ride the reservation so a successful promote via opPlace can inherit them without re-shipping the payload — see fsm.go opPlace preserve-on-empty rules.

Returns ErrReservationConflict when the slot is already placed or held by a different owner; ErrNameConflict when redacted.Name is held cluster-wide. Either should map to 4xx at the API surface; transport errors stay 5xx (caller may retry).

func (*Cluster) ResolveCustomDomain added in v0.3.1

func (c *Cluster) ResolveCustomDomain(hostname string) (string, bool)

ResolveCustomDomain answers the TLS-ask probe from the local FSM index in O(1). Returns sandboxID, true when the hostname is bound somewhere in the cluster; false otherwise. hostname is canonicalized inside the FSM.

func (*Cluster) RetireNodeStorage added in v0.6.1

func (c *Cluster) RetireNodeStorage(ctx context.Context, nodeID, actor, reason string, attestedAt time.Time) error

RetireNodeStorage records the attestation in the replicated control plane.

func (*Cluster) RevokeNodeStorageRetirement added in v0.6.1

func (c *Cluster) RevokeNodeStorageRetirement(ctx context.Context, nodeID string) error

RevokeNodeStorageRetirement withdraws the attestation cluster-wide.

func (*Cluster) SecretsOf

func (c *Cluster) SecretsOf(sandboxID string) PlacementSecrets

SecretsOf returns a copy of the provider handle paired with SpecOf's spec.

func (*Cluster) SelectPlacement

func (c *Cluster) SelectPlacement(req capacity.Request) (PlacementTarget, error)

SelectPlacement chooses an owner node for a new sandbox using power-of-two- choices. The algorithm samples up to 2 random alive members (including self) and picks the one with the most CPU+memory headroom relative to the requested resources. This gives near-optimal load balancing without any global coordination — at scale it converges to within a small constant factor of the truly-optimal choice while costing O(1) per placement.

Falls back to self if no alive members are visible yet (just-bootstrapped node) or if no other member can satisfy req. Self always loses ties because "place locally" is the existing single-node behavior; we only forward when a peer is genuinely better.

func (*Cluster) SelectPlacementForCreate added in v0.6.1

func (c *Cluster) SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

SelectPlacementForCreate is SelectPlacement plus the bounded seal recipient set for sandboxID. The server-side member of the pair: no candidate slice crosses a process boundary here, because there is no boundary to cross.

func (*Cluster) SelectPlacementWithCandidates added in v0.6.1

func (c *Cluster) SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

SelectPlacementWithCandidates is SelectPlacement plus the filtered candidate slice the router uses to record the secret recipient set at reserve time (plans/secrets-hardening §3d-1). The target is always one of candidates.

func (*Cluster) SelfAPIURL

func (c *Cluster) SelfAPIURL() string

func (*Cluster) SelfNodeID

func (c *Cluster) SelfNodeID() string

func (*Cluster) SetLocalTemplateCatalogProvider added in v0.6.1

func (c *Cluster) SetLocalTemplateCatalogProvider(fn func() ([]string, bool))

SetLocalTemplateCatalogProvider registers the all-lifecycle template catalogue callback used by administrative item routing.

func (*Cluster) SetLocalTemplateIDsProvider added in v0.4.1

func (c *Cluster) SetLocalTemplateIDsProvider(fn func() ([]string, bool))

SetLocalTemplateIDsProvider registers the Phase 6 PR-D callback used by the capacity lease cache to overlay Firecracker template inventory onto local heartbeats. Idempotent; passing nil unregisters. Wired by main.go after the service is attached. Single-node deployments and nodes without Firecracker enabled never call this and degrade to nil-provider behaviour (no LocalTemplateIDs on the snapshot, peers see the legacy "unknown, allow" placement).

func (*Cluster) SetLocalWasmModuleIDsProvider added in v0.5.3

func (c *Cluster) SetLocalWasmModuleIDsProvider(fn func() ([]string, bool))

SetLocalWasmModuleIDsProvider registers the WASM module inventory callback.

func (*Cluster) SetNodeDrainState

func (c *Cluster) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

SetNodeDrainState marks nodeID as drained (drained=true) or restores it to the SelectPlacement candidate pool (drained=false). Goes through raft so every node — including the drained node itself — sees the new state on its next placement decision. Idempotent.

Drain is a no-op on existing placements: a drained node continues serving the sandboxes it already owns. Combine with the dead-owner reconciler (kill the process) to force ownership transfer, or use a separate evacuate command (not implemented) to move work proactively.

func (*Cluster) SpecOf

func (c *Cluster) SpecOf(sandboxID string) *models.CreateSandboxRequest

SpecOf returns a deep-copy of the replicated spec for sandboxID, or nil if none is recorded. The returned spec is REDACTED — registry passwords and mount credentials are stripped at write time. Use SecretsOf to retrieve the matching secret ref and service.OpenClusterSecretsForNode to merge credentials back in. Callers may safely mutate the returned struct (it shares no memory with the FSM).

func (*Cluster) StorageObligations added in v0.6.1

func (c *Cluster) StorageObligations(context.Context) ([]StorageObligationView, error)

StorageObligations serves the job view from the local FSM. The expected reporters are the members gossip reports alive that can own sandboxes.

func (*Cluster) SubscribePlacement

func (c *Cluster) SubscribePlacement(ctx context.Context) <-chan struct{}

SubscribePlacement returns a buffered (cap=1) wake channel that fires after every FSM apply on this node. Cancelling ctx removes the subscriber. See the Client.SubscribePlacement contract for semantics.

func (*Cluster) UpdatePlacementSecretRecipients added in v0.6.1

func (c *Cluster) UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

UpdatePlacementSecretRecipients commits a replacement seal recipient set (and optional new provider handle after reseal). Preserves IncarnationID. expectedIncarnationID / expectedSealGeneration CAS against the live placement.

func (*Cluster) UpsertSpec

func (c *Cluster) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

UpsertSpec replicates a sandbox spec mutation (resize, lifecycle change) without changing ownership. Idempotent; nil spec + empty secrets is a no-op. Safe to call from any node — applyCommand forwards to the leader as needed.

spec MUST be redacted; empty secrets preserves the previously replicated handle (resize/lifecycle never change credentials).

func (*Cluster) VolumeAttachmentCount added in v0.5.4

func (c *Cluster) VolumeAttachmentCount(_ context.Context, tenant, id string) (int, error)

func (*Cluster) VolumeByID added in v0.5.4

func (c *Cluster) VolumeByID(_ context.Context, tenant, id string) (models.Volume, error)

func (*Cluster) VolumeByName added in v0.5.4

func (c *Cluster) VolumeByName(_ context.Context, tenant, name string) (models.Volume, error)

func (*Cluster) VolumeDelete added in v0.5.4

func (c *Cluster) VolumeDelete(ctx context.Context, tenant, id string) error

func (*Cluster) VolumeExistsForSource added in v0.5.4

func (c *Cluster) VolumeExistsForSource(_ context.Context, source string) (bool, error)

func (*Cluster) VolumeUpsert added in v0.5.4

func (c *Cluster) VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)

func (*Cluster) VolumesForTenant added in v0.5.4

func (c *Cluster) VolumesForTenant(_ context.Context, tenant string) ([]models.Volume, error)

func (*Cluster) WatchPlacementChanges added in v0.6.1

func (c *Cluster) WatchPlacementChanges(ctx context.Context, fn PlacementChangeHandler) bool

WatchPlacementChanges streams this server's own change log to fn, in-process (no network): a full view first, then deltas, resyncing on resnapshot. It runs until ctx ends.

type ClusterTLS

type ClusterTLS struct {
	// contains filtered or unexported fields
}

ClusterTLS is the loaded TLS material the daemon uses for cluster-internal channels (raft transport + leader-forwarded HTTP applies). Built once at boot from the on-disk files and shared by the raft StreamLayer and the internal HTTPS listener / client.

func (*ClusterTLS) NodeID added in v0.6.1

func (t *ClusterTLS) NodeID() string

NodeID returns the node-bound identity extracted from the local node cert.

type DrainStateResponse

type DrainStateResponse struct {
	Drained bool `json:"drained"`
}

type DrainedNodesReader added in v0.6.1

type DrainedNodesReader interface {
	DrainedNodes() map[string]bool
}

DrainedNodesReader is implemented by clients that can return the drained set in one read. It is deliberately NOT part of Client: callers type-assert and fall back to per-node IsNodeDrained, so no test double has to grow a method it never exercises.

type DrainedNodesResponse added in v0.6.1

type DrainedNodesResponse struct {
	Drained []string `json:"drained"`
}

DrainedNodesResponse is the wire shape of PublicInternalDrainedNodesPath.

type Endpoint

type Endpoint struct {
	NodeID      string
	InternalURL string
	APIURL      string
}

Endpoint binds a peer identity to its cluster-internal mTLS URL. APIURL is response metadata; ForwardHTTP never uses it as a downgrade.

type ExposedPortRoute

type ExposedPortRoute struct {
	Protocol  string `json:"protocol"`
	HostPort  int    `json:"host_port,omitempty"`
	PublicURL string `json:"public_url,omitempty"`
}

Placement is one row of the FSM's placement map. Spec is the replicated sandbox creation request used by the new owner to re-materialize a sandbox after its previous owner died (see owner_watcher.go). Spec is a pointer so older snapshots written before spec replication still decode cleanly.

SecretRef/SecretVersion identify the secret provider record that can rehydrate the redacted Spec on the owner or an authorized recovery target. They are a handle, not secret material — raw sealed bytes have no field anywhere in cluster state, so Raft can never fan secret material out to every participant.

ExposedPortRoute is the replicated routing intent for one exposed sandbox port. Protocol drives which Caddy surface is used. HostPort is populated for raw TCP so every node can bind/proxy the same cluster-wide port.

type IngressRouteOwner

type IngressRouteOwner struct {
	NodeID        string `json:"node_id"`
	APIURL        string `json:"api_url,omitempty"`
	InternalURL   string `json:"internal_url,omitempty"`
	DataPlaneHost string `json:"data_plane_host,omitempty"`
}

type IngressShardFilterCache added in v0.6.1

type IngressShardFilterCache struct {
	// contains filtered or unexported fields
}

IngressShardFilterCache retains only the last topology for one reconciler. Capacity heartbeats and endpoint changes do not change HRW ownership. Compare the sorted live ingress IDs, not the entire membership (or a lossy hash).

func (*IngressShardFilterCache) ForNode added in v0.6.1

func (c *IngressShardFilterCache) ForNode(members []Member, nodeID, selfRole string) PlacementShardFilter

type IngressShardRoute

type IngressShardRoute struct {
	SandboxID  string              `json:"sandbox_id"`
	Shard      int                 `json:"shard"`
	ShardCount int                 `json:"shard_count"`
	Owners     []IngressRouteOwner `json:"owners"`
	// RingVersion identifies the membership view this answer was computed
	// from. Route installation and route lookup hash the same ordered set of
	// ingress node IDs, so two nodes that disagree about membership hand out
	// routes for different rings — the upstream can be pointed at a node that
	// never installed the shard. Publishing the version makes that divergence
	// observable (compare across ingress nodes) instead of silent.
	RingVersion string `json:"ring_version,omitempty"`
}

func IngressRouteForSandbox

func IngressRouteForSandbox(members []Member, sandboxID string) IngressShardRoute

type LocalSandboxState

type LocalSandboxState struct {
	ID           string
	Spec         *models.CreateSandboxRequest
	Secrets      PlacementSecrets
	ExposedPorts map[int]ExposedPortRoute
	// CustomHostnames is the per-sandbox bound hostname set known to the
	// local store. Carried through AssertOwnership so a sandbox created before
	// the FSM learned about its hostnames (e.g. cluster mode enabled after the
	// sandbox already had custom domains) backfills the replicated set on
	// boot — without this a failover-recreate would lose the user's TLS
	// matchers.
	CustomHostnames []string
}

LocalSandboxState is one entry in the boot-time AssertOwnership payload. Carrying Spec + ExposedPorts (rather than just an ID) lets the boot replay backfill the FSM with everything a future failover-recreate needs — without this, sandboxes that pre-date cluster mode (or pre-date the spec/ports replication features) would never gain a replicated spec until their next mutating call.

Spec MUST be redacted (no plaintext registry password / mount credentials) before being handed to AssertOwnership; Secrets carries the provider ref that the new owner re-merges on recreate. cmd/sandboxd takes care of this via service.SealAndDistribute + RedactClusterSecrets so the cluster layer never sees plaintext. Secrets may be empty when the sandbox has no secrets to ship.

type Member

type Member struct {
	NodeID string `json:"node_id"`
	// NodeName is the operator-friendly display label (SB_NODE_NAME). Empty
	// for peers running pre-NodeName builds; dashboards fall back to NodeID.
	NodeName      string `json:"node_name,omitempty"`
	APIURL        string `json:"api_url"`
	DataPlaneHost string `json:"data_plane_host,omitempty"`
	RaftAddr      string `json:"raft_addr,omitempty"`
	// InternalURL is the required cluster-internal mTLS endpoint used for peer
	// RPC and leader-forwarded applies.
	InternalURL string `json:"internal_url,omitempty"`
	// Role is the peer's gossiped SB_NODE_ROLE. Empty for older builds that
	// pre-date the field; callers treat empty as the legacy "mixed" default.
	Role string `json:"role,omitempty"`
	// PublicHost is the peer's gossiped public ingress address
	// (config.EffectivePublicHost). Aggregated by Cluster.IngressTargets to
	// answer the DNS-helper API. Empty for peers without a public host set
	// or running pre-PublicHost builds.
	PublicHost string            `json:"public_host,omitempty"`
	Alive      bool              `json:"alive"`
	Capacity   capacity.Snapshot `json:"capacity"`
	// CapacityUpdatedUnix is when this node's last capacity heartbeat was
	// observed by the scheduler. CapacityStale means the last heartbeat is
	// missing or too old for placement admission.
	CapacityUpdatedUnix int64 `json:"capacity_updated_unix,omitempty"`
	CapacityStale       bool  `json:"capacity_stale,omitempty"`
}

Member is a snapshot of a peer's gossiped state.

func IdentityMembers added in v0.6.1

func IdentityMembers(c Client) []Member

IdentityMembers is the membership view for callers that need only identity, role, liveness or endpoints — never capacity or artifact inventories.

Identity/liveness gossip is already fleet-wide and local, so these callers have the answer in-process. Members() on an agent is a control-plane round trip that redistributes the whole membership record (~850 B per peer, ~1.7 MB at 2,000 nodes) including capacity snapshots and template/module inventories the caller is about to throw away. Local gossip first; the control plane is the fallback for a node whose gossip is not up yet.

func IngressRingMembers added in v0.6.1

func IngressRingMembers(c Client) []Member

IngressRouteForSandbox returns the ingress owner set an upstream router should target for sandboxID. Small ingress tiers all own every sandbox route; very large ingress tiers return the primary shard owner plus one failover replica so a single ingress death does not create a traffic vacuum. IngressRingMembers is the single membership view the shard-aware ingress path may hash. Reconciliation (which shards this node installs) and the route lookup (which node the upstream is sent to) must agree: an Agent's Members() is a control-plane snapshot while LocalMembers() is local gossip, and hashing one on the install side and the other on the lookup side routes traffic to nodes that never installed the shard. Local gossip is the preferred source — it is what the installer can act on — with the control-plane view as the fallback for a node whose gossip is not up yet.

type NodeStorageRetirement added in v0.6.1

type NodeStorageRetirement struct {
	NodeID string `json:"node_id"`
	Actor  string `json:"actor,omitempty"`
	Reason string `json:"reason,omitempty"`
	// AttestedUnixNano is the fence discharge compares copy provenance
	// against, in NANOSECONDS. Second granularity would round the attestation
	// backwards, so a copy distributed a few hundred milliseconds before it
	// would read as newer than the attestation and stay pinned forever.
	AttestedUnixNano int64 `json:"attested_unix_nano"`
}

NodeStorageRetirement is an operator's attestation that a node's storage was destroyed, replicated through the placement FSM.

It has to be cluster state, not a row on the node that served the API call: the deletion obligations an attestation discharges live in the delete outbox of whichever node owns the secret, and an operator's request reaches an arbitrary ingress or server. A node-local table meant the owners never learned, and list/revoke through a different entry node saw a different set.

The payload is tiny administrative metadata — one row per decommissioned node — so replicating it costs nothing like the placement state does.

func (NodeStorageRetirement) AttestedAt added in v0.6.1

func (r NodeStorageRetirement) AttestedAt() time.Time

AttestedAt is the fence discharge decisions compare obligation provenance against.

type NodeStorageRetirementsResponse added in v0.6.1

type NodeStorageRetirementsResponse struct {
	Retirements []NodeStorageRetirement `json:"retirements"`
	// Authoritative distinguishes "no attestations" from "could not ask".
	// A discharge is irreversible evidence-wise, so a non-authoritative
	// answer must never be read as an empty set.
	Authoritative bool `json:"authoritative,omitempty"`
}

NodeStorageRetirementsResponse is the agent-facing read.

type Noop

type Noop struct {
	// contains filtered or unexported fields
}

Noop is the single-node-mode Client implementation. Every method behaves as if this node owns every sandbox and is the only cluster member. Used when cfg.EnableCluster is false so callsites can be unconditional.

func NewNoop

func NewNoop(nodeID, apiURL, publicHost string) *Noop

NewNoop returns a single-node Client. nodeID and apiURL are reported back for observability but never actually used for routing. publicHost is the operator-configured public ingress address (config.EffectivePublicHost) that IngressTargets reports as the single-member target. Empty when the daemon runs in IP-only mode (no SB_PUBLIC_HOST / SB_DOMAIN), which also means custom domains are disabled.

func (*Noop) AddCustomDomain added in v0.3.1

func (n *Noop) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error

func (*Noop) AddExposedPort

func (n *Noop) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error

func (*Noop) ApplyEncoded

func (n *Noop) ApplyEncoded(ctx context.Context, payload []byte) error

func (*Noop) AssertOwnership

func (n *Noop) AssertOwnership(ctx context.Context, local []LocalSandboxState) error

func (*Noop) AttachInternalHandler

func (n *Noop) AttachInternalHandler(h http.Handler)

AttachInternalHandler is a no-op for single-node mode — there's no mTLS listener to wire into, so nothing to do.

func (*Noop) AuditACLForSandbox added in v0.6.1

func (n *Noop) AuditACLForSandbox(context.Context, string, string) (AuditACL, bool, error)

func (*Noop) AuditOwnerRef added in v0.6.1

func (n *Noop) AuditOwnerRef(context.Context, string) (string, bool, error)

func (*Noop) AuthoritativePlacementsByIDs added in v0.6.1

func (n *Noop) AuthoritativePlacementsByIDs(context.Context, []string) (map[string]Placement, error)

func (*Noop) BeginDeletePlacementExact added in v0.6.1

func (n *Noop) BeginDeletePlacementExact(context.Context, string, string, string) error

func (*Noop) CancelReservation

func (n *Noop) CancelReservation(ctx context.Context, sandboxID string) error

func (*Noop) ClaimOrphan

func (n *Noop) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Noop) Close

func (n *Noop) Close() error

func (*Noop) CustomDomainsOf added in v0.3.1

func (n *Noop) CustomDomainsOf(sandboxID string) []string

func (*Noop) DeletePlacement

func (n *Noop) DeletePlacement(_ context.Context, sandboxID string) error

func (*Noop) DeletePlacementExact added in v0.6.1

func (n *Noop) DeletePlacementExact(_ context.Context, sandboxID, _, incarnationID string) error

func (*Noop) DeleteVolumeAttachmentsForSandbox added in v0.5.4

func (n *Noop) DeleteVolumeAttachmentsForSandbox(_ context.Context, sandboxID, incarnationID string) error

func (*Noop) ExposedPortsOf

func (n *Noop) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute

func (*Noop) FetchSandboxAuditFromPeer added in v0.6.1

func (n *Noop) FetchSandboxAuditFromPeer(context.Context, string, string, int, string, string, string) (AuditPeerPage, error)

FetchSandboxAuditFromPeer on Noop always fails — single-node mode has no peers.

func (*Noop) ForwardHTTP

func (n *Noop) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)

func (*Noop) IngressTargets added in v0.3.1

func (n *Noop) IngressTargets() models.IngressTarget

IngressTargets reports the single-node deployment's public address as the DNS target. Empty publicHost (IP-only mode) returns the Unknown source so the service layer can surface a clean 412 rather than fake records.

func (*Noop) IsNodeDrained

func (n *Noop) IsNodeDrained(nodeID string) bool

func (*Noop) Leader

func (n *Noop) Leader() string

func (*Noop) LocalMembers added in v0.6.1

func (n *Noop) LocalMembers() []Member

LocalMembers mirrors Members in single-node mode.

func (*Noop) LookupMember added in v0.6.1

func (n *Noop) LookupMember(id string) (Member, bool)

LookupMember resolves the single Noop member when id matches.

func (*Noop) Members

func (n *Noop) Members() []Member

func (*Noop) OwnerOf

func (n *Noop) OwnerOf(sandboxID string) (OwnerInfo, error)

func (*Noop) OwnerOfName

func (n *Noop) OwnerOfName(name string) (string, OwnerInfo, error)

func (*Noop) PeerInternalHTTPClient added in v0.6.1

func (n *Noop) PeerInternalHTTPClient() *http.Client

func (*Noop) PlacementOf

func (n *Noop) PlacementOf(sandboxID string) (Placement, bool)

PlacementOf has no record in single-node mode — there's no FSM. Returns the zero Placement and false so callers fall back to the local sandbox row.

func (*Noop) PlacementPage

func (*Noop) PlacementVersion

func (n *Noop) PlacementVersion() uint64

PlacementVersion always returns 0 in single-node mode — there's no FSM and no need to wake an ingress reconciler that isn't running.

func (*Noop) Placements

func (n *Noop) Placements() []Placement

func (*Noop) PlacementsByIDs added in v0.6.1

func (n *Noop) PlacementsByIDs(ids []string) map[string]Placement

PlacementsByIDs is empty in single-node mode (no FSM).

func (*Noop) PlacementsForShards

func (n *Noop) PlacementsForShards(PlacementShardFilter) []Placement

func (*Noop) PruneAuditACL added in v0.6.1

func (n *Noop) PruneAuditACL(context.Context, time.Time) error

func (*Noop) PutVolumeAttachments added in v0.5.4

func (n *Noop) PutVolumeAttachments(_ context.Context, attachments []models.VolumeAttachment) error

func (*Noop) ReassignPlacement added in v0.5.3

func (n *Noop) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error

func (*Noop) RecordPlacement

func (n *Noop) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Noop) RemoveCustomDomain added in v0.3.1

func (n *Noop) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

func (*Noop) RemoveExposedPort

func (n *Noop) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error

func (*Noop) RemoveMember added in v0.2.2

func (n *Noop) RemoveMember(ctx context.Context, nodeID string, force bool) error

func (*Noop) ReserveOnTarget

func (n *Noop) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error

func (*Noop) ResolveCustomDomain added in v0.3.1

func (n *Noop) ResolveCustomDomain(hostname string) (string, bool)

func (*Noop) SecretsOf

func (n *Noop) SecretsOf(sandboxID string) PlacementSecrets

func (*Noop) SelectPlacement

func (n *Noop) SelectPlacement(req capacity.Request) (PlacementTarget, error)

func (*Noop) SelectPlacementForCreate added in v0.6.1

func (n *Noop) SelectPlacementForCreate(req capacity.Request, sandboxID string, recipientBackups int) (PlacementTarget, []string, error)

func (*Noop) SelectPlacementWithCandidates added in v0.6.1

func (n *Noop) SelectPlacementWithCandidates(req capacity.Request) (PlacementTarget, []Member, error)

func (*Noop) SelfAPIURL

func (n *Noop) SelfAPIURL() string

func (*Noop) SelfNodeID

func (n *Noop) SelfNodeID() string

func (*Noop) SetNodeDrainState

func (n *Noop) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error

func (*Noop) SpecOf

func (n *Noop) SpecOf(sandboxID string) *models.CreateSandboxRequest

func (*Noop) SubscribePlacement

func (n *Noop) SubscribePlacement(ctx context.Context) <-chan struct{}

SubscribePlacement returns nil in single-node mode. Selecting on a nil channel never proceeds, so an ingress reconciler that select{}s on this channel + a slow ticker just behaves as if the cluster never has placement events — which is exactly the truth.

func (*Noop) UpdatePlacementSecretRecipients added in v0.6.1

func (n *Noop) UpdatePlacementSecretRecipients(ctx context.Context, sandboxID string, recipients []string, secrets PlacementSecrets, expectedIncarnationID, expectedOwnerNodeID string, expectedSealGeneration int64) error

func (*Noop) UpsertSpec

func (n *Noop) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error

func (*Noop) VolumeAttachmentCount added in v0.5.4

func (n *Noop) VolumeAttachmentCount(_ context.Context, tenant, id string) (int, error)

func (*Noop) VolumeByID added in v0.5.4

func (n *Noop) VolumeByID(_ context.Context, tenant, id string) (models.Volume, error)

func (*Noop) VolumeByName added in v0.5.4

func (n *Noop) VolumeByName(_ context.Context, tenant, name string) (models.Volume, error)

func (*Noop) VolumeDelete added in v0.5.4

func (n *Noop) VolumeDelete(_ context.Context, tenant, id string) error

func (*Noop) VolumeExistsForSource added in v0.5.4

func (n *Noop) VolumeExistsForSource(_ context.Context, source string) (bool, error)

func (*Noop) VolumeUpsert added in v0.5.4

func (n *Noop) VolumeUpsert(_ context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)

func (*Noop) VolumesForTenant added in v0.5.4

func (n *Noop) VolumesForTenant(_ context.Context, tenant string) ([]models.Volume, error)

type OwnedRecoveryRequest added in v0.6.1

type OwnedRecoveryRequest struct {
	Limit     int    `json:"limit,omitempty"`
	PageToken string `json:"page_token,omitempty"`
}

OwnedRecoveryRequest asks the control plane for the recovery work assigned to the CALLING node. The owner is never taken from the body: the handler uses the mTLS-authenticated peer identity, so a node can only ever ask for its own placements.

type OwnedRecoveryResponse added in v0.6.1

type OwnedRecoveryResponse struct {
	Placements    []Placement `json:"placements"`
	NextPageToken string      `json:"next_page_token,omitempty"`
	// Authoritative distinguishes "this node owns no recovery work" from
	// "the control plane could not answer". The watcher must not treat the
	// second as the first.
	Authoritative bool `json:"authoritative,omitempty"`
}

OwnedRecoveryResponse carries full placements — spec and secret handle included — for the caller's own failover-recreate sandboxes. It is the worker-side equivalent of the FSM read a server-role owner watcher does locally; without it a dedicated worker has no way to learn that a placement was reassigned to it.

type OwnerInfo

type OwnerInfo struct {
	NodeID string
	APIURL string
	// InternalURL is the owner's required cluster-internal mTLS URL.
	InternalURL string
	IsSelf      bool
}

OwnerInfo is returned by OwnerOf.

type Placement

type Placement struct {
	SandboxID           string              `json:"sandbox_id"`
	OwnerNodeID         string              `json:"owner_node_id"`
	OwnerAPIURL         string              `json:"owner_api_url"`
	OwnerDataPlaneHost  string              `json:"owner_data_plane_host,omitempty"`
	OwnerState          PlacementOwnerState `json:"owner_state,omitempty"`
	OrphanedOwnerNodeID string              `json:"orphaned_owner_node_id,omitempty"`
	OrphanedUnix        int64               `json:"orphaned_unix,omitempty"`
	Version             uint64              `json:"version"`
	CreatedUnix         int64               `json:"created_unix"`
	UpdatedUnix         int64               `json:"updated_unix"`
	// Name is the small, hot copy of Spec.Name used to rebuild the cluster-wide
	// uniqueness index without loading the larger recovery spec payload.
	Name string `json:"name,omitempty"`
	// RecoveryRef points at the out-of-snapshot recovery payload for this row.
	// Spec/secret fields are hydrated from that store only for point lookups and
	// recreate flows.
	RecoveryRef string                       `json:"-"`
	Spec        *models.CreateSandboxRequest `json:"spec,omitempty"`
	// PublicTraffic mirrors Spec.AllowPublicTraffic on the HOT row.
	//
	// Spec is split into the recovery store on write and redacted from every
	// paged/point read, so a dedicated ingress node — which has no FSM and
	// reads only those pages — never saw it. The ingress route builder skips
	// any placement whose public-traffic flag it cannot see, so on a hetero
	// cluster it installed no L4 route for any remote sandbox: raw TCP
	// exposures were unreachable (T18, UC-34). HTTP kept working only through
	// the per-request ingress proxy fallback. One bool rides the row instead.
	//
	// false means "not public, or recorded by a build that predates this
	// field"; placementAllowsPublicTraffic still consults Spec when present,
	// so a legacy row behaves exactly as before until its next write.
	PublicTraffic bool   `json:"public_traffic,omitempty"`
	SecretRef     string `json:"secret_ref,omitempty"`
	SecretVersion int    `json:"secret_version,omitempty"`
	// SecretRecipients is the seal recipient set recorded at reserve time
	// (owner + N backups). The create target seals to this set and must not
	// recompute it. It is empty only when the placement has no replicated
	// credential payload.
	SecretRecipients []string `json:"secret_recipients,omitempty"`
	// IncarnationID uniquely tags this placement lifetime for secret refs /
	// seal bindings. Minted at reserve and preserved on reassign/promote.
	IncarnationID string `json:"incarnation_id,omitempty"`
	// SecretSealGeneration is the last Raft-coordinated seal generation for
	// this placement (reseal CAS). It is 0 only when no secret was sealed.
	SecretSealGeneration int64 `json:"secret_seal_generation,omitempty"`
	// OwnerRef is the control-plane tenant account (tenancy). Distinct from
	// OwnerNodeID (which cluster node hosts the sandbox).
	OwnerRef string `json:"owner_ref,omitempty"`
	// AuditNodeIDs is the bounded history of nodes that have owned this
	// sandbox and may therefore hold local audit evidence. It lets audit reads
	// remain O(owner changes), rather than fanning out to every worker after a
	// failover or delete. AuditNodesTruncated forces the reader to fall back to
	// full worker fan-out so the bound can never create a silent evidence gap.
	AuditNodeIDs        []string `json:"audit_node_ids,omitempty"`
	AuditNodesTruncated bool     `json:"audit_nodes_truncated,omitempty"`

	ExposedPorts      map[int]string           `json:"exposed_ports,omitempty"`
	ExposedPortRoutes map[int]ExposedPortRoute `json:"exposed_port_routes,omitempty"`
	// CustomHostnames is the replicated set of user-bound hostnames pointing at
	// this sandbox. Kept sorted and lower-cased so snapshot bytes are stable
	// across rebuilds and every node derives the same Caddy matcher order. The
	// FSM additionally maintains a hostname→sandboxID index so the TLS-ask
	// resolver and ingress reconciler can look up ownership in O(1) from any
	// node, including the ingress-only ones that don't run the local SQLite
	// custom-domains table. Hostnames are added/removed via the
	// opAddCustomDomain/opRemoveCustomDomain raft commands; opPlace preserves
	// the existing slice the same way ExposedPorts is preserved so a
	// re-place/reassign cannot erase domains a prior raft entry installed.
	CustomHostnames []string `json:"custom_hostnames,omitempty"`
	// State is empty for materialized placements, PlacementStateReserved for
	// capacity-only intents, and PlacementStateDeleting while lifecycle-wide
	// finalization owns the row. Reserved and deleting states are eligible for
	// TTL-driven reconciliation; opPlace promotes a reservation to empty.
	State PlacementState `json:"state,omitempty"`
	// ExpiresUnix bounds reserved and deleting states. The leader reconciler
	// cancels expired reservations and completes abandoned exact-lifecycle
	// deletes after their finalization lease expires.
	ExpiresUnix int64 `json:"expires_unix,omitempty"`
}

ExposedPorts is the legacy replicated set of port→protocol intents. Newer code also fills ExposedPortRoutes with HostPort/PublicURL metadata so remote ingress nodes can install owner-aware data-plane routes. Both fields are kept so older snapshots still restore cleanly.

func (Placement) IsDeleting added in v0.6.1

func (p Placement) IsDeleting() bool

IsDeleting reports whether lifecycle-wide finalization owns the placement.

func (Placement) IsOrphaned

func (p Placement) IsOrphaned() bool

IsOrphaned reports whether the placement has no active owner.

func (Placement) IsReserved

func (p Placement) IsReserved() bool

IsReserved reports whether p is a reservation awaiting promotion.

type PlacementChange added in v0.6.1

type PlacementChange struct {
	// Index is the Raft index of the newest change seen for this id in the
	// served range. For a delete, a client drops its row only if the row's
	// Version is not above Index.
	Index     uint64     `json:"index"`
	SandboxID string     `json:"sandbox_id"`
	Deleted   bool       `json:"deleted,omitempty"`
	Placement *Placement `json:"placement,omitempty"`
}

PlacementChange is one sandbox's latest state since a cursor.

type PlacementChangeHandler added in v0.6.1

type PlacementChangeHandler func(full []Placement, changes []PlacementChange)

PlacementChangeHandler receives placement changes in order. full is non-nil after a (re)sync and is then the COMPLETE view; changes follow incrementally. Handlers must not block for long.

type PlacementChangeWatcher added in v0.6.1

type PlacementChangeWatcher interface {
	WatchPlacementChanges(ctx context.Context, fn PlacementChangeHandler) bool
}

PlacementChangeWatcher streams placement changes to a consumer that keeps its own index (the ingress route responder). It is implemented by the Agent (from its delta feed) and by a server Cluster (from its own change log, in-process).

type PlacementChangesResponse added in v0.6.1

type PlacementChangesResponse struct {
	Changes []PlacementChange `json:"changes"`
	// Next is the cursor to send as since on the next call.
	Next uint64 `json:"next"`
	// Resnapshot means the log no longer covers since. The client must
	// page-walk /internal/placements/page and then resume from Next.
	Resnapshot bool   `json:"resnapshot,omitempty"`
	Floor      uint64 `json:"floor"`
}

PlacementChangesResponse answers GET /v1/cluster/internal/placements/changes.

type PlacementLookupResponse

type PlacementLookupResponse struct {
	SandboxID string    `json:"sandbox_id"`
	Placement Placement `json:"placement"`
	Owner     OwnerInfo `json:"owner"`
	Orphaned  bool      `json:"orphaned"`
}

type PlacementOwnerState

type PlacementOwnerState string

PlacementOwnerState records the ownership lifecycle independently from the reservation lifecycle. Empty means the placement has an active owner.

const (
	PlacementOwnerStateActive   PlacementOwnerState = ""
	PlacementOwnerStateOrphaned PlacementOwnerState = "orphaned"
)

type PlacementPageRequest

type PlacementPageRequest struct {
	Limit       int                  `json:"limit,omitempty"`
	PageToken   string               `json:"page_token,omitempty"`
	ShardFilter PlacementShardFilter `json:"shard_filter,omitempty"`
	// OwnerRef, when set, returns only placements for that tenant account.
	OwnerRef string `json:"owner_ref,omitempty"`
	// OwnerNodeID, when set, returns only placements this cluster node owns.
	// Distinct from OwnerRef (tenant account). It is the read path a worker's
	// reconcile sweep uses to enumerate its own rows: at 100k sandboxes the
	// unfiltered view is ~72 MB, which every worker would otherwise pull on
	// every sweep. Reserved and orphaned rows are excluded — the FSM's owner
	// index only tracks materialized placements.
	OwnerNodeID string `json:"owner_node_id,omitempty"`
}

func (PlacementPageRequest) Normalize

type PlacementPageResponse

type PlacementPageResponse struct {
	Placements    []Placement `json:"placements"`
	NextPageToken string      `json:"next_page_token,omitempty"`
	// Authoritative is true when the page was produced by a ready FSM / control
	// plane. Empty Authoritative pages mean "tenant (or fleet) has zero rows"
	// — not "placement view unavailable". Agents set this false on CP errors.
	Authoritative bool `json:"authoritative,omitempty"`
	// SkippedSandboxIDs names rows that could not be delivered because one row
	// alone exceeds the response byte budget. They are neither present nor
	// absent, and a caller that makes cleanup decisions from this view (route
	// GC deletes routes for placements it cannot see) must treat them as
	// unavailable rather than deleted.
	SkippedSandboxIDs []string `json:"skipped_sandbox_ids,omitempty"`
}

type PlacementReservation added in v0.2.2

type PlacementReservation struct {
	SandboxID string
	Target    PlacementTarget
	Redacted  *models.CreateSandboxRequest
	Secrets   PlacementSecrets
	TTL       time.Duration
}

PlacementReservation is one item in a leader-side batch reservation request. Redacted MUST have plaintext credentials stripped before the caller passes it in; Secrets carries only a provider handle (and optional Recipients for the reserve-time seal set).

type PlacementSecrets

type PlacementSecrets struct {
	Ref     string `json:"ref,omitempty"`
	Version int    `json:"version,omitempty"`
	// Recipients is the seal recipient set chosen at reserve time for
	// failover.policy=recreate. It rides opReserve onto Placement.SecretRecipients
	// (node IDs only — never ciphertext). Reserved promotes normally repeat the
	// same set; boot ownership replay carries the durable local set so rebuilding
	// a missing placement cannot silently downgrade HA to one holder.
	Recipients []string `json:"recipients,omitempty"`
	// OwnerRef is the control-plane tenant account for this sandbox. Not
	// secret material — rides Place/Reserve so failover recreate preserves
	// tenancy when the owner-watcher uses an unscoped internal context.
	OwnerRef string `json:"owner_ref,omitempty"`
	// IncarnationID tags this placement lifetime for secret binding. Minted at
	// reserve and preserved across reassign.
	IncarnationID string `json:"incarnation_id,omitempty"`
	// SealGeneration is the seal generation last coordinated via Raft
	// (reseal / recipient expansion).
	SealGeneration int64 `json:"seal_generation,omitempty"`
}

PlacementSecrets is the recoverability handle paired with a redacted placement spec. It carries a secret provider reference only — the encrypted secret payload lives behind the service secret provider, never in Raft state. There is deliberately no field for raw sealed bytes: the raft log and FSM snapshots are not erasable per-sandbox, so making the field unrepresentable is what enforces the "no secret material in the log" rule.

type PlacementShardFilter

type PlacementShardFilter struct {
	ShardCount int   `json:"shard_count,omitempty"`
	Shards     []int `json:"shards,omitempty"`
	// None selects NO shards at all. It exists because the zero value already
	// means "all shards": a node with no public-route work of its own had no
	// way to say so, and the difference between "nothing" and "the whole
	// fleet" is a full placement download per node on every reconcile tick.
	None bool `json:"none,omitempty"`
}

PlacementShardFilter asks the control plane for only a subset of placement shards. Empty Shards means "all shards" for compatibility with existing callers; ShardCount <= 0 uses DefaultPlacementShardCount.

func IngressShardFilterForNode

func IngressShardFilterForNode(members []Member, nodeID, selfRole string) PlacementShardFilter

IngressShardFilterForNode returns the placement shard subset a single ingress-capable node should reconcile. Small ingress tiers replicate the full public route table to each ingress node so ordinary DNS round-robin / TCP load balancers work for every sandbox. Very large ingress tiers shard the table and require a shard-aware upstream router.

selfRole is this node's own SB_NODE_ROLE. A node that does not serve ingress gets NoPlacementShards: it has no public-route work, and synthesizing a ring membership for it (the old behavior when it was absent from the ingress ids) handed a dedicated worker either a slice of unrelated shards or, at small ingress counts, the entire placement map.

func NoPlacementShards added in v0.6.1

func NoPlacementShards() PlacementShardFilter

NoPlacementShards is the filter a node with no public-route responsibility asks with.

func (PlacementShardFilter) Normalize

Normalize returns a deterministic, de-duplicated copy of f.

type PlacementState

type PlacementState string

PlacementState distinguishes a reservation (capacity held, no docker yet) from a placement (sandbox materialized). Empty defaults to Placed so pre-reservation snapshots restore correctly: every old row is a real placement, never a pending reservation.

const (
	PlacementStatePlaced   PlacementState = "" // empty = legacy/placed
	PlacementStateReserved PlacementState = "reserved"
	// PlacementStateDeleting is a distributed lifecycle fence. The current
	// owner has committed delete intent, so failover/recreate and mutations
	// must stop while durable secret/artifact cleanup completes.
	PlacementStateDeleting PlacementState = "deleting"
)

type PlacementTarget

type PlacementTarget struct {
	NodeID        string
	APIURL        string
	DataPlaneHost string
	// InternalURL is the peer's required cluster-internal mTLS URL.
	InternalURL string
	IsSelf      bool
}

PlacementTarget is returned by SelectPlacement.

type ReassignStuckRequest added in v0.6.1

type ReassignStuckRequest struct {
	SandboxID string `json:"sandbox_id"`
	// IncarnationID fences the request against a placement that has already
	// moved on since this node last read it.
	IncarnationID string `json:"incarnation_id,omitempty"`
}

ReassignStuckRequest asks the leader to move a placement this node has failed to recreate. Target selection stays on the FSM side: it needs drain state, pending reservations and capacity leases that an Agent does not hold.

type RecoveryBlob added in v0.2.2

type RecoveryBlob struct {
	Ref           string                       `json:"ref"`
	SandboxID     string                       `json:"sandbox_id"`
	Spec          *models.CreateSandboxRequest `json:"spec,omitempty"`
	SecretRef     string                       `json:"secret_ref,omitempty"`
	SecretVersion int                          `json:"secret_version,omitempty"`
}

type SandboxRecreateReporter added in v0.5.6

type SandboxRecreateReporter interface {
	RecreateSandboxReport(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets PlacementSecrets, exposedPorts map[int]ExposedPortRoute) (attempted bool, err error)
}

SandboxRecreateReporter is the optional outcome-reporting form of SandboxRecreator. attempted is false only for a successful steady-state no-op where the sandbox and its replicated routes were already present. Failures always report attempted=true so retry/error metrics retain the signal that drives stuck-placement reassignment.

type SandboxRecreator

type SandboxRecreator interface {
	RecreateSandbox(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets PlacementSecrets, exposedPorts map[int]ExposedPortRoute) error
}

SandboxRecreator is the cluster's escape hatch back into the service layer for the failover-recreate path. The owner watcher (see owner_watcher.go) invokes RecreateSandbox for any FSM placement that points to self but has no corresponding local sandbox — typically because the previous owner died and the dead-owner reconciler reassigned the placement here.

exposedPorts carries the replicated port routing intents the previous owner had recorded; the implementation is expected to re-issue ExposePort for each entry after the create succeeds. Raw TCP entries include the original HostPort so the recreated owner can preserve the public endpoint when the port is available on the new node.

Implementations MUST be idempotent: the watcher polls and may invoke RecreateSandbox multiple times for the same id while a previous attempt is still in flight or after an unexpected restart.

secrets is the provider ref that can rehydrate the redacted spec. Legacy sealed bytes may be present only for old placement rows. The implementation is expected to resolve and re-merge it before instantiating the container — without this step, recreated sandboxes would lose access to the user's private registry / mount credentials.

type SecretPeerMinACKPusher added in v0.6.1

type SecretPeerMinACKPusher interface {
	PushSecretBlobToAnyPeer(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)
}

SecretPeerMinACKPusher is implemented by the production cluster clients. Keeping it separate preserves the small full-fanout test seam while letting the synchronous HA-create path stop immediately after its first backup ACK.

type SecretPeerPusher added in v0.6.1

type SecretPeerPusher interface {
	PushSecretBlobToPeers(ctx context.Context, blob secrets.SecretBlob, recipients []string) (ackedNodes []string, err error)
	DeleteSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, generation int64) (acked []string, err error)
	ProbeSecretOnPeers(ctx context.Context, sandboxID, incarnationID string, recipients []string, minGeneration int64) (holding []string, err error)
}

SecretPeerPusher is the narrow seam Service uses for async fan-out so tests can inject a fake without a full Cluster.

type SelectPlacementRequest

type SelectPlacementRequest struct {
	Request capacity.Request `json:"request"`
	// SandboxID and RecipientBackups ask the control plane to pick the seal
	// recipients itself and return only those. Without them the server falls
	// back to returning the full candidate slice, which is what an agent
	// running the previous build still expects — keep both paths until every
	// node in a rolling upgrade sends the id.
	SandboxID        string `json:"sandbox_id,omitempty"`
	RecipientBackups int    `json:"recipient_backups,omitempty"`
	// TargetOnly asks for the chosen node and nothing else. Build, template,
	// JS-bundle and local-image routing all want a single target, but the
	// only shape that existed without a SandboxID also serialized every
	// eligible worker back to the caller — O(fleet) bytes for a one-field
	// answer. An older server ignores the flag and simply returns the
	// candidates the caller then discards, so this is rolling-upgrade safe.
	TargetOnly bool `json:"target_only,omitempty"`
}

type SelectPlacementResponse

type SelectPlacementResponse struct {
	Target PlacementTarget `json:"target"`
	// Candidates is the legacy shape: every eligible worker, serialized to the
	// caller so IT could pick secret recipients. At 2,000 nodes that made each
	// create's response O(fleet) in bytes, allocations and control-plane CPU,
	// for an answer of at most a handful of node ids. Populated only when the
	// request did not carry a SandboxID.
	Candidates []Member `json:"candidates,omitempty"`
	// Recipients is the bounded answer: owner first, then the chosen backups.
	Recipients []string `json:"recipients,omitempty"`
	Error      string   `json:"error,omitempty"`
}

type StorageObligation added in v0.6.1

type StorageObligation struct {
	NodeID         string `json:"node_id"`
	OpenedUnixNano int64  `json:"opened_unix_nano,omitempty"`
}

StorageObligation is the open job for one draining node.

type StorageObligationReport added in v0.6.1

type StorageObligationReport struct {
	Reporter string `json:"reporter"`
	// Seq increases on every report from this owner. The FSM ignores a report
	// whose Seq is not newer than the one it holds, so a retried or
	// out-of-order delivery can never resurrect an outbox row the owner has
	// already cleared.
	Seq uint64 `json:"seq"`
	// ObservedUnixNano is the owner's own clock when it took the snapshot.
	// Informational only: staleness is judged on ReceivedUnixNano.
	ObservedUnixNano int64 `json:"observed_unix_nano,omitempty"`
	// Owed maps node id -> number of outbox rows still owing that node a
	// delete. Empty is a real report ("I owe nothing"), not an absent one.
	Owed map[string]int `json:"owed,omitempty"`
	// ReceivedUnixNano is stamped by the leader when the batch is proposed,
	// never by Apply: every replica must reach the same state from the log.
	ReceivedUnixNano int64 `json:"received_unix_nano,omitempty"`
}

StorageObligationReport is one owner's current snapshot of the deletes it still owes. It REPLACES that owner's previous report.

type StorageObligationReporter added in v0.6.1

type StorageObligationReporter struct {
	NodeID     string    `json:"node_id"`
	Owed       int       `json:"owed"`
	ReportedAt time.Time `json:"reported_at,omitempty"`
	Stale      bool      `json:"stale"`
}

StorageObligationReporter is one owner's contribution to a job, as the operator sees it.

type StorageObligationReporterClient added in v0.6.1

type StorageObligationReporterClient interface {
	ReportStorageObligations(ctx context.Context, report StorageObligationReport) error
}

StorageObligationReporterClient is implemented by clients that can submit an owner report. Optional (type-asserted) so no test double must grow it.

type StorageObligationView added in v0.6.1

type StorageObligationView struct {
	NodeID   string    `json:"node_id"`
	OpenedAt time.Time `json:"opened_at,omitempty"`
	// Holders is how many placements still list the node as a secret
	// recipient — copies the reseal has not moved yet.
	Holders int `json:"holders"`
	// PendingDeletes sums the owners' reported outbox rows owed to the node.
	PendingDeletes int `json:"pending_deletes"`
	// Reporters are the owners that report owing this node something.
	Reporters []StorageObligationReporter `json:"reporters,omitempty"`
	// StaleReporters are owners expected to report whose last report is too
	// old (or absent). Their outbox is unknown, so PendingDeletes may be low.
	StaleReporters []StorageObligationReporter `json:"stale_reporters,omitempty"`
	// Complete is false whenever a reporter is stale: the counts are then a
	// floor, never an all-clear.
	Complete bool `json:"complete"`
	// ReadyForAttestation: no copies left, no deletes owed, every reporter
	// fresh. The job still stays open until an operator attests.
	ReadyForAttestation bool `json:"ready_for_attestation"`
}

StorageObligationView is what GET /v1/cluster/storage-retirements shows for an open job.

type StorageObligationsReader added in v0.6.1

type StorageObligationsReader interface {
	StorageObligations(ctx context.Context) ([]StorageObligationView, error)
}

StorageObligationsReader is implemented by clients that can serve the job view.

type StorageObligationsResponse added in v0.6.1

type StorageObligationsResponse struct {
	Obligations []StorageObligationView `json:"obligations"`
}

StorageObligationsResponse is the internal wire shape agents read.

type VolumeQueryResponse added in v0.5.4

type VolumeQueryResponse struct {
	Volume  *models.Volume  `json:"volume,omitempty"`
	Volumes []models.Volume `json:"volumes,omitempty"`
	Exists  bool            `json:"exists,omitempty"`
	Count   int             `json:"count,omitempty"`
}

VolumeQueryResponse is the wire shape of the internal volume read endpoint. Exactly one of Volume / Volumes / Exists is meaningful per request kind.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL