Documentation
¶
Overview ¶
Package cluster implements Phase-1 cluster mode for sandboxd.
Architecture (Phase 1):
Membership: SWIM gossip via hashicorp/memberlist. Gossip carries identity and role metadata only so memberlist's 512-byte NodeMeta limit cannot strip Raft addresses.
Capacity heartbeats: server-role nodes fetch authenticated capacity.Snapshot payloads from worker-capable peers and require a fresh heartbeat before placement can target that worker.
Placement map: server-role nodes run a small Raft FSM (hashicorp/raft) holding sandbox_id -> owner_node_id plus replicated recovery metadata. Mutations happen on the leader; server reads are local from the FSM. Worker/ingress-only nodes run Agent instead: they gossip identity and receive owner API forwards, but all placement reads/writes go to the server quorum over authenticated RPC. They do not store the FSM and do not join Raft as non-voters.
Owner-sharded execution: once a sandbox is placed on node N, all of its state and lifecycle stays on N. The local SQLite store is unchanged. Cross-node API calls (toolbox, sessions, port forwards) are transparently reverse-proxied to the owner via internal/cluster.ForwardHTTP.
No central control plane: any node can accept any request. Mutating requests for sandbox X are forwarded to X's owner; CreateSandbox forwards to the placement target chosen by power-of-two-choices.
Single-node mode (cfg.EnableCluster = false) uses Noop so that callsites can stay unconditional.
Index ¶
- Constants
- Variables
- func CanOwnSandboxRole(role string) bool
- func CanServeControlPlaneRole(role string) bool
- func CanServeIngressRole(role string) bool
- func ExportWasmMigrateLocal(export func(w io.Writer) (string, error)) ([]byte, string, error)
- func ExposedPortRoutesForPlacement(p Placement) map[int]ExposedPortRoute
- func IsMixedArchitectureRole(role string) bool
- func LargeClusterTopologyError(members []Member) error
- func LiveMemberCount(members []Member) int
- func PlacementShardForSandbox(sandboxID string, count int) int
- func PostWasmMigrateImport(ctx context.Context, c Client, target Member, sandboxID, cloneGen string, ...) error
- func RecordOwnerForwardStale()
- func StreamWasmMigrateExport(ctx context.Context, c Client, owner OwnerInfo, sandboxID string, w io.Writer) (cloneGen string, err error)
- func ValidateRecoveryPayloadSize(sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
- type Agent
- func (a *Agent) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error
- func (a *Agent) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error
- func (a *Agent) ApplyEncoded(ctx context.Context, payload []byte) error
- func (a *Agent) AssertOwnership(ctx context.Context, local []LocalSandboxState) error
- func (a *Agent) AttachInternalHandler(h http.Handler)
- func (a *Agent) CancelReservation(ctx context.Context, sandboxID string) error
- func (a *Agent) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (a *Agent) Close() error
- func (a *Agent) CustomDomainsOf(sandboxID string) []string
- func (a *Agent) DeletePlacement(ctx context.Context, sandboxID string) error
- func (a *Agent) DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID string) error
- func (a *Agent) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
- func (a *Agent) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)
- func (a *Agent) IngressTargets() models.IngressTarget
- func (a *Agent) IsNodeDrained(nodeID string) bool
- func (a *Agent) Leader() string
- func (a *Agent) Members() []Member
- func (a *Agent) OwnerOf(sandboxID string) (OwnerInfo, error)
- func (a *Agent) OwnerOfName(name string) (string, OwnerInfo, error)
- func (a *Agent) PlacementOf(sandboxID string) (Placement, bool)
- func (a *Agent) PlacementPage(req PlacementPageRequest) PlacementPageResponse
- func (a *Agent) PlacementVersion() uint64
- func (a *Agent) Placements() []Placement
- func (a *Agent) PlacementsForShards(filter PlacementShardFilter) []Placement
- func (a *Agent) PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error
- func (a *Agent) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error
- func (a *Agent) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (a *Agent) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error
- func (a *Agent) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error
- func (a *Agent) RemoveMember(ctx context.Context, nodeID string, force bool) error
- func (a *Agent) ReplicateJSBundle(ctx context.Context, owner string, req models.CreateJSBundleRequest) error
- func (a *Agent) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, ...) error
- func (a *Agent) ResolveCustomDomain(hostname string) (string, bool)
- func (a *Agent) SecretsOf(sandboxID string) PlacementSecrets
- func (a *Agent) SelectPlacement(req capacity.Request) (PlacementTarget, error)
- func (a *Agent) SelfAPIURL() string
- func (a *Agent) SelfNodeID() string
- func (a *Agent) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error
- func (a *Agent) SpecOf(sandboxID string) *models.CreateSandboxRequest
- func (a *Agent) SubscribePlacement(context.Context) <-chan struct{}
- func (a *Agent) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (a *Agent) VolumeAttachmentCount(ctx context.Context, tenant, id string) (int, error)
- func (a *Agent) VolumeByID(ctx context.Context, tenant, id string) (models.Volume, error)
- func (a *Agent) VolumeByName(ctx context.Context, tenant, name string) (models.Volume, error)
- func (a *Agent) VolumeDelete(ctx context.Context, tenant, id string) error
- func (a *Agent) VolumeExistsForSource(ctx context.Context, source string) (bool, error)
- func (a *Agent) VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)
- func (a *Agent) VolumesForTenant(ctx context.Context, tenant string) ([]models.Volume, error)
- type Client
- type Cluster
- func (c *Cluster) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error
- func (c *Cluster) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error
- func (c *Cluster) ApplyEncoded(ctx context.Context, payload []byte) error
- func (c *Cluster) AssertOwnership(ctx context.Context, local []LocalSandboxState) error
- func (c *Cluster) AttachInternalHandler(h http.Handler)
- func (c *Cluster) AttachRecreator(r SandboxRecreator)
- func (c *Cluster) CancelReservation(ctx context.Context, sandboxID string) error
- func (c *Cluster) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (c *Cluster) Close() error
- func (c *Cluster) CustomDomainsOf(sandboxID string) []string
- func (c *Cluster) DeletePlacement(ctx context.Context, sandboxID string) error
- func (c *Cluster) DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID string) error
- func (c *Cluster) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
- func (c *Cluster) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)
- func (c *Cluster) HealthyForReads() bool
- func (c *Cluster) IngressTargets() models.IngressTarget
- func (c *Cluster) IsNodeDrained(nodeID string) bool
- func (c *Cluster) Leader() string
- func (c *Cluster) LeaderAPIURL() string
- func (c *Cluster) Members() []Member
- func (c *Cluster) OwnerOf(sandboxID string) (OwnerInfo, error)
- func (c *Cluster) OwnerOfName(name string) (string, OwnerInfo, error)
- func (c *Cluster) PlacementOf(sandboxID string) (Placement, bool)
- func (c *Cluster) PlacementPage(req PlacementPageRequest) PlacementPageResponse
- func (c *Cluster) PlacementVersion() uint64
- func (c *Cluster) Placements() []Placement
- func (c *Cluster) PlacementsForShards(filter PlacementShardFilter) []Placement
- func (c *Cluster) PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error
- func (c *Cluster) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error
- func (c *Cluster) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (c *Cluster) RecoveryBlob(ctx context.Context, ref string) (RecoveryBlob, bool, error)
- func (c *Cluster) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error
- func (c *Cluster) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error
- func (c *Cluster) RemoveMember(ctx context.Context, nodeID string, force bool) error
- func (c *Cluster) ReplicateJSBundle(ctx context.Context, owner string, req models.CreateJSBundleRequest) error
- func (c *Cluster) ReserveBatchOnTargets(ctx context.Context, reservations []PlacementReservation) error
- func (c *Cluster) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, ...) error
- func (c *Cluster) ResolveCustomDomain(hostname string) (string, bool)
- func (c *Cluster) SecretsOf(sandboxID string) PlacementSecrets
- func (c *Cluster) SelectPlacement(req capacity.Request) (PlacementTarget, error)
- func (c *Cluster) SelfAPIURL() string
- func (c *Cluster) SelfNodeID() string
- func (c *Cluster) SetLocalTemplateIDsProvider(fn func() ([]string, bool))
- func (c *Cluster) SetLocalWasmModuleIDsProvider(fn func() ([]string, bool))
- func (c *Cluster) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error
- func (c *Cluster) SpecOf(sandboxID string) *models.CreateSandboxRequest
- func (c *Cluster) SubscribePlacement(ctx context.Context) <-chan struct{}
- func (c *Cluster) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (c *Cluster) VolumeAttachmentCount(_ context.Context, tenant, id string) (int, error)
- func (c *Cluster) VolumeByID(_ context.Context, tenant, id string) (models.Volume, error)
- func (c *Cluster) VolumeByName(_ context.Context, tenant, name string) (models.Volume, error)
- func (c *Cluster) VolumeDelete(ctx context.Context, tenant, id string) error
- func (c *Cluster) VolumeExistsForSource(_ context.Context, source string) (bool, error)
- func (c *Cluster) VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)
- func (c *Cluster) VolumesForTenant(_ context.Context, tenant string) ([]models.Volume, error)
- type ClusterTLS
- type DrainStateResponse
- type Endpoint
- type ExposedPortRoute
- type IngressRouteOwner
- type IngressShardRoute
- type LocalSandboxState
- type Member
- type Noop
- func (n *Noop) AddCustomDomain(ctx context.Context, sandboxID, hostname string) error
- func (n *Noop) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error
- func (n *Noop) ApplyEncoded(ctx context.Context, payload []byte) error
- func (n *Noop) AssertOwnership(ctx context.Context, local []LocalSandboxState) error
- func (n *Noop) AttachInternalHandler(h http.Handler)
- func (n *Noop) CancelReservation(ctx context.Context, sandboxID string) error
- func (n *Noop) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (n *Noop) Close() error
- func (n *Noop) CustomDomainsOf(sandboxID string) []string
- func (n *Noop) DeletePlacement(ctx context.Context, sandboxID string) error
- func (n *Noop) DeleteVolumeAttachmentsForSandbox(_ context.Context, sandboxID string) error
- func (n *Noop) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
- func (n *Noop) ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)
- func (n *Noop) IngressTargets() models.IngressTarget
- func (n *Noop) IsNodeDrained(nodeID string) bool
- func (n *Noop) Leader() string
- func (n *Noop) Members() []Member
- func (n *Noop) OwnerOf(sandboxID string) (OwnerInfo, error)
- func (n *Noop) OwnerOfName(name string) (string, OwnerInfo, error)
- func (n *Noop) PlacementOf(sandboxID string) (Placement, bool)
- func (n *Noop) PlacementPage(PlacementPageRequest) PlacementPageResponse
- func (n *Noop) PlacementVersion() uint64
- func (n *Noop) Placements() []Placement
- func (n *Noop) PlacementsForShards(PlacementShardFilter) []Placement
- func (n *Noop) PutVolumeAttachments(_ context.Context, attachments []models.VolumeAttachment) error
- func (n *Noop) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error
- func (n *Noop) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (n *Noop) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error
- func (n *Noop) RemoveExposedPort(ctx context.Context, sandboxID string, port int) error
- func (n *Noop) RemoveMember(ctx context.Context, nodeID string, force bool) error
- func (n *Noop) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, ...) error
- func (n *Noop) ResolveCustomDomain(hostname string) (string, bool)
- func (n *Noop) SecretsOf(sandboxID string) PlacementSecrets
- func (n *Noop) SelectPlacement(req capacity.Request) (PlacementTarget, error)
- func (n *Noop) SelfAPIURL() string
- func (n *Noop) SelfNodeID() string
- func (n *Noop) SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error
- func (n *Noop) SpecOf(sandboxID string) *models.CreateSandboxRequest
- func (n *Noop) SubscribePlacement(ctx context.Context) <-chan struct{}
- func (n *Noop) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, ...) error
- func (n *Noop) VolumeAttachmentCount(_ context.Context, tenant, id string) (int, error)
- func (n *Noop) VolumeByID(_ context.Context, tenant, id string) (models.Volume, error)
- func (n *Noop) VolumeByName(_ context.Context, tenant, name string) (models.Volume, error)
- func (n *Noop) VolumeDelete(_ context.Context, tenant, id string) error
- func (n *Noop) VolumeExistsForSource(_ context.Context, source string) (bool, error)
- func (n *Noop) VolumeUpsert(_ context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)
- func (n *Noop) VolumesForTenant(_ context.Context, tenant string) ([]models.Volume, error)
- type OwnerInfo
- type Placement
- type PlacementLookupResponse
- type PlacementOwnerState
- type PlacementPageRequest
- type PlacementPageResponse
- type PlacementReservation
- type PlacementSecrets
- type PlacementShardFilter
- type PlacementState
- type PlacementTarget
- type RecoveryBlob
- type SandboxRecreator
- type SelectPlacementRequest
- type SelectPlacementResponse
- type VolumeQueryResponse
Constants ¶
const ( PublicInternalApplyPath = "/v1/cluster/internal/apply" PublicInternalPlacementPath = "/v1/cluster/internal/placement/" PublicInternalPlacementByNamePath = "/v1/cluster/internal/placement-by-name/" PublicInternalPlacementsPath = "/v1/cluster/internal/placements" PublicInternalPlacementsQueryPath = "/v1/cluster/internal/placements/query" PublicInternalPlacementsPagePath = "/v1/cluster/internal/placements/page" PublicInternalRecoveryPath = "/v1/cluster/internal/recovery/" PublicInternalSelectPlacementPath = "/v1/cluster/internal/select-placement" PublicInternalVolumePath = "/v1/cluster/internal/volume" PublicInternalDrainStatePath = "/v1/cluster/internal/drain/" PublicInternalClusterLeaderPath = "/v1/cluster/leader" )
const ( // CreateBackpressureRetryAfterSeconds is the public Retry-After hint for // queue/concurrency backpressure. Keep it short: these rejects are caused by // transient in-flight create fan-in, not by a long operator action. CreateBackpressureRetryAfterSeconds = 5 CapacityRetryAfterSeconds = 30 // InvalidTopologyRetryAfterSeconds is the hint for ErrInvalidTopology // rejects, which only clear when an operator reshapes the cluster // (promoting servers or adding workers). A long back-off keeps clients // from busy-retrying while the human change is in flight. InvalidTopologyRetryAfterSeconds = 300 )
const ( DefaultPlacementPageLimit = 1000 MaxPlacementPageLimit = 5000 // MaxReplicatedIngressRouteNodes is the largest ingress tier where every // ingress-capable node keeps the full public route table. Ordinary DNS // round-robin / TCP load balancers can only work if any advertised ingress // node can answer for any sandbox. MaxReplicatedIngressRouteNodes = 10 )
const ( // PublicWasmMigratePath is the operator-facing live-migration endpoint (§4.4). PublicWasmMigratePath = "/v1/cluster/wasm-migrate" // PublicInternalWasmMigratePath prefixes per-sandbox internal export/import routes. PublicInternalWasmMigratePath = "/v1/cluster/internal/wasm-migrate/" // WasmMigrateCloneGenHeader carries the §4.8 clone-generation token on export. WasmMigrateCloneGenHeader = "X-Aerol-Clone-Generation" // WasmMigrateTarMediaType is the streamed mem.snap tarball content type. WasmMigrateTarMediaType = "application/vnd.aerolvm.wasm-migrate.v1+tar" )
const DefaultPlacementShardCount = 16384
DefaultPlacementShardCount is the stable shard space for placement-index reads. The value intentionally stays independent from the number of ingress nodes: ingress ownership changes remap shard IDs to nodes, but a sandbox's shard ID remains stable as the fleet scales up and down.
const InternalAPIPath = "/internal/apply"
InternalAPIPath is the URL path served by the cluster-internal mTLS listener. Distinct from the public /v1/cluster/internal/apply route so a misconfigured load balancer can't accidentally tunnel public traffic into the cluster-internal channel.
const MaxMixedClusterNodes = 10
MaxMixedClusterNodes is the largest live cluster size allowed to keep the legacy mixed/hybrid convenience topology. Above this, production clusters must run dedicated server, worker, and ingress tiers.
Variables ¶
var ( // ErrVolumeQuotaExceeded is returned by opUpsertVolume when inserting a new // row would exceed the tenant's configured volume-count cap. ErrVolumeQuotaExceeded = errors.New("cluster: tenant volume quota exceeded") // ErrUnknownVolume is returned by volume reads/deletes when no replicated row // exists for the (tenant, id). ErrUnknownVolume = errors.New("cluster: unknown volume") // ErrVolumeInUse is returned when a delete races with or follows a sandbox // attachment that still points at the volume. ErrVolumeInUse = errors.New("cluster: volume is still attached") )
var ErrCapacityExceeded = errors.New("cluster: target capacity exceeded after pending reservations")
ErrCapacityExceeded is returned when an opReserve apply finds that the chosen target no longer has headroom for the request once concurrent in-flight reservations are added to its gossiped reserved totals. Routers translate this to 503 so the client retries; the next SelectPlacement will see the new reservation and pick a different node.
var ErrCreateBackpressure = errors.New("cluster: create backpressure")
ErrCreateBackpressure is returned when the leader-side create queue for a worker is full. This is intentionally distinct from ErrCapacityExceeded: capacity means "pick another worker or wait for resources"; backpressure means "too many creates are already in-flight to this worker, retry shortly."
var ErrCustomHostnameConflict = errors.New("cluster: custom hostname already in use")
ErrCustomHostnameConflict is returned when an opAddCustomDomain entry asks the FSM to claim a hostname already held by a different sandbox. Maps to the same 409 the local SQLite custom-domains insert returns — the FSM is the cluster-wide tiebreaker that catches the race where two sandboxes on different owners try to claim the same hostname concurrently. The API/service layer surfaces this as models.ErrCustomDomainConflict after reading the raft Apply result.
var ErrHostPortReserved = errors.New("cluster: tcp host port already reserved")
ErrHostPortReserved is returned when the FSM rejects a raw-TCP exposure because another placement already owns the requested cluster-wide host port. Callers can retry with another candidate instead of surfacing a random expose failure to the user.
var ErrInvalidTopology = errors.New("cluster: invalid topology")
ErrInvalidTopology is returned when the live cluster shape violates a production topology invariant. API layers translate this to 503 so clients retry after the operator fixes membership instead of treating it as a malformed request.
var ErrLastVoter = errors.New("cluster: cannot remove last raft voter")
ErrLastVoter prevents an explicit removal from deleting the last voting raft server and leaving the cluster with no quorum path.
var ErrMemberStillAlive = errors.New("cluster: raft member is still alive")
ErrMemberStillAlive protects operators from accidentally removing a live control-plane node. Stop the node first, or pass force through the public lifecycle API when intentionally retiring a live member.
var ErrNameConflict = errors.New("cluster: sandbox name already in use")
ErrNameConflict is returned when the placement FSM rejects an opPlace / opUpsertSpec because a different sandbox already owns the requested Name. Sandbox names are unique cluster-wide; without this check, two concurrent creates landing on different owners could each succeed locally and present ambiguous name-based lookups to facades like Daytona that resolve sandboxes by name. Callers handle this by rolling back the local create and surfacing 409 Conflict to the user.
var ErrNoPlacementTarget = errors.New("cluster: no worker placement target available")
ErrNoPlacementTarget is returned when no alive worker-capable node can accept a new sandbox. This is distinct from "self wins": a pure server or ingress node must not silently fall back to local Docker ownership.
var ErrNotLeader = errors.New("cluster: not raft leader")
ErrNotLeader is returned by mutating Cluster operations when this node is not the Raft leader. Callers should retry against the leader (which Client resolves automatically by following raft leadership).
var ErrOrphanClaimConflict = errors.New("cluster: orphaned placement belongs to a different previous owner")
ErrOrphanClaimConflict is returned when a node tries to reclaim an orphaned placement that was explicitly orphaned from a different previous owner. This prevents a returning node from stealing another dead node's sandbox just because it has a stale local row with the same ID.
var ErrOrphaned = errors.New("cluster: sandbox owner is dead, placement orphaned")
ErrOrphaned is returned by OwnerOf when a placement exists but its owner has been auto-evicted and the dead-owner reconciler has cleared the pointer. This is the terminal state for default non-HA sandboxes; callers should surface 410 Gone. Sandboxes that opted into failover.policy=recreate are reassigned before the remaining dead-owner placements are orphaned.
var ErrRecoveryPayloadTooLarge = errors.New("cluster: recovery payload exceeds inline size limit")
ErrRecoveryPayloadTooLarge is returned when a redacted spec + secret handle encodes past inlineRecoveryMaxBytes. It surfaces at create validation (the user-facing 400) and again defensively at command encode time — hitting the latter without the former means a service-side mutation grew the spec after validation, which is a bug, not a routing case.
var ErrReservationConflict = errors.New("cluster: sandbox already placed or reserved")
ErrReservationConflict is returned when opReserve tries to reserve a sandbox ID that already has a non-expired placement (placed or actively reserved by a different owner). Indicates either a router racing a completed sandbox or a router with a stale view.
var ErrUnknownMember = errors.New("cluster: unknown raft member")
ErrUnknownMember is returned when an operator asks to remove a node that is not present in the current raft configuration.
var ErrUnknownSandbox = errors.New("cluster: unknown sandbox placement")
ErrUnknownSandbox is returned by OwnerOf when no placement record exists for the given sandbox ID. Callers should treat this as "owned locally" only when they have just-created the sandbox and not yet committed its placement.
Functions ¶
func CanOwnSandboxRole ¶
CanOwnSandboxRole reports whether a gossiped node role may own sandboxes. Empty is treated as worker-capable for rolling upgrades from builds that did not advertise role metadata.
func CanServeControlPlaneRole ¶
CanServeControlPlaneRole reports whether a gossiped node role may host the authoritative Raft/FSM control plane. Empty is treated as server-capable for rolling upgrades from builds that did not advertise role metadata.
func CanServeIngressRole ¶
CanServeIngressRole reports whether a gossiped node role may host public ingress routes. Empty is treated as ingress-capable for rolling upgrades from builds that did not advertise role metadata.
func ExportWasmMigrateLocal ¶
ExportWasmMigrateLocal buffers export output for tests.
func ExposedPortRoutesForPlacement ¶
func ExposedPortRoutesForPlacement(p Placement) map[int]ExposedPortRoute
func IsMixedArchitectureRole ¶
IsMixedArchitectureRole reports whether a role value runs more than one production tier. Empty legacy metadata is treated as mixed because old nodes behaved as server+worker+ingress.
func LargeClusterTopologyError ¶
LargeClusterTopologyError enforces the production topology contract: clusters above MaxMixedClusterNodes must have dedicated server, worker, and ingress tiers. Small clusters keep the legacy convenience behavior.
func LiveMemberCount ¶
LiveMemberCount counts live members with a usable node ID.
func PlacementShardForSandbox ¶
PlacementShardForSandbox maps sandboxID to a stable shard ID in [0, count).
func PostWasmMigrateImport ¶
func PostWasmMigrateImport(ctx context.Context, c Client, target Member, sandboxID, cloneGen string, body io.Reader) error
PostWasmMigrateImport pushes a mem.snap tarball to target.
func RecordOwnerForwardStale ¶
func RecordOwnerForwardStale()
func StreamWasmMigrateExport ¶
func StreamWasmMigrateExport(ctx context.Context, c Client, owner OwnerInfo, sandboxID string, w io.Writer) (cloneGen string, err error)
StreamWasmMigrateExport pulls a mem.snap tarball from owner into w.
func ValidateRecoveryPayloadSize ¶
func ValidateRecoveryPayloadSize(sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
ValidateRecoveryPayloadSize reports whether the recovery record for (sandboxID, spec, secrets) fits in a raft log entry. Exported so the service layer can reject oversized specs with a clean validation error before any admission/container work happens. Encodes the exact bytes the FSM's recovery store persists, so validation and apply cannot disagree.
Types ¶
type Agent ¶
type Agent struct {
// contains filtered or unexported fields
}
Agent is the worker/ingress-side cluster client. It deliberately does not start Raft, does not create a placement FSM, and never joins the raft configuration as a non-voter. It gossips identity/addresses, serves local capacity heartbeats, and delegates every authoritative placement read/write to server-role nodes.
func (*Agent) AddCustomDomain ¶
AddCustomDomain forwards to the cluster Apply pipe. hostname is canonicalized here (the public Cluster wrapper does the same on the other path); the FSM then enforces cluster-wide uniqueness.
func (*Agent) AddExposedPort ¶
func (*Agent) ApplyEncoded ¶
func (*Agent) AssertOwnership ¶
func (a *Agent) AssertOwnership(ctx context.Context, local []LocalSandboxState) error
func (*Agent) AttachInternalHandler ¶
func (*Agent) CancelReservation ¶
func (*Agent) ClaimOrphan ¶
func (a *Agent) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
func (*Agent) CustomDomainsOf ¶
CustomDomainsOf returns the hostnames bound to sandboxID. Agent doesn't run the placement FSM locally, so this rides the same remote placement lookup the existing ExposedPortsOf path uses — failure modes match: a transient network blip yields nil, which the ingress reconciler treats as "no custom matchers right now" until the placement subscription wakes a re-read.
func (*Agent) DeletePlacement ¶
func (*Agent) DeleteVolumeAttachmentsForSandbox ¶
func (*Agent) ExposedPortsOf ¶
func (a *Agent) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
func (*Agent) ForwardHTTP ¶
func (*Agent) IngressTargets ¶
func (a *Agent) IngressTargets() models.IngressTarget
IngressTargets aggregates live ingress-role members' PublicHost values. Agents have no FSM but Members() already falls back to the local gossip view when the control plane is unreachable, so the same aggregator works for both cases.
func (*Agent) IsNodeDrained ¶
func (*Agent) PlacementPage ¶
func (a *Agent) PlacementPage(req PlacementPageRequest) PlacementPageResponse
func (*Agent) PlacementVersion ¶
func (*Agent) Placements ¶
func (*Agent) PlacementsForShards ¶
func (a *Agent) PlacementsForShards(filter PlacementShardFilter) []Placement
func (*Agent) PutVolumeAttachments ¶
func (*Agent) ReassignPlacement ¶
func (*Agent) RecordPlacement ¶
func (a *Agent) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
func (*Agent) RemoveCustomDomain ¶
func (*Agent) RemoveExposedPort ¶
func (*Agent) RemoveMember ¶
func (*Agent) ReplicateJSBundle ¶
func (a *Agent) ReplicateJSBundle(ctx context.Context, owner string, req models.CreateJSBundleRequest) error
ReplicateJSBundle mirrors *Cluster's for worker/ingress agents: an upload can land on any node (caddy routes /v1/js-bundles to whichever answers), so the fan-out must work from an agent too.
func (*Agent) ReserveOnTarget ¶
func (a *Agent) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error
func (*Agent) ResolveCustomDomain ¶
ResolveCustomDomain returns ("", false) on agent nodes. A reverse-by-hostname lookup endpoint isn't wired through yet (the TLS-ask handler is intended to live alongside a node that runs raft locally — server/ingress roles). See task #17 for adding a remote lookup if agent-role ingress becomes a supported topology.
func (*Agent) SecretsOf ¶
func (a *Agent) SecretsOf(sandboxID string) PlacementSecrets
func (*Agent) SelectPlacement ¶
func (a *Agent) SelectPlacement(req capacity.Request) (PlacementTarget, error)
func (*Agent) SelfAPIURL ¶
func (*Agent) SelfNodeID ¶
func (*Agent) SetNodeDrainState ¶
func (*Agent) SubscribePlacement ¶
func (*Agent) UpsertSpec ¶
func (a *Agent) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
func (*Agent) VolumeAttachmentCount ¶
func (*Agent) VolumeByID ¶
func (*Agent) VolumeByName ¶
func (*Agent) VolumeDelete ¶
func (*Agent) VolumeExistsForSource ¶
func (*Agent) VolumeUpsert ¶
type Client ¶
type Client interface {
// SelfNodeID returns this node's stable cluster identifier.
SelfNodeID() string
// SelfAPIURL returns this node's externally-reachable API base URL.
SelfAPIURL() string
// OwnerOf returns the node currently owning sandboxID, or
// ErrUnknownSandbox if no placement record exists.
OwnerOf(sandboxID string) (OwnerInfo, error)
// OwnerOfName resolves a cluster-wide sandbox name to its sandbox ID and
// current owner. Names are indexed from replicated specs and exist for
// facade APIs that accept either ID or name. Returns ErrUnknownSandbox if
// no placement record claims name.
OwnerOfName(name string) (string, OwnerInfo, error)
// SelectPlacement chooses a node to host a new sandbox with the given
// resource request. In single-node mode it always returns self.
SelectPlacement(req capacity.Request) (PlacementTarget, error)
// RecordPlacement commits sandboxID -> self into the FSM along with the
// (optional) creation spec used to re-materialize the sandbox after a
// failover. Idempotent — re-recording with the same owner is a no-op, and
// passing spec=nil preserves any spec that was previously replicated for
// this id (so a boot-time replay can't erase a richer record written by
// the original CreateSandbox call).
//
// spec MUST be redacted (no plaintext credentials) before being passed in;
// secrets is the provider handle produced by the service layer. Passing an
// empty handle preserves any previously replicated ref (mirrors the
// spec-preservation rule — a boot-time replay that has the spec but not
// the secrets can't erase the original create's recoverability handle).
RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
// ClaimOrphan promotes an orphaned placement back to self. It succeeds
// only when the placement is currently orphaned and either has no recorded
// previous owner (legacy row) or was orphaned from this node. It preserves
// any replicated spec/secrets when the caller passes nil.
ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
// UpsertSpec replaces the replicated spec for sandboxID without touching
// ownership. Mutating handlers (resize, lifecycle) call this after a
// successful local mutation so the FSM stays current — otherwise a
// failover-recreated sandbox would revert to its create-time shape.
//
// spec MUST be redacted. An empty secrets handle preserves the previously
// replicated ref — resize/lifecycle never touch credentials, so passing an
// empty handle is the right choice for those callers; the "real" secrets
// are only re-shipped when the user re-runs create or rotates them
// explicitly.
UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
// SpecOf returns the most-recently-replicated CreateSandboxRequest for
// sandboxID, or nil if no spec is recorded (pre-cluster sandbox, or no
// placement). The returned spec is REDACTED — no plaintext secrets. Use
// SecretsOf to retrieve the matching provider handle if you need to
// reconstruct the full spec for a recreate.
SpecOf(sandboxID string) *models.CreateSandboxRequest
// SecretsOf returns the secret provider handle paired with SpecOf's spec.
// Empty when the sandbox was created without any private-registry / mount
// credentials, or when the placement predates secret replication.
SecretsOf(sandboxID string) PlacementSecrets
// AddExposedPort records intent that sandboxID has port exposed. Raw TCP
// routes include HostPort so every ingress node can bind/proxy the same
// cluster-wide endpoint. Idempotent when the route metadata is unchanged.
AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error
// RemoveExposedPort drops a port intent from the placement. Idempotent.
RemoveExposedPort(ctx context.Context, sandboxID string, port int) error
// ExposedPortsOf returns a copy of the replicated port route map for
// sandboxID, or nil if none. Used by the recreator and ingress reconciler.
ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
// AddCustomDomain records intent that sandboxID owns hostname (already
// canonicalized lower-case) cluster-wide. Returns ErrCustomHostnameConflict
// when a different sandbox already holds it. Idempotent for the same
// (sandbox, hostname) pair so retries are safe.
AddCustomDomain(ctx context.Context, sandboxID, hostname string) error
// RemoveCustomDomain drops hostname from sandboxID's set. No-op when the
// hostname isn't claimed by this sandbox (the local row may already be
// gone after a Delete) so callers can retry without special-casing.
RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error
// CustomDomainsOf returns a copy of the replicated hostname set for
// sandboxID, or nil when no placement exists or the set is empty. The
// ingress reconciler uses it to populate Caddy matchers on every node.
CustomDomainsOf(sandboxID string) []string
// ResolveCustomDomain answers "which sandbox owns this hostname?" from
// the local FSM in O(1). Used by the TLS-ask handler on ingress nodes
// that may not own the sandbox row themselves. hostname is matched
// case-insensitively. Returns sandboxID, true when known.
ResolveCustomDomain(hostname string) (string, bool)
// DeletePlacement removes sandboxID from the FSM. Idempotent.
DeletePlacement(ctx context.Context, sandboxID string) error
// ReserveOnTarget writes opReserve into the FSM holding capacity + name
// for target before the body is forwarded to it. The reservation carries
// the redacted spec + secret ref so the target's RecordPlacement promote
// step can run with nil Spec/empty Secrets and inherit them
// atomically. ttl bounds how long the reservation can hold capacity if
// the target never promotes (the leader's GC sweep cancels expired rows
// at ~5s tick cadence). Spec MUST already be redacted — the raft log
// must NOT carry plaintext credentials.
ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error
// CancelReservation drops a pending reservation from the FSM. No-op on
// missing or already-promoted (Placed) rows, so router rollback / TTL GC
// / late successful promote can race harmlessly.
CancelReservation(ctx context.Context, sandboxID string) error
// SetNodeDrainState marks nodeID as drained (excluded from
// SelectPlacement) or restores it to the candidate pool. Idempotent on
// both edges. The mark lives in the FSM and survives the drained node
// going away — the operator's intent must outlast the process they're
// about to stop.
SetNodeDrainState(ctx context.Context, nodeID string, drained bool) error
// ReassignPlacement moves sandboxID to target without touching the
// replicated spec or port intents. Used by WASM live migration once the
// receiving node has imported a §4.8.1 checkpoint.
ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error
// Platform-volume metadata replication. These keep a Daytona volume's
// id/name/source consistent cluster-wide so get/list/delete-by-id survive the
// tenant's API ownership moving between nodes. The data itself is
// deterministic in S3/NFS and is never replicated.
//
// VolumeUpsert is an idempotent get-or-create: a duplicate (tenant, name)
// converges on the existing row. It returns the canonical row and whether
// this call created it, or ErrVolumeQuotaExceeded.
VolumeUpsert(ctx context.Context, v models.Volume, maxPerTenant int) (models.Volume, bool, error)
// VolumeDelete removes the (tenant, id) row, or returns ErrUnknownVolume.
VolumeDelete(ctx context.Context, tenant, id string) error
// VolumeByID / VolumeByName resolve a single row or return ErrUnknownVolume.
VolumeByID(ctx context.Context, tenant, id string) (models.Volume, error)
VolumeByName(ctx context.Context, tenant, name string) (models.Volume, error)
// VolumesForTenant lists the tenant's rows, newest first.
VolumesForTenant(ctx context.Context, tenant string) ([]models.Volume, error)
// VolumeExistsForSource reports whether any replicated row points at source
// (the reclaim worker's cluster-wide live-data guard).
VolumeExistsForSource(ctx context.Context, source string) (bool, error)
// VolumeAttachmentCount returns the cluster-wide live attachment count for a
// volume. Put/Delete keep this index in the same raft log as volume metadata
// so Daytona delete cannot miss a sandbox that lives on another worker.
VolumeAttachmentCount(ctx context.Context, tenant, id string) (int, error)
PutVolumeAttachments(ctx context.Context, attachments []models.VolumeAttachment) error
DeleteVolumeAttachmentsForSandbox(ctx context.Context, sandboxID string) error
// RemoveMember explicitly removes nodeID from the raft configuration after
// marking it drained and orphaning any placements it owned. Unknown raft
// members return ErrUnknownMember. Live members require force=true so an
// operator cannot accidentally cut out a healthy server by typo.
RemoveMember(ctx context.Context, nodeID string, force bool) error
// IsNodeDrained reports whether nodeID is currently marked drained.
// Reads the local FSM (no network hop). Used by observability endpoints
// — placement scoring uses an internal accessor that takes one lock for
// the whole sweep.
IsNodeDrained(nodeID string) bool
// ApplyEncoded is the receiving end of leader-forwarded raft writes. The
// internal API endpoint pipes the request body through here on the leader
// so any owner-side mutating call (Record/Upsert/Add/Remove/Delete) made on
// a follower can transparently land on the leader's raft. Returns
// ErrNotLeader if leadership has shifted; the forwarder retries.
ApplyEncoded(ctx context.Context, payload []byte) error
// AssertOwnership ensures the FSM lists self as owner for every entry in
// local, and backfills any missing Spec / ExposedPorts so failover-recreate
// works for sandboxes that pre-date the spec-replication features. Used at
// boot. Idempotent.
AssertOwnership(ctx context.Context, local []LocalSandboxState) error
// ForwardHTTP reverse-proxies r to the given peer, copying response back to
// w. Used by the API layer when OwnerOf != self. When target.InternalURL is
// non-empty AND this node has its own TLS material loaded, the proxy rides
// the cert-pinned mTLS channel; otherwise it falls back to target.APIURL
// with PAT-only auth (the legacy public-API path).
ForwardHTTP(target Endpoint, w http.ResponseWriter, r *http.Request)
// AttachInternalHandler wires the API server's HTTP handler into the
// cluster-internal mTLS listener so peers can reverse-proxy owner API calls
// over the cert-pinned channel (not just leader-forwarded raft applies).
// No-op for Noop and for Cluster instances with SB_CLUSTER_TLS_DIR unset.
// Safe to call exactly once after construction; subsequent calls overwrite.
AttachInternalHandler(h http.Handler)
// Members returns a snapshot of all known cluster members.
Members() []Member
// IngressTargets aggregates live ingress-role nodes' gossiped PublicHost
// values into the set of public addresses users must point DNS at for
// custom domains. Hostnames and raw IPs are partitioned via net.ParseIP;
// duplicates are removed; output ordering is stable so the API response
// is byte-identical across calls when membership is unchanged. Source
// reflects whether the cluster published a hostname, IPs, both, or
// nothing usable (see models.IngressTargetSource*). Service-layer
// IngressDNSTarget wraps this; clients hit GET /v1/ingress/dns.
IngressTargets() models.IngressTarget
// Placements returns the local FSM's hot placement snapshot. Recovery
// payloads (Spec/secrets) are omitted; use PlacementOf/SpecOf for point
// lookups that need them.
Placements() []Placement
// PlacementsForShards returns only placements whose sandbox ID belongs to
// one of the requested placement shards. Ingress nodes use this instead of
// pulling the full global placement map; server nodes serve it from the
// FSM shard index and agent nodes delegate it to a server-role control
// plane peer. Empty Shards means "all placements." Returned rows are hot
// rows without Spec/secrets.
PlacementsForShards(filter PlacementShardFilter) []Placement
// PlacementPage returns a bounded global placement-index page sorted by
// sandbox ID. It is the scalable read path for operators/control planes
// that need to enumerate large clusters without asking every worker for
// its local DB. Returned rows are hot rows without Spec/secrets.
PlacementPage(req PlacementPageRequest) PlacementPageResponse
// PlacementOf returns the full Placement record for sandboxID and true,
// or a zero Placement and false if no record exists. Operator/debug
// endpoints use this for convergence-status reads where the per-aspect
// getters (OwnerOf/ExposedPortsOf) would lose the placement.Version
// needed to compute "is this sandbox's route installed yet on this node".
PlacementOf(sandboxID string) (Placement, bool)
// PlacementVersion is the FSM's monotonic apply counter — bumps on every
// raft log entry the FSM applied. Exposed for metrics/observability and
// as a tie-breaker for tests; the ingress reconciler now uses
// SubscribePlacement to wake on apply rather than polling this counter.
// Zero means "no version data" (Noop or fresh cluster).
PlacementVersion() uint64
// SubscribePlacement returns a buffered (cap=1) channel that receives a
// signal after every FSM apply on this node. The channel is fed directly
// from FSM.Apply, so a leader-side commit reaches every node's
// subscribers as soon as raft delivers the log entry — no poll interval.
//
// Multiple applies between reads collapse into one wake (cap=1, drops on
// full). Cancel ctx (or the returned cancel func) to deregister; both
// are safe to call multiple times. Callers MUST tolerate spurious wakes —
// the channel says "something changed in the FSM," not "the placement
// you care about changed."
//
// In single-node mode (Noop) the returned channel never fires; this is
// safe to use in a select{} alongside a ticker because Go's select treats
// a never-firing nil channel as permanently un-ready, not an error.
SubscribePlacement(ctx context.Context) <-chan struct{}
// Leader returns the node ID of the current Raft leader, empty if none.
Leader() string
// Close shuts down the cluster cleanly.
Close() error
}
Client is the surface the rest of the daemon (Service, API handlers) interacts with. Both *Cluster and *Noop satisfy it so callsites stay unconditional.
type Cluster ¶
type Cluster struct {
// contains filtered or unexported fields
}
Cluster is the multi-node Client implementation. Construct with New.
func New ¶
New constructs the server-role Cluster for cfg.EnableCluster=true. Caller takes ownership of Close. For worker/ingress-only roles call NewAgent; for cfg.EnableCluster=false call NewNoop.
func (*Cluster) AddCustomDomain ¶
AddCustomDomain replicates a sandbox→hostname binding through raft so every node can answer the TLS-ask probe and install the matching Caddy matcher. hostname is canonicalized (trim + lower) here so callers don't have to remember to do it themselves — the local SQLite layer already trims. Idempotent for the same (sandbox, hostname) pair; returns ErrCustomHostnameConflict when the hostname is held by a different sandbox.
func (*Cluster) AddExposedPort ¶
func (c *Cluster) AddExposedPort(ctx context.Context, sandboxID string, port int, route ExposedPortRoute) error
AddExposedPort replicates a port-exposure intent. Idempotent when the route metadata is unchanged. Safe to call from any node.
func (*Cluster) ApplyEncoded ¶
ApplyEncoded is the receiving side of leader-forwarded raft writes. It decodes-validates the payload (so a malformed body never reaches the FSM) and applies it locally. Returns ErrNotLeader if leadership has changed since the forwarder picked us — the caller is expected to retry.
func (*Cluster) AssertOwnership ¶
func (c *Cluster) AssertOwnership(ctx context.Context, local []LocalSandboxState) error
AssertOwnership reconciles local sandbox state against the cluster FSM at boot. Used at boot. Idempotent. Best-effort callers should log errors and retry later.
Three-way decision per local row, with the FSM as the source of truth for ownership (never overwrite an existing non-self owner):
**No FSM placement.** Write opPlace claiming self as owner, with spec + secret ref so the new placement is born recoverable. This is the normal case for sandboxes created before AssertOwnership last ran.
**FSM owner == self.** No ownership change. Backfill missing Spec / secret ref via opUpsertSpec (closes the pre-spec-replication gap) and replay ExposedPort intents via opAddExposedPort (no-ops when already recorded).
**FSM orphaned from self.** Reclaim via opClaimOrphan, then replay the same spec/port backfill. This is the false-positive dead-owner recovery path: gossip/Raft marked us dead, but the local sandbox never left.
**FSM owner != self, or orphaned from another owner.** **Do NOT reclaim.** This is the failover-recovery/stale-local-row case. Calling RecordPlacement here would overwrite another owner or steal a different dead node's orphan. Instead: log loudly and leave the FSM alone. service.reconcileStaleOwnership handles destroying stale local copies on its next pass when the placement has an active different owner.
Edge cases:
- Owner unknown to gossip yet (fresh boot, gossip not converged): we still defer. Treating "presumed alive" as the safe default keeps us from racing a peer whose announce just hasn't reached us. The next reconcile pass reclassifies.
- Owner is dead in gossip: still defer until the leader has committed the orphan transition. Claiming before the FSM records which previous owner lost the row would recreate the overwrite hazard.
func (*Cluster) AttachInternalHandler ¶
AttachInternalHandler wires the public API mux into the cluster-internal mTLS listener so peers can reverse-proxy owner API calls over the cert-pinned channel. No-op when this node has no TLS material loaded (SB_CLUSTER_TLS_DIR empty) — there's no listener to attach to. Called once from cmd/sandboxd after the API server is constructed; the order avoids a service→cluster→api construction cycle.
func (*Cluster) AttachRecreator ¶
func (c *Cluster) AttachRecreator(r SandboxRecreator)
AttachRecreator wires the service-layer recreate hook used by the owner watcher. Called once from cmd/sandboxd/main after both service.New and cluster.New have returned. Safe to call concurrently with the watcher loop.
func (*Cluster) CancelReservation ¶
CancelReservation deletes a pending reservation for sandboxID. Safe to call at any rollback point: opCancelReserve only fires when the row is State == Reserved, so a stale cancel after a successful promote is a no-op. Idempotent; calling on a never-reserved id is also a no-op.
func (*Cluster) ClaimOrphan ¶
func (c *Cluster) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
ClaimOrphan commits sandboxID -> self only if the existing placement is currently orphaned and was orphaned from this node (or predates the previous-owner metadata). This is the false-positive recovery path for a node that was marked dead by gossip but never actually lost its local sandbox.
func (*Cluster) CustomDomainsOf ¶
CustomDomainsOf returns a sorted copy of the hostnames bound to sandboxID, or nil when none. Used by the ingress reconciler so a peer-owned sandbox still gets its TLS matchers installed in the local Caddy.
func (*Cluster) DeletePlacement ¶
DeletePlacement removes sandboxID from the placement map. Idempotent.
func (*Cluster) DeleteVolumeAttachmentsForSandbox ¶
func (*Cluster) ExposedPortsOf ¶
func (c *Cluster) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
ExposedPortsOf returns a copy of the replicated port route map. Returns nil if no placement exists or no ports are recorded.
func (*Cluster) ForwardHTTP ¶
ForwardHTTP reverse-proxies r to target and writes the response into w. The PAT bearer header is preserved (httputil.ReverseProxy passes through headers by default; we only strip hop-by-hop ones).
Channel selection: when this node has TLS material loaded AND target.InternalURL is non-empty, the proxy rides the cluster-internal mTLS channel — the receiving node's mTLS listener serves the same v1 API mux as the public port (see AttachInternalHandler), so handlers run identically. Falling back to target.APIURL (public, PAT-only) when either side lacks TLS keeps mixed/legacy clusters working.
Forwarding loops are detected by the X-Cluster-Forwarded header and returned as 421 Misdirected Request so clients retry against a refreshed placement.
IMPORTANT: a hard network/TLS failure on the internal channel must NOT silently fall back to the public path — that would defeat the cert-pinned security promise B3 is meant to enforce. The reverse proxy's ErrorHandler surfaces such failures as 502 to the original caller, who can then retry against a different peer.
func (*Cluster) HealthyForReads ¶
HealthyForReads is true once the FSM has caught up to the leader's last log index — i.e. our local OwnerOf reads are not stale by more than a single round trip. Used by EnsureClusterReady.
func (*Cluster) IngressTargets ¶
func (c *Cluster) IngressTargets() models.IngressTarget
IngressTargets aggregates live ingress-role members' gossiped PublicHost values. See aggregateIngressTargets for the partition / dedup / ordering rules — those are pinned by ingress_targets_test.go.
func (*Cluster) IsNodeDrained ¶
IsNodeDrained reports the FSM's view of nodeID's drain state. Read from the local FSM with no network hop.
func (*Cluster) LeaderAPIURL ¶
LeaderAPIURL returns the API URL of the current leader, or empty if unknown. Used by the API wrapper to forward mutating raft writes to the leader.
func (*Cluster) Members ¶
Members returns gossip-known members (self included), enriched with fresh capacity heartbeats where available.
func (*Cluster) OwnerOf ¶
OwnerOf reads the placement map from the local FSM (no network round-trip). Returns ErrUnknownSandbox if no row exists, or ErrOrphaned if the placement exists but has no active owner (auto-orphaned after the owning node died — see voter_autojoin / dead-owner reconciler).
func (*Cluster) OwnerOfName ¶
OwnerOfName resolves a replicated sandbox Name to its placement owner. This is intentionally a local FSM read just like OwnerOf; the name index is maintained inside the FSM apply path so it tracks the authoritative placement map exactly.
func (*Cluster) PlacementOf ¶
PlacementOf returns the FSM record for sandboxID. Goes through f.get so the returned Placement is deep-cloned and safe to mutate without aliasing live FSM state (same guarantee as Placements).
func (*Cluster) PlacementPage ¶
func (c *Cluster) PlacementPage(req PlacementPageRequest) PlacementPageResponse
func (*Cluster) PlacementVersion ¶
PlacementVersion returns the FSM's monotonic apply counter — bumps on every committed raft log entry. Exposed for metrics and as a tie-breaker for tests; the ingress reconciler uses SubscribePlacement to wake on apply rather than polling this counter.
func (*Cluster) Placements ¶
func (*Cluster) PlacementsForShards ¶
func (c *Cluster) PlacementsForShards(filter PlacementShardFilter) []Placement
func (*Cluster) PutVolumeAttachments ¶
func (*Cluster) ReassignPlacement ¶
func (c *Cluster) ReassignPlacement(ctx context.Context, sandboxID string, target PlacementTarget) error
ReassignPlacement moves sandboxID to target via opReassign.
func (*Cluster) RecordPlacement ¶
func (c *Cluster) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
RecordPlacement commits sandboxID -> self into the FSM via raft along with the (optional) creation spec. Idempotent. Safe to call from any node: applyCommand transparently forwards to the current leader if we're a follower. Passing spec=nil preserves a previously-recorded spec — see fsm.go opPlace handling.
spec MUST be redacted before being passed in; secrets is the provider handle the caller produces via service.PutClusterSecretsForRecipient. Passing an empty handle preserves a previously-recorded handle.
func (*Cluster) RecoveryBlob ¶
func (*Cluster) RemoveCustomDomain ¶
RemoveCustomDomain releases the binding. Idempotent — a stale call after a successful DeletePlacement is a no-op.
func (*Cluster) RemoveExposedPort ¶
RemoveExposedPort drops a replicated port-exposure intent. Idempotent.
func (*Cluster) RemoveMember ¶
RemoveMember explicitly retires nodeID from the raft configuration. It is an operator lifecycle command, not gossip failure detection: the caller should drain the node first, then stop/terminate it, then remove it from raft.
Before RemoveServer, we persist a drain mark and orphan any placements owned by the retiring node so surviving nodes stop routing new work there and clients get a clear 410 for lost sandboxes under the current non-HA policy.
func (*Cluster) ReplicateJSBundle ¶
func (c *Cluster) ReplicateJSBundle(ctx context.Context, owner string, req models.CreateJSBundleRequest) error
ReplicateJSBundle fans a just-uploaded JS bundle out to every other live cluster member. Isolate's bundle store is per-node with no cluster distribution, so without this an isolate create placed on a node other than the upload node fails "bundle not found". Best-effort: an unreachable peer is returned as an error for the caller to log, but the bundle is already stored locally so replication failure is non-fatal to the upload. Idempotent (the content-addressed store dedups a re-push) and loop-safe (the replicated POST carries HeaderJSBundleReplicated so the receiver stores it without fanning out again). No-op with no peers (single-node / lone node).
func (*Cluster) ReserveBatchOnTargets ¶
func (c *Cluster) ReserveBatchOnTargets(ctx context.Context, reservations []PlacementReservation) error
ReserveBatchOnTargets commits multiple reservation intents in one Raft entry. It is intended for create-burst frontends that have already selected targets and want the leader to admit the whole chunk against the same capacity view. The public one-create API still uses ReserveOnTarget, but scale harnesses and bulk create callers use this to avoid one Raft log entry per sandbox.
func (*Cluster) ReserveOnTarget ¶
func (c *Cluster) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error
ReserveOnTarget commits a capacity-and-name reservation for sandboxID owned by target before any docker side effect runs. The router calls this after SelectPlacement so the cluster has intent recorded the instant the router forwards the body to target. ttl bounds how long the reservation holds the slot before the leader GC sweep cancels it; pick large enough to cover the slowest expected image pull on the target.
redacted MUST be stripped of plaintext credentials (call service.RedactClusterSecrets); secrets is the secret provider handle. Both ride the reservation so a successful promote via opPlace can inherit them without re-shipping the payload — see fsm.go opPlace preserve-on-empty rules.
Returns ErrReservationConflict when the slot is already placed or held by a different owner; ErrNameConflict when redacted.Name is held cluster-wide. Either should map to 4xx at the API surface; transport errors stay 5xx (caller may retry).
func (*Cluster) ResolveCustomDomain ¶
ResolveCustomDomain answers the TLS-ask probe from the local FSM index in O(1). Returns sandboxID, true when the hostname is bound somewhere in the cluster; false otherwise. hostname is canonicalized inside the FSM.
func (*Cluster) SecretsOf ¶
func (c *Cluster) SecretsOf(sandboxID string) PlacementSecrets
SecretsOf returns a copy of the provider handle paired with SpecOf's spec.
func (*Cluster) SelectPlacement ¶
func (c *Cluster) SelectPlacement(req capacity.Request) (PlacementTarget, error)
SelectPlacement chooses an owner node for a new sandbox using power-of-two- choices. The algorithm samples up to 2 random alive members (including self) and picks the one with the most CPU+memory headroom relative to the requested resources. This gives near-optimal load balancing without any global coordination — at scale it converges to within a small constant factor of the truly-optimal choice while costing O(1) per placement.
Falls back to self if no alive members are visible yet (just-bootstrapped node) or if no other member can satisfy req. Self always loses ties because "place locally" is the existing single-node behavior; we only forward when a peer is genuinely better.
func (*Cluster) SelfAPIURL ¶
func (*Cluster) SelfNodeID ¶
func (*Cluster) SetLocalTemplateIDsProvider ¶
SetLocalTemplateIDsProvider registers the Phase 6 PR-D callback used by the capacity lease cache to overlay Firecracker template inventory onto local heartbeats. Idempotent; passing nil unregisters. Wired by main.go after the service is attached. Single-node deployments and nodes without Firecracker enabled never call this and degrade to nil-provider behaviour (no LocalTemplateIDs on the snapshot, peers see the legacy "unknown, allow" placement).
func (*Cluster) SetLocalWasmModuleIDsProvider ¶
SetLocalWasmModuleIDsProvider registers the WASM module inventory callback.
func (*Cluster) SetNodeDrainState ¶
SetNodeDrainState marks nodeID as drained (drained=true) or restores it to the SelectPlacement candidate pool (drained=false). Goes through raft so every node — including the drained node itself — sees the new state on its next placement decision. Idempotent.
Drain is a no-op on existing placements: a drained node continues serving the sandboxes it already owns. Combine with the dead-owner reconciler (kill the process) to force ownership transfer, or use a separate evacuate command (not implemented) to move work proactively.
func (*Cluster) SpecOf ¶
func (c *Cluster) SpecOf(sandboxID string) *models.CreateSandboxRequest
SpecOf returns a deep-copy of the replicated spec for sandboxID, or nil if none is recorded. The returned spec is REDACTED — registry passwords and mount credentials are stripped at write time. Use SecretsOf to retrieve the matching secret ref and service.OpenClusterSecretsForNode to merge credentials back in. Callers may safely mutate the returned struct (it shares no memory with the FSM).
func (*Cluster) SubscribePlacement ¶
SubscribePlacement returns a buffered (cap=1) wake channel that fires after every FSM apply on this node. Cancelling ctx removes the subscriber. See the Client.SubscribePlacement contract for semantics.
func (*Cluster) UpsertSpec ¶
func (c *Cluster) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
UpsertSpec replicates a sandbox spec mutation (resize, lifecycle change) without changing ownership. Idempotent; nil spec + empty secrets is a no-op. Safe to call from any node — applyCommand forwards to the leader as needed.
spec MUST be redacted; empty secrets preserves the previously replicated handle (resize/lifecycle never change credentials).
func (*Cluster) VolumeAttachmentCount ¶
func (*Cluster) VolumeByID ¶
func (*Cluster) VolumeByName ¶
func (*Cluster) VolumeDelete ¶
func (*Cluster) VolumeExistsForSource ¶
func (*Cluster) VolumeUpsert ¶
type ClusterTLS ¶
type ClusterTLS struct {
// contains filtered or unexported fields
}
ClusterTLS is the loaded TLS material the daemon uses for cluster-internal channels (raft transport + leader-forwarded HTTP applies). Built once at boot from the on-disk files and shared by the raft StreamLayer and the internal HTTPS listener / client.
type DrainStateResponse ¶
type DrainStateResponse struct {
Drained bool `json:"drained"`
}
type Endpoint ¶
Endpoint pairs a peer's optional cluster-internal mTLS URL with its public API URL. Passed to ForwardHTTP so the forwarder can transparently pick the cert-pinned internal channel when both ends have TLS material, falling back to the public APIURL (PAT-authenticated) otherwise.
type ExposedPortRoute ¶
type ExposedPortRoute struct {
Protocol string `json:"protocol"`
HostPort int `json:"host_port,omitempty"`
PublicURL string `json:"public_url,omitempty"`
}
Placement is one row of the FSM's placement map. Spec is the replicated sandbox creation request used by the new owner to re-materialize a sandbox after its previous owner died (see owner_watcher.go). Spec is a pointer so older snapshots written before spec replication still decode cleanly.
SecretRef/SecretVersion identify the secret provider record that can rehydrate the redacted Spec on the owner or an authorized recovery target. They are a handle, not secret material — raw sealed bytes have no field anywhere in cluster state, so Raft can never fan secret material out to every participant.
ExposedPortRoute is the replicated routing intent for one exposed sandbox port. Protocol drives which Caddy surface is used. HostPort is populated for raw TCP so every node can bind/proxy the same cluster-wide port.
type IngressRouteOwner ¶
type IngressShardRoute ¶
type IngressShardRoute struct {
SandboxID string `json:"sandbox_id"`
Shard int `json:"shard"`
ShardCount int `json:"shard_count"`
Owners []IngressRouteOwner `json:"owners"`
}
func IngressRouteForSandbox ¶
func IngressRouteForSandbox(members []Member, sandboxID string) IngressShardRoute
IngressRouteForSandbox returns the ingress owner set an upstream router should target for sandboxID. Small ingress tiers all own every sandbox route; very large ingress tiers return the stable shard owner.
type LocalSandboxState ¶
type LocalSandboxState struct {
ID string
Spec *models.CreateSandboxRequest
Secrets PlacementSecrets
ExposedPorts map[int]ExposedPortRoute
// CustomHostnames is the per-sandbox bound hostname set known to the
// local store. Carried through AssertOwnership so a sandbox created before
// the FSM learned about its hostnames (e.g. cluster mode enabled after the
// sandbox already had custom domains) backfills the replicated set on
// boot — without this a failover-recreate would lose the user's TLS
// matchers.
CustomHostnames []string
}
LocalSandboxState is one entry in the boot-time AssertOwnership payload. Carrying Spec + ExposedPorts (rather than just an ID) lets the boot replay backfill the FSM with everything a future failover-recreate needs — without this, sandboxes that pre-date cluster mode (or pre-date the spec/ports replication features) would never gain a replicated spec until their next mutating call.
Spec MUST be redacted (no plaintext registry password / mount credentials) before being handed to AssertOwnership; Secrets carries the provider ref that the new owner re-merges on recreate. cmd/sandboxd takes care of this via service.PutClusterSecretsForRecipient + RedactClusterSecrets so the cluster layer never sees plaintext. Secrets may be empty when the sandbox has no secrets to ship.
type Member ¶
type Member struct {
NodeID string `json:"node_id"`
// NodeName is the operator-friendly display label (SB_NODE_NAME). Empty
// for peers running pre-NodeName builds; dashboards fall back to NodeID.
NodeName string `json:"node_name,omitempty"`
APIURL string `json:"api_url"`
DataPlaneHost string `json:"data_plane_host,omitempty"`
RaftAddr string `json:"raft_addr,omitempty"`
// InternalURL is the cluster-internal mTLS endpoint used for raft
// leader-forward applies. Empty when the peer is running without
// SB_CLUSTER_TLS_DIR — the forwarder falls back to APIURL with PAT-only
// auth in that case.
InternalURL string `json:"internal_url,omitempty"`
// Role is the peer's gossiped SB_NODE_ROLE. Empty for older builds that
// pre-date the field; callers treat empty as the legacy "mixed" default.
Role string `json:"role,omitempty"`
// PublicHost is the peer's gossiped public ingress address
// (config.EffectivePublicHost). Aggregated by Cluster.IngressTargets to
// answer the DNS-helper API. Empty for peers without a public host set
// or running pre-PublicHost builds.
PublicHost string `json:"public_host,omitempty"`
Alive bool `json:"alive"`
Capacity capacity.Snapshot `json:"capacity"`
// CapacityUpdatedUnix is when this node's last capacity heartbeat was
// observed by the scheduler. CapacityStale means the last heartbeat is
// missing or too old for placement admission.
CapacityUpdatedUnix int64 `json:"capacity_updated_unix,omitempty"`
CapacityStale bool `json:"capacity_stale,omitempty"`
}
Member is a snapshot of a peer's gossiped state.
type Noop ¶
type Noop struct {
// contains filtered or unexported fields
}
Noop is the single-node-mode Client implementation. Every method behaves as if this node owns every sandbox and is the only cluster member. Used when cfg.EnableCluster is false so callsites can be unconditional.
func NewNoop ¶
NewNoop returns a single-node Client. nodeID and apiURL are reported back for observability but never actually used for routing. publicHost is the operator-configured public ingress address (config.EffectivePublicHost) that IngressTargets reports as the single-member target. Empty when the daemon runs in IP-only mode (no SB_PUBLIC_HOST / SB_DOMAIN), which also means custom domains are disabled.
func (*Noop) AddCustomDomain ¶
func (*Noop) AddExposedPort ¶
func (*Noop) AssertOwnership ¶
func (n *Noop) AssertOwnership(ctx context.Context, local []LocalSandboxState) error
func (*Noop) AttachInternalHandler ¶
AttachInternalHandler is a no-op for single-node mode — there's no mTLS listener to wire into, so nothing to do.
func (*Noop) CancelReservation ¶
func (*Noop) ClaimOrphan ¶
func (n *Noop) ClaimOrphan(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
func (*Noop) CustomDomainsOf ¶
func (*Noop) DeletePlacement ¶
func (*Noop) DeleteVolumeAttachmentsForSandbox ¶
func (*Noop) ExposedPortsOf ¶
func (n *Noop) ExposedPortsOf(sandboxID string) map[int]ExposedPortRoute
func (*Noop) ForwardHTTP ¶
func (*Noop) IngressTargets ¶
func (n *Noop) IngressTargets() models.IngressTarget
IngressTargets reports the single-node deployment's public address as the DNS target. Empty publicHost (IP-only mode) returns the Unknown source so the service layer can surface a clean 412 rather than fake records.
func (*Noop) IsNodeDrained ¶
func (*Noop) PlacementOf ¶
PlacementOf has no record in single-node mode — there's no FSM. Returns the zero Placement and false so callers fall back to the local sandbox row.
func (*Noop) PlacementPage ¶
func (n *Noop) PlacementPage(PlacementPageRequest) PlacementPageResponse
func (*Noop) PlacementVersion ¶
PlacementVersion always returns 0 in single-node mode — there's no FSM and no need to wake an ingress reconciler that isn't running.
func (*Noop) Placements ¶
func (*Noop) PlacementsForShards ¶
func (n *Noop) PlacementsForShards(PlacementShardFilter) []Placement
func (*Noop) PutVolumeAttachments ¶
func (*Noop) ReassignPlacement ¶
func (*Noop) RecordPlacement ¶
func (n *Noop) RecordPlacement(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
func (*Noop) RemoveCustomDomain ¶
func (*Noop) RemoveExposedPort ¶
func (*Noop) RemoveMember ¶
func (*Noop) ReserveOnTarget ¶
func (n *Noop) ReserveOnTarget(ctx context.Context, sandboxID string, target PlacementTarget, redacted *models.CreateSandboxRequest, secrets PlacementSecrets, ttl time.Duration) error
func (*Noop) ResolveCustomDomain ¶
func (*Noop) SecretsOf ¶
func (n *Noop) SecretsOf(sandboxID string) PlacementSecrets
func (*Noop) SelectPlacement ¶
func (n *Noop) SelectPlacement(req capacity.Request) (PlacementTarget, error)
func (*Noop) SelfAPIURL ¶
func (*Noop) SelfNodeID ¶
func (*Noop) SetNodeDrainState ¶
func (*Noop) SubscribePlacement ¶
SubscribePlacement returns nil in single-node mode. Selecting on a nil channel never proceeds, so an ingress reconciler that select{}s on this channel + a slow ticker just behaves as if the cluster never has placement events — which is exactly the truth.
func (*Noop) UpsertSpec ¶
func (n *Noop) UpsertSpec(ctx context.Context, sandboxID string, spec *models.CreateSandboxRequest, secrets PlacementSecrets) error
func (*Noop) VolumeAttachmentCount ¶
func (*Noop) VolumeByID ¶
func (*Noop) VolumeByName ¶
func (*Noop) VolumeExistsForSource ¶
func (*Noop) VolumeUpsert ¶
type OwnerInfo ¶
type OwnerInfo struct {
NodeID string
APIURL string
// InternalURL is the owner's cluster-internal mTLS URL. Empty when the
// owner has no TLS material (mixed/legacy cluster). Owner API forwarding
// uses this in preference to APIURL so the cross-node hop is cert-pinned.
InternalURL string
IsSelf bool
}
OwnerInfo is returned by OwnerOf.
type Placement ¶
type Placement struct {
SandboxID string `json:"sandbox_id"`
OwnerNodeID string `json:"owner_node_id"`
OwnerAPIURL string `json:"owner_api_url"`
OwnerDataPlaneHost string `json:"owner_data_plane_host,omitempty"`
OwnerState PlacementOwnerState `json:"owner_state,omitempty"`
OrphanedOwnerNodeID string `json:"orphaned_owner_node_id,omitempty"`
OrphanedUnix int64 `json:"orphaned_unix,omitempty"`
Version uint64 `json:"version"`
CreatedUnix int64 `json:"created_unix"`
UpdatedUnix int64 `json:"updated_unix"`
// Name is the small, hot copy of Spec.Name used to rebuild the cluster-wide
// uniqueness index without loading the larger recovery spec payload.
Name string `json:"name,omitempty"`
// RecoveryRef points at the out-of-snapshot recovery payload for this row.
// Spec/secret fields are hydrated from that store only for point lookups and
// recreate flows.
RecoveryRef string `json:"-"`
Spec *models.CreateSandboxRequest `json:"spec,omitempty"`
SecretRef string `json:"secret_ref,omitempty"`
SecretVersion int `json:"secret_version,omitempty"`
ExposedPorts map[int]string `json:"exposed_ports,omitempty"`
ExposedPortRoutes map[int]ExposedPortRoute `json:"exposed_port_routes,omitempty"`
// CustomHostnames is the replicated set of user-bound hostnames pointing at
// this sandbox. Kept sorted and lower-cased so snapshot bytes are stable
// across rebuilds and every node derives the same Caddy matcher order. The
// FSM additionally maintains a hostname→sandboxID index so the TLS-ask
// resolver and ingress reconciler can look up ownership in O(1) from any
// node, including the ingress-only ones that don't run the local SQLite
// custom-domains table. Hostnames are added/removed via the
// opAddCustomDomain/opRemoveCustomDomain raft commands; opPlace preserves
// the existing slice the same way ExposedPorts is preserved so a
// re-place/reassign cannot erase domains a prior raft entry installed.
CustomHostnames []string `json:"custom_hostnames,omitempty"`
// State is empty for materialized placements (the historical schema) and
// PlacementStateReserved for capacity-only intents that have not yet been
// promoted by a successful local create. Reservations are eligible for
// TTL-driven GC; opPlace transitions a reservation back to empty.
State PlacementState `json:"state,omitempty"`
// ExpiresUnix is meaningful only when State == PlacementStateReserved;
// the leader GC sweep cancels rows whose ExpiresUnix < now.
ExpiresUnix int64 `json:"expires_unix,omitempty"`
}
ExposedPorts is the legacy replicated set of port→protocol intents. Newer code also fills ExposedPortRoutes with HostPort/PublicURL metadata so remote ingress nodes can install owner-aware data-plane routes. Both fields are kept so older snapshots still restore cleanly.
func (Placement) IsOrphaned ¶
IsOrphaned reports whether the placement has no active owner.
func (Placement) IsReserved ¶
IsReserved reports whether p is a reservation awaiting promotion.
type PlacementLookupResponse ¶
type PlacementOwnerState ¶
type PlacementOwnerState string
PlacementOwnerState records the ownership lifecycle independently from the reservation lifecycle. Empty means the placement has an active owner.
const ( PlacementOwnerStateActive PlacementOwnerState = "" PlacementOwnerStateOrphaned PlacementOwnerState = "orphaned" )
type PlacementPageRequest ¶
type PlacementPageRequest struct {
Limit int `json:"limit,omitempty"`
PageToken string `json:"page_token,omitempty"`
ShardFilter PlacementShardFilter `json:"shard_filter,omitempty"`
}
func (PlacementPageRequest) Normalize ¶
func (r PlacementPageRequest) Normalize() PlacementPageRequest
type PlacementPageResponse ¶
type PlacementReservation ¶
type PlacementReservation struct {
SandboxID string
Target PlacementTarget
Redacted *models.CreateSandboxRequest
Secrets PlacementSecrets
TTL time.Duration
}
PlacementReservation is one item in a leader-side batch reservation request. Redacted MUST have plaintext credentials stripped before the caller passes it in; Secrets carries only a provider handle or legacy sealed bytes.
type PlacementSecrets ¶
type PlacementSecrets struct {
Ref string `json:"ref,omitempty"`
Version int `json:"version,omitempty"`
}
PlacementSecrets is the recoverability handle paired with a redacted placement spec. It carries a secret provider reference only — the encrypted secret payload lives behind the service secret provider, never in Raft state. There is deliberately no field for raw sealed bytes: the raft log and FSM snapshots are not erasable per-sandbox, so making the field unrepresentable is what enforces the "no secret material in the log" rule.
type PlacementShardFilter ¶
type PlacementShardFilter struct {
ShardCount int `json:"shard_count,omitempty"`
Shards []int `json:"shards,omitempty"`
}
PlacementShardFilter asks the control plane for only a subset of placement shards. Empty Shards means "all shards" for compatibility with existing callers; ShardCount <= 0 uses DefaultPlacementShardCount.
func IngressShardFilterForNode ¶
func IngressShardFilterForNode(members []Member, nodeID string) PlacementShardFilter
IngressShardFilterForNode returns the placement shard subset a single ingress-capable node should reconcile. Small ingress tiers replicate the full public route table to each ingress node so ordinary DNS round-robin / TCP load balancers work for every sandbox. Very large ingress tiers shard the table and require a shard-aware upstream router.
func (PlacementShardFilter) Normalize ¶
func (f PlacementShardFilter) Normalize() PlacementShardFilter
Normalize returns a deterministic, de-duplicated copy of f.
type PlacementState ¶
type PlacementState string
PlacementState distinguishes a reservation (capacity held, no docker yet) from a placement (sandbox materialized). Empty defaults to Placed so pre-reservation snapshots restore correctly: every old row is a real placement, never a pending reservation.
const ( PlacementStatePlaced PlacementState = "" // empty = legacy/placed PlacementStateReserved PlacementState = "reserved" )
type PlacementTarget ¶
type PlacementTarget struct {
NodeID string
APIURL string
DataPlaneHost string
// InternalURL is the peer's cluster-internal mTLS URL (e.g. https://10.0.0.5:7002).
// Empty when the peer is running without SB_CLUSTER_TLS_DIR or hasn't yet
// gossiped its advertise URL. Cross-node create-forwarding uses this in
// preference to APIURL so the hop rides the cert-pinned channel.
InternalURL string
IsSelf bool
}
PlacementTarget is returned by SelectPlacement.
type RecoveryBlob ¶
type SandboxRecreator ¶
type SandboxRecreator interface {
RecreateSandbox(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets PlacementSecrets, exposedPorts map[int]ExposedPortRoute) error
}
SandboxRecreator is the cluster's escape hatch back into the service layer for the failover-recreate path. The owner watcher (see owner_watcher.go) invokes RecreateSandbox for any FSM placement that points to self but has no corresponding local sandbox — typically because the previous owner died and the dead-owner reconciler reassigned the placement here.
exposedPorts carries the replicated port routing intents the previous owner had recorded; the implementation is expected to re-issue ExposePort for each entry after the create succeeds. Raw TCP entries include the original HostPort so the recreated owner can preserve the public endpoint when the port is available on the new node.
Implementations MUST be idempotent: the watcher polls and may invoke RecreateSandbox multiple times for the same id while a previous attempt is still in flight or after an unexpected restart.
secrets is the provider ref that can rehydrate the redacted spec. Legacy sealed bytes may be present only for old placement rows. The implementation is expected to resolve and re-merge it before instantiating the container — without this step, recreated sandboxes would lose access to the user's private registry / mount credentials.
type SelectPlacementRequest ¶
type SelectPlacementResponse ¶
type SelectPlacementResponse struct {
Target PlacementTarget `json:"target"`
Error string `json:"error,omitempty"`
}
type VolumeQueryResponse ¶
type VolumeQueryResponse struct {
Volume *models.Volume `json:"volume,omitempty"`
Volumes []models.Volume `json:"volumes,omitempty"`
Exists bool `json:"exists,omitempty"`
Count int `json:"count,omitempty"`
}
VolumeQueryResponse is the wire shape of the internal volume read endpoint. Exactly one of Volume / Volumes / Exists is meaningful per request kind.
Source Files
¶
- agent.go
- agent_volumes.go
- capacity_lease.go
- client.go
- client_volumes.go
- cluster.go
- dead_owner.go
- exposed_ports.go
- failover_policy.go
- forward.go
- fsm.go
- fsm_volumes.go
- gossip.go
- ingress_targets.go
- internal_server.go
- internal_transport.go
- jsbundle_replication.go
- metrics.go
- noop.go
- owner_watcher.go
- placement.go
- raft.go
- raft_recovery.go
- raft_tls.go
- recovery_replication.go
- recovery_store.go
- shards.go
- tls.go
- topology.go
- voter_autojoin.go
- wasm_migrate.go