service

package
v0.6.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: MIT Imports: 94 Imported by: 0

Documentation

Index

Constants

View Source
const (
	EgressStatusActive      = "active"
	EgressStatusHeld        = "held"
	EgressStatusUnavailable = "unavailable"
	// EgressStatusUnenforced: the sandbox's runtime enforces its policy with
	// the host firewall, which is off on this node, so nothing filters its
	// traffic. Only a sandbox accepted under SB_ALLOW_UNENFORCED_NETWORK_POLICY,
	// or created before the firewall was turned off, shows it.
	EgressStatusUnenforced = "unenforced"
)

Egress status shown on GET (egress_status).

View Source
const (
	// ExecLedgerScope is the request_idempotency scope for exec rows. Its value
	// is load-bearing for upgrades: renaming it would orphan every pending row
	// a running fleet already holds, so it stays "runloop.exec" even though the
	// mechanism is no longer Runloop-specific.
	ExecLedgerScope = "runloop.exec"
	// ExecLedgerTTL is how long a settled execution replays, counted from when
	// it settled, and how long a named execution's claim holds.
	ExecLedgerTTL = 24 * time.Hour
	// ExecObligationHold is a plain execution's pending hold: longer than any
	// sandbox lives, so the row stays until its watcher settles it instead of
	// lapsing on a timer (which would let a retry run the command again).
	ExecObligationHold = 100 * 365 * 24 * time.Hour
)

The execution ledger is the durable identity of an execution, kept in the shared request_idempotency table (one row per execution, scope ExecLedgerScope). It outlives the disposable output cache, so a retried command id joins or replays the original execution whatever happened to the cache. The row-state policy lives here (D14) rather than in a facade, so any surface can get durable exec dedup without re-deriving it; the Runloop facade still owns the fingerprint scheme (which bytes identify an execution) and the per-execution watcher that settles a row once its toolbox session is gone.

Row states, for a plain execution:

pending — claimed; the command was (or is about to be) submitted, and its
          toolbox session may still exist. A pending row is an obligation,
          not a lock: it never lapses on a timer (a lapsed row would be
          swept, or re-claimed by a retry that runs the command again), and
          only its watcher settles it, once the session is confirmed gone.
ready   — settled: the session is gone. target_id holds the ExecOutcome, so
          the exit status survives a lost output cache, or marks the
          execution lost when its session vanished with no result.

A row is deleted early only by the request that claimed it, and only when the toolbox refused the command outright: then nothing can have run.

A named-shell execution's row goes ready as soon as the command is submitted, with the execution id as target_id: the shell keeps the command's state, so the ledger only has to remember which command a retry maps to. Its pending state is an ordinary lock that lapses after ExecLedgerTTL.

View Source
const (
	CustodyWrappedKeyStore = "customer key store (wrapped)"
	CustodyPlaintextFile   = "plaintext file"
	CustodyEnvVar          = "env var"
	CustodyLocalServerKey  = "local (server key)"
	CustodyNotConfigured   = "not configured"
)

Custody report answers (§9).

View Source
const (
	CustodyWrappedSoftwareToken = "customer key store (wrapped, software token)"
	CustodyPKCS11SoftwareToken  = "pkcs11 (software token)"
)

The answers for a PKCS#11 slot that is not hardware (SoftHSM2): the key sits in a file on this host, so the report does not claim an HSM (§2).

View Source
const (
	KindKeyRevoked   = "key_revoked"
	KindKeyRecovered = "key_recovered"
)

Secret-audit kinds of a revoke and a recovery: new Kind values, no new field (§8).

View Source
const (
	KindKeyRotationStaged   = "key_rotation_staged"
	KindKeyRotationPromoted = "key_rotation_promoted"
	KindKeyRotationRetired  = "key_rotation_retired"
)

Secret-audit kinds of a rotation step, written by each node when it takes the step: a new Kind value, no new field (§8).

View Source
const (
	ToolboxSurfaceV1      = "v1"
	ToolboxSurfaceDaytona = "daytona"
	ToolboxSurfaceE2B     = "e2b"
	ToolboxSurfaceRunloop = "runloop"
)

Surfaces a toolbox call can come from.

View Source
const KindIdentityTokenIssued = "identity.token_issued"

KindIdentityTokenIssued is the audit record of one token delivered to a sandbox (D1.10): sandbox, owner (the team), and in Reason the role tag, key id, token id, expiry and audience. Never the token. A new Kind and no new field, like the key-rotation records, so every build verifies it.

View Source
const KindKeyCustodyChanged = "key_custody_changed"

KindKeyCustodyChanged is the secret-audit record of a custody change between boots (S7): a new Kind value, no new field, so older binaries re-marshal the record byte for byte (§8).

View Source
const TemplateArtifactSchemaVersion = 1

TemplateArtifactSchemaVersion is the wire/storage schema of the manifest.json that travels inside the OCI image alongside the three artifact files. PR 6-B.2 (consumer side) reads this and refuses unknown versions; bumping it means a coordinated push/pull rollout.

Variables

View Source
var (
	// ErrAPITokenInvalid is a create request that fails validation (400);
	// the message names the field.
	ErrAPITokenInvalid = errors.New("invalid token request")
	// ErrAPITokenNameTaken is a create whose name another token in the team
	// already holds (409). Two live tokens never share a name.
	ErrAPITokenNameTaken = errors.New("token name already used in this team")
	// ErrAPITokenClusterVersion is a token write before every Raft server
	// runs a build that applies token ops (409).
	ErrAPITokenClusterVersion = errors.New("not every server supports API tokens yet: upgrade every server first")
	// ErrAPITokensUnavailable is a token write whose outcome is unknown or
	// that could not reach the leader (503). After a create, the token may
	// exist without its secret: list tokens, find it by name, revoke it by
	// id, then create it again.
	ErrAPITokensUnavailable = errors.New("API token store unavailable")
)
View Source
var (
	// ErrIdentityDisabled is an identity create on a deployment that is not
	// an issuer (400, D1.1 failure handling).
	ErrIdentityDisabled = errors.New("identity needs SB_IDENTITY_ISSUER_URL")
	// ErrIdentityInvalid is a malformed identity option or role (400).
	ErrIdentityInvalid = cloudidentity.ErrInvalid
	// ErrIdentityRuntime is identity on a runtime with no file a cloud SDK
	// could read (400, D1.9).
	ErrIdentityRuntime = errors.New("identity is not supported on the wasm and isolate runtimes")
	// ErrIdentityRoleNotFound is a tag not on the owner's allow list: 404 on
	// the roles API.
	ErrIdentityRoleNotFound = store.ErrIdentityRoleNotFound
	// ErrIdentityRoleDenied is a create asking for a tag not on the allow
	// list, or by a caller below its minimum role, or a roles call for a team
	// the caller is not bound to (403, D1.5).
	ErrIdentityRoleDenied = errors.New("identity role tag not allowed")
	// ErrIdentityUnavailable means the keys or the allow list could not be
	// read or written right now, or a cluster member still runs a build
	// without identity (503, retry).
	ErrIdentityUnavailable = errors.New("cloud identity is unavailable")
	// ErrIdentityKeysConflict is a key-set write another writer beat; the
	// service re-reads and decides again, so callers only see it wrapped in
	// ErrIdentityUnavailable after repeated losses.
	ErrIdentityKeysConflict = store.ErrIdentityKeysConflict
)
View Source
var (
	// ErrBuildInvalid is a build request the daemon will not run: no
	// Dockerfile, a missing push credential, or (for a tenant) a Dockerfile
	// naming an image only this node has.
	ErrBuildInvalid = errors.New("invalid image build")
	// ErrBuildNotImplemented is a build needing context uploads, which the
	// daemon has no resolver for yet.
	ErrBuildNotImplemented = errors.New("image build context uploads are not supported on this daemon")
	// ErrBuilderUnavailable is a daemon started without an image builder.
	ErrBuilderUnavailable = errors.New("image builder is not configured on this daemon")
	// ErrBuildTimeout is a build or push that ran past the build timeout.
	ErrBuildTimeout = errors.New("image build exceeded timeout")
	// ErrBuildFailed is the engine failing a build or a push.
	ErrBuildFailed = errors.New("image build failed")
)

Build errors. The API surfaces map each to one status (apihttp.WriteStoreAwareError), so a build fails the same way everywhere.

View Source
var (
	// ErrSandboxNotRunning is returned by PauseSandbox when the sandbox is in
	// a state that cannot be paused (anything but started or already stopped).
	ErrSandboxNotRunning = errors.New("sandbox is not running")
	// ErrSandboxNotResumable is returned by ResumeSandbox when the sandbox is
	// in a state that cannot be resumed (anything but stopped or already
	// started).
	ErrSandboxNotResumable = errors.New("sandbox is not resumable")
)

Lifecycle state-transition errors. A facade words them for its own SDK (apihttp.Classify maps both to 409); the sentinels are the precondition a pause/resume could not meet, decided once here instead of in each facade (D7).

View Source
var (
	// ErrTokenVisibilityInvalid is a 400: an unknown visibility value.
	ErrTokenVisibilityInvalid = errors.New("invalid port visibility")
	// ErrTokenVisibilityUnavailable is a 400: this deployment, runtime or
	// protocol cannot gate a port behind a token.
	ErrTokenVisibilityUnavailable = errors.New("visibility \"token\" is not available here")
	// ErrTokenVisibilityConflict is a 409: a custom domain targets the
	// port, which would be a second, ungated way in.
	ErrTokenVisibilityConflict = errors.New("a custom domain targets this port")
)
View Source
var (
	// ErrSandboxTokenInvalid is a 400: the request asked for a token the
	// server will not issue (no or too long a lifetime, a bad scope).
	ErrSandboxTokenInvalid = errors.New("invalid sandbox token request")
	// ErrSandboxTokensUnavailable is a 503: this node has no server key to
	// sign with (none configured, or a key-store revoke dropped it).
	ErrSandboxTokensUnavailable = errors.New("sandbox tokens are unavailable: this server has no server key loaded")
	// ErrSandboxTokenRevoked is a 401: a good token used on its sandbox's
	// toolbox after that sandbox's tokens were revoked. The words match
	// the public-port gate's refusal for the same token.
	ErrSandboxTokenRevoked = errors.New("sandbox token refused: revoked")
)
View Source
var (
	ErrClusterSecretOriginatorDenied     = errors.New("cluster secret originator is not authoritative")
	ErrClusterSecretPlacementUnavailable = errors.New("cluster secret placement is unavailable")
)
View Source
var (

	// ErrSecretAuditBusy is returned when this node's audit read slots are
	// all taken for longer than secretAuditQueryAdmitWait. It maps to 429 +
	// Retry-After: the caller should back off, not wait out its deadline in
	// a queue that would only answer with a 504.
	ErrSecretAuditBusy = errors.New("secret audit read capacity is exhausted on this node; retry shortly")

	// ErrSecretAuditChainBroken is returned for every local read once the
	// indexer has seen the hash chain fail to link. It clears only with a
	// restart, which re-verifies the whole file (and refuses to boot on it
	// under strict mode).
	ErrSecretAuditChainBroken = errors.New("local secret audit chain failed verification; evidence on this node is not served until it is repaired")
)
View Source
var AutoImportDisabledError = errors.New("auto-import disabled")

AutoImportDisabledError is returned by callsites that want to distinguish "feature off" from "feature on, but failed". main() can switch on it to suppress alerts.

View Source
var ErrClusterFinalizationUnavailable = errors.New("cluster placement finalization temporarily unavailable")

ErrClusterFinalizationUnavailable means local runtime teardown completed but the owner+incarnation-fenced placement could not be durably removed. The local sandbox row is intentionally retained as a retry/reconcile anchor.

View Source
var ErrCreateInProgress = errors.New("a create of this sandbox is still running on this node; retry shortly")

ErrCreateInProgress is answered to a create that waited out createLeaseWait behind another create of the same sandbox ID on this node. Nothing was created or changed; the same request retried later replays or creates.

View Source
var ErrCreatePolicyInvalid = errors.New("create policy file is invalid; creates are refused until it is fixed")

ErrCreatePolicyInvalid refuses every create and resize while the create policy file (SB_CREATE_POLICY_FILE) is set but has not loaded since boot: its limits are unknown, so the only safe answer is no (503, code create_policy_invalid). A bad reload keeps the last good file instead and never lands here.

View Source
var ErrCreateRollbackPending = fmt.Errorf("%w: the previous attempt failed and its rollback has not finished", ErrCreateInProgress)

ErrCreateRollbackPending answers a create of an id whose earlier create was reported failed and whose rollback could not be finished now either: replaying that sandbox would hand the client one that is about to be destroyed. It is an ErrCreateInProgress, so the API answers 503 with Retry-After, and a retry after the rollback has finished creates afresh.

View Source
var ErrCustomDomainNotSupported = models.ErrCustomDomainNotSupported

ErrCustomDomainNotSupported is re-exported from pkg/models so callers that already import internal/service (every handler) don't need a second import just to errors.Is against the sentinel. Same pattern other service-layer sentinels (ErrSandboxManuallyStopped) follow.

View Source
var ErrCustomDomainPerSandboxCap = models.ErrCustomDomainPerSandboxCap

ErrCustomDomainPerSandboxCap — see pkg/models comment.

View Source
var ErrCustomDomainPortMismatch = store.ErrCustomDomainPortMismatch

ErrCustomDomainPortMismatch surfaces the store-layer port-mismatch sentinel under the service-layer sentinel that the API handler already maps to 409. See store.ErrCustomDomainPortMismatch for the underlying rule.

View Source
var ErrCustomDomainProtocolConflict = models.ErrCustomDomainProtocolConflict

ErrCustomDomainProtocolConflict — see pkg/models comment.

View Source
var ErrEgressApplyFailedHeld = errors.New("egress policy stored but not applied; the sandbox is held until it can be")

ErrEgressApplyFailedHeld means a policy update is stored but could not be made live. A running container sandbox is held without egress (D16) until a retry of the idempotent PUT or the reconcile pass applies it (503 apply_failed_held).

View Source
var ErrEgressFloorNotReady = errors.New("this node cannot keep sandboxes off the cloud metadata service right now; creates on this runtime are refused until it can")

ErrEgressFloorNotReady refuses a container or Firecracker create while the egress deny floor is not installed for that runtime: the sandbox could reach the cloud metadata service and the node's credentials (503, egress_floor_not_ready; the supervisor retries every few seconds).

View Source
var ErrEgressGatewayRequired = fmt.Errorf("hostname egress filtering needs the egress gateway on this node: %w", models.ErrRuntimeNotImplemented)

ErrEgressGatewayRequired means a policy needs the egress gateway (hostname entries or learn mode) and this node can't attach the sandbox to one.

View Source
var ErrEgressGatewayUnavailable = errors.New("egress gateway unavailable")

ErrEgressGatewayUnavailable means the gateway could not attach the sandbox (unreachable, version skew, a failed nft write). The create rolls back and the API answers 503: a 2xx would claim a policy that is not live.

View Source
var ErrEgressInjectRecreate = errors.New("recreate the sandbox with this inject rule: only env keys withheld at create can be injected")

ErrEgressInjectRecreate is a live policy change that injects an env key the sandbox already holds in clear (409, P3-2): only keys withheld at create can be protected.

View Source
var ErrEgressInspectRecreate = errors.New("recreate the sandbox with inspect rules: trusting the egress gateway's CA is set up at create")

ErrEgressInspectRecreate is a live policy change that adds an inspect rule to a container sandbox created without one: trusting the node's CA is set up at create (409).

View Source
var ErrEgressLearnConflict = errors.New("sandbox is in egress learn mode; switch it to enforce with the native policy API before setting lists")

ErrEgressLearnConflict is a facade update that sets lists on a learn-mode sandbox (409): learn mode needs empty lists, and switching it to enforce is the owner's call, through the native API.

View Source
var ErrEgressOperatorConfigInvalid = errors.New("egress operator file is invalid; creates are refused until it is fixed")

ErrEgressOperatorConfigInvalid refuses every create while the operator file (SB_EGRESS_OPERATOR_FILE, plans/egress-domain-filtering.md §5.10) is present but invalid at boot: the default policy and the ceiling are unknown, so the only safe answer is no (503, code egress_operator_config_invalid). An invalid reload keeps the last good file instead and never lands here.

View Source
var ErrEgressPolicyBusy = errors.New("sandbox is changing state; retry the egress policy update shortly")

ErrEgressPolicyBusy means the sandbox is mid-create or mid-transition, so a policy written now could be overtaken by the request it was created with (409, retry shortly).

View Source
var ErrEgressProcidRefused = errors.New("egress: no per-binary rules to trace for this sandbox")

ErrEgressProcidRefused refuses an executable lookup for a sandbox that isn't a started, gateway-mode sandbox with per-binary rules on this node.

View Source
var ErrEgressProfileCapExceeded = errors.New("egress profile update would exceed a referencing sandbox's hostname cap")

ErrEgressProfileCapExceeded is a profile PUT that would push a referencing sandbox past the 1024-hostname union cap (409, naming the sandboxes), so a re-apply never fails on caps.

View Source
var ErrEgressProfileInUse = store.ErrEgressProfileInUse

ErrEgressProfileInUse is a DELETE of a profile live sandboxes reference (409).

View Source
var ErrEgressProfileNotFound = store.ErrEgressProfileNotFound

ErrEgressProfileNotFound is a profile GET for a name the owner doesn't have (404).

View Source
var ErrEgressProfileUnavailable = errors.New("egress profiles are unavailable")

ErrEgressProfileUnavailable means the profiles a policy references could not be read right now (503, retry).

View Source
var ErrEgressProfilesConflict = errors.New("sandbox references egress profiles; clear them with the native policy API before blocking all egress")

ErrEgressProfilesConflict is a facade block-all on a sandbox that references egress profiles (409).

View Source
var ErrEgressSelfTestFailed = fmt.Errorf("egress hostname filtering is not available on this node (gateway self-test failed): %w", models.ErrRuntimeNotImplemented)

ErrEgressSelfTestFailed refuses a gateway-mode create on a node whose per-bridge self-test failed (plans/egress-domain-filtering.md §5.3 step 4, T41): redirected traffic would never reach the gateway there, so the sandbox would have no egress at all. 501, like any node that can't offer hostname filtering; in a cluster the node also stops advertising egress_gateway_ready, so placement goes elsewhere.

View Source
var ErrEgressSpecCommitFailed = errors.New("egress policy update not committed to the cluster; nothing changed")

ErrEgressSpecCommitFailed means a policy update could not be committed to the cluster's replicated spec, so nothing changed (503 spec_commit_failed). Committing first and strictly (D10) is what keeps a failover from resurrecting the old policy.

View Source
var ErrExecRequestTooLarge = errors.New("exec request body is too large")

ErrExecRequestTooLarge refuses a buffered exec body sandboxd will not read whole to apply the operator's maximum. It maps to 413.

View Source
var ErrIPv6FenceNotReady = errors.New("this node cannot keep IPv6 off its sandbox networks right now; creates on this runtime are refused until it can")

ErrIPv6FenceNotReady refuses a container or Firecracker create while the node's IPv6 fence is not installed for its runtime (503, ipv6_fence_not_ready; the node retries every few seconds).

View Source
var ErrInvalidClusterSecretBlob = errors.New("invalid cluster secret blob")

ErrInvalidClusterSecretBlob is a client-fixable peer-push body (ref / version / recipients shape). Mapped to HTTP 400 by the internal handler.

View Source
var ErrNetworkPolicyUnenforced = errors.New("network policy refused: the host firewall is off on this node")

ErrNetworkPolicyUnenforced is a create, update or start that needs the host firewall on a node where it is off (501 network_policy_unenforced). On the docker, containerd and Firecracker runtimes block-all, the CIDR lists and the byte-cap blocks are host firewall rules, and hostname rules, profiles and learn mode rely on them too: the sandbox boots behind a block-all until the egress gateway attaches it, and a failed attach or apply holds it with a firewall DROP. With the firewall off (SB_ENABLE_NETWORK_RULES=false, or not a Linux host) every one of those is a silent no-op, so the policy would be stored, shown on GET, and not enforced. WASM and isolate filter egress in their own host-side mediator and are never refused.

View Source
var ErrNodeGuardNotReady = errors.New("this node cannot keep sandboxes off its own services right now; creates on this runtime are refused until it can")

ErrNodeGuardNotReady refuses a container or Firecracker create while the node guard is not installed for that runtime: the sandbox could reach every service on the node (503, node_guard_not_ready; the supervisor retries every few seconds).

View Source
var ErrNodeStorageRetirementAlive = errors.New("cluster: node is alive; storage retirement requires a decommissioned node")

ErrNodeStorageRetirementAlive refuses an attestation for a node gossip still reports as alive. A live node can ACK; attesting its disk destroyed would discard a real obligation.

View Source
var ErrOrgProfileInvalid = errors.New("org egress profile is not defined")

ErrOrgProfileInvalid is an org: reference the operator file doesn't define (400 at create and policy PUT; a hold with reason org_profile_invalid on a running sandbox, §5.10 PC-3).

View Source
var ErrOwnershipFenced = errors.New("cluster: this node cannot reach the control plane to confirm it still owns the sandbox; it is paused until it can")

ErrOwnershipFenced is the answer for a sandbox this node has not been able to prove it still owns: it is frozen until the control plane answers.

View Source
var ErrPreferredHostPortUnavailable = errors.New("preferred host port unavailable on this node; exposure parked")

ErrPreferredHostPortUnavailable is returned by exposePort when a TCP replay supplied a specific preferredHostPort that's already reserved (cluster-wide or on this node) and so cannot be re-bound. The allocator deliberately does NOT silently fall through to a fresh random port: cluster-stable TCP endpoints are the entire point of B6 — clients addressing host:40123 must not be invisibly rerouted to host:55555 after a failover-recreate. Park is the policy; the FSM record (with the original HostPort) stays intact and the watcher / operator surfaces the parked state instead of mutating the contract behind the client's back.

View Source
var ErrPublicTrafficDisabled = errors.New("public traffic is disabled for this sandbox; pass allow_public_traffic=true at create or expose a port to opt in")

ErrPublicTrafficDisabled is returned when a still-private sandbox (allow_public_traffic=false) is asked for a custom-domain route. It is NOT returned by expose_port — expose is the opt-in lever that flips the sandbox public (enableSandboxPublicTraffic); custom domains alone stay refused so that attaching a domain never silently publishes a sandbox the caller kept private. The sandbox stays reachable through the toolbox proxy and SSH gateway.

View Source
var ErrReservedPlacementGone = errors.New("reserved placement is no longer authoritative")

ErrReservedPlacementGone answers a reserved create whose reservation this node no longer holds: a failed create of the same id cancelled it (a retry that waited for that create lands here), it expired, or it names another owner. Nothing was created. The client's retry, through any node, reserves afresh, so the API answers 503 with Retry-After rather than a 500.

View Source
var ErrSandboxManuallyStopped = errors.New("sandbox manually stopped (do not wake)")

ErrSandboxManuallyStopped is returned by the wake helper when the sandbox was stopped via the manual StopSandbox path. Surfaces 409 at the control-plane proxy so the caller knows to call StartSandbox explicitly instead of triggering an implicit resume.

View Source
var ErrSandboxStoppedMidRequest = fmt.Errorf("%w: it stopped while the request was in flight", ErrSandboxNotRunning)

ErrSandboxStoppedMidRequest ends a toolbox request whose sandbox stopped, exited or was destroyed while the request waited on it. It wraps ErrSandboxNotRunning, so every surface answers 409 and words it for its SDK. 409 is deliberate: the SDKs do not resend an exec on a 409 (they do on a 503), and whether the command ran is unknown.

View Source
var (
	// ErrSecretAuditIndexIncomplete refuses an unbounded all-worker scan when
	// Raft cannot identify the nodes that owned the sandbox's evidence.
	ErrSecretAuditIndexIncomplete = errors.New("secret audit node index is unavailable or truncated")
)
View Source
var ErrTeamQuotaExceeded = models.ErrTeamQuotaExceeded

ErrTeamQuotaExceeded is a create, resize or start a team quota refuses (429).

View Source
var ErrTeamQuotaNotFound = models.ErrTeamQuotaNotFound

ErrTeamQuotaNotFound is a read of a team with no quota (404).

View Source
var ErrTeamQuotaUnavailable = errors.New("team quotas are unavailable")

ErrTeamQuotaUnavailable means the quota store could not be reached (503).

View Source
var ErrWakeCircuitOpen = errors.New("wake circuit open")

ErrWakeCircuitOpen is returned when the per-sandbox wake breaker is tripped (D3). Surfaces 503 + Retry-After: 60 at the control-plane proxy so a sandbox stuck in a cold-start failure loop does not thrash StartSandbox on every request for the full open window.

View Source
var ErrWarmPoolLimit = store.ErrWarmPoolLimit

ErrWarmPoolLimit is a PUT of a new entry past models.MaxWarmPools (409).

View Source
var ErrWarmPoolNotFound = store.ErrWarmPoolNotFound

ErrWarmPoolNotFound is a GET of an image with no entry (404).

View Source
var ErrWarmPoolsUnavailable = errors.New("warm pools are unavailable")

ErrWarmPoolsUnavailable means the replicated entries could not be read or written right now: no leader, an unfinished rolling upgrade, or the server tier out of reach (503, retry).

View Source
var ErrWasmCustomDomainPortNotExposed = errors.New("wasm custom domain target_port is not an exposed http port")

ErrWasmCustomDomainPortNotExposed is returned when a WASM sandbox custom domain targets a guest HTTP port that is not exposed while other HTTP ports already are. Attaching before the first expose_port is allowed.

Functions

func AuditCallerFromContext added in v0.6.8

func AuditCallerFromContext(ctx context.Context) *auditlog.Caller

AuditCallerFromContext is the caller a record written under ctx names, before the rolling-upgrade gate:

  • the request's principal, when the API edge attached an Access: kind and id, the display name when it adds something, the team a team-bound credential acted for, and Via, the edge node a forwarded request came through (from the verified mTLS peer);
  • except for a call the SSH gateway bridged for a sandbox key holder (the edge attached an auditlog.SSHGatewayCall after checking where it came from): the gateway sends the operator token because that is the credential its loopback hop has, but the operator did nothing, so the caller is sandbox_key:<id>, via the gateway's node;
  • otherwise the system:<subsystem> principal background work tagged;
  • otherwise system:daemon, for work reached through an untagged entry point.

Use (*Service).auditCaller to fill a record: it also applies the gate.

func ContextWithSecretAuditCorrelation added in v0.6.1

func ContextWithSecretAuditCorrelation(ctx context.Context, correlationID string) context.Context

ContextWithSecretAuditCorrelation attaches correlationID for audited loads (include_env, etc.). Empty id is a no-op. It is the same context key the API edge fills for every authenticated request and the cluster forward helper sends to peers, so one call's records share one id on every node.

func GenerateSandboxID added in v0.2.1

func GenerateSandboxID() (string, error)

GenerateSandboxID is the exported entry point for the cluster handler's reservation-first create path: the router (Node A) needs to mint a sandbox ID before opReserve so the chosen target (Node T) can accept the forward with X-Cluster-Create-ID and run CreateSandboxWithID against the same reservation. Routes through the package-private generateSandboxID so the format stays in lockstep with the local create path.

func HoldSecretAuditQuerySlotsForTest added in v0.6.1

func HoldSecretAuditQuerySlotsForTest() (release func())

HoldSecretAuditQuerySlotsForTest occupies every local audit read slot so a caller can observe the fail-fast busy path. The returned func releases them.

func HoldSecretAuditVerifyForTest added in v0.6.1

func HoldSecretAuditVerifyForTest() (release func())

HoldSecretAuditVerifyForTest makes VerifySecretAuditChain report busy until the returned func runs.

func HostMediatedNodePorts added in v0.6.9

func HostMediatedNodePorts(cfg config.Config) []uint16

HostMediatedNodePorts are the node ports WASM and isolate sandboxes may still reach on the node's non-loopback addresses: ingress and the escape hatch. The gateway ports and DNS are for bridged sandboxes; a host process resolves names itself.

func ImageRequiresLocalPlacement added in v0.2.1

func ImageRequiresLocalPlacement(req models.CreateSandboxRequest) bool

ImageRequiresLocalPlacement reports a create that must run on the node receiving it, because its image exists only there. A pushed snapshot's committed image is the exception: a node without it starts the registry copy instead (see NormalizeCreateImageDistribution).

func IngressInstalledVersion added in v0.2.1

func IngressInstalledVersion() uint64

IngressInstalledVersion returns the highest placement.Version this node's ingress reconciler has finished installing routes for. The /v1/cluster/ placements/{id} convergence-status read compares this against the per- placement Version to answer "has this node bound the cluster-stable TCP route yet?" without reaching into Caddy state. Zero means the reconciler has not yet run a successful pass (fresh boot) — treat as "not converged" for any non-zero placement.Version.

func IsBareSnapshotName added in v0.6.12

func IsBareSnapshotName(image string) bool

IsBareSnapshotName reports an image that can only be a snapshot name a node without the snapshot's row resolves across the cluster: no registry host, no tag or digest, and not a local-only ref (a build or a committed snapshot tag, which the cluster create router routes by their own keys).

func KeyStoreOptions added in v0.6.9

func KeyStoreOptions(cfg config.Config, nodeID string, pkcs11 secrets.DataKeyWrapper) secrets.ProviderOptions

KeyStoreOptions is the provider options every key-store client in sandboxd is built from: the daemon at boot, the service, and `sandboxd key`.

func KeyStoreRef added in v0.6.9

func KeyStoreRef(cfg config.Config) string

KeyStoreRef names the key the provider wraps with, for the wrapped file and the custody report.

func NormalizeCreateEgressProfiles added in v0.6.6

func NormalizeCreateEgressProfiles(req *models.CreateSandboxRequest)

NormalizeCreateEgressProfiles pins each bare builtin:<name> in a create to this node's newest version, so the spec the cluster replicates carries the pinned form and a failover rebuilds the same list (CEO D11). The owner node calls it after any forward, so the pin is its own catalogue's. Other references are left for the create to resolve.

func NormalizeCreateFailover added in v0.2.2

func NormalizeCreateFailover(req *models.CreateSandboxRequest) error

func NormalizeCreateLifecycle added in v0.6.12

func NormalizeCreateLifecycle(req *models.CreateSandboxRequest, now time.Time) error

NormalizeCreateLifecycle turns a create's lifecycle.destroy_at (L6) into destroy_at_age, measured from now, the sandbox's creation. It runs where a create request enters: createSandbox, and the cluster create paths before they record the replicated spec, so the spec only ever carries the age. It works on a copy: the caller's request struct is not changed behind its back. A past time, or destroy_at with destroy_at_age, is a 400.

func OwnerRefForCreate added in v0.6.1

func OwnerRefForCreate(ctx context.Context) string

OwnerRefForCreate is the exported form of ownerRefForCreate for API layers that need to stamp PlacementSecrets.OwnerRef at reserve time.

func PublishKeyCustody added in v0.6.9

func PublishKeyCustody(cfg config.Config, facts CustodyFacts)

PublishKeyCustody publishes the custody report as the gauge.

func RecomputeChainHead added in v0.6.1

func RecomputeChainHead(path string) (head, eventID string, err error)

RecomputeChainHead scans secrets.jsonl and verifies every EventHash against HashEvent(PrevHash, ev) with PrevHash linkage. Does not trust secrets.tip. Returns the verified tip head and its event ID (genesis/"0" when empty).

func RecordCreateReservationState added in v0.2.1

func RecordCreateReservationState(state string)

func RecordExpiredReservations added in v0.2.1

func RecordExpiredReservations(count int)

func RecordFacadeIdempotencyConflict added in v0.2.1

func RecordFacadeIdempotencyConflict(scope string)

func RecordFacadeIdempotencyReplay added in v0.2.1

func RecordFacadeIdempotencyReplay(scope string)

func RecordPromoteRetract added in v0.5.6

func RecordPromoteRetract(result string)

RecordPromoteRetract increments aerolvm_cluster_promote_retract_total after an overlapped create/promote failure path finishes its sync retract. result is typically "ok", "delete_placement_failed", or "delete_secrets_failed".

func RecordRouteMiss added in v0.2.1

func RecordRouteMiss(reasons ...string)

RecordRouteMiss bumps the route-miss counter from the API layer. Exported so pkg/api/v1 can wire it in without exposing the expvar directly (keeps the metric name owned by this package). Service has no logical role here — this is a package-level counter the v1 wrap layer pokes when it observes the no-usable-URL case.

func RecordToolboxCall added in v0.6.12

func RecordToolboxCall(ctx context.Context, path string, status int)

RecordToolboxCall counts one toolbox call.

func RedactClusterSecrets added in v0.2.1

func RedactClusterSecrets(req models.CreateSandboxRequest) models.CreateSandboxRequest

RedactClusterSecrets returns a copy of req with credentials stripped — safe to replicate via raft. The Registry field's Server/Username are preserved (not secret) but Password is cleared; mount Credentials maps are dropped per-entry. Maps and slices that the caller might mutate are deep-copied so the original req is left untouched.

Env is always cleared from the Raft spec and rides in the provider bag instead (§5c / T9).

Mount Source and Options are kept on purpose. They are not secrets by contract — the read API returns them, and models.MountSpec.Validate refuses credential-shaped names in them at intake (an rclone connection-string parameter, an extra_args flag, an NFS option, an invented options key) — and the replicated spec must carry them to re-run the mount on the new owner. Everything the adapters treat as a credential rides in Credentials, which is sealed. Do not scrub Options here: a value removed from the spec but absent from the sealed bag would break the recreate it exists for.

Lives next to Put/Open because the two are always called as a pair: put returns the provider handle, redact returns the safe-to-replicate spec, and writing one without the other would either leak secrets (no redact) or lose them on failover (no put).

func SetIngressRouteLag added in v0.2.1

func SetIngressRouteLag(fsmVersion uint64)

SetIngressRouteLag is the post-tick hook the reconciler calls with the FSM's current PlacementVersion. Lag is computed as max(0, fsmVersion - ingressPlacementVersionMax). Computed here (not at recordIngressReconcile) so callers that only have the FSM version can still publish the lag without needing the reconciler's maxVersion.

func SnapshotNeedsPush added in v0.2.4

func SnapshotNeedsPush(snapshot *models.SandboxSnapshot) bool

SnapshotNeedsPush reports whether a snapshot row is a candidate for background push. Only local_only snapshots benefit — anything already in a remote registry is already distributable, and pushing it would just double-store the image.

func TemplateNeedsPush added in v0.4.1

func TemplateNeedsPush(tpl *models.Template) bool

TemplateNeedsPush reports whether a template row is a candidate for background AOCR push. Only `ready` rows qualify — `unhealthy` is the Phase 6 PR-A "snapshot was corrupt at load time" terminal state, and pushing a known-corrupt snapshot would just propagate the corruption to every peer. The reconciler enforces this defensively too.

func ToolboxCallEnded added in v0.6.9

func ToolboxCallEnded(ctx context.Context) error

ToolboxCallEnded reports why a request registered with TrackToolboxCall ended early: ErrSandboxStoppedMidRequest when its sandbox stopped, nil otherwise (it is still running, or the caller went away).

func ValidateClusterIsolateBundleRef added in v0.6.1

func ValidateClusterIsolateBundleRef(req models.CreateSandboxRequest) error

ValidateClusterIsolateBundleRef enforces the cluster-wide placement contract for isolate bundles. Uploaded bundles are worker-local, so cluster callers must use the node-bound reference returned by POST /v1/js-bundles. Keep this policy here so every API facade applies the same rule before placement.

func ValidateSnapshotRefArch added in v0.5.4

func ValidateSnapshotRefArch(ref, hostArch string) error

ValidateSnapshotRefArch rejects foreign-arch AOCR refs at resume/pull time. Defense-in-depth for homogeneous per-arch clusters (see plans/arm64-firecracker-hosts.md D5).

func WithIdempotencyKey added in v0.6.9

func WithIdempotencyKey(ctx context.Context, key string) context.Context

WithIdempotencyKey names the client's Idempotency-Key for the create on ctx, so a refusal can say which key was reused.

func WithToolboxSurface added in v0.6.12

func WithToolboxSurface(ctx context.Context, surface string) context.Context

WithToolboxSurface marks ctx with the API surface its toolbox calls come from. Each facade sets it once at its entry point.

func WithoutPortProbe added in v0.6.12

func WithoutPortProbe(ctx context.Context) context.Context

WithoutPortProbe marks an expose whose port cannot be listening yet, so exposePort skips its best-effort probe. E2B's mcp option exposes the gateway port at create time, but the E2B SDK starts the gateway only after the create returns: the probe could never succeed there, and a dropped SYN made it wait out portProbeTimeout (2 s) on every such create.

Types

type ACMEBudget added in v0.3.1

type ACMEBudget struct {
	// contains filtered or unexported fields
}

ACMEBudget is the daemon-wide brake on new-cert issuance for custom hostnames (plan §5b / OV5A). Let's Encrypt enforces a per-account `new-orders` limit (300 / 3h rolling). One misconfigured tenant adding 100 custom domains can drain that window for the whole cluster, so every TLSAsk hit that would trigger a new ACME order (status != ready) passes through Reserve; the hot path is the wildcard fast-fail and the negative cache, not this bucket.

Semantics:

  • Sliding window over the LE rolling limit (capacity attempts per window). At capacity*fraction (default 0.8 — 240/300) we refuse, returning a Retry-After equal to the time the oldest in-window attempt ages out. The 20% headroom buys the in-flight renewals (cert-magic kicks them off ~30 days before expiry) survival room.
  • Per-node bucket, NOT raft-synced. LE limits are per-account, not per-IP, and certmagic-s3's distributed lock already serialises simultaneous issuance for the same hostname across nodes; a node- local bucket is the right granularity for "this node is about to burn the account."
  • Observation, not metering: a single Reserve call corresponds to one TLSAsk that COULD trigger issuance. Caddy may rate-limit further before actually opening an order, and certmagic dedupes simultaneous orders for the same host. Treat the bucket as an upper bound on issuance rate, not the exact count.

func NewACMEBudget added in v0.3.1

func NewACMEBudget(cfg ACMEBudgetConfig) *ACMEBudget

NewACMEBudget builds a budget. A nil result is safe: budget == nil implies "no daemon-wide brake" (Reserve degrades to always-allow), so the TLSAsk hot path stays single-branch when the feature is off.

func (*ACMEBudget) InWindow added in v0.3.1

func (b *ACMEBudget) InWindow() int

InWindow returns the current in-window attempt count. Test-only — production callers should not gate on this number; use Reserve.

func (*ACMEBudget) Reserve added in v0.3.1

func (b *ACMEBudget) Reserve(sandboxID, hostname string) (bool, time.Duration)

Reserve records an attempt and returns (true, 0) if the new attempt keeps the bucket below threshold, or (false, retryAfter) if it would cross. sandboxID is logged on denial so the operator can spot the burner. retryAfter is the time until the oldest in-window attempt ages out — the earliest moment another attempt could succeed.

A nil *ACMEBudget always allows: makes the call site one-liner.

func (*ACMEBudget) Threshold added in v0.3.1

func (b *ACMEBudget) Threshold() int

Threshold returns the denial line. Stable for the life of the budget.

type ACMEBudgetConfig added in v0.3.1

type ACMEBudgetConfig struct {
	Capacity int
	Window   time.Duration
	Fraction float64
	Logger   *slog.Logger
}

ACMEBudgetConfig captures the operator-facing knobs.

type APIAuditPolicy added in v0.6.8

type APIAuditPolicy struct {
	// Enabled is false with SB_AUDIT_API_RECORDS=off, or while the cluster
	// audit gate is closed (an older member could not verify the records).
	Enabled bool
	// AllReads records plain GETs too (SB_AUDIT_API_RECORDS=all).
	AllReads bool
	// ExecArgv records the argv of exec calls (SB_AUDIT_EXEC_COMMAND=full).
	ExecArgv bool
	// Durable makes token create, token revoke and sandbox delete wait for
	// their record to reach disk (SB_AUDIT_API_DURABLE).
	Durable bool
}

APIAuditPolicy is what the API edge needs to decide which calls to record and how.

type APICall added in v0.6.8

type APICall struct {
	// Method and Route are the HTTP method and the matched route pattern
	// without its method ("/v1/sandboxes/{id}/exec").
	Method string
	Route  string
	Status int
	// SandboxID is the sandbox the call acted on; "" for none.
	SandboxID string
	// IncarnationID and OwnerRef are the sandbox's lifecycle and owner when
	// the edge resolved them before the handler ran (a delete forgets them
	// once it succeeds). Empty means resolve now.
	IncarnationID string
	OwnerRef      string
	// Target, Digest, Argv and Surface fill the record's request (see
	// auditlog.Request).
	Target  string
	Digest  string
	Argv    []string
	Surface string
	// Grant is the role and team a token create handed out.
	Grant *auditlog.Grant
	// Denied is a refused call (403, or a 400/403 from authentication).
	Denied bool
	// Durable waits for the record to reach disk (SB_AUDIT_API_DURABLE).
	Durable bool
}

APICall is one call the API edge asks the service to record.

type AutoImportConfig added in v0.2.3

type AutoImportConfig struct {
	// Enabled gates the whole feature. When false the post-pull path
	// short-circuits and the auto-importer is never built.
	Enabled bool
	// HooksBaseURL is the root URL of the AOCR hooks service (no trailing
	// slash, e.g. `https://aocr.aerol.ai`). The importer appends
	// `/v1/internal/imports`.
	HooksBaseURL string
	// ClusterID identifies this sandboxd cluster to AOCR. Goes into the
	// import request body and constrains the cluster PAT's scope.
	ClusterID string
	// ClusterPAT is the bearer token presented to the ImportAPI. Treat as
	// secret; do not log. Sourced from a file path at startup, not env.
	ClusterPAT string
	// RetentionSuffix is appended to the target tag in the cluster
	// namespace (e.g. `--idle-90d`). Empty means use the ImportAPI's
	// server-side default (`--idle-90d`).
	RetentionSuffix string
	// RequestTimeout caps a single ImportAPI call. The mount-from-repo
	// flow is normally sub-second (no byte transfer) but the timeout
	// guards against an upstream that hangs on layer enumeration.
	RequestTimeout time.Duration
}

AutoImportConfig wires the post-pull auto-import path to a specific AOCR hooks deployment. A zero value disables auto-import; the rest of the service runs exactly as before. main() is expected to validate that the PAT + cluster ID are set when Enabled=true.

See `plans/cluster-mirror-and-snapshot-distribution.md` F21 for the full motivation. Short version: the first time a private upstream image is pulled on a sandbox with `failover.policy: recreate`, sandboxd asks AOCR to re-mount the just-cached bytes under `cluster/<id>/_imported/<host>/<repo>`. Future recreates pull from the cluster namespace with a cluster PAT — surviving an upstream credential rotation that would otherwise break the F17 wrap-creds path.

func (AutoImportConfig) Validate added in v0.2.3

func (c AutoImportConfig) Validate() error

Validate enforces the consistency rules the rest of the service relies on. Returns nil for a disabled config so callers can validate unconditionally.

type AutoImportPendingStore added in v0.2.3

type AutoImportPendingStore interface {
	ListAutoImportPendingIDs(ctx context.Context) ([]string, error)
	Get(ctx context.Context, id string) (*models.Sandbox, error)
	SetAutoImportPending(ctx context.Context, id string, pending bool) error
}

AutoImportPendingStore is the narrow store surface the reconciler needs. Pulled out as an interface so the reconciler can be tested without instantiating a full SQLite store — and so future store backends can satisfy it without growing the reconciler's dependency surface.

type AutoImportReconcileStats added in v0.2.3

type AutoImportReconcileStats struct {
	Scanned   int
	Succeeded int
	Failed    int
	Skipped   int
}

AutoImportReconcileStats is what RunOnce reports back. Used by tests and by the metrics-collection wrapper in main.go.

type AutoImportReconciler added in v0.2.3

type AutoImportReconciler struct {
	// contains filtered or unexported fields
}

AutoImportReconciler walks the rows the post-pull path flagged `auto_import_pending = true` and re-tries them. One ticker per node; fan-out inside a single tick is capped to keep a recovery storm from saturating the local Docker daemon or AOCR.

The reconciler does not own a goroutine — `RunOnce` is the unit of work, and a thin wrapper in main.go schedules it on a ticker. That keeps the type fully testable (table-driven, no time.Tick) and lets the operator trigger an immediate retry from a debug endpoint if they want.

func NewAutoImportReconciler added in v0.2.3

func NewAutoImportReconciler(importer *AutoImporter, store AutoImportPendingStore, specs AutoImportSpecResolver, logger *slog.Logger, maxInFlight int) *AutoImportReconciler

NewAutoImportReconciler builds the reconciler. Returns nil if the importer is nil (auto-import disabled) so callers can `if r == nil { return }` cheaply. maxInFlight clamps fan-out per tick; pass 0 for the safe default (4).

func (*AutoImportReconciler) RunOnce added in v0.2.3

RunOnce processes one sweep of the auto-import pending queue. Blocks until all per-sandbox attempts have returned (success, failure, or context cancel). The bounded semaphore caps in-flight HTTP calls; the AOCR side is happy with the load but the local Docker network stack is not infinitely parallel.

func (*AutoImportReconciler) SetSpecMutator added in v0.2.3

func (r *AutoImportReconciler) SetSpecMutator(m AutoImportSpecMutator)

SetSpecMutator installs the optional write-back hook. Calling again replaces the previous one; pass nil to disable. Separated from the constructor so existing call sites (and tests) keep working unchanged — the mutator is a strict upgrade, not a required dependency.

type AutoImportRequest added in v0.2.3

type AutoImportRequest struct {
	UpstreamHost   string
	UpstreamRepo   string
	UpstreamTag    string
	UpstreamDigest string
}

AutoImportRequest is the per-sandbox import payload. Fields mirror what AOCR `ImportAPI` (hooks/src/controllers/ImportAPI.ts) expects on the wire; the JSON marshaller below pins the contract.

type AutoImportResult added in v0.2.3

type AutoImportResult struct {
	RegistryRef    string
	AlreadyPresent bool
}

AutoImportResult is what comes back. RegistryRef is the *cluster-side* pull reference the caller should write onto the sandbox spec (replacing the user's upstream ref for future recreates). AlreadyPresent=true means the destination tag already existed at the right digest — the importer treats that as success but the caller may want to suppress the spec mutation to avoid an unnecessary Raft write.

type AutoImportSpecMutator added in v0.2.3

type AutoImportSpecMutator interface {
	MarkImported(ctx context.Context, sandboxID string, registryRef string)
}

AutoImportSpecMutator is the optional write-back hook. After a successful import, the reconciler calls MarkImported so the replicated spec carries the new cluster registry ref and `aocr_imported` distribution mode. Future failovers then pull from the cluster namespace with the cluster PAT, fully decoupled from the original upstream credential.

Implementations must be safe to call on a non-leader node: the leader- forwarding lives below this surface (cluster.UpsertSpec proposes through the Raft FSM regardless of caller node). Errors are absorbed inside the implementation — the reconciler always treats import success as a clean retryOutcome even if the mutator can't reach the leader, because the imported bytes are already in AOCR.

A nil mutator disables write-through cleanly; the reconciler logs the new ref and moves on. This is the standalone-mode default.

type AutoImportSpecResolver added in v0.2.3

type AutoImportSpecResolver interface {
	GetSandboxSpec(sandboxID string) (*models.CreateSandboxRequest, bool)
}

AutoImportSpecResolver lets the reconciler look up the replicated CreateSandboxRequest that owns a given sandbox. The reconciler needs this because the spec carries the original upstream ref + the failover policy (the gate for whether import should happen at all), but the sandbox row only carries the runtime view. In standalone mode this is backed by an in-memory map; in cluster mode by the Raft FSM.

Returns nil, false when no replicated spec exists — the reconciler treats that as "drop the flag, nothing to retry" rather than as an error, since a sandbox without a spec cannot meaningfully be recreated elsewhere anyway.

type AutoImporter added in v0.2.3

type AutoImporter struct {
	// contains filtered or unexported fields
}

AutoImporter is the leaf dependency that actually talks to AOCR. It holds no per-sandbox state; instantiate once at boot and reuse. Safe for concurrent use (it's just an http.Client wrapper).

func NewAutoImporter added in v0.2.3

func NewAutoImporter(cfg AutoImportConfig) (*AutoImporter, error)

NewAutoImporter builds an importer. Returns nil if cfg.Enabled is false so callers can do `if importer == nil { return }` on the post-pull path without an extra config check.

func (*AutoImporter) Endpoint added in v0.2.3

func (a *AutoImporter) Endpoint() string

Endpoint is exposed for log/debug use. Returns "" for a nil importer so callers can log it unconditionally.

func (*AutoImporter) Import added in v0.2.3

Import calls the AOCR ImportAPI once. Idempotent on (upstream_digest, cluster_id, target_tag): re-posting an already-imported sandbox returns AlreadyPresent=true without an error. Network/HTTP failures return a wrapped error so the caller can decide whether to log+flag or surface.

type BuildImageRequest added in v0.6.7

type BuildImageRequest struct {
	Dockerfile    string
	ContextHashes []string
	// BaseShortcut lets a Dockerfile that is only `FROM <image>` resolve to
	// that image without building: the engine pulls it like any image.
	BaseShortcut bool
	// Push, when set, copies the built image to a registry.
	Push *ImagePush
}

BuildImageRequest is one Dockerfile build.

type BuiltImage added in v0.6.7

type BuiltImage struct {
	// Image is the image to start: the built tag, or the base image under
	// BaseShortcut.
	Image string
	// Fresh reports that this call built Image (not the cache, not the
	// shortcut): the tag to hand to DiscardBuiltImage if the next step
	// fails.
	Fresh bool
	// Pushed is the registry ref the build was pushed to, if any.
	Pushed string
}

BuiltImage is a build's result.

type CustodyFacts added in v0.6.9

type CustodyFacts struct {
	SoftwareToken bool
}

CustodyFacts are what config cannot say: only the running PKCS#11 token knows whether its slot is hardware.

type CustodyRow added in v0.6.9

type CustodyRow struct {
	Class   string
	Custody string
	// InScope is false for keys the plan does not move to the key store;
	// they are listed so the report never overstates (§9).
	InScope bool
}

CustodyRow is one key class and where it lives.

func CustodyReport added in v0.6.9

func CustodyReport(cfg config.Config, facts CustodyFacts) []CustodyRow

CustodyReport is where each key class lives on this node, from config and facts: no key-store call, no database (§9).

type CustomDomainCacheEvicter added in v0.3.1

type CustomDomainCacheEvicter interface {
	EvictNegativeCache(hostname string)
}

CustomDomainCacheEvicter is the narrow surface the service needs from the ingressproxy TLS-ask handler so that AddCustomDomain can punch a hole in the negative cache immediately when a new hostname is attached. Kept as an interface (rather than importing pkg/api/ingressproxy directly) because ingressproxy already depends on internal/service — pulling the dependency the other way would create an import cycle. cmd/sandboxd wires the real handler in via AttachCustomDomainCacheEvicter once both packages are built.

type DNSResolver added in v0.4.1

type DNSResolver interface {
	LookupTXT(ctx context.Context, name string) ([]string, error)
}

DNSResolver is an interface to allow mocking of net.Resolver in tests.

type DefaultDNSResolver added in v0.4.1

type DefaultDNSResolver struct{}

DefaultDNSResolver is the default implementation using net.DefaultResolver.

func (*DefaultDNSResolver) LookupTXT added in v0.4.1

func (d *DefaultDNSResolver) LookupTXT(ctx context.Context, name string) ([]string, error)

LookupTXT implements DNSResolver using the system's default resolver.

type DurableSecretAuditSink added in v0.6.1

type DurableSecretAuditSink interface {
	SecretAuditSink
	EmitDurable(SecretAuditEvent) error
}

DurableSecretAuditSink accepts an event only after its bytes have reached durable storage. Security-boundary ingest endpoints use this interface so a 202 response never means merely "queued in process memory".

type ExecOutcome added in v0.6.7

type ExecOutcome struct {
	Exit      int   `json:"exit"`
	StdoutLen int64 `json:"stdout_len"`
	StderrLen int64 `json:"stderr_len"`
	Lost      bool  `json:"lost,omitempty"`
}

ExecOutcome is what the ledger keeps of a settled plain execution: the exit status and how many bytes each stream produced, so a replay after the output itself is gone still reports real offsets and an honest truncation instead of an empty, successful-looking result. Lost marks an execution whose session ended before any result was seen (its sandbox restarted, or its submission failed): what the command did is unknown, so it is reported as gone, never as an exit status.

func DecodeExecOutcome added in v0.6.7

func DecodeExecOutcome(target string) (outcome ExecOutcome, ok bool)

DecodeExecOutcome reads a plain execution's target_id. ok is false for a row with no outcome yet, or one too corrupt to trust.

func (ExecOutcome) Encode added in v0.6.7

func (o ExecOutcome) Encode() string

Encode renders the outcome for storage in a row's target_id.

type GetSandboxOptions added in v0.6.1

type GetSandboxOptions struct {
	IncludeEnv bool
	// CorrelationID is attached to the audited include_env load when set
	// (typically from X-Correlation-ID / X-Request-ID).
	CorrelationID string
}

GetSandboxOptions controls public Get/List env materialization (D9).

type HostPortForwarder added in v0.6.1

type HostPortForwarder interface {
	Ensure(hostPort int, t hostport.Target) error
	Remove(hostPort int) error
	PruneUnasserted() error
	Reconcile(desired map[int]hostport.Target) error
}

HostPortForwarder is the kernel forwarding the index writer drives for raw TCP host ports (*hostport.Forwarder on Linux).

type IPv6Fence added in v0.6.9

type IPv6Fence interface {
	Ready(runtime string) error
}

IPv6Fence is what the create gate asks: nil when the node's IPv6 fence covers a runtime's sandboxes (ipv6fence.Supervisor). Container, gVisor and Firecracker sandboxes sit on a host bridge or TAP, and every host rule between them is IPv4, so without the fence a sandbox that gets IPv6 up reaches its neighbours and the node over link-local addresses.

type ImageBuilder added in v0.6.7

type ImageBuilder interface {
	BuildImage(ctx context.Context, req docker.BuildImageRequest) error
	ImageExists(ctx context.Context, imageRef string) (bool, error)
	PushImage(ctx context.Context, req docker.PushImageRequest) (string, error)
	// RefreshTag bumps a tag's last-tag time, so the built-image janitor
	// does not remove a tag a build just handed out from its cache.
	RefreshTag(ctx context.Context, fullRef string) error
	RemoveImage(ctx context.Context, imageRef string) error
}

ImageBuilder compiles Dockerfiles into local images: the Docker engine client, or the containerd engine's BuildKit builder. Every API surface builds through Service.BuildImage, never through the builder directly, so a build has one owner rule, one tag scheme and one set of errors.

type ImageDistributionProvider added in v0.2.1

type ImageDistributionProvider interface {
	ClassifyImage(ctx context.Context, image string) (models.ImageDistributionMetadata, error)
}

ImageDistributionProvider is the control-plane contract for image availability. Providers may verify digests or talk to an external registry cache, but the core daemon only needs the resulting placement metadata.

type ImagePush added in v0.6.7

type ImagePush struct {
	Registry string
	Tag      string
	Auth     models.RegistryAuth
}

ImagePush is where a build is pushed. The credential lives only for the request: it goes to the engine and is never stored.

type IngressRouting added in v0.6.1

type IngressRouting struct {
	Owner       *routedns.OwnerTable
	Ingress     *routedns.IngressIndex // nil unless this node is a cluster ingress
	SelfNodeID  string
	ClusterMode bool
	// contains filtered or unexported fields
}

IngressRouting is an engaged routing mode: the tables the host router reads, plus what Commit needs.

func (*IngressRouting) Stop added in v0.6.1

func (r *IngressRouting) Stop()

Stop ends the responder, the index watcher and the redirect listener.

type IngressRoutingOptions added in v0.6.1

type IngressRoutingOptions struct {
	Static caddy.StaticRouteSpec
	// Forwarder is required: under the flag no Caddy tcp-port servers
	// remain, so without kernel forwarding raw TCP exposures would go dark.
	Forwarder HostPortForwarder
	// RedirectAddr is the listener that REDIRECTed host ports land on (wake
	// and WASM/isolate mediator targets), on workers. It must listen on
	// every address: REDIRECT rewrites the destination to the receiving
	// interface's address.
	RedirectAddr string
	// ProbeTimeout bounds the responder bind, each resolver probe, and the
	// wait for the first ingress index snapshot. Zero means 60s. Boot is
	// noisy: bootstrap restarts sandboxd back to back, so the previous
	// process may still hold the port, and install.sh restarts
	// systemd-resolved. A 5s budget refused the flag on every node of a
	// live cluster.
	ProbeTimeout time.Duration
	// SystemResolver is the resolver caddy-l4 dials through (nil: the
	// system resolver). Tests inject one.
	SystemResolver *net.Resolver
}

IngressRoutingOptions wires the routing mode.

type OwnerLeaseVerdict added in v0.6.12

type OwnerLeaseVerdict int

OwnerLeaseVerdict is what the owner's last authoritative read says about a forward that carries an owner epoch (cluster.HeaderOwnerEpoch).

const (
	// OwnerLeaseUnknown: the read cannot decide (no read, too old, no
	// entry, an older epoch than the forward's, the sandbox is fenced, the
	// flag is off). The router looks the sandbox up, as without E3b.
	OwnerLeaseUnknown OwnerLeaseVerdict = iota
	// OwnerLeaseOwned: the read proves this node owns the sandbox at the
	// forward's epoch. Two authoritative readings agree, so the router
	// serves without a lookup.
	OwnerLeaseOwned
	// OwnerLeaseNewer: the read holds a newer epoch than the forward's: the
	// edge's view predates an owner change this node already saw.
	OwnerLeaseNewer
)

type PortAccess added in v0.6.12

type PortAccess struct {
	// Started is IsSandboxStarted's answer: a warm sandbox skips the
	// cold-start body buffering and admission.
	Started bool
	// TokenGated means the port was exposed with visibility "token": the
	// request needs a sandbox token for this port.
	TokenGated bool
	// TokensNotBefore is the sandbox's tokens_not_before: a port token
	// issued before it was revoked (sandboxtoken.Claims.CheckNotBefore).
	TokensNotBefore int64
}

PortAccess is what the ingress proxy needs to know about a sandbox port before it does anything else with a request.

type PortEndpoint added in v0.2.4

type PortEndpoint struct {
	URL string
	// MaskRequestHost is the value the loopback ingress proxy must rewrite the
	// upstream Host header to before re-dialing the container. Empty = keep the
	// proxy's default (the upstream URL host). This is the serverless wake-path
	// enforcement point for E2B network.maskRequestHost; the direct Caddy route
	// enforces the same value via caddy.HTTPRouteOptions. Both read the one
	// sandbox row, so they can't disagree.
	MaskRequestHost string
}

PortEndpoint is the upstream the ingress proxy dials for a wake-aware exposed-port request. URL has the form http://{containerIP}:{port}.

type RouteKind added in v0.2.4

type RouteKind int

RouteKind is the protocol surface a callsite is choosing a shape for. HTTP and L4 (TCP/TLS) consult different bypass flags so each can be rolled out independently — Phase 1 ships HTTP behind SB_HTTP_WAKE_DIRECT_BYPASS_ENABLED; Phase 2 ships L4 behind SB_L4_WAKE_DIRECT_BYPASS_ENABLED. Within each kind the decision tree is identical (Status + WakeArmed + ContainerIP) — only the "is bypass enabled?" predicate differs.

const (
	RouteKindHTTP RouteKind = iota
	RouteKindL4
)

type RouteShape added in v0.2.4

type RouteShape int

RouteShape is the published Caddy shape for a (sandbox, exposed-port). One shape is live at a time per exposure; transitions are driven by status changes (Start/Stop/Die/Reconcile), never by per-request work.

See plans/warm-direct-route-bypass.md D7/D8 — this enum + chooseRouteShape are the single source of truth so HTTP, TCP, TLS, reconcile, and event callsites cannot disagree about what shape should be live for a given sandbox row.

const (
	// RouteShapeNone means no route should be published. Destroyed
	// sandboxes and disarmed-stopped serverless sandboxes fall through
	// to Caddy's fallback (404).
	RouteShapeNone RouteShape = iota
	// RouteShapeDirect publishes the upstream as ContainerIP:port.
	// Used for non-serverless sandboxes always, and for serverless
	// sandboxes only while warm (HTTPWakeDirectBypassEnabled=true).
	RouteShapeDirect
	// RouteShapeWake publishes the upstream as the sandboxd loopback
	// ingress proxy (or per-exposure unix socket for TLS). Used for
	// serverless sandboxes while Stopped+armed, and for serverless
	// sandboxes always when HTTPWakeDirectBypassEnabled=false.
	RouteShapeWake
)

func (RouteShape) String added in v0.2.4

func (r RouteShape) String() string

type SecretAuditCoverage added in v0.6.1

type SecretAuditCoverage struct {
	Answered []string `json:"answered"`
	Missing  []string `json:"missing"`
	Partial  bool     `json:"partial"`
}

SecretAuditCoverage reports which members answered a fan-out read. Partial is true when Missing is non-empty — never silently truncate history.

type SecretAuditEvent added in v0.6.1

type SecretAuditEvent = auditlog.Event

SecretAuditEvent is one audit record (secret-open by default, or host-mediated egress). Never carry plaintext, credentials, PII, or wrapped error strings (those may embed caller input). Destination must never include credentials.

type SecretAuditPage added in v0.6.1

type SecretAuditPage struct {
	Events     []SecretAuditEvent  `json:"events"`
	Coverage   SecretAuditCoverage `json:"coverage"`
	NextCursor string              `json:"next_cursor,omitempty"`
}

SecretAuditPage is the GET /v1/sandboxes/{id}/audit response body.

type SecretAuditQuery added in v0.6.1

type SecretAuditQuery struct {
	Limit  int    // default 100, max 1000
	Cursor string // exclusive lower bound time (RFC3339Nano), or empty = from start
	// Kind filters by event kind ("egress", "secret_open"). Empty = all kinds.
	Kind string
	// Caller filters by the record's caller: its id ("tok_7Hq2…",
	// "operator", "system:lifecycle") or its display name, ignoring case.
	// Empty = every caller. Gap markers always pass, as for Kind.
	Caller string
	// IncarnationID scopes history to one placement lifetime. When empty and
	// the sandbox still has a live placement, ListSecretAuditLocal defaults
	// it from Placement.IncarnationID.
	IncarnationID string
}

SecretAuditQuery pages local/cluster secret-audit history for one sandbox.

type SecretAuditSink added in v0.6.1

type SecretAuditSink interface {
	Emit(SecretAuditEvent)
}

SecretAuditSink receives secret-read events. Emit must be non-blocking. A nil sink is a no-op and must not panic.

type SecretAuditVerification added in v0.6.1

type SecretAuditVerification struct {
	OK         bool   `json:"ok"`
	Generation string `json:"generation,omitempty"`
	Head       string `json:"head"`
	EventID    string `json:"event_id,omitempty"`
	Records    int64  `json:"records"`
	// Redacted counts retention stubs among the records: expired records
	// whose payload retention removed in place because a newer record stood
	// in front of them. The node never produces one inside the retention
	// window, so a stub younger than SB_SECRET_AUDIT_RETENTION_DAYS is
	// evidence of a hand-edited file.
	Redacted int64 `json:"redacted"`
	Bytes    int64 `json:"bytes"`
	// WriterTipMatches reports whether the verified head equals the tip the
	// writer held at the snapshot, i.e. the file and the process agree.
	WriterTipMatches bool   `json:"writer_tip_matches"`
	IndexReady       bool   `json:"index_ready"`
	IndexLagBytes    int64  `json:"index_lag_bytes"`
	DurationMS       int64  `json:"duration_ms"`
	Error            string `json:"error,omitempty"`
}

SecretAuditVerification is the report of one on-demand full-chain verification of the local audit log.

type SecretHolders added in v0.6.1

type SecretHolders struct {
	SandboxID     string `json:"sandbox_id"`
	IncarnationID string `json:"incarnation_id"`
	// Ref is the sealed row's address. Useful when correlating with audit
	// records, which reference secrets by ref rather than by sandbox.
	Ref string `json:"ref"`
	// Holders is the recipient set: the nodes that should hold a sealed copy.
	// Sorted so a caller can compare two reads without normalising.
	Holders []string `json:"holders"`
	// SealGeneration fences a reseal. Two reads with the same holders but
	// different generations mean a reseal happened in between, which is the
	// difference between "nothing changed" and "everything was rewritten".
	SealGeneration int64 `json:"seal_generation"`
	Version        int   `json:"version"`
	// PendingPut is the fan-out obligation: peers that have not yet
	// acknowledged a sealed copy. PendingDelete is the teardown obligation:
	// peers whose copy has not yet been removed. Without these a caller cannot
	// tell a converged fan-out from one still in flight, nor a completed
	// delete from one whose tombstones are still outstanding.
	PendingPut    []string `json:"pending_put,omitempty"`
	PendingDelete []string `json:"pending_delete,omitempty"`
}

SecretHolders is the operator-visible view of which nodes hold a sandbox's sealed cluster secret.

It exists because every /v1/cluster/internal/* route is mTLS-gated and there is no list verb on the peer secret path, so an operator — or the integration suite, which authenticates with a PAT and has no client certificate — had no way to observe the recipient set at all. Fan-out, failover and reseal are all defined in terms of that set, so it could not be asserted from outside the cluster's own peer protocol.

It deliberately carries NO ciphertext and no plaintext: only who holds a copy, at which generation, and what is still owed. Adding the payload here would turn an observability read into a second exfiltration path for the thing the whole subsystem exists to protect.

type Service

type Service struct {
	// contains filtered or unexported fields
}

func New

func New(cfg config.Config, logger *slog.Logger, db *store.Store, runtimeDriver runtime.Runtime, eventsClient docker.EventsSource, caddyClient *caddy.Client, cipher *secrets.Cipher, mountManager *mounts.Manager, admitter *capacity.Admitter) *Service

func (*Service) APIAuditPolicy added in v0.6.8

func (s *Service) APIAuditPolicy() APIAuditPolicy

APIAuditPolicy reports the api-record policy right now. The gate part can change at runtime (the last old member leaves), so the edge asks per call.

func (*Service) APIRecordsConfigured added in v0.6.8

func (s *Service) APIRecordsConfigured() bool

APIRecordsConfigured reports whether api records can ever be written on this node (SB_AUDIT_API_RECORDS is not off). The edge installs its hook only then, so "off" costs nothing per request.

func (*Service) APITokenLookup added in v0.6.8

func (s *Service) APITokenLookup() identity.TokenLookup

APITokenLookup is the token index the apiToken authenticator reads: the single-node set, this server's FSM, or this node's mirror. A cluster node with neither is never fresh, so every token is refused with 503 rather than checked against a node-local list.

func (*Service) AddCustomDomain added in v0.3.1

func (s *Service) AddCustomDomain(ctx context.Context, sandboxID, hostname string, targetPort int) error

AddCustomDomain attaches a new public hostname to an existing sandbox. The caller is responsible for sandbox-scoped authorization (the API layer); this helper validates the hostname against the deployment's base domain, enforces the IRON RULE (no coexistence with protocol=tcp/tls exposures), persists the row, re-PATCHes the Caddy route to include the new hostname, and rolls back the store insert on Caddy failure. On success, the negative cache entry for the hostname is punched so the next ACME ask hits the resolver instead of being refused for the cached 60s.

targetPort selects the in-container port the hostname routes to. Zero is the sentinel meaning "use the toolbox port" (the pre-target-port default, preserving existing behavior). Set once at attach time; changing the port requires detach + re-add so in-flight traffic can't silently redirect.

Returns:

  • ErrCustomDomainNotSupported (412) when the deployment doesn't support custom domains.
  • ErrCustomDomainInvalid (400) when the hostname fails validation.
  • ErrCustomDomainInvalidTargetPort (400) when targetPort is outside [0, 65535].
  • ErrCustomDomainProtocolConflict (409) when the sandbox already exposes tcp/tls ports.
  • ErrCustomDomainPerSandboxCap (409) when the sandbox already holds the cap.
  • ErrCustomDomainPortMismatch (409) when re-adding an existing hostname with a different port.
  • store.ErrCustomDomainConflict (409) when the hostname is owned by a different sandbox.
  • store.ErrNotFound (404) when the sandbox does not exist.

func (*Service) ApplyEgressFloor added in v0.6.9

func (s *Service) ApplyEgressFloor(ctx context.Context)

ApplyEgressFloor installs the deny floor with no operator file: the cloud metadata endpoints alone. The daemon calls it at boot when SB_EGRESS_OPERATOR_FILE is unset, so the node's instance credentials are out of every sandbox's reach on every node, not only where an operator file is configured.

func (*Service) ApplyLifecyclePatch added in v0.6.7

func (s *Service) ApplyLifecyclePatch(ctx context.Context, id string, patch func(*models.Sandbox) (models.Lifecycle, bool, error)) (*models.Sandbox, bool, error)

ApplyLifecyclePatch atomically reads the sandbox under the per-sandbox lifecycle lock, lets patch compute the desired Lifecycle from the current state, and writes it back. patch returns the desired Lifecycle and whether anything changed — returning (_, false, nil) leaves the sandbox untouched (the E2B reconnect rule that never shortens a longer timeout). This replaces the facades' own read→mutate→UpdateLifecycle, whose window raced: two field patches on the same sandbox could each read the same base and the later write would drop the earlier field (D7).

func (*Service) ApplyNodeGuard added in v0.6.9

func (s *Service) ApplyNodeGuard(ctx context.Context)

ApplyNodeGuard installs the guard on every runtime. The daemon calls it at boot, before the API serves, so no sandbox is created before it.

func (*Service) AttachCluster added in v0.2.1

func (s *Service) AttachCluster(c cluster.Client)

AttachCluster swaps in a cluster.Client. Called from cmd/sandboxd/main after service.New when SB_ENABLE_CLUSTER=true. Idempotent.

func (*Service) AttachCustomDomainCacheEvicter added in v0.3.1

func (s *Service) AttachCustomDomainCacheEvicter(e CustomDomainCacheEvicter)

AttachCustomDomainCacheEvicter is the wire-up hook called from cmd/sandboxd once the TLSAskHandler is built. Passing nil is allowed (and the default) — negative-cache eviction is best-effort; the cache entry expires on its own TTL and the worst case is a 60s delay before the first ACME ask sees the freshly-added hostname.

func (*Service) AttachSnapshotPusher added in v0.2.4

func (s *Service) AttachSnapshotPusher(pusher *SnapshotPusher, reconciler *SnapshotPushReconciler)

AttachSnapshotPusher wires in the optional AOCR snapshot-push pipeline. pusher must be non-nil to activate the feature; a nil pusher is a no-op and leaves the service in legacy local-only snapshot mode. reconciler may be nil — when nil, kick-after-create is disabled (useful in tests that assert on the initial persisted state before any reconcile runs). Called once from main() after cfg.SnapshotPushEnabled validation.

func (*Service) AttachTemplateArtifactPuller added in v0.4.1

func (s *Service) AttachTemplateArtifactPuller(p *TemplateArtifactPuller)

AttachTemplateArtifactPuller wires in the consumer-side puller. nil puller leaves the feature off and EnsureTemplateLocal becomes a fast no-op — the right behaviour for single-node deployments and for nodes where SnapshotPushEnabled=false (no AOCR configured). Called once from main() after the docker client is up.

func (*Service) AttachTemplateArtifactPusher added in v0.4.1

func (s *Service) AttachTemplateArtifactPusher(pusher *TemplateArtifactPusher, reconciler *TemplateArtifactPushReconciler)

AttachTemplateArtifactPusher wires in the optional AOCR template push pipeline (Phase 6 PR 6-B.1). pusher must be non-nil to activate the feature; a nil pusher leaves new template rows in push_state='active' forever (the reconciler only picks 'pending'/'error') — backward- compatible with single-node deployments and with cfg.EnableCluster=false. reconciler may be nil for tests that assert on initial persisted state. Called once from main() after cfg.SnapshotPushEnabled validation.

func (*Service) AttachWasmCheckpointPusher added in v0.5.3

func (s *Service) AttachWasmCheckpointPusher(pusher WasmCheckpointStore)

AttachWasmCheckpointPusher wires AOCR push for durable WASM checkpoints. Called from main when SnapshotPushEnabled is true.

func (*Service) AuditExportHealthy added in v0.6.1

func (s *Service) AuditExportHealthy(ctx context.Context) error

AuditExportHealthy reports the configured connector's last observed outcome; nil when none is configured.

func (*Service) AuditIdentityForSandbox added in v0.6.8

func (s *Service) AuditIdentityForSandbox(sandboxID string) (incarnationID, ownerRef string)

AuditIdentityForSandbox is the lifecycle id and owner a record about sandboxID is stamped with. The edge resolves it before a call that may end the lifecycle (a delete), so the record still finds the sandbox's history afterwards. A cache hit after the sandbox's first record.

func (*Service) AuthorizeSandboxAuditAccess added in v0.6.1

func (s *Service) AuthorizeSandboxAuditAccess(ctx context.Context, sandboxID, incarnationID string) (string, error)

AuthorizeSandboxAuditAccess checks tenant/operator scope for GET …/audit without requiring a local sandbox row. Ingress and non-owner workers resolve OwnerRef from Placement.OwnerRef / the placement owner (secure peer meta) so fan-out can stay on the entry node (plans/secrets-hardening E1b + P1 ingress).

After sandbox deletion, retained history stays readable via sandbox_audit_acl (and the Raft-replicated Placement.OwnerRef while placement remains). Missing placement ownership metadata fails closed; no ID-only peer lookup can cross an ID-reuse boundary. ACL rows are incarnation- scoped; authorize requires an incarnation match (from query, else live placement). There is no legacy any-incarnation fallback.

func (*Service) BeginSandboxCreate added in v0.6.10

func (s *Service) BeginSandboxCreate(ctx context.Context, id string, digests []string) (context.Context, func(), error)

BeginSandboxCreate takes id's create lease for a caller that runs steps of its own around the create (the cluster overlap's seal and promote, a facade's state row), so a second create of id waits for all of them, not only for CreateSandboxWithID inside. Within one create, the outer take holds the lease and CreateSandboxWithID's inner take joins it.

It refuses a keyed create whose request differs from the one running (ErrIdempotencyKeyReused) or from the one id's sandbox was made with, and returns ErrCreateInProgress or ctx's error when the wait gives up. A create without digests (no key, or no server key) is not checked. The returned ctx is the one to create with; release gives the lease back and must run after every step that can roll the create back.

func (*Service) BuildImage added in v0.6.7

func (s *Service) BuildImage(ctx context.Context, req BuildImageRequest) (BuiltImage, error)

BuildImage builds a Dockerfile into a local image tagged for its caller.

The tag is content-addressed (a retry, or the same Dockerfile again, is served from the cache), bound to this node in a cluster (the create that uses it is routed back here), and, for a user token, under the caller's owner namespace: one tenant's build is never another's tag, so neither starting it by name nor the build cache crosses tenants.

A user token's build resolves every image it names from its registry (docker.BuildImageRequest.ForcePull), so a Dockerfile — FROM, COPY --from, a base image's ONBUILD triggers, a short image id — can never read another tenant's snapshot or build off this node. A Dockerfile that names such an image outright is refused up front with a clear error rather than failing a pull. Operators build as before.

func (*Service) Capacity

func (s *Service) Capacity() capacity.Snapshot

Capacity returns the admitter's current snapshot. Returns the zero value when no admitter is configured (e.g. in tests).

Phase 6 PR-D: overlays LocalTemplateIDs from a 5s-TTL cache so peers fetching /v1/capacity see which Firecracker templates this node can already serve without paying a remote AOCR pull. The cache shields SQLite from the gossip cadence (single-writer; MaxOpenConns=1).

func (*Service) CheckCreatePolicy added in v0.6.12

func (s *Service) CheckCreatePolicy(req models.CreateSandboxRequest) error

CheckCreatePolicy holds a create to the operator's create policy before a cluster places it. In a cluster the router picks a node first, and a create too big for every node was answered 503 "no placement target" before the owner's own check could name the rule it broke (UC-410 live, 2026-11-06). It judges a copy with the same defaults and runtime resolution createSandbox applies, so the router and the owner judge the same create; the owner still checks again. In memory only.

func (*Service) CheckNetworkPolicy added in v0.6.6

CheckNetworkPolicy answers "would a sandbox created with these egress fields reach this destination?" (plans/egress-domain-filtering.md P2-9, CEO D5) with the same matcher every enforcement point uses. It sees the policy a create would get: with no egress fields, the operator's default policy applies. A ceiling breach is reported in OutsideCeiling rather than refused, so a caller can see why a create would fail. Pure and read-only: no sandbox, no I/O.

func (*Service) ClaimExecLedger added in v0.6.7

func (s *Service) ClaimExecLedger(ctx context.Context, fingerprint string, named bool) (*models.IdempotentRequestRecord, bool, error)

ClaimExecLedger claims an execution's row. named=false uses the obligation hold (the row stays until its watcher settles it); named=true uses a lapsing lock (the shell, not the ledger, owns the command). acquired=false means another request — possibly in a previous process — owns it, and the returned record says whether it has settled.

func (*Service) ClaimIdempotentRequest added in v0.1.7

func (s *Service) ClaimIdempotentRequest(ctx context.Context, scope, fingerprint string, now time.Time, pendingTTL time.Duration) (*models.IdempotentRequestRecord, bool, error)

func (*Service) ClearClusterForTest added in v0.5.2

func (s *Service) ClearClusterForTest()

ClearClusterForTest drops the attached cluster client so API handlers can exercise their "cluster disabled" branches. Production always has at least cluster.Noop from New().

func (*Service) CloseSecretAuditSink added in v0.6.1

func (s *Service) CloseSecretAuditSink()

CloseSecretAuditSink flushes and stops the file writer. Idempotent. Stops retention first and awaits it so Prune cannot race Close's channel close.

func (*Service) Cluster added in v0.2.1

func (s *Service) Cluster() cluster.Client

Cluster returns the attached cluster.Client. Always non-nil.

func (*Service) ClusterArtifactCatalog added in v0.6.1

func (s *Service) ClusterArtifactCatalog(ctx context.Context, req cluster.ArtifactCatalogRequest) (cluster.ArtifactCatalogPage, bool)

ClusterArtifactCatalog reads one page of the replicated catalogue. Returns ok=false when this node is not clustered or the control plane could not answer — the caller then falls back to asking peers directly, which is the pre-catalogue behavior and never reports an artifact as absent.

func (*Service) ClusterEnabled added in v0.6.1

func (s *Service) ClusterEnabled() bool

ClusterEnabled distinguishes a configured cluster from the single-node Noop client, which intentionally implements the same interface.

func (*Service) ClusterTopologyError added in v0.2.2

func (s *Service) ClusterTopologyError() error

ClusterTopologyError returns a production-topology violation for the current live member set. It is intentionally a runtime check so rolling membership, old nodes that still gossip empty roles, and explicit hybrid roles are all evaluated from the same source of truth the scheduler uses.

func (*Service) CommitIngressProxyRouting added in v0.6.1

func (s *Service) CommitIngressProxyRouting(ctx context.Context, r *IngressRouting, ownerReasserted bool) error

CommitIngressProxyRouting removes what the static routes replace, in one Caddy load. ownerReasserted says whether the boot reconcile re-asserted every local exposure. Only then is it safe to prune kernel rules nobody asserted: a failed pass would otherwise cut live sessions.

func (*Service) CompleteExecLedger added in v0.6.7

func (s *Service) CompleteExecLedger(ctx context.Context, fingerprint, target string) error

CompleteExecLedger marks the row ready with target (an encoded ExecOutcome, or a named execution's id), opening the replay window. It detaches from the caller's context so a settling watcher completes even past a request's end.

func (*Service) CompleteIdempotentRequest added in v0.1.7

func (s *Service) CompleteIdempotentRequest(ctx context.Context, scope, fingerprint, targetID string, now time.Time, replayTTL time.Duration) error

func (*Service) ConfigureAuditExporter added in v0.6.1

func (s *Service) ConfigureAuditExporter() error

ConfigureAuditExporter builds the connector named by SB_AUDIT_EXPORT_BACKEND (plans/audit-export-connectors.md) and installs it behind the shared tailer. A noop backend installs nothing: the open-source default keeps evidence on local disk and says so. Configuration errors fail boot; a backend that cannot reach its receiver does not — it lags, visibly.

func (*Service) ConfigureHTTPAuditExporter added in v0.6.1

func (s *Service) ConfigureHTTPAuditExporter()

ConfigureHTTPAuditExporter is the pre-connector entry point; it now resolves SB_SECRET_AUDIT_EXPORT_URL to the webhook backend. Kept for callers that predate ConfigureAuditExporter; configuration errors are logged, not fatal.

func (*Service) ConfigureSecretProvider added in v0.6.1

func (s *Service) ConfigureSecretProvider(ctx context.Context) error

ConfigureSecretProvider selects the secrets.Provider from cfg.SecretProvider. For an external provider it builds the key-store client behind the per-node limiter, runs a wrap/unwrap canary (E4 lite), and fails open unless cfg.SecretProviderStrictBoot. Safe to call when the provider is already local — it is a no-op refresh for that case.

func (*Service) CountSnapshots added in v0.6.7

func (s *Service) CountSnapshots(ctx context.Context, sourceSandboxID string) (int, error)

CountSnapshots counts the snapshots ListSnapshotPage would walk for the caller with the same source filter.

func (*Service) CreateAPIToken added in v0.6.8

CreateAPIToken mints a token. Only an admin reaches it (token.write). Names are unique per team: a retry or a concurrent duplicate gets ErrAPITokenNameTaken, so the secret of the first create was shown once and is never shown again.

func (*Service) CreateJSBundle added in v0.5.6

func (s *Service) CreateJSBundle(ctx context.Context, req models.CreateJSBundleRequest) (*models.JSBundle, error)

CreateJSBundle stores a bundle under the caller's identity and returns its catalogue view. Idempotent: re-uploading identical bytes yields the same digest with no error; re-using a name repoints it. The digest is the stable reference a create request passes as module_ref ("sha256:<digest>").

func (*Service) CreatePlatformVolume added in v0.5.4

func (s *Service) CreatePlatformVolume(ctx context.Context, name string) (*models.Volume, error)

CreatePlatformVolume creates (or, idempotently, returns) the named volume for the calling tenant. Duplicate creates converge on the existing row rather than erroring, so the operation is safe under retry and concurrent duplicate calls (repo idempotency rule).

func (*Service) CreateRequestDigests added in v0.6.9

func (s *Service) CreateRequestDigests(facade string, canonical []byte) ([]string, error)

CreateRequestDigests returns the keyed digests of a create request's canonical encoding, one per server key, the current key's first, written "hmac:<key id>:<hex>". canonical must not depend on map order (encoding/json sorts map keys). No server key configured means no digest: such a server replays a keyed create as before, unchecked.

func (*Service) CreateSandbox

func (*Service) CreateSandboxWithID added in v0.2.1

func (s *Service) CreateSandboxWithID(ctx context.Context, req models.CreateSandboxRequest, id string) (*models.CreateSandboxResponse, error)

CreateSandboxWithID is the failover-recreate entry point: it behaves like CreateSandbox but uses the supplied ID instead of generating a fresh one. Idempotent at the cluster boundary — if a sandbox with this ID already exists locally we return the existing record without touching docker. Used by the cluster owner watcher to re-materialize a sandbox after its previous owner died, and by every create that derives its ID (Idempotency-Key, E2B, Runloop).

It holds the ID's create lease (create_lease.go) from before it looks for the row until the create's rollback has run, so a second create of the ID on this node waits for this one and then replays its sandbox, or creates afresh when this one failed, instead of running a second create whose failure would undo this one's reservation and mounts.

func (*Service) CreateSnapshot added in v0.1.7

func (s *Service) CreateSnapshot(ctx context.Context, sandboxID string, req models.CreateSandboxSnapshotRequest) (*models.SandboxSnapshot, error)

CreateSnapshot commits the sandbox container into a reusable local image. Idempotency is by snapshot name: repeated requests for the same sandbox + name return the stored snapshot metadata, while a different sandbox trying to claim the same name is rejected with a conflict.

func (*Service) CreateSnapshotWithAlias added in v0.6.7

func (s *Service) CreateSnapshotWithAlias(ctx context.Context, sandboxID string, req models.CreateSandboxSnapshotRequest, alias models.SnapshotAlias) (*models.SandboxSnapshot, bool, error)

CreateSnapshotWithAlias commits a snapshot and records the facade alias that addresses it, in one call — the create-then-alias sequence E2B and Runloop each used to re-implement, each with its own compensating delete (D8). The alias's SnapshotName is filled in from the committed snapshot, so a caller passes only its own Alias / Facade / ExtraNames / StateJSON. The alias is part of the snapshot: an id that resolves to nothing is useless, so a failed alias write rolls the commit back — but only when this call created it (a replayed retry leaves the existing row alone and just re-asserts the alias, which is idempotent). Snapshot creation stays serialized by the service's snapshotMu inside CreateSnapshotWithOwnership.

func (*Service) CreateSnapshotWithOwnership added in v0.1.7

func (s *Service) CreateSnapshotWithOwnership(ctx context.Context, sandboxID string, req models.CreateSandboxSnapshotRequest) (*models.SandboxSnapshot, bool, error)

CreateSnapshotWithOwnership commits a sandbox image and reports whether this call created the native snapshot row. Callers that add companion metadata can use the flag to avoid rolling back a snapshot that already existed.

func (*Service) CreateTemplate added in v0.4.1

func (s *Service) CreateTemplate(ctx context.Context, req models.CreateTemplateRequest) (*models.Template, error)

CreateTemplate accepts a build request, persists a PENDING row, and kicks off the background build. The handler renders the returned row as 202 — callers poll GET /v1/templates/{id} to observe the READY/FAILED transition. Build errors do NOT bubble back through the API; they land in the row's last_error so the operator sees them on poll. This matches the snapshot-push reconciler's async shape.

func (*Service) CreateWasmModule added in v0.5.3

func (s *Service) CreateWasmModule(ctx context.Context, req models.CreateWasmModuleRequest) (*models.WasmModule, error)

CreateWasmModule resolves module_ref on this host and upserts the catalogue row in the caller's own namespace (a tenant's account, or "" for operator and internal callers). The id defaults to the module's content digest, so two owners registering identical bytes each get their own row: rows in other namespaces are never read here, which is what keeps another owner's module from surfacing as a conflict. Idempotent when the caller re-registers an id it already holds for the same module_ref; the same id bound to a different module_ref in the caller's namespace returns ErrWasmModuleIDConflict.

func (*Service) CustomDomainDNS added in v0.3.1

func (s *Service) CustomDomainDNS(ctx context.Context, sandboxID string) (models.CustomDomainDNSRecords, error)

CustomDomainDNS expands a sandbox's attached custom domains into the ready-to-paste DNS record set users must create at their provider. Returns ErrCustomDomainNotSupported (412) when the deployment can't accept custom domains at all — symmetric with ListCustomDomains so callers handle both endpoints the same way. Empty Records is returned (with the resolved Target attached) when the sandbox exists but has no custom domains; that lets the SDK still surface the cluster target without separately calling IngressDNSTarget.

func (*Service) DeleteByOwner added in v0.5.1

func (s *Service) DeleteByOwner(ctx context.Context, ownerRef string) error

DeleteByOwner destroys every sandbox owned by ownerRef. Terminal and idempotent: a sandbox already gone (ErrNotFound) counts as success, so a re-tick over a partially-deleted set converges. Errors on other sandboxes are collected and returned so the loop retries the remainder.

func (*Service) DeleteClusterSecrets added in v0.2.1

func (s *Service) DeleteClusterSecrets(ctx context.Context, sandboxID, incarnationID string) error

func (*Service) DeleteClusterSecretsForAuthoritativePlacement added in v0.6.1

func (s *Service) DeleteClusterSecretsForAuthoritativePlacement(ctx context.Context, sandboxID string) error

DeleteClusterSecretsForAuthoritativePlacement is the rollback finalizer for a failed reserved/promote create that may not have persisted a sandbox row. It retains the placement until this exact lifecycle has been tombed and its remote cleanup journal is durable.

func (*Service) DeleteClusterSecretsLocal added in v0.6.1

func (s *Service) DeleteClusterSecretsLocal(ctx context.Context, sandboxID, incarnationID string, generation int64, peerNodeID string) error

DeleteClusterSecretsLocal applies an authenticated peer DELETE with generation gating and a local tombstone so delayed PUTs cannot resurrect deleted credentials. In cluster mode the mTLS peer must be either the authoritative owner or a recipient recorded on this exact local ciphertext lifecycle. Standalone callers retain the local-only behavior.

func (*Service) DeleteEgressProfile added in v0.6.6

func (s *Service) DeleteEgressProfile(ctx context.Context, name string) error

DeleteEgressProfile removes one of the caller's profiles; one a live sandbox references is refused with 409. Deleting a profile that is gone succeeds.

func (*Service) DeleteIdempotentRequest added in v0.1.7

func (s *Service) DeleteIdempotentRequest(ctx context.Context, scope, fingerprint string) error

func (*Service) DeleteIdentityRole added in v0.6.12

func (s *Service) DeleteIdentityRole(ctx context.Context, team, tag string) error

DeleteIdentityRole takes a tag off a team's allow list. Deleting a tag already gone succeeds. Sandboxes created with it keep their current token until it expires (at most one hour) and get no new one.

func (*Service) DeleteJSBundle added in v0.5.6

func (s *Service) DeleteJSBundle(ctx context.Context, digest string) error

DeleteJSBundle removes a bundle the caller owns, refusing when a live sandbox still pins its digest (mirrors DeleteWasmModule — a referenced artifact is not garbage).

func (*Service) DeletePlatformVolume added in v0.5.4

func (s *Service) DeletePlatformVolume(ctx context.Context, id string) error

DeletePlatformVolume removes a volume, but only when no live sandbox still has it attached — otherwise the backing data would vanish out from under a running workload. Returns models.ErrPlatformVolumeInUse (→ 409) when attachers remain, models.ErrNotFound when the id is unknown.

The store writes a pending cleanup ledger row before removing the metadata row, so backend bytes retain durable reconciliation coordinates even if a request-path cleanup worker or operator lifecycle policy is delayed.

func (*Service) DeleteSnapshot added in v0.1.7

func (s *Service) DeleteSnapshot(ctx context.Context, idOrName string) error

func (*Service) DeleteSnapshotAlias added in v0.1.7

func (s *Service) DeleteSnapshotAlias(ctx context.Context, alias string) error

DeleteSnapshotAlias removes an alias whose snapshot the caller may see.

func (*Service) DeleteTeamQuota added in v0.6.12

func (s *Service) DeleteTeamQuota(ctx context.Context, team string) error

DeleteTeamQuota removes a team's quota. Removing one that is already gone succeeds, so a retried DELETE converges.

func (*Service) DeleteTemplate added in v0.4.1

func (s *Service) DeleteTemplate(ctx context.Context, id string) error

DeleteTemplate refuses when an active sandbox still references the template — yanking the rootfs out from under a running Firecracker guest would surface as a delayed I/O error inside the guest, hours after the operator's action. The 409 forces the operator to destroy the sandbox first. Best-effort artifact cleanup runs before the row delete: a half-cleaned dir on disk is fine (GC catches it), a missing row with an orphan dir is the painful case (no one will ever clean it). Pending rows are also rejected — a delete that races the build goroutine would leave the goroutine writing to a directory the operator believed was gone.

func (*Service) DeleteWarmPool added in v0.6.12

func (s *Service) DeleteWarmPool(ctx context.Context, image string) error

DeleteWarmPool removes the entry for an image; each server's pool for it drains. One that is already gone counts as deleted.

func (*Service) DeleteWasmModule added in v0.5.3

func (s *Service) DeleteWasmModule(ctx context.Context, id string) error

DeleteWasmModule removes the caller's own catalogue row when no sandbox that resolves through it remains. Delete acts only inside the caller's namespace: another owner's row with the same id is out of reach, and an operator-published ("") row is read-only to a tenant, so both 404 like an id that does not exist.

func (*Service) DestroySandbox

func (s *Service) DestroySandbox(ctx context.Context, id string) error

DestroySandbox deletes a sandbox inside the lifecycle boundary, so a start or a restart on failure in flight cannot write the row back after the delete (Upsert inserts a missing row).

With no local row, a cluster node may still own the sandbox's placement: one reassigned to it by a failover that its owner watcher has not recreated yet. destroyUnmaterializedPlacement deletes that placement instead of answering 404. It runs after the lifecycle lock is released, because it takes the create lease, and the lock order is create lease, then lifecycle lock (a create's rollback destroys under its lease).

func (*Service) DiscardBuiltImage added in v0.6.7

func (s *Service) DiscardBuiltImage(ctx context.Context, tag string)

DiscardBuiltImage undoes a fresh build (BuiltImage.Fresh) whose next step failed, so no sandbox or snapshot came of it. The tag goes to the pending-image GC rather than being removed here: a concurrent request for the same build may have been handed the tag from the cache meanwhile, and the GC removes it only once nothing uses it.

func (*Service) DrainWasmSandboxes added in v0.5.3

func (s *Service) DrainWasmSandboxes(ctx context.Context) error

DrainWasmSandboxes checkpoints passivatable/durable live WASM sandboxes during graceful shutdown (plans/wasm-runtime.md §4.3).

func (*Service) EgressAuditObserver added in v0.6.1

func (s *Service) EgressAuditObserver() func(sandboxID, network, destination string)

EgressAuditObserver returns a non-blocking callback for wasm/isolate mediators. Safe to install when attribution is disabled (returns a no-op).

func (*Service) EgressDenialObserver added in v0.6.6

func (s *Service) EgressDenialObserver() func(sandboxID, destination, reason string)

EgressDenialObserver is the callback for denials decided inside sandboxd rather than by the gateway (the isolate egress proxy; P1-7, H5): it adds to aerolvm_egress_denied_total and writes the denial to the audit log when attribution is on. It never blocks the request path: the audit write is handed to a goroutine like the success observer's.

func (*Service) EgressGatewayReady added in v0.6.6

func (s *Service) EgressGatewayReady() bool

EgressGatewayReady reports whether this node can attach a hostname-filtered sandbox right now. It is advertised in the capacity heartbeat, so cluster placement sends gateway-mode creates only to nodes where it holds (plans/egress-domain-filtering.md CEO D20). Two atomic loads: it runs on every heartbeat and every /v1/capacity read.

func (*Service) EgressProcidAuthorize added in v0.6.6

func (s *Service) EgressProcidAuthorize(ctx context.Context, sandboxID string) (int, []string, error)

EgressProcidAuthorize answers the gateway's executable lookup (review finding 16) for one sandbox: the pid of its init process from sandboxd's own record, and the paths its rules list, the only ones it may be asked about. The gateway names a sandbox, never a pid.

func (*Service) EgressStatus added in v0.6.6

func (s *Service) EgressStatus(ctx context.Context, sb *models.Sandbox) string

EgressStatus reports a sandbox's egress status for GET: "" for a sandbox outside gateway mode and not held, otherwise active, held or unavailable. A hold shows on every runtime: a policy that failed to apply, or profiles that can't be resolved, hold WASM, isolate and CIDR sandboxes too. A policy on a runtime whose host firewall is off is unenforced, ahead of all of those: neither an attach nor a hold filters anything there.

func (*Service) EnableBlockPrivateByDefault added in v0.6.12

func (s *Service) EnableBlockPrivateByDefault()

EnableBlockPrivateByDefault turns on SB_EGRESS_BLOCK_PRIVATE_BY_DEFAULT. It reads the resolvers sandboxes are handed (the same files the node guard reads) once, here at boot, so a create never touches the disk: a resolver inside a blocked range, such as an AWS VPC resolver at 10.x.0.2 or 172.31.0.2, gets a /32 allow entry, or every default sandbox would lose DNS. A resolver change on the host is picked up at the next restart.

func (*Service) EnsureClusterReady added in v0.2.1

func (s *Service) EnsureClusterReady(ctx context.Context) error

EnsureClusterReady blocks until the cluster has elected a leader, mirroring the EnsureLayer4Ready single-flight latch shape. Single-node mode latches immediately. The API wrapper calls this before any RecordPlacement so a just-booted node doesn't 503 a CreateSandbox while raft is still catching up.

func (*Service) EnsureEgressGatewayReady added in v0.6.6

func (s *Service) EnsureEgressGatewayReady(ctx context.Context) error

EnsureEgressGatewayReady connects to the gateway and pushes the node's full state: the sandbox bridges, then a Sync of every gateway-mode sandbox. Same atomic.Bool + mutex single-flight shape as EnsureLayer4Ready; called best-effort at daemon start and lazily before the first attach. A failure leaves the latch unset.

func (*Service) EnsureIdentityKeysReady added in v0.6.12

func (s *Service) EnsureIdentityKeysReady(ctx context.Context) error

EnsureIdentityKeysReady makes sure a signing key exists: the deployment's first key is made by whichever node gets here first (the daemon start, or the first JWKS fetch or token after a start that could not reach the store or the leader). Concurrent nodes race through the key set's generation CAS, so exactly one key is made.

func (*Service) EnsureLayer4Ready added in v0.1.4

func (s *Service) EnsureLayer4Ready(ctx context.Context) error

EnsureLayer4Ready bootstraps the caddy-l4 app under a single-flight mutex and latches success. Safe to call from boot AND from each L4 exposure path: the atomic fast-path turns it into a single load on the steady state, and a failed boot is recovered by the very next TCP/TLS expose call instead of surfacing as a confusing "layer4 app missing" error from caddy.

func (*Service) EnsureNetstatsReady added in v0.1.7

func (s *Service) EnsureNetstatsReady(ctx context.Context) error

EnsureNetstatsReady boots the per-sandbox network byte-counter poller under a single-flight latch. Same lazy-bootstrap shape as EnsureLayer4Ready: caller pays the bootstrap cost only once across the daemon's lifetime, and the poller goroutine survives until ctx — typically the daemon's signal context — is cancelled.

Failure here is non-fatal: callers log and continue. Without the poller, network counters stay at zero and quotas never trigger; both Create and reads of /network/usage still work.

func (*Service) EnsureSandboxAwakeForHTTP added in v0.2.4

func (s *Service) EnsureSandboxAwakeForHTTP(ctx context.Context, id string) (*models.Sandbox, error)

EnsureSandboxAwakeForHTTP is the single chokepoint the control-plane HTTP proxies (v1 toolbox, daytona toolbox, e2b runtime) call before resolving the toolbox endpoint. It returns the current sandbox row so the caller can route to ContainerIP without re-fetching.

Behavior:

  • sandbox started → fast-path return (no lock, no metric latency observation; only a requests_total tick).
  • sandbox stopped + (gate off OR Lifecycle.Serverless=false OR WakeArmed=false) → ErrSandboxManuallyStopped. The proxy maps this to 409 so the SDK can explicitly call StartSandbox.
  • sandbox stopped + eligible to wake → take the per-sandbox wake lock, recheck under the lock (a concurrent flight may have finished), enforce the circuit breaker, then call StartSandbox with a 15s budget. On admission failure or start error the breaker counter advances; 5 failures inside 60s trip it for 60s. Success resets the counter.

Same single-flight pattern as EnsureLayer4Ready, but per-sandbox: the wave of HTTP requests waking the same sandbox collapses to one StartSandbox; waves to different sandboxes proceed concurrently.

func (*Service) EnsureTemplateLocal added in v0.4.1

func (s *Service) EnsureTemplateLocal(ctx context.Context, tpl *models.Template) error

EnsureTemplateLocal guarantees that <templatesDir>/<id>/ exists with rootfs.ext4 + snapshot.{memory,state} on this node, fetching from AOCR via the attached puller when missing. Idempotent: callers that hit the latch fast-path pay one atomic load.

Mirrors EnsureLayer4Ready exactly:

  • atomic.Bool per-template latch for the lock-free steady-state path
  • sync.Mutex per-template single-flight collapses concurrent first callers into one pull
  • failure leaves the latch unset so the next call retries

Returns nil immediately when:

  • puller is not attached (single-node mode, or push/pull disabled)
  • the template has no registry_ref (never pushed; this node must be the producer and the files are presumed local already)
  • the files are already on disk (cold-start on the producer node)

func (*Service) EvacuateLocalWasmSandboxesForDrain added in v0.5.3

func (s *Service) EvacuateLocalWasmSandboxesForDrain(ctx context.Context) error

EvacuateLocalWasmSandboxesForDrain checkpoints and migrates passivatable/durable WASM sandboxes off this node during cluster drain (§4.4).

func (*Service) ExecLedgerOutcome added in v0.6.7

func (s *Service) ExecLedgerOutcome(ctx context.Context, fingerprint string) (outcome ExecOutcome, owed, ok bool)

ExecLedgerOutcome returns the recorded outcome of a finished (or lost) plain execution. owed reports that its session may still exist (the row is not ready yet), so a watcher has to see the cleanup through.

func (*Service) ExportWasmMigration added in v0.5.3

func (s *Service) ExportWasmMigration(ctx context.Context, sandboxID string, w io.Writer) (cloneGen string, err error)

ExportWasmMigration streams a §4.8.1 mem.snap tarball for sandboxID.

func (*Service) ExposePort

func (s *Service) ExposePort(ctx context.Context, id string, port int, protocol string) (models.ExposePortResponse, error)

ExposePort publishes a sandbox container port through one of three caddy surfaces, selected by protocol:

  • "" / "http": existing Caddy HTTP reverse-proxy route, returns https://<id>-<port>.<domain> (or the path-mode equivalent).
  • "tcp": allocates a parent-host TCP port from the [SB_L4_PORT_RANGE_START, SB_L4_PORT_RANGE_END] pool, points caddy-l4 at it, and returns tcp://<public-host>:<host-port>. This is what unblocks native Postgres / Redis / MySQL DSNs in the spawn-postgres docs.
  • "tls": adds a TLS-SNI route to the shared layer4 server. Requires --domain (so the SNI hostname has a place to resolve) and a non-empty SB_L4_TLS_LISTEN. Returns tls://<id>-<port>.<domain>:<l4-port>.

func (*Service) ExposePortWithVisibility added in v0.6.12

func (s *Service) ExposePortWithVisibility(ctx context.Context, id string, port int, protocol, visibility string) (models.ExposePortResponse, error)

ExposePortWithVisibility is ExposePort with a visibility (A2): "public", "token" (the public URL needs a sandbox token for this port), or "" (a new exposure is public; a re-expose keeps what it has). Only an explicit value changes a port's visibility, so a retry from a client that does not know about visibility can never open a gated port.

func (*Service) FenceOwnershipForTest added in v0.6.12

func (s *Service) FenceOwnershipForTest(ids ...string)

FenceOwnershipForTest marks ids fenced without touching the runtime, for tests outside this package that exercise the 503.

func (*Service) FinishRollbackPending added in v0.6.12

func (s *Service) FinishRollbackPending(ctx context.Context, id string) error

FinishRollbackPending makes one attempt, for a retry of a create that found its earlier attempt's row marked, to run that attempt's pending destroy. nil means nothing of it is left and the retry may create afresh; otherwise the error is an ErrCreateRollbackPending and the row stays marked for reconcile. The caller holds the id's create lease, so no create of the id can run meanwhile. The attempt is bounded by createRollbackTimeout, the budget a create's own rollback gets; each control-plane call inside it is bounded by the destroy path's own timeouts.

func (*Service) FireWebhook added in v0.5.1

func (s *Service) FireWebhook(_ context.Context, ownerRef, level string) error

FireWebhook records that an owner crossed a notify-level directive. The webhook URL and secret live entirely in the managed contract (never in the open-source build), so the host's part of the contract is just to surface the crossing in its audit log; the managed control plane owns any outbound notification. Called once per transition by the enforcement loop, so this is already idempotent per crossing.

func (*Service) ForceReconcileHTTPWakeShape added in v0.2.4

func (s *Service) ForceReconcileHTTPWakeShape(ctx context.Context) error

ForceReconcileHTTPWakeShape republishes the HTTP route for every serverless sandbox's HTTP exposures using the current HTTPWakeDirectBypassEnabled value. Called exactly once at daemon startup when the rollback marker shows the previous run had bypass=true and the current run has bypass=false: without this, every serverless sandbox that was last written under bypass=true keeps its direct route pointing straight at the container IP, bypassing the wake-aware ingress proxy entirely — i.e. the rollback knob would not actually roll back. Conversely, on a fresh enable (false→true) routes will be rewritten lazily on the next Start/Stop, so no force-pass is needed in that direction (D5 of plans/warm-direct-route-bypass.md).

Idempotent: chooseRouteShape is a pure function of (cfg, sandbox), so repeated calls converge to the same route set. Best-effort per sandbox — one failed Caddy write logs and the loop continues so a single broken row does not block the rest from being rolled back.

func (*Service) GCUnreferencedJSBundles added in v0.5.6

func (s *Service) GCUnreferencedJSBundles(ctx context.Context) (int, error)

GCUnreferencedJSBundles is the single sweep used by the loop, operators, and tests.

func (*Service) GetCompatState added in v0.1.7

func (s *Service) GetCompatState(ctx context.Context, sandboxID, facade string) (*models.SandboxCompatState, error)

func (*Service) GetDeletedSandboxView added in v0.6.7

func (s *Service) GetDeletedSandboxView(ctx context.Context, sandboxID, facade string) (string, bool, error)

GetDeletedSandboxView returns the stored terminal view for a deleted sandbox, or ("", false, nil) when there is none, the caller may not see it, or it is past the replay window. The owner check lives here, in the shared helper, under the same rule scopedGet applies to live rows (D13): a user token sees only its own account's tombstones; operator / internal callers see any.

func (*Service) GetEgressProfile added in v0.6.6

func (s *Service) GetEgressProfile(ctx context.Context, name string) (*models.EgressProfile, error)

GetEgressProfile returns one of the caller's profiles, a built-in one (builtin:<name> is its newest version, builtin:<name>@<version> that one) or one of the operator's org profiles, which every tenant can read.

func (*Service) GetExecLedger added in v0.6.7

func (s *Service) GetExecLedger(ctx context.Context, fingerprint string) (*models.IdempotentRequestRecord, error)

GetExecLedger reads one execution row.

func (*Service) GetIdempotentRequest added in v0.1.7

func (s *Service) GetIdempotentRequest(ctx context.Context, scope, fingerprint string) (*models.IdempotentRequestRecord, error)

func (*Service) GetJSBundle added in v0.5.6

func (s *Service) GetJSBundle(ctx context.Context, digest string) (*models.JSBundle, error)

GetJSBundle returns one bundle by digest, refusing digests the caller does not own (a 404, not a 403, so a user token cannot probe others' digests).

func (*Service) GetNetworkLearned added in v0.6.6

func (s *Service) GetNetworkLearned(ctx context.Context, id string) (*models.NetworkLearned, error)

GetNetworkLearned returns what a sandbox reached while in learn mode, and the allow list that would have allowed it. A recording stays readable after the sandbox switches to enforce, until it is destroyed.

func (*Service) GetNetworkUsage added in v0.1.7

func (s *Service) GetNetworkUsage(ctx context.Context, id string) (*models.NetworkUsage, error)

GetNetworkUsage returns the current cumulative byte counters and configured limits for a sandbox. Callers handle ErrNotFound translation.

Best-effort lazy bootstrap of the netstats poller: if boot's EnsureNetstatsReady failed (cold-start race against the docker daemon, etc.) this call retries it under the same single-flight latch. Failure is logged and swallowed — the caller still gets back whatever counters the store has, and the next call will retry again.

func (*Service) GetPlatformVolume added in v0.5.4

func (s *Service) GetPlatformVolume(ctx context.Context, id string) (*models.Volume, error)

GetPlatformVolume returns the tenant's volume by id, or models.ErrNotFound.

func (*Service) GetPlatformVolumeByName added in v0.5.4

func (s *Service) GetPlatformVolumeByName(ctx context.Context, name string) (*models.Volume, error)

GetPlatformVolumeByName returns the tenant's volume by name, or models.ErrNotFound. The name is sanitized so lookups match how it was stored.

func (*Service) GetSandbox

func (s *Service) GetSandbox(ctx context.Context, id string) (*models.Sandbox, error)

func (*Service) GetSandboxWithOptions added in v0.6.1

func (s *Service) GetSandboxWithOptions(ctx context.Context, id string, opts GetSandboxOptions) (*models.Sandbox, error)

GetSandboxWithOptions returns a sandbox. Env is omitted unless IncludeEnv is set; that opt-in path loads via loadEnv and is audited.

func (*Service) GetSnapshot added in v0.1.7

func (s *Service) GetSnapshot(ctx context.Context, idOrName string) (*models.SandboxSnapshot, error)

GetSnapshot resolves a snapshot by name or image ID. Another tenant's snapshot reads as store.ErrNotFound for a user token, the same 404 a cross-owner sandbox id gets.

func (*Service) GetSnapshotAlias added in v0.1.7

func (s *Service) GetSnapshotAlias(ctx context.Context, alias string) (*models.SnapshotAlias, error)

GetSnapshotAlias returns the alias row when its snapshot is visible to the caller, store.ErrNotFound otherwise.

func (*Service) GetTeamQuota added in v0.6.12

func (s *Service) GetTeamQuota(ctx context.Context, team string) (*models.TeamQuota, error)

GetTeamQuota returns a team's quota and what the team holds now.

func (*Service) GetTemplate added in v0.4.1

func (s *Service) GetTemplate(ctx context.Context, id string) (*models.Template, error)

GetTemplate is the read path behind GET /v1/templates/{id}. Returns the store row unchanged — status is whatever the build goroutine last wrote, so a freshly-created template reads as PENDING until the goroutine flips it.

func (*Service) GetWarmPool added in v0.6.12

func (s *Service) GetWarmPool(ctx context.Context, image string) (*models.WarmPool, error)

GetWarmPool returns one entry with its status.

func (*Service) GetWasmModule added in v0.5.3

func (s *Service) GetWasmModule(ctx context.Context, id string) (*models.WasmModule, error)

func (*Service) HasLocalSandbox added in v0.6.9

func (s *Service) HasLocalSandbox(ctx context.Context, id string) bool

HasLocalSandbox reports whether this node's store holds a row for id, unscoped (the handler that serves the row applies owner scoping). Only a definite miss is false: any other store error is true, so the caller serves locally and the handler surfaces that error instead of a guessed absence. The cluster router asks it only while the placement view is catching up.

func (*Service) HasLocalSealedSecretGeneration added in v0.6.1

func (s *Service) HasLocalSealedSecretGeneration(ctx context.Context, sandboxID, incarnationID string, minGeneration int64) (bool, error)

HasLocalSealedSecretGeneration reports whether this node holds a sealed row with seal_generation >= minGeneration (peer HEAD probe target).

func (*Service) Health

func (s *Service) Health(ctx context.Context) (models.HealthStatus, error)

func (*Service) HoldsCommittedSnapshotImage added in v0.6.9

func (s *Service) HoldsCommittedSnapshotImage(ctx context.Context, image string) (bool, error)

HoldsCommittedSnapshotImage reports whether this node's catalogue holds the snapshot committed under image (a docker.SnapshotImageTag, any spelling). It answers for the node, not the caller: whether the caller may start that snapshot is still NormalizeCreateImageDistribution's call. The cluster create router asks it first so a create for a snapshot this node holds stays here, exactly as before, whatever the cluster's snapshot-home index says.

func (*Service) HoldsSnapshotNamed added in v0.6.12

func (s *Service) HoldsSnapshotNamed(ctx context.Context, name string) (bool, error)

HoldsSnapshotNamed reports whether this node's catalogue has a snapshot of this name in the caller's namespace (any spelling). The cluster create router asks it before it sends a create by snapshot name to the snapshot's home, since a node holding the row resolves the name itself; the home asks it when that create arrives.

func (*Service) IdentityDiscovery added in v0.6.12

func (s *Service) IdentityDiscovery() ([]byte, error)

IdentityDiscovery is the OIDC discovery document.

func (*Service) IdentityEnabled added in v0.6.12

func (s *Service) IdentityEnabled() bool

IdentityEnabled reports whether this deployment is an OIDC issuer.

func (*Service) IdentityIssuer added in v0.6.12

func (s *Service) IdentityIssuer() string

IdentityIssuer is the issuer URL, or "" when off.

func (*Service) IdentityJWKS added in v0.6.12

func (s *Service) IdentityJWKS(ctx context.Context) ([]byte, error)

IdentityJWKS is the public key set: the active key and every key rotated out less than two hours ago (D1.3). It holds public keys only.

func (*Service) ImageBuildTimeout added in v0.6.12

func (s *Service) ImageBuildTimeout() time.Duration

ImageBuildTimeout is the most one image build, or one push, may take on this node (SB_IMAGE_BUILD_TIMEOUT). 0 means no limit.

func (*Service) ImageBuilderConfigured added in v0.6.7

func (s *Service) ImageBuilderConfigured() bool

ImageBuilderConfigured reports whether builds can run on this daemon.

func (*Service) ImportWasmMigration added in v0.5.3

func (s *Service) ImportWasmMigration(ctx context.Context, sandboxID, expectedCloneGen string, r io.Reader) (err error)

ImportWasmMigration accepts a streamed mem.snap tarball on the receiving node, promotes the sandbox row to passivated, and reassigns cluster ownership to self.

func (*Service) IngressDNSTarget added in v0.3.1

func (s *Service) IngressDNSTarget() models.IngressTarget

IngressDNSTarget returns the cluster's public ingress address as the SDK would render it to a user (CNAME target hostname, raw A/AAAA IP list, or both). Delegates to cluster.Client so single-node, cluster, and agent deployments all answer through the same aggregator. This call does NOT gate on EnableCustomDomains: knowing the target is useful BEFORE attaching any domain (e.g. an SDK consumer rendering "point your DNS here first"), and the operator information returned (a hostname or IP they already chose to advertise on memberlist) is the same data anyone running `dig <sandbox-url>` could discover.

func (*Service) IsSandboxStarted added in v0.2.4

func (s *Service) IsSandboxStarted(ctx context.Context, id string) (bool, error)

IsSandboxStarted is the preflight check the wake-aware ingress proxy uses to skip request-body buffering for warm sandboxes. It returns (true, nil) only when the sandbox row exists and its status is Started; any other status (Stopped, Destroyed, in-flight create) returns (false, nil). store.ErrNotFound is returned unwrapped so the proxy can map it to 404 without a second store hit.

Hot path: a short-TTL cache (warmCache) lets repeated warm requests to the same sandbox skip the SQLite read. The cache holds positives only ("seen Started in the last warmCacheTTL"); a miss always falls through to s.store.Get so cold/stopped/destroyed sandboxes never short-circuit. Lifecycle transitions (StartSandbox success, every stop path, every destroy path) explicitly invalidate the entry.

func (*Service) IssueEgressAuditCapability added in v0.6.1

func (s *Service) IssueEgressAuditCapability(sandboxID, incarnationID string, ttl time.Duration) (string, error)

IssueEgressAuditCapability mints a per-sandbox HMAC capability for wasm workers. Scoped so a capability for sandbox A cannot forge events for B.

func (*Service) IssueEgressAuditCapabilityForSandbox added in v0.6.1

func (s *Service) IssueEgressAuditCapabilityForSandbox(sandboxID string, ttl time.Duration) (string, string, error)

IssueEgressAuditCapabilityForSandbox resolves the current placement incarnation and returns only the scoped capability needed by a worker.

func (*Service) IssueSandboxToken added in v0.6.12

func (s *Service) IssueSandboxToken(ctx context.Context, sandboxID string, req models.CreateSandboxTokenRequest) (models.SandboxToken, error)

IssueSandboxToken signs a token for sandboxID with the scope req asks for. The caller must see the sandbox (owner scope); the route already required sandbox.exec, the most a token grants. Not idempotent by nature: every call mints a new token with a new id, and nothing is stored, so a retried call leaks nothing but a second short-lived token the caller never received.

func (*Service) KeyRevoked added in v0.6.9

func (s *Service) KeyRevoked() bool

KeyRevoked reports whether the customer's key store revoked the server key and this node stopped processing their data.

func (*Service) ListAPITokens added in v0.6.8

func (s *Service) ListAPITokens(ctx context.Context) (models.APITokenList, error)

ListAPITokens returns every token, sorted by team and name, expired ones flagged. Never a secret or a hash.

func (*Service) ListCompatState added in v0.1.7

func (s *Service) ListCompatState(ctx context.Context, facade string) (map[string]models.SandboxCompatState, error)

ListCompatState returns the facade state for the caller's sandboxes. A user token only ever joins this against its own (owner-scoped) sandbox list, so a scoped caller reads just its own rows — not every tenant's — which both keeps the owner check in this shared helper (D13) and avoids a fleet-wide scan on each list at 100k sandboxes. Operator / internal callers (no owner scope) get the full map, as the fleet-list paths need.

func (*Service) ListCustomDomains added in v0.3.1

func (s *Service) ListCustomDomains(ctx context.Context, sandboxID string) ([]models.CustomDomain, error)

ListCustomDomains returns the canonical rows for one sandbox. Empty slice (nil) when none are attached. Same deployment-capability gate as Add/Remove: disabled-feature or IP-mode returns ErrCustomDomainNotSupported (412) rather than an empty list, so callers can distinguish "this sandbox has no domains" from "this cluster does not do custom domains".

func (*Service) ListEgressProfiles added in v0.6.6

func (s *Service) ListEgressProfiles(ctx context.Context, cursor string, limit int) (*models.EgressProfileList, error)

ListEgressProfiles returns a page of the caller's profiles by name.

func (*Service) ListIdempotentRequests added in v0.6.7

func (s *Service) ListIdempotentRequests(ctx context.Context, scope, state, afterFingerprint string, limit int) ([]*models.IdempotentRequestRecord, error)

ListIdempotentRequests returns one page of a scope's rows in one state, after the cursor fingerprint (see the store method). Used by claimers that recover outstanding work after a restart; a service with no store (embedded wiring) has none.

func (*Service) ListIdentityRoles added in v0.6.12

func (s *Service) ListIdentityRoles(ctx context.Context, team, after string, limit int) (*models.IdentityRoleList, error)

ListIdentityRoles pages a team's allow list by tag.

func (*Service) ListJSBundles added in v0.5.6

func (s *Service) ListJSBundles(ctx context.Context) ([]*models.JSBundle, error)

ListJSBundles returns the caller's stored bundles.

func (*Service) ListMounts

func (s *Service) ListMounts(ctx context.Context, sandboxID string) ([]models.MountSpecRedacted, error)

ListMounts returns the redacted mount config for a sandbox. Credentials are never included in the response — they are write-only via CreateSandbox.

func (*Service) ListNodeStorageRetirements added in v0.6.1

func (s *Service) ListNodeStorageRetirements(ctx context.Context) ([]store.NodeStorageRetirement, error)

ListNodeStorageRetirements exposes the attestation set for operators.

func (*Service) ListOwnedSnapshots added in v0.6.7

func (s *Service) ListOwnedSnapshots(ctx context.Context, sourceSandboxID, aliasFacade string) ([]store.SnapshotPageRow, error)

ListOwnedSnapshots returns every snapshot the caller may see — optionally narrowed to one source sandbox and joined with one facade's alias rows — by paging the store's owner-scoped range scan. It exists so a facade that needs its own sort order or page-number pagination (Daytona, E2B) can materialize just the caller's snapshots instead of calling ListSnapshots, which reads every tenant's rows and filters in Go (D9). Rows come back in the store's keyset order (name ASC); the caller re-sorts as its wire API requires. Pass aliasFacade "" when the caller does not join aliases.

func (*Service) ListPendingExecLedger added in v0.6.7

func (s *Service) ListPendingExecLedger(ctx context.Context, after string, limit int) ([]*models.IdempotentRequestRecord, error)

ListPendingExecLedger returns one page of pending execution rows after the given fingerprint, for a restart-time recovery sweep.

func (*Service) ListPlatformVolumes added in v0.5.4

func (s *Service) ListPlatformVolumes(ctx context.Context) ([]models.Volume, error)

ListPlatformVolumes returns all of the tenant's volumes.

func (*Service) ListSandboxes

func (s *Service) ListSandboxes(ctx context.Context, tagFilter map[string]string) ([]*models.Sandbox, error)

ListSandboxes returns sandboxes whose Tags match every entry in tagFilter. A nil or empty filter returns every sandbox on this node. Filtering happens in-memory after the store read because Tags is JSON-encoded; pushing the filter into SQL via json_extract is a follow-up once row counts make the extra hop worth it. The filter exists so an external control plane can ask "give me the sandboxes belonging to user X" without round-tripping every sandbox in the cluster (see plans/multi-tenancy-via-control-plane.md). Env is always omitted (D9); use ListSandboxesWithOptions to opt in.

func (*Service) ListSandboxesByName added in v0.6.5

func (s *Service) ListSandboxesByName(ctx context.Context, name string, tagFilter map[string]string, opts GetSandboxOptions) ([]*models.Sandbox, error)

sandboxMatchesTags returns true iff every key in want is present on sb.Tags with the same value. An empty want matches everything (caller short-circuits). ListSandboxesByName returns the caller's sandbox called name as a list of zero or one, so a lookup by name and a list share one wire shape (the v1 GET /sandboxes?name= filter). Names are unique per owner and the lookup runs in the caller's namespace (ResolveSandboxIDByName), so another tenant's name reads as an empty list rather than revealing that it exists. A tag filter still applies to the one row.

func (*Service) ListSandboxesWithOptions added in v0.6.1

func (s *Service) ListSandboxesWithOptions(ctx context.Context, tagFilter map[string]string, opts GetSandboxOptions) ([]*models.Sandbox, error)

ListSandboxesWithOptions is ListSandboxes with optional IncludeEnv (audited).

func (*Service) ListSecretAudit added in v0.6.1

func (s *Service) ListSecretAudit(ctx context.Context, sandboxID string, opts SecretAuditQuery) (SecretAuditPage, error)

ListSecretAudit returns local events merged with a live fan-out to reachable peers. Coverage.Missing lists peers that timed out or failed — never silent.

Scale bounds:

  • target the bounded Raft-retained owner history when complete
  • fail explicitly when history is absent/truncated; never scan all workers
  • skip pure ingress roles (they do not host sandboxes)
  • bounded parallel peer fetches under a short global deadline

func (*Service) ListSecretAuditLocal added in v0.6.1

func (s *Service) ListSecretAuditLocal(ctx context.Context, sandboxID string, opts SecretAuditQuery) (events []SecretAuditEvent, nextCursor string, err error)

ListSecretAuditLocal answers one page of sandboxID's history from the authoritative local JSONL. With the read index ready the cost is O(page): the index names the lines the page needs and only those are read and verified. Without it (index disabled, rebuilding, or disagreeing with the file) the page comes from a scan of the file, which is correct but O(retained events) — the metrics below say which path served.

Integrity on this path is per returned record (its hash must match its own content and prev_hash); whole-chain verification is the job of boot, retention, and VerifySecretAuditChain, not of every page read.

func (*Service) ListSnapshotAliases added in v0.1.7

func (s *Service) ListSnapshotAliases(ctx context.Context, facade string) (map[string]models.SnapshotAlias, error)

ListSnapshotAliases returns the facade's alias rows whose snapshots the caller may see. A user token's list is filtered in the store, so other tenants' rows are never loaded.

func (*Service) ListSnapshotPage added in v0.6.7

func (s *Service) ListSnapshotPage(ctx context.Context, sourceSandboxID, aliasFacade, afterName string, limit int) ([]store.SnapshotPageRow, error)

ListSnapshotPage returns one name-ordered page of the snapshots the caller may see, optionally narrowed to one source sandbox and joined with one facade's alias rows. It pages in the store, so a list costs one bounded range scan however large the catalogue is.

func (*Service) ListSnapshots added in v0.1.7

func (s *Service) ListSnapshots(ctx context.Context) ([]*models.SandboxSnapshot, error)

ListSnapshots returns the snapshots the caller may see: every row for operator and internal callers, the caller's own account's for a user token.

func (*Service) ListTemplates added in v0.4.1

func (s *Service) ListTemplates(ctx context.Context) ([]*models.Template, error)

ListTemplates is the read path behind GET /v1/templates. Returns rows in newest-first order matching the snapshot list shape.

func (*Service) ListWarmPools added in v0.6.12

func (s *Service) ListWarmPools(ctx context.Context) (*models.WarmPoolList, error)

ListWarmPools returns every entry, by image, with its status.

func (*Service) ListWasmModules added in v0.5.3

func (s *Service) ListWasmModules(ctx context.Context) ([]*models.WasmModule, error)

func (*Service) LocalReadyTemplateIDs added in v0.4.1

func (s *Service) LocalReadyTemplateIDs(ctx context.Context) []string

LocalReadyTemplateIDs returns only the template IDs from LocalReadyTemplateInventory. Callers that need to distinguish authoritative empty inventory from legacy/unknown should use the richer method.

func (*Service) LocalReadyTemplateInventory added in v0.4.1

func (s *Service) LocalReadyTemplateInventory(ctx context.Context) ([]string, bool)

LocalReadyTemplateInventory returns the IDs of Firecracker templates whose artifacts are usable on this host plus whether that inventory is authoritative. Cached for 5s — peers gossip /v1/capacity at this cadence and a per-tick SQLite hit would starve the create path's writes (single-writer DB). On error the cache is left untouched and the previous value is returned, matching the "stale is better than wrong" contract the rest of the heartbeat path uses.

func (*Service) LocalReadyWasmModuleInventory added in v0.5.3

func (s *Service) LocalReadyWasmModuleInventory(ctx context.Context) ([]string, bool)

LocalReadyWasmModuleInventory returns module_ref values present in the local wasm_modules catalogue plus whether that inventory is authoritative.

func (*Service) LocalSlotPools added in v0.6.9

func (s *Service) LocalSlotPools(ctx context.Context) []capacity.SlotPool

LocalSlotPools reports this node's bounded per-sandbox slot pools and how many slots each has free. Service.Capacity and the cluster's own lease overlay it on the capacity snapshot, which is how placement learns that a node with CPU and memory to spare is still full for a runtime.

The pools, and the runtimes whose creates draw on each:

  • containerd netns pool (containerd engine with the native netns pool): docker (runc) and gVisor share it. Free counts free and prewarmed rows; rows held by running or stopped sandboxes, warm-parked containers and slots parked as releasing after a failed teardown are not free. Held to the 1,023 ports of its bridge (aerolvm0) when seeded past them.
  • docker bridge pool (docker engine): docker (runc) and gVisor share the sandbox bridge. The admitter counts it (capacity.SlotLimit), so it costs no store read.
  • Firecracker TAP pool: unallocated /30 slots (warm VMMs hold theirs).
  • WASM instances, only when SB_WASM_MAX_INSTANCES caps them.

Nothing is reported for isolate's egress slots, which are per tenant group rather than per node; an isolate create refuses itself when its group's pool is full.

On a store error it returns the last good answer: a one-off SQLite hiccup must not make the node advertise "no slot limit" or "no slots".

func (*Service) LocalTemplateCatalogInventory added in v0.6.1

func (s *Service) LocalTemplateCatalogInventory(ctx context.Context) ([]string, bool)

LocalTemplateCatalogInventory returns every template row on this worker, including pending and failed rows. It shares the ready-inventory refresh so capacity heartbeats still perform only one small SQLite query per TTL.

func (*Service) MarkArtifactCatalogDirty added in v0.6.1

func (s *Service) MarkArtifactCatalogDirty(kind string)

MarkArtifactCatalogDirty records that this node's inventory of a kind changed. It is deliberately cheap — no I/O, no Raft — so every mutation can call it; the reconciler does the publishing.

func (*Service) MarkImported added in v0.2.3

func (s *Service) MarkImported(ctx context.Context, sandboxID string, registryRef string)

MarkImported is the spec write-back invoked by the AutoImportReconciler after a successful import. It flips the replicated spec's distribution mode to `aocr_imported` and points ImageRegistryRef at the new cluster- side ref so any future failover/recreate pulls from the cluster namespace (cluster PAT) instead of the original upstream.

Best-effort by contract: a missing spec, a non-leader UpsertSpec rejection, or a Raft commit error all log+return without surfacing the failure. The bytes are already in AOCR; the original ref still resolves through the mirror; the next mutation will refresh the FSM. This is the AutoImportSpecMutator implementation referenced by the reconciler.

func (*Service) MarkRollbackPending added in v0.6.12

func (s *Service) MarkRollbackPending(ctx context.Context, id string) error

MarkRollbackPending records that the create of id was reported failed and its rollback has not finished. A cluster create's rollback calls it before it destroys, so neither a destroy that fails nor a crash before it can let the row be published. A missing row is nothing to roll back, not an error.

func (*Service) MarkSnapshotCorrupt added in v0.4.1

func (s *Service) MarkSnapshotCorrupt(ctx context.Context, templateID, reason string) error

MarkSnapshotCorrupt transitions a template from ready to unhealthy and, on the first such transition for a given corruption event, kicks an async snapshot rebuild. Subsequent concurrent calls for the same template return nil with no rebuild kicked — the idempotent UPDATE ... WHERE status='ready' is the gating primitive.

reason is logged + persisted to the snapshot_error column so the operator-facing GET /v1/templates/{id} payload carries the human-readable detail.

The method swallows store/rebuild errors after logging — the caller (driver.Create's failure path / warm-spawn notifier) is already returning its own user-facing error. The notification is a best-effort hint to the service layer; failing to honour it leaves the system no worse than today (next corruption observation retries).

func (*Service) MigrateWasmSandbox added in v0.5.3

func (s *Service) MigrateWasmSandbox(ctx context.Context, sandboxID, destDir string) (string, string, error)

MigrateWasmSandbox packages a boundary checkpoint for handoff to a sibling node.

func (*Service) MigrateWasmSandboxToNode added in v0.5.3

func (s *Service) MigrateWasmSandboxToNode(ctx context.Context, sandboxID, targetNodeID string) (resp *WasmMigrateResponse, err error)

MigrateWasmSandboxToNode orchestrates export on the current owner and import on targetNodeID (plans/wasm-runtime.md §4.4).

func (*Service) NormalizeCreateEgressDefault added in v0.6.6

func (s *Service) NormalizeCreateEgressDefault(req *models.CreateSandboxRequest)

NormalizeCreateEgressDefault writes the operator's default policy into a create that says nothing about egress, before placement and before the owner records the spec. The cluster replicates the caller's copy of the request, not the one createSandbox defaults, and a failover replay skips the default so it can't pick up a file edited since; without this a defaulted sandbox would come back open after failover. Placement also has to see a hostname default to send the create to a gateway-ready node.

func (*Service) NormalizeCreateImageDistribution added in v0.2.1

func (s *Service) NormalizeCreateImageDistribution(ctx context.Context, req *models.CreateSandboxRequest) error

NormalizeCreateImageDistribution resolves snapshot aliases and fills the create request's image-distribution metadata before placement decisions are made. The method is intentionally side-effect free: it does not pull, push, or verify image contents. Its result is a fixed point: a request it has already resolved resolves to itself, so callers that run it twice get the same answer and the same authorization.

func (*Service) NoteLocalOwnersChanged added in v0.6.8

func (s *Service) NoteLocalOwnersChanged()

NoteLocalOwnersChanged nudges the flood-mode evaluator after a local create or delete. Non-blocking: a nudge already pending covers this one.

func (*Service) OnEgressOperatorChange added in v0.6.6

func (s *Service) OnEgressOperatorChange(op *operator.Operator)

OnEgressOperatorChange is the watcher's change callback: it refreshes the drift metric and the host-firewall floor, and wakes the profile re-apply pass so an edited or removed org profile reaches this node's sandboxes (§5.10 PC-3; each node re-applies its own, paced like any profile change). The gateway and the WASM workers watch the file themselves (the gateway's heartbeat reports what it applied); the control-port guard follows on the next supervisor tick.

func (*Service) OnKeyRevoked added in v0.6.9

func (s *Service) OnKeyRevoked(closeAll func() int)

OnKeyRevoked registers a closer run when the key is revoked: the SSH gateway's CloseAll. It returns how many sessions it closed.

func (*Service) OpenClusterSecretsForNode added in v0.2.1

func (s *Service) OpenClusterSecretsForNode(ctx context.Context, sandboxID string, redacted models.CreateSandboxRequest, placement cluster.PlacementSecrets, nodeID string) (out models.CreateSandboxRequest, err error)

OpenClusterSecretsForNode resolves a replicated secret handle and merges the decrypted credentials back into a redacted spec. The handle (Ref/Version) is the only carrier — placements never embed sealed bytes.

sandboxID is preferred for the audit event; when empty it is parsed from the incarnation-scoped placement ref. Every open is fenced to that placement lifetime before the provider can resolve retained ciphertext.

func (*Service) OwnerLeaseCheck added in v0.6.12

func (s *Service) OwnerLeaseCheck(id string, epoch uint64, hinted bool, now time.Time) OwnerLeaseVerdict

OwnerLeaseCheck compares a forward's owner epoch with the last authoritative read. A forward that came from a fresh edge lookup (hinted false) may lean on a read as old as the lease itself: the edge's lookup is the fresh evidence and the read only has to agree with it. A cache-routed forward (hinted true) needs a read under ownerLeaseHintedMaxAge.

func (*Service) OwnershipFenced added in v0.6.12

func (s *Service) OwnershipFenced(id string) bool

OwnershipFenced reports whether id is held by the ownership fence. It is what clusterroute.Wrap asks before serving a sandbox locally.

func (*Service) PauseSandbox added in v0.6.7

func (s *Service) PauseSandbox(ctx context.Context, id string) (*models.Sandbox, error)

PauseSandbox stops a running sandbox inside the lifecycle boundary (lifecycle_boundary.go) so the state check and the stop are one critical section. An already-stopped sandbox is a no-op (a retried pause is idempotent); any other state is ErrSandboxNotRunning. This is the single definition of "can this be paused" the E2B pause and Runloop suspend handlers used to each carry (D7).

func (*Service) PortAccess added in v0.6.12

func (s *Service) PortAccess(ctx context.Context, id string, port int) (PortAccess, error)

PortAccess answers IsSandboxStarted, the port's gate and the sandbox's token not-before from one read: the warm cache when it holds this sandbox's port list, else the sandbox row (which carries its exposed ports and the not-before). It adds no store read to the ingress path. An entry cached without a port list (warmCacheSet) is never trusted for the gate, so a gated port can never read as open from it, and a revoke drops the entry (RevokeSandboxTokens), so a revoked token is never let in by a not-before cached before it.

func (*Service) PruneSecretAudit added in v0.6.1

func (s *Service) PruneSecretAudit(ctx context.Context) error

PruneSecretAudit drops audit state older than SB_SECRET_AUDIT_RETENTION_DAYS. The JSONL rewrite is skipped when the file sink is unavailable, but retained post-delete ACL metadata is still pruned.

Zero retention means "retain nothing beyond the crash buffer": post-delete ACL rows go at the next sweep and the JSONL keeps one day for the export tailer to catch up. It never means forever.

func (*Service) PushWasmModule added in v0.5.3

func (s *Service) PushWasmModule(ctx context.Context, name, tag, username, token string, data io.Reader) (*models.PushWasmModuleResponse, error)

PushWasmModule is a STATELESS proxy: it validates a BYO .wasm upload (core-wasip1, size) and forwards it to the registry under the caller's own per-tenant credentials, returning the oci:// ref for a later create. The daemon never stores or serves the bytes — AOCR is the registry, this is just orchestration convenience for clients without oras/docker.

func (*Service) PutEgressProfile added in v0.6.6

func (s *Service) PutEgressProfile(ctx context.Context, name string, req models.EgressProfileRequest) (*models.EgressProfile, error)

PutEgressProfile creates or replaces one of the caller's profiles. The same body twice is a no-op. A change is checked against the operator's ceiling and against the hostname cap of every sandbox that references the profile, then stored; referencing sandboxes pick it up from the re-apply pass, which this call wakes.

func (*Service) PutIdentityRole added in v0.6.12

func (s *Service) PutIdentityRole(ctx context.Context, team, tag string, req models.IdentityRoleRequest) (*models.IdentityRole, error)

PutIdentityRole puts a role tag on a team's allow list, or changes its minimum role. The same body twice is a no-op (idempotent PUT).

func (*Service) PutTeamQuota added in v0.6.12

func (s *Service) PutTeamQuota(ctx context.Context, team string, req models.TeamQuotaRequest) (*models.TeamQuota, error)

PutTeamQuota creates or replaces a team's quota. The same body twice is a no-op (same generation). Lowering a quota below what the team holds removes nothing: it stops the team from growing until it is back under.

func (*Service) PutWarmPool added in v0.6.12

func (s *Service) PutWarmPool(ctx context.Context, image string, req models.WarmPoolRequest) (*models.WarmPool, error)

PutWarmPool creates or replaces the entry for an image. The same body twice is a no-op. A change reaches this node's pool at once and every other node's within one sync interval.

func (*Service) ReFanoutClusterSecrets added in v0.6.1

func (s *Service) ReFanoutClusterSecrets(ctx context.Context) error

ReFanoutClusterSecrets rebuilds in-memory holder counts from local cluster_secrets rows and asynchronously re-pushes multi-recipient blobs to peers. Call once after cluster attach / ownership replay on worker boot so failover_ready is not stuck false forever after a restart.

func (*Service) ReFanoutClusterSecretsForNodes added in v0.6.1

func (s *Service) ReFanoutClusterSecretsForNodes(ctx context.Context, nodeIDs map[string]struct{}) error

ReFanoutClusterSecretsForNodes is ReFanoutClusterSecrets restricted to the secrets whose recipient set contains one of nodeIDs. A nil/empty set means "every local secret" (boot, ownership replay).

Membership churn uses the restricted form. A flap otherwise makes every node re-push every secret it holds and ask the Raft leader for a placement snapshot per page — at 2,000 nodes that is one member restart turning into a fleet-wide retransmit and a leader thundering herd, to repair copies that were never lost.

func (*Service) RebuildTemplateSnapshot added in v0.4.1

func (s *Service) RebuildTemplateSnapshot(ctx context.Context, templateID string) error

RebuildTemplateSnapshot reruns the snapshot phase against an existing rootfs. Synchronous (the caller — typically kickSnapshotRebuild — provides its own goroutine + timeout). Exported because future phases (snapshot rotation in PR 6-D, operator-triggered repair via an API endpoint) will call it directly.

Precondition: template must exist and be in status=unhealthy. (ready_no_snapshot rebuilds need a different entry point — they failed during build and may not have a valid rootfs path; that's PR 6-D territory.)

State transitions on success: unhealthy → snapshotting → ready. On snapshotter failure: unhealthy → snapshotting → ready_no_snapshot. The intermediate snapshotting state is intentional — it matches the kickTemplateBuild semantics so operators see a uniform state machine on the dashboard.

func (*Service) RecheckServerKey added in v0.6.9

func (s *Service) RecheckServerKey(ctx context.Context) string

RecheckServerKey runs one re-check and acts on its result, which it returns. Only a deliberate signal from the key store counts toward a revoke: a 403, an outage or a retired version never does (§10).

func (*Service) Reconcile

func (s *Service) Reconcile(ctx context.Context) error

func (*Service) ReconcileArtifactCatalog added in v0.6.1

func (s *Service) ReconcileArtifactCatalog(ctx context.Context)

ReconcileArtifactCatalog republishes any kind whose local inventory has moved since its last accepted publication. Called from the maintenance tick and at boot.

func (*Service) ReconcileClusterIngress added in v0.2.1

func (s *Service) ReconcileClusterIngress(ctx context.Context) error

func (*Service) ReconcileSecretDeleteOutbox added in v0.6.1

func (s *Service) ReconcileSecretDeleteOutbox(ctx context.Context) error

ReconcileSecretDeleteOutbox retries durable peer DELETEs after boot / crash and on the periodic ticker. Work is dispatched through the bounded delete fan-out pool so a large outbox cannot serialize the reconciler for minutes.

Without cluster mode there is no transport to the peers these rows name and never will be; see retireStandaloneSecretOutbox for what happens to them.

func (*Service) ReconcileSecretPutOutbox added in v0.6.1

func (s *Service) ReconcileSecretPutOutbox(ctx context.Context) error

ReconcileSecretPutOutbox retries durable create-path peer PUTs left when the in-memory fan-out queue was saturated or an async push partially failed. Work is dispatched through a bounded worker pool mirroring delete reconcile so a large outbox cannot serialize the reconciler for minutes.

func (*Service) ReconstructWakeArmedIfNeeded added in v0.2.4

func (s *Service) ReconstructWakeArmedIfNeeded(ctx context.Context, sandbox *models.Sandbox)

ReconstructWakeArmedIfNeeded is the D1 reconstruction rule from plans/serverless-sandbox-http-wake.md, extended to also self-heal Caddy state. A serverless sandbox can land in `stopped + wake_armed=false` on a node that does not own the arming history — typically after a cluster owner change (the old owner's SQLite wake_armed bit doesn't migrate) or after a daemon restart that lost the in-memory expected-stop bookkeeping and saw a crash event arrive before the row got an armed value. In those cases the conservative default is the same as the involuntary-stop policy: arm wake so the next HTTP request resurrects the sandbox.

It ALSO re-upserts wake routes when wake_armed=true is already set. Caddy's admin-API routes are dynamic and do not survive a Caddy restart (only certs persist via S3 storage), so the daemon must treat wake_armed=true as a goal state to reassert against Caddy on every reconcile pass, not as a witness that routes still exist. Without this, a Caddy restart between auto-stop and the next wake request leaves the wildcard 404 fallback serving requests forever even though the DB looks correct.

Reconstruction is a no-op unless ALL of:

  • cfg.EnableServerless is on
  • sandbox is Lifecycle.Serverless
  • sandbox is in Stopped status
  • the sandbox has at least one exposed port that can receive an inbound wake request or connection

Caddy-first per D5: install wake routes for every exposure before flipping the store bit. If every wake route install fails we leave wake_armed unchanged (false stays false, true stays true) so route state and the bit remain in sync — the next reconcile pass will retry. The store write is skipped when wake_armed is already true so steady-state reconcile passes do not churn the row.

func (*Service) RecordAPICall added in v0.6.8

func (s *Service) RecordAPICall(ctx context.Context, call APICall)

RecordAPICall writes one kind "api" record for call, attributed to the caller in ctx. It never blocks on disk unless call.Durable is set and durable records are configured.

func (*Service) RecordAuthDenied added in v0.6.8

func (s *Service) RecordAuthDenied(ctx context.Context, method, route, reason, source string, count int64)

RecordAuthDenied writes one auth_denied record for a refused credential from source. The API edge rate-limits it to one per source address per minute and passes how many 401s that record stands for.

func (*Service) RecordDeletedSandboxView added in v0.6.7

func (s *Service) RecordDeletedSandboxView(ctx context.Context, sandboxID, facade, ownerRef, viewJSON string) error

RecordDeletedSandboxView durably stores the terminal body a facade should replay for a sandbox it just deleted, so a retried delete or a GET of the gone sandbox answers with it instead of 404 — the durable, cross-restart replacement for a per-process tombstone map (D6). ownerRef is the deleted sandbox's owner; GetDeletedSandboxView scopes reads to it.

func (*Service) RecordFailoverRefused added in v0.6.12

func (s *Service) RecordFailoverRefused(r cluster.FailoverRefusal)

RecordFailoverRefused writes the audit record for a failover the cluster refused (cluster.FailoverRefusalRecorder). The leader calls it from its dead-owner tick, so it never blocks on the disk: Emit queues.

func (*Service) RecordIdentityConfig added in v0.6.8

func (s *Service) RecordIdentityConfig(sum identity.Summary) bool

RecordIdentityConfig writes one identity_config record when sum's hash differs from the last one this node recorded, then remembers sum. It reports whether there is nothing left to do: false means "not recorded yet, call again later", which happens while the cluster audit gate is closed (a member still runs a release that cannot verify the record, §5.8) or when the durable write failed. An unchanged file records nothing. A file changed twice while the gate was closed yields one record from the last recorded file to the current one.

func (*Service) RecordKeyCustody added in v0.6.9

func (s *Service) RecordKeyCustody()

RecordKeyCustody writes a key_custody_changed record when this boot's server-key custody differs from the last boot's, then remembers it. It catches a downgrade to a plaintext key, by `sandboxd key unwrap` or by hand, that would otherwise go unnoticed. The first boot that records custody has nothing to compare with and records nothing.

func (*Service) RecordSSHSession added in v0.6.8

func (s *Service) RecordSSHSession(ctx context.Context, sandboxID, session string)

RecordSSHSession writes the record of an SSH session opened through the gateway (CEO review D3). The key that opened it belongs to the sandbox, not to a person, so the caller is the sandbox key; whoever created the sandbox is found through its create record. session is the toolbox session name it attached to ("" for a one-shot exec).

The gateway calls it for a session on its own node. For a session it bridged to another node the owner calls it, under the request that opened the session there (pkg/api), and the caller names the gateway's node as Via, the way a forwarded call's does.

func (*Service) RecreateSandbox added in v0.2.1

func (s *Service) RecreateSandbox(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets cluster.PlacementSecrets, exposedPorts map[int]cluster.ExposedPortRoute) error

RecreateSandbox satisfies cluster.SandboxRecreator. The cluster owner watcher invokes this for any FSM placement that points to self. If the sandbox already exists locally, we still replay the replicated port intents: a previous recreate attempt may have created the container and then failed while restoring Caddy/L4 ingress.

secrets is the provider handle that can rehydrate the redacted spec; we resolve and re-merge it here via OpenClusterSecretsForNode so the recreated container can pull from the same private registry / mount the same external storage. Two walls historically blocked cross-node open: the sealed cluster_secrets row lived only on the sealing node, and the envelope was recipient-bound to that node alone. Recipient-set sealing plus mandatory peer fan-out fix both. A decrypt failure is fatal to this attempt; ErrRecipientDenied is permanent for this node (the owner watcher must not reassign-churn the fleet).

Port replay tries every port but returns an error when any replay failed so the owner watcher keeps retrying and can eventually reassign the placement. ExposePort is idempotent, so a partial replay is safe to resume.

Only placements whose create spec opted into failover.policy=recreate reach this path; default sandboxes remain non-HA and are orphaned on owner death.

It knows no takeover time (the owner watcher calls RecreateSandboxReport), so a reused local row keeps its tokens_not_before.

func (*Service) RecreateSandboxReport added in v0.5.6

func (s *Service) RecreateSandboxReport(ctx context.Context, id string, spec models.CreateSandboxRequest, secrets cluster.PlacementSecrets, exposedPorts map[int]cluster.ExposedPortRoute, ownerSinceUnixNano int64) (bool, error)

RecreateSandboxReport implements cluster.SandboxRecreateReporter. attempted is false when a watcher tick finds an already-materialized sandbox and all idempotent route replay work is already healthy. This keeps the failover recreate counter from becoming a constant five-second polling rate.

ownerSinceUnixNano is the placement's OwnerSinceUnixNano (0: no takeover known). A local row this node kept from an earlier ownership is reused, not recreated, so it does not get a new row's not-before; its tokens_not_before is raised to the takeover instead (raiseTokensNotBeforeToTakeover).

func (*Service) RedactClusterSecretsConfigured added in v0.6.1

func (s *Service) RedactClusterSecretsConfigured(req models.CreateSandboxRequest) models.CreateSandboxRequest

RedactClusterSecretsConfigured applies the current secret redaction contract.

func (*Service) RegisterSnapshot added in v0.1.7

func (s *Service) RegisterSnapshot(ctx context.Context, snapshot *models.SandboxSnapshot) (*models.SandboxSnapshot, error)

RegisterSnapshot persists a snapshot row whose Image was resolved out-of-band — either a pre-existing registry image the caller supplied by name, or a freshly built local tag produced by the image builder (e.g. the daytona facade's buildInfo path). It does NOT call docker.CreateSnapshot; the image is assumed to already be runnable. Idempotency is by snapshot name; a re-register with matching image is treated as a no-op so SDK retries don't fail. A different image under the same name is a conflict.

The image is authorized as the caller now, the same way a create from it would be: a user token cannot record a pointer to an image it could not start, such as another tenant's snapshot named directly or through registered names of its own.

func (*Service) RegisterSnapshotFromDockerfile added in v0.6.7

func (s *Service) RegisterSnapshotFromDockerfile(ctx context.Context, snapshot *models.SandboxSnapshot, build BuildImageRequest) (*models.SandboxSnapshot, error)

RegisterSnapshotFromDockerfile builds a Dockerfile and registers the result as a snapshot, discarding a fresh build when the registration fails.

func (*Service) RegisterWarmPool added in v0.6.12

func (s *Service) RegisterWarmPool(p warmPoolApplier, runtime string)

RegisterWarmPool hands a local warm pool to the entry sync. Call it before StartWarmPoolSync; a nil pool is ignored.

func (*Service) RekickUnhealthyTemplatesAtStart added in v0.4.1

func (s *Service) RekickUnhealthyTemplatesAtStart(ctx context.Context)

RekickUnhealthyTemplatesAtStart sweeps every firecracker_templates row in status='unhealthy' and re-kicks the snapshot rebuild for each.

Closes the crash-mid-rebuild gap: MarkSnapshotCorrupt flips the row then spawns an async kickSnapshotRebuild goroutine. If sandboxd dies between the row flip and the rebuild completing, the row is stuck unhealthy on restart and no caller re-kicks it — every subsequent create against that template fails with a confusing "template unhealthy" error until an operator intervenes.

Designed to be called once at daemon start, after the snapshotter + CID-allocator wiring has landed (without those, RebuildTemplateSnapshot would refuse with a clear error and the rebuild would never happen). Returns nil and only logs on store errors — a daemon that can't list templates should still start, since the rest of the service may be healthy and the scanner re-runs on the next process restart.

Idempotency: safe under repeated daemon starts. The CAS in MarkTemplateUnhealthy is not re-entered (the row is already unhealthy), and RebuildTemplateSnapshot re-checks the row's state before doing any work, so a row that another caller already recovered between this list and the rebuild kick short-circuits cleanly.

func (*Service) ReleaseExecLedger added in v0.6.7

func (s *Service) ReleaseExecLedger(ctx context.Context, fingerprint string) error

ReleaseExecLedger drops a claim whose command the toolbox refused, so a retry with the same id starts it fresh. It detaches from the caller's context: a client that disconnected mid-submit is exactly when this runs, and the claim must still be freed promptly.

func (*Service) RemoveCustomDomain added in v0.3.1

func (s *Service) RemoveCustomDomain(ctx context.Context, sandboxID, hostname string) error

RemoveCustomDomain detaches a hostname. Order is store-first then Caddy. If the store delete succeeds but the Caddy patch fails, the hostname is gone from the store while Caddy may still serve it until the next reconcile pass converges routing. Cross-sandbox removal returns store.ErrNotFound (the store layer scopes the DELETE to (sandbox, hostname)), so a tenant cannot rip a domain out of someone else's route by guessing the hostname.

func (*Service) RepairLayer4Ready added in v0.5.5

func (s *Service) RepairLayer4Ready(ctx context.Context) error

func (*Service) ReplayClusterOwnership added in v0.2.4

func (s *Service) ReplayClusterOwnership(ctx context.Context) (int, error)

ReplayClusterOwnership pushes local sandbox truth into the cluster placement FSM. It is the local-store -> global-index repair path used at boot and by the periodic reconciler after Docker confirms a container still exists.

func (*Service) ReplayReservations

func (s *Service) ReplayReservations(ctx context.Context)

ReplayReservations re-populates the admitter from persistent state. Without this, after a daemon restart the admitter sees zero reservations and the host can be overcommitted on the first wave of new sandboxes. Destroyed AND stopped sandboxes are skipped — neither holds host CPU/RAM (the stop path releases the slot, and StartSandbox re-Admits on the way back up), so counting them here would re-introduce the overcommit-budget bug we fixed when stop began releasing capacity. Best-effort: a store error is logged, not returned, since admission control degrading to "unaware" is preferable to refusing to boot.

func (*Service) RequestDigests added in v0.6.9

func (s *Service) RequestDigests(purpose string, data []byte) ([]secrets.KeyedDigest, error)

RequestDigests returns keyed digests of a request under the server key ring, the current key's first (secrets.Cipher.Digests). A facade that stores or logs a value derived from a request that carries secrets, such as an idempotency fingerprint over a create's env, uses these instead of a plain hash, which anyone holding the value could test guesses against. Every node that shares the server key derives the same digest.

func (*Service) RequestTemplateRebuild added in v0.4.1

func (s *Service) RequestTemplateRebuild(ctx context.Context, templateID string) (*models.Template, error)

RequestTemplateRebuild is the operator-triggered entry point for re-running the snapshot phase of a template. It is the service-layer half of POST /v1/templates/{id}/rebuild.

State semantics:

  • ready: delegate to MarkSnapshotCorrupt(reason="operator-requested"). The store-side CAS (UPDATE...WHERE status='ready') makes the transition idempotent under concurrent operator requests — N parallel callers collapse to one ready→unhealthy transition and one rebuild kick. The N-1 callers that lose the race see the post- transition row on the re-read below and get a successful 202.
  • unhealthy: a rebuild is already in flight (or will be re-kicked by RekickUnhealthyTemplatesAtStart on next daemon restart). Returning the row as-is is the simplest UX — operator polls GET /v1/templates/{id} for the transition back to ready.
  • anything else: ErrTemplateNotRebuildable. See the sentinel's docstring for the reasoning per state.

The 202-with-current-template response shape mirrors the rest of the async template lifecycle (POST /v1/templates also returns 202 with a PENDING row), so SDK callers can reuse their existing "poll until status=ready" code path.

func (*Service) ResealServerKeyRows added in v0.6.9

func (s *Service) ResealServerKeyRows(ctx context.Context) (map[string]int, error)

ResealServerKeyRows moves every value sealed under the server key onto its current ring key (§13 step 3) and returns how many values each key opens afterwards. Each write is a compare-and-swap on the bytes read, so a concurrent write wins and its value is counted under the key it was read on until the next pass looks again: the count never reaches 0 early. With one key in the ring there is nothing to move and nothing is read.

func (*Service) ReservedSecretBinding added in v0.6.1

func (s *Service) ReservedSecretBinding(ctx context.Context, sandboxID string) (binding cluster.PlacementSecrets, err error)

ReservedSecretBinding returns the one leader-confirmed identity used by both legs of an overlapped reserved create. Resolving once before either leg starts prevents a follower-cache race from giving the sandbox row and its sealed credentials different incarnations.

func (*Service) ResizeSandbox

func (s *Service) ResizeSandbox(ctx context.Context, id string, req models.ResizeSandboxRequest) (*models.Sandbox, error)

ResizeSandbox changes a sandbox's CPU, memory or disk.

One resize of a sandbox runs at a time, from the read below to the quota settle at the end (resizeLocks); resizes of different sandboxes still run together. Before, two resizes of one sandbox each read the old size, the later one replaced the earlier one's quota claim while the earlier one was still changing the runtime, and a create could be admitted into room the earlier one then used: a 4-CPU team ended up holding 5.

The team quota is held for the whole resize at the larger of the old and new size (admitTeamQuotaResize) and settled at the size the store shows at the end, so a downsize frees its room and a failed resize gives back only what it took. If the runtime changed but the store write failed, the runtime is put back first: only then is the hold undone. If that fails too the hold stays at the larger size, so the team is never charged less than the sandbox may hold.

func (*Service) ResolveOwnedWasmModule added in v0.6.8

func (s *Service) ResolveOwnedWasmModule(ctx context.Context, id string) (path, digest string, ok bool)

ResolveOwnedWasmModule is the owner-scoped create-time catalogue lookup wired into the module resolver (step 3 of ref resolution). A BYO catalogue id resolves in the booting owner's namespace first, then to an operator-published ("") row, so a tenant cannot boot a sandbox from another tenant's catalogued module by naming its id, and an own row shadows an operator row with the same id. Only "ready" rows with a real path resolve.

func (*Service) ResolvePlatformVolumesForReplication added in v0.5.4

func (s *Service) ResolvePlatformVolumesForReplication(ctx context.Context, req *models.CreateSandboxRequest) error

ResolvePlatformVolumesForReplication rewrites platform-volume references into concrete MountSpecs for the cluster-replicated create spec. That preserves the already-resolved tenant prefix across failover/recreate, where there is no original user auth context to derive the tenant from.

It writes nothing: no volume row, no quota charge, no attachment. Those belong to createSandbox's resolvePlatformVolumes, which also deletes a row it created when the create fails. So a name with no row yet is not an error here. The reserved cluster create calls this BEFORE its local create (the seal leg runs alongside the create and needs the concrete mounts), and on a volume's first use that create is what makes the row. The mount is then derived from firstUseVolumeRow, the same coordinate the create freezes into the new row.

func (*Service) ResolveSandboxIDByName added in v0.1.7

func (s *Service) ResolveSandboxIDByName(ctx context.Context, name string) (string, error)

ResolveSandboxIDByName looks up the caller's sandbox with the given name. Names are unique per owner, so the lookup runs in the namespace a create by this caller would use: a user token's account, or "" for operator/PAT and internal callers. Another owner's name reads as ErrNotFound, never as a conflict, so a user token cannot learn which names other tenants hold. Empty name returns ErrNotFound (handled inside the store).

func (*Service) RestoreByOwner added in v0.5.1

func (s *Service) RestoreByOwner(ctx context.Context, ownerRef string) error

RestoreByOwner restarts the owner's fleet-suspended sandboxes and clears the marker. Only sandboxes carrying the marker are touched — a user-initiated stop during suspension is respected. Start happens before the marker is cleared so a failed start retries next tick (marker preserved); an already-running marked sandbox just has its marker cleared. Idempotent.

func (*Service) ResumeSandbox added in v0.6.7

func (s *Service) ResumeSandbox(ctx context.Context, id string) (*models.Sandbox, error)

ResumeSandbox starts a stopped sandbox inside the lifecycle boundary. An already-started sandbox is a no-op (a retried resume is idempotent); any other state is ErrSandboxNotResumable. This is the single definition of "can this be resumed" the E2B connect and Runloop resume handlers used to each carry (D7).

func (*Service) ResyncEgressGateway added in v0.6.6

func (s *Service) ResyncEgressGateway(ctx context.Context) error

ResyncEgressGateway forces a full Sync (gateway restart, table loss).

func (*Service) RetireNodeStorage added in v0.6.1

func (s *Service) RetireNodeStorage(ctx context.Context, nodeID, actor, reason string) error

RetireNodeStorage records an operator attestation that nodeID's storage was destroyed. Refused while gossip reports the node alive.

func (*Service) RevocableContext added in v0.6.9

func (s *Service) RevocableContext(parent context.Context) (context.Context, context.CancelFunc)

RevocableContext is ctx, also cancelled when the key is revoked: a toolbox stream opened before the revoke holds a token opened before it, so it must end with it (§10). Already revoked, it is cancelled at once.

func (*Service) RevokeAPIToken added in v0.6.8

func (s *Service) RevokeAPIToken(ctx context.Context, id string) error

RevokeAPIToken deletes a token at once. A token already gone is not an error: revoke is idempotent, so a retried or repeated revoke is a 204. The evidence of who revoked it is the api record of the call (T5).

func (*Service) RevokeNodeStorageRetirement added in v0.6.1

func (s *Service) RevokeNodeStorageRetirement(ctx context.Context, nodeID string) (bool, error)

RevokeNodeStorageRetirement withdraws an attestation.

func (*Service) RevokeSandboxTokens added in v0.6.12

func (s *Service) RevokeSandboxTokens(ctx context.Context, sandboxID string) (models.RevokeSandboxTokensResponse, error)

RevokeSandboxTokens refuses every token issued for sandboxID so far: it moves the sandbox's tokens_not_before to the next second and returns it. A token issued after the call works (IssueSandboxToken lifts one issued in the same second up to the not-before). The route is owner-forwarded and needs sandbox.exec, the same as issuing; a sandbox token cannot call it (the edge holds a token to the toolbox).

Idempotent in effect: the store keeps the larger not-before, so a retry or a second call refuses the same tokens plus any issued in between, and never brings one back. Nothing else changes: no route, no other row, no cluster state, so there is nothing to roll back. The cost of the check it arms is no extra read: the toolbox route reads this row on the owner already (enforceOwner), and a gated port reads it with its gate (PortAccess), whose warm-cache entry is dropped here.

func (*Service) RollbackIngressProxyRouting added in v0.6.1

func (s *Service) RollbackIngressProxyRouting(ctx context.Context, fwd HostPortForwarder) error

RollbackIngressProxyRouting undoes a previous boot's routing mode in one Caddy load. It removes the static routes and re-asserts every per-sandbox route through the Caddy writer, then removes the kernel host-port rules (Caddy's tcp-port servers own the ports again). It runs the full boot reconcile inside the batch: that is the one complete re-assert of every route shape.

func (*Service) RotateIdentityKey added in v0.6.12

func (s *Service) RotateIdentityKey(ctx context.Context, ifActive string) (*models.IdentityKeyRotation, error)

RotateIdentityKey makes a new active signing key; the old one stays in the JWKS for two hours (section 4). ifActive, when set, makes the call idempotent under retries: it rotates only while that key is still the active one, and otherwise answers with the current state.

func (*Service) RoundTripToolbox added in v0.5.3

func (s *Service) RoundTripToolbox(ctx context.Context, sandboxID, method, path string, query url.Values, body io.Reader, headers http.Header) (*http.Response, error)

RoundTripToolbox performs an HTTP round trip to the sandbox toolbox. The caller must close the response body: the request stays registered (see TrackToolboxCall) until it does. When the sandbox stops while the request waits on it, the error, or the error reading the body, is ErrSandboxStoppedMidRequest.

func (*Service) RunJSBundleGCLoop added in v0.5.6

func (s *Service) RunJSBundleGCLoop(ctx context.Context, interval time.Duration)

RunJSBundleGCLoop periodically deletes content-addressed bundles that no live isolate sandbox pins and that no catalogue name points at (plans/isolate-runtime.md Phase 2 leftover). Interval <= 0 disables.

func (*Service) SSHSessionsRecordedByOwner added in v0.6.9

func (s *Service) SSHSessionsRecordedByOwner() bool

SSHSessionsRecordedByOwner reports whether the SSH gateway may leave the record of a session it bridges to another node to that node, the owner (cluster.SSHSessionRecordGate). False on a single node, which bridges nothing, and in cluster mode until every live member runs a build that writes it; the gateway then records the session itself, as before.

func (*Service) SandboxNodeAllowlist added in v0.6.9

func (s *Service) SandboxNodeAllowlist() []netrules.NodePort

SandboxNodeAllowlist is what sandboxes on bridges and TAP subnets may still reach on their own node (see the list above), sorted and deduplicated.

func (*Service) SealAndDistribute added in v0.6.1

func (s *Service) SealAndDistribute(ctx context.Context, sandboxID string, req models.CreateSandboxRequest, recipients []string) (cluster.PlacementSecrets, error)

SealAndDistribute seals to recipients, stores locally, and fans out when the sandbox is recreate-HA and len(recipients) > 1. Boot-path note: default creates are unchanged (seal-to-self, no fan-out). HA creates: local seal + optional bounded sync wait for ≥1 peer ACK (SB_SECRET_FANOUT_MIN_ACK_WAIT, default 2s) to shrink GAP-1; remaining peers / retries continue asynchronously. A zero-ACK HA create is retracted. Every caller is a create, so the seal may take the key-store slot kept for creates: a restart storm of opens cannot queue it.

func (*Service) SecretHoldersForSandbox added in v0.6.1

func (s *Service) SecretHoldersForSandbox(ctx context.Context, sandboxID string) (*SecretHolders, error)

SecretHoldersForSandbox reports the recipient set for a sandbox's current incarnation.

store.ErrNotFound (which the API maps to 404) distinguishes the two "no holders" cases that matter: an UNKNOWN sandbox 404s, while a known sandbox with no sealed secret returns an empty holder set. Collapsing them would make a test for "secrets were removed on delete" pass against a typo'd id.

func (*Service) SecretRecipientBackupCount added in v0.6.1

func (s *Service) SecretRecipientBackupCount() int

SecretRecipientBackupCount returns the configured backup count (default 2).

func (*Service) SecretRecipientsForSeal added in v0.6.1

func (s *Service) SecretRecipientsForSeal(sandboxID string) []string

SecretRecipientsForSeal returns the recorded Placement.SecretRecipients when present, otherwise [self]. It is only used by local-only create paths; HA reserved creates use SealAndDistributeReserved's leader-confirmed binding.

func (*Service) SelectReplacementRecipients added in v0.6.1

func (s *Service) SelectReplacementRecipients(sandboxID string, maxBackups int) []string

SelectReplacementRecipients picks owner + up to maxBackups alive worker/mixed members using the same HRW policy as create-time reserve. Empty when cluster identity or live candidates are unavailable.

func (*Service) ServeToolboxReverseProxy added in v0.5.3

func (s *Service) ServeToolboxReverseProxy(ctx context.Context, sandboxID string, w http.ResponseWriter, r *http.Request, path string) error

ServeToolboxReverseProxy forwards a toolbox request to the sandbox toolbox. WASM sandboxes are served in-process via the driver's ToolboxHost.

It returns an error only before anything is written to w. When the sandbox stops while the request waits on it, that error is ErrSandboxStoppedMidRequest (toolbox_calls.go), so the caller answers 409 at once instead of a 502 after the connection times out.

func (*Service) SetAuditExporter added in v0.6.1

func (s *Service) SetAuditExporter(ex controlplane.AuditExporter)

SetAuditExporter installs the off-node audit batch exporter.

func (*Service) SetContainerdRuntime added in v0.5.6

func (s *Service) SetContainerdRuntime(r runtime.Runtime)

SetContainerdRuntime registers the containerd engine driver. Called from pkg/daemon when SB_CONTAINER_ENGINE=containerd.

func (*Service) SetCreatePolicy added in v0.6.12

func (s *Service) SetCreatePolicy(w *createpolicy.Watcher)

SetCreatePolicy wires the create-policy watcher (nil: no file, today's behavior). The daemon runs the watcher's poll loop.

func (*Service) SetDNSResolver added in v0.4.1

func (s *Service) SetDNSResolver(resolver DNSResolver)

SetDNSResolver sets the DNS resolver for the service (used for testing from other packages).

func (*Service) SetDockerAuxClient added in v0.5.6

func (s *Service) SetDockerAuxClient(c *docker.Client)

SetDockerAuxClient wires docker-only auxiliary APIs (built-image GC, live stats) that are not part of the EventsSource seam.

func (*Service) SetEgressGateway added in v0.6.6

func (s *Service) SetEgressGateway(api egress.API, bridges func(context.Context) []egress.Bridge)

SetEgressGateway wires the gateway client and the bridge discovery. A nil api leaves the feature off (Noop).

func (*Service) SetEgressGatewayRestarter added in v0.6.9

func (s *Service) SetEgressGatewayRestarter(fn func(context.Context) error)

SetEgressGatewayRestarter replaces the restart of a gateway that predates clear_inspect_ca (tests).

func (*Service) SetEgressOperator added in v0.6.6

func (s *Service) SetEgressOperator(w *operator.Watcher)

SetEgressOperator wires the operator-file watcher (nil: no file, today's behavior). The daemon runs the watcher's poll loop.

func (*Service) SetEgressSelfTest added in v0.6.6

func (s *Service) SetEgressSelfTest(pn egress.ProbeNet)

SetEgressSelfTest wires the probe network builder. nil (tests, non-Linux) skips the self-test.

func (*Service) SetEventsSource added in v0.5.6

func (s *Service) SetEventsSource(src docker.EventsSource)

SetEventsSource wires the daemon event monitor + netstats PID lookup seam.

func (*Service) SetExecLedgerTarget added in v0.6.7

func (s *Service) SetExecLedgerTarget(ctx context.Context, fingerprint, target string) error

SetExecLedgerTarget records a plain execution's outcome on the still-pending row, before its session is torn down, so the exit status is never the session's only copy. Unlike CompleteExecLedger it does not detach: the watcher that calls it owns the context's lifetime.

func (*Service) SetFirecrackerRuntime added in v0.4.1

func (s *Service) SetFirecrackerRuntime(r runtime.Runtime)

SetFirecrackerRuntime registers the second runtime driver. Called by main.go after construction, only when cfg.EnableFirecracker is true. Passing nil clears the registration (used by tests). Once set, the CreateSandbox dispatch routes runtime="firecracker" requests through this driver instead of returning a "not enabled" error.

Why a setter rather than a New() parameter: the existing service.New signature is consumed by ~20 test files, and growing it to carry a nil-able second runtime would force a fan-out edit across every one. A setter keeps the constructor stable and confines the wire-up to main.go where the daemon-config flag actually lives.

func (*Service) SetFleetAdmitter added in v0.5.1

func (s *Service) SetFleetAdmitter(a controlplane.Admitter)

SetFleetAdmitter installs the managed create-gate. The managed daemon calls this once at boot from the control-plane provider; the open-source build never does, so fleetAdmitter stays nil and createSandbox skips the gate.

func (*Service) SetIPv6Fence added in v0.6.9

func (s *Service) SetIPv6Fence(f IPv6Fence)

SetIPv6Fence wires the fence the create gate asks. The daemon calls it after the fence's first sync, before the API serves.

func (*Service) SetIdempotentRequestTarget added in v0.6.7

func (s *Service) SetIdempotentRequestTarget(ctx context.Context, scope, fingerprint, targetID string) error

SetIdempotentRequestTarget stores a pending row's result without moving it to ready (see the store method).

func (*Service) SetImageBuilder added in v0.6.7

func (s *Service) SetImageBuilder(builder ImageBuilder)

SetImageBuilder wires the engine's image builder. A daemon without one answers every build with ErrBuilderUnavailable.

func (*Service) SetIsolateBundleStore added in v0.5.6

func (s *Service) SetIsolateBundleStore(bundleStore *jsbundle.Store)

SetIsolateBundleStore registers the content-addressed bundle store. Called from pkg/daemon when cfg.EnableIsolate is true (the same store instance the isolate driver's resolver reads).

func (*Service) SetIsolateRuntime added in v0.5.6

func (s *Service) SetIsolateRuntime(r runtime.Runtime)

SetIsolateRuntime registers the V8-isolate driver. Called from pkg/daemon when cfg.EnableIsolate is true.

func (*Service) SetKeyStoreWrapper added in v0.6.9

func (s *Service) SetKeyStoreWrapper(w secrets.DataKeyWrapper)

SetKeyStoreWrapper hands the service the key-store client the daemon built before the server key was opened, so one client serves every use.

func (*Service) SetNetworkLimits added in v0.1.7

func (s *Service) SetNetworkLimits(ctx context.Context, id string, bytesInLimit, bytesOutLimit int64) (*models.NetworkUsage, error)

SetNetworkLimits writes new caps (0 = unlimited) and re-evaluates the quota state. If the new limit moves the sandbox back under-quota, the matching ingress/egress block is cleared. If it leaves the sandbox over, the matching block is (re-)applied.

Egress clears are conditional: NetworkBlockAll uses the same DOCKER-USER row as the quota egress block, so we must not lift it here when the operator's blanket egress block is still on. See pkg/docker/netrules commentary on the shared rule.

func (*Service) SetOwnerLeaseForTest added in v0.6.12

func (s *Service) SetOwnerLeaseForTest(owned map[string]uint64, at time.Time)

SetOwnerLeaseForTest installs an owned-epoch reading taken at at, for tests outside this package that exercise the router's fast path.

func (*Service) SetPKCS11Helper added in v0.6.9

func (s *Service) SetPKCS11Helper(w secrets.DataKeyWrapper)

SetPKCS11Helper hands ConfigureSecretProvider the running PKCS#11 helper (pkg/daemon starts it: the vendor library never loads into sandboxd).

func (*Service) SetTemplateBuilder added in v0.4.1

func (s *Service) SetTemplateBuilder(b TemplateBuilder)

SetTemplateBuilder is the bootstrap-time wiring hook called once from main.go after the daemon has constructed pkg/oci.Builder. Idempotent: passing nil disables template create (the handler returns 503) without tearing down existing template rows.

func (*Service) SetTemplateCIDAllocator added in v0.4.1

func (s *Service) SetTemplateCIDAllocator(a TemplateCIDAllocator)

SetTemplateCIDAllocator wires the per-template CID reservation seam. Nil disables the snapshot phase (same effect as a nil snapshotter): without a reserved CID the snapshot would clash with the next sandbox slot allocation. The setter is separate so main.go can wire the two independently — handy when one of them is being swapped out.

func (*Service) SetTemplateSnapshotter added in v0.4.1

func (s *Service) SetTemplateSnapshotter(snap TemplateSnapshotter)

SetTemplateSnapshotter wires the Phase 3 snapshot-capture seam. Nil disables the second build phase — new templates still get a rootfs but land in status=ready_no_snapshot, and CreateSandbox cold-boots from them. Useful for hosts where the snapshotter is misbehaving.

func (*Service) SetUsageReporter added in v0.5.1

func (s *Service) SetUsageReporter(r controlplane.Reporter)

SetUsageReporter installs the control-plane usage reporter. The managed daemon calls this once at boot; the open-source build never does, so usageReporter stays nil and every emit path is a no-op that allocates nothing.

func (*Service) SetVolumeReclaimer added in v0.5.4

func (s *Service) SetVolumeReclaimer(r VolumeReclaimer)

SetVolumeReclaimer installs the backend reclaimer. Called once during daemon wiring before the worker starts; nil disables byte reclaim.

func (*Service) SetWasmModuleResolver added in v0.5.3

func (s *Service) SetWasmModuleResolver(r WasmModuleResolver)

SetWasmModuleResolver wires the pkg/wasmmod.Resolver used by CreateWasmModule and lazy create paths.

func (*Service) SetWasmRuntime added in v0.5.3

func (s *Service) SetWasmRuntime(r runtime.Runtime)

SetWasmRuntime registers the WASM driver. Called from pkg/daemon when cfg.EnableWasm is true.

func (*Service) SetWasmWarmPool added in v0.5.5

func (s *Service) SetWasmWarmPool(p WasmWarmPoolNotifier)

SetWasmWarmPool wires the warm pool into module registration.

func (*Service) SetWitness added in v0.6.1

func (s *Service) SetWitness(w controlplane.Witness)

SetWitness installs the control-plane audit witness used to ship hash-chain heads off-node. Open-source builds leave this nil/noop.

func (*Service) SnapshotPushReconciler added in v0.2.4

func (s *Service) SnapshotPushReconciler() *SnapshotPushReconciler

SnapshotPushReconciler exposes the reconciler so main.go can drive it from a ticker. Returns nil when snapshot push is disabled — the ticker wrapper should no-op cleanly in that case.

func (*Service) StartAPITokens added in v0.6.8

func (s *Service) StartAPITokens(ctx context.Context) error

StartAPITokens readies the token index when identity is on: it loads a single node's tokens into memory, or starts the mirror on a cluster node that holds no Raft state, and starts the expiry sweep. With identity off it does nothing: no load, no mirror, no polls (L8). Call it after AttachCluster and before the API server is built.

Boot cost: one SELECT of api_tokens on a single node; nothing on a Raft server; one background goroutine on any other cluster node. Nothing runs on CreateSandbox.

func (*Service) StartAuditIngestServer added in v0.6.1

func (s *Service) StartAuditIngestServer(ctx context.Context) error

StartAuditIngestServer binds 127.0.0.1:SB_AUDIT_INGEST_PORT (0 = ephemeral) and publishes only SB_AUDIT_INGEST_PORT. The master signing token remains in the daemon; workers receive sandbox-scoped capabilities over IPC.

func (*Service) StartBuiltImageGC added in v0.1.7

func (s *Service) StartBuiltImageGC(ctx context.Context)

StartBuiltImageGC launches the periodic janitor that removes locally-built images (BuiltImageNamespace, i.e. "aerolvm-build/*") that are no longer referenced by any active sandbox AND were created more than the configured TTL ago. Without this, two failure modes leak images forever:

  • POST /v1/images/build called standalone (no follow-up CreateSandbox).
  • Build succeeded, CreateSandbox failed AND the daytona facade's inline rollback couldn't reach the daemon (e.g. server-side panic, dropped connection between build success and rollback call).

The TTL keeps the janitor from racing the dominant build+create flow: an image built moments ago must clear ImageBuildGCTTL before it's eligible, so a transient network hiccup between build and create can't have the janitor yanking an image the client is about to consume.

No-op if ImageBuildGCEnabled is false or ImageBuildGCInterval <= 0.

func (*Service) StartCaddyCoalescer added in v0.2.4

func (s *Service) StartCaddyCoalescer(ctx context.Context)

StartCaddyCoalescer starts the periodic-drain goroutine. Called once from cmd/sandboxd/main.go after svc.New on worker nodes. Flush-only callers (installHTTPPortRoute today) don't strictly need this — Flush drives its own drain — but the ticker is the safety net for any future Enqueue (fire-and-forget) callsites and for ops that get stranded when a Flush caller's ctx cancels mid-drain.

func (*Service) StartClusterIngressReconcile added in v0.2.1

func (s *Service) StartClusterIngressReconcile(ctx context.Context)

func (*Service) StartDiskLimitWatch added in v0.6.9

func (s *Service) StartDiskLimitWatch(ctx context.Context)

StartDiskLimitWatch runs the disk watchdog every SB_DISK_LIMIT_CHECK_INTERVAL; zero leaves it off.

func (*Service) StartEventMonitor

func (s *Service) StartEventMonitor(ctx context.Context)

StartEventMonitor launches the Docker event consumer goroutine. It is the realtime counterpart to Reconcile() — when a container dies or is destroyed out-of-band, this loop updates the DB and tears down routes within ~1s instead of waiting for the next reconcile tick. An OOM kill inside a container is recorded, not treated as a stop (sandbox_oom.go).

func (*Service) StartIngressProxyRouting added in v0.6.1

func (s *Service) StartIngressProxyRouting(ctx context.Context, opts IngressRoutingOptions) (*IngressRouting, error)

StartIngressProxyRouting engages the routing mode's in-memory half. It returns (nil, nil) when the flag is off. On error nothing is left running and the Caddy writer stays in place.

func (*Service) StartKeyCustody added in v0.6.9

func (s *Service) StartKeyCustody(ctx context.Context)

StartKeyCustody runs the key-custody loop until ctx ends (eng D2: one owner of the server key's lifecycle). Every SB_SECRET_PROVIDER_RECHECK_INTERVAL it re-reads the server-key ring, so a `sandboxd key rotate` step reaches the node without a restart (§13): with wrapped custody as part of the re-check that unwraps the file again, with custody=file by reading the ring file. While a rotation is in progress the same tick reseals rows onto the current key. With any external provider it also runs the rewrap sweep now (the API is already serving) and then every hour after a random start delay of up to an hour, so a fleet restarted together does not sweep in lockstep (eng D19).

The re-check has a loop of its own: a sweep or reseal pass can take minutes (a key-store call per stale row, a hung call up to its timeout), and a revoke must not wait for one. The passes share a second loop, one at a time; each re-check asks it for a reseal pass.

func (*Service) StartL4RedirectListener added in v0.6.1

func (s *Service) StartL4RedirectListener(ctx context.Context, addr string) (net.Listener, error)

StartL4RedirectListener serves raw-TCP host ports the kernel can't forward directly (plans/ingress-proxy-routing.md §3.5): stopped/wake sandboxes and WASM/isolate loopback mediators. hostport installs "-j REDIRECT --to-ports <this listener>" for them.

A redirected connection carries no PROXY header, which the Caddy-fed wake listener relied on. The original host port is read from the kernel with SO_ORIGINAL_DST instead. A connection made to this listener directly (not redirected) reports this listener's own port, matches no exposure, and is closed.

func (*Service) StartL4WakeProxy added in v0.2.4

func (s *Service) StartL4WakeProxy(ctx context.Context) error

StartL4WakeProxy starts the loopback TCP wake listener used by raw-TCP serverless routes. TLS-SNI wake uses per-exposure Unix sockets installed by ensureTLSWakeListener, but those listeners are also closed from this context.

func (*Service) StartLifecycleSweep

func (s *Service) StartLifecycleSweep(ctx context.Context)

func (*Service) StartLiveUsageSampler added in v0.5.1

func (s *Service) StartLiveUsageSampler(ctx context.Context)

StartLiveUsageSampler launches the opt-in live CPU/memory sampler if, and only if, all of these hold: an interval is configured (> 0), a usage reporter is wired (managed build), and a Docker client is available to query. Under the open-source build (no reporter) or the default config (interval 0) it returns immediately and starts no goroutine — so this is genuinely off unless an operator opts in on a managed node.

The sampler is deliberately additive to the reserved axes: reserved gives the provisioned envelope from the row (no I/O), live gives measured consumption from one cheap stats read per running Docker sandbox per tick. Firecracker sandboxes are skipped (the Docker stats endpoint doesn't know them); their reserved axes still flow.

func (*Service) StartOwnershipLease added in v0.6.12

func (s *Service) StartOwnershipLease(ctx context.Context)

StartOwnershipLease begins the lease and runs its two loops: an authoritative read every lease/3, and an expiry check every lease/15 (a local clock comparison; it touches the runtime only when the lease has run out). A no-op in single-node mode, and when already started.

func (*Service) StartPendingImageGC added in v0.3.1

func (s *Service) StartPendingImageGC(ctx context.Context)

StartPendingImageGC launches the periodic janitor that removes images scheduled by destroy paths (DestroySandbox, reconcile destroyed-branch, handleDestroyEvent). It piggybacks on ImageBuildGCEnabled / Interval / TTL: same enable switch, same cadence, same TTL — so an operator who wants to disable image GC entirely flips one knob, and tightening the TTL applies to both pulled images (this janitor) and built images (StartBuiltImageGC). No-op when disabled or Interval <= 0.

func (*Service) StartReconcileLoop

func (s *Service) StartReconcileLoop(ctx context.Context)

func (*Service) StartSandbox

func (s *Service) StartSandbox(ctx context.Context, id string) (*models.Sandbox, error)

StartSandbox starts a sandbox inside the lifecycle boundary (lifecycle_boundary.go): it waits for any start, stop or restart of the same sandbox already in flight, and a stop that comes during it waits for it in turn.

func (*Service) StartSecretDeleteOutboxReconcile added in v0.6.1

func (s *Service) StartSecretDeleteOutboxReconcile(ctx context.Context)

StartSecretDeleteOutboxReconcile runs the secret-lifecycle maintenance loop: periodic peer-delete and peer-put retries so offline recipients are not permanently abandoned after the boot pass, tombstone and audit-ACL pruning, holder possession refresh, and the stale-ciphertext retirement scan. When membership gains a newly-alive node, reconcile runs immediately and secrets are re-fanout so holders re-ACK after rejoin.

It runs in every mode. Standalone, the cluster-only steps are no-ops and the outbox/ciphertext steps retire what a node without peers can never finish (retireStandaloneSecretOutbox, runSecretRetirementScan) — the four lifecycle tables must never depend on cluster mode being on to be garbage collected.

func (*Service) StartTemplateGC added in v0.4.1

func (s *Service) StartTemplateGC(ctx context.Context)

StartTemplateGC launches the periodic janitor that drops unreferenced templates. Shape mirrors StartBuiltImageGC verbatim: ticker + context-cancellation, per-tick bounded sweep context, best-effort log on failures. No-op when SB_ENABLE_FIRECRACKER=false (templates can't exist on those nodes) or when the GC enable knob is off — the latter gives operators a way to pause cleanup during a forensic window without recompiling.

func (*Service) StartVolumeReclaim added in v0.5.4

func (s *Service) StartVolumeReclaim(ctx context.Context)

StartVolumeReclaim launches the backend-reclaim janitor that drains the pending_volume_deletions ledger. No-op when platform volumes are disabled, no reclaimer was wired, or the interval is non-positive — the metadata delete path is unaffected either way, only byte reclaim pauses.

func (*Service) StartWarmPoolSync added in v0.6.12

func (s *Service) StartWarmPoolSync(ctx context.Context)

StartWarmPoolSync applies the entries to this node's pools now, then on every change: at once after a write on this node, and, in a cluster, within one poll of a write anywhere. A node with no warm pool skips it.

func (*Service) StartWasmDurablePushSweep added in v0.5.3

func (s *Service) StartWasmDurablePushSweep(ctx context.Context)

StartWasmDurablePushSweep retries AOCR push for durable checkpoints missing registry metadata.

func (*Service) StartWasmModuleGC added in v0.5.3

func (s *Service) StartWasmModuleGC(ctx context.Context)

StartWasmModuleGC launches the periodic wasm_modules catalogue janitor.

func (*Service) StartWasmPeriodicCheckpoint added in v0.5.3

func (s *Service) StartWasmPeriodicCheckpoint(ctx context.Context)

StartWasmPeriodicCheckpoint launches optional boundary checkpoint sweeps.

func (*Service) StopAuditIngestServer added in v0.6.1

func (s *Service) StopAuditIngestServer()

StopAuditIngestServer shuts down the loopback ingest listener.

func (*Service) StopByOwner added in v0.5.1

func (s *Service) StopByOwner(ctx context.Context, ownerRef string) error

StopByOwner stops every currently-running sandbox owned by ownerRef, marking each one fleet-suspended first so RestoreByOwner knows to bring it back. A sandbox already stopped (by its user, or by a prior tick) is left untouched, so repeated calls do not thrash. Errors are collected and returned joined so the enforcement loop retries the whole owner next tick; the marker-then-stop order means a stop that fails still leaves the sandbox marked and running, and the retry re-attempts it.

func (*Service) StopCaddyCoalescer added in v0.2.4

func (s *Service) StopCaddyCoalescer()

StopCaddyCoalescer drains any pending op and exits the Run goroutine. No-op when StartCaddyCoalescer was never called — caddyCoalescer.Stop blocks on c.done which is only closed by Run, so calling it without a prior Start would hang shutdown.

func (*Service) StopSandbox

func (s *Service) StopSandbox(ctx context.Context, id string) (*models.Sandbox, error)

StopSandbox is the operator-initiated stop (API surface, manual). It is a thin wrapper over stopSandboxInternal that pins the stop mode to manual: wake_armed is always cleared, so a serverless sandbox stopped via this path stays down until the operator explicitly starts it again. See serverless.go for the full wake-arming policy. It runs inside the lifecycle boundary: a restart on failure already starting the sandbox finishes first, and this stop then stops it.

func (*Service) StorageObligations added in v0.6.1

func (s *Service) StorageObligations(ctx context.Context) ([]cluster.StorageObligationView, error)

StorageObligations is the operator view of open decommission jobs. Served from the local FSM on a server, or by one call to a server from an agent — never by asking the owners.

func (*Service) SuperviseEgressFloor added in v0.6.9

func (s *Service) SuperviseEgressFloor(ctx context.Context, interval time.Duration)

SuperviseEgressFloor keeps the floor in place: it retries a runtime where installing it failed every interval, and every nodeGuardReassertInterval checks that it is still whole on the others and puts it back if a host firewall reload or a flush took it away.

func (*Service) SuperviseEgressGateway added in v0.6.6

func (s *Service) SuperviseEgressGateway(ctx context.Context, interval time.Duration)

SuperviseEgressGateway re-runs the gateway bootstrap whenever the latch is down (at startup before the gateway is up, and after its event stream breaks on a gateway restart), and re-attaches held sandboxes once it is back. The lazy retry on create is not enough in cluster mode, where placement stops sending gateway-mode creates to a node that isn't ready, so nothing else would bring it back. One atomic load per tick while ready; logs only when the failure changes.

func (*Service) SuperviseEgressProfiles added in v0.6.6

func (s *Service) SuperviseEgressProfiles(ctx context.Context)

SuperviseEgressProfiles re-applies changed egress profiles to this node's sandboxes until ctx ends: each owner re-applies its own referencing sandboxes, with no cross-node calls beyond reading the profiles (D21).

func (*Service) SuperviseIdentityTokens added in v0.6.12

func (s *Service) SuperviseIdentityTokens(ctx context.Context)

SuperviseIdentityTokens is the refresher (D1.7): every five minutes it re-pushes the token of every running sandbox on this node with less than thirty minutes left. Each node refreshes only its own sandboxes.

func (*Service) SuperviseNodeGuard added in v0.6.9

func (s *Service) SuperviseNodeGuard(ctx context.Context, interval time.Duration)

SuperviseNodeGuard keeps the guard in place: it installs it on a runtime where it failed, re-installs it when the allowlist changes (the gateway came up, a resolver changed), and re-asserts it every nodeGuardReassertInterval in case a host firewall reload or a flush took it away.

func (*Service) SweepExpiredAPITokens added in v0.6.8

func (s *Service) SweepExpiredAPITokens(ctx context.Context) int

SweepExpiredAPITokens deletes tokens that expired more than 30 days ago: on a single node from SQLite and memory, in a cluster through Raft from the leader only. It returns how many went.

func (*Service) SweepKeys added in v0.6.9

func (s *Service) SweepKeys(ctx context.Context) (int, error)

SweepKeys moves this node's stale rows and its server-key file onto the sealing provider's current key and returns how many stay stale. Each node sweeps only its own copies, every node reaching the same key store, so there is no fan-out. A row is read without a lock, the key store is called with no lock held, and only the conditional write takes the sandbox's secret lock (compare-and-swap): a reseal, a delete or a newer peer copy that lands in between wins and the row is skipped (Section 4).

func (*Service) TemplateArtifactPushReconciler added in v0.4.1

func (s *Service) TemplateArtifactPushReconciler() *TemplateArtifactPushReconciler

TemplateArtifactPushReconciler exposes the reconciler so main.go can drive it from a ticker. Returns nil when template push is disabled.

func (*Service) TemplatePushStore added in v0.6.1

func (s *Service) TemplatePushStore(inner TemplateArtifactPushStore) TemplateArtifactPushStore

TemplatePushStore wraps the raw store the push reconciler writes through so every push-metadata change also invalidates the replicated catalogue.

The reconciler is wired independently of the Service and writes RegistryRef, PushDigest and PushState straight to SQLite. Those fields are part of the row the catalogue publishes, so without this seam a successful push left the fleet-visible DTO saying "pending" with an empty registry reference — and nothing marked the kind dirty, so no maintenance pass ever repaired it. Wrapping the STORE rather than adding a callback means a future writer on this path is covered by construction.

func (*Service) ToolboxTarget

func (s *Service) ToolboxTarget(ctx context.Context, id string) (ToolboxEndpoint, error)

func (*Service) TouchSandbox

func (s *Service) TouchSandbox(ctx context.Context, id string) error

TouchSandbox bumps last_active_at for id with per-sandbox debounce. The wake-aware ingress proxy and the toolbox/session/runtime proxies call this on every request, so it must be cheap under high per-sandbox RPS — see touch_coalescer.go for the rationale and the debounce window. Callers that need a guaranteed immediate flush (lifecycle transitions, tests) should drop to s.store.Touch directly.

func (*Service) TrackToolboxCall added in v0.6.9

func (s *Service) TrackToolboxCall(ctx context.Context, sandboxID string) (context.Context, func())

TrackToolboxCall registers a toolbox request to sandboxID for its lifetime. The returned context ends with ErrSandboxStoppedMidRequest when the sandbox stops or is destroyed; release must be called once the request is over (it is safe to call more than once). Register before reading the sandbox's row (see the comment at the top of this file).

func (*Service) UnexposePort

func (s *Service) UnexposePort(ctx context.Context, id string, port int) error

func (*Service) UnsealRegistry added in v0.2.1

func (s *Service) UnsealRegistry(ctx context.Context, sandboxID string, sealed []byte) (auth *models.RegistryAuth, err error)

UnsealRegistry decrypts a previously sealed RegistryAuth. Returns nil/nil when the input is empty (no credentials persisted). Exported for the boot-time backfill in cmd/sandboxd that rebuilds CreateSandboxRequest from the persisted Sandbox row. sandboxID is audit-only (ref registry:{id}).

func (*Service) UpdateLifecycle

func (s *Service) UpdateLifecycle(ctx context.Context, id string, l models.Lifecycle) (*models.Sandbox, error)

UpdateLifecycle replaces the lifecycle timers on an existing sandbox. Full-replacement semantics: pass zero in any field to clear that timer. The sweep picks up the new values on its next tick (within ~1 minute), so a tightened deadline can fire as soon as the next sweep runs.

func (*Service) UpdateNetworkLists added in v0.6.6

func (s *Service) UpdateNetworkLists(ctx context.Context, id string, blockAll bool, allowOut, denyOut []string) (*models.NetworkPolicy, error)

UpdateNetworkLists replaces a sandbox's block-all flag and allow and deny lists and keeps its profile references, for facades whose APIs can't express profiles (E2B updateNetwork, D19). The references are read under the same per-sandbox lock as the write, so a concurrent native PUT can't be undone. Block-all on a sandbox that references profiles is a 409: clearing them is the owner's call, through the native API.

func (*Service) UpdateNetworkPolicy added in v0.6.6

func (s *Service) UpdateNetworkPolicy(ctx context.Context, id string, req models.NetworkPolicyRequest) (*models.NetworkPolicy, error)

UpdateNetworkPolicy replaces a sandbox's egress policy live (plans/egress-domain-filtering.md §5.8). Steps, under a per-sandbox mutex: validate with the create grammar and the operator ceiling; in a cluster, commit the patched spec first and strictly (a failure returns 503 and changes nothing, D10); write the store; apply the transition, tightening before loosening. A 2xx means the policy is stored, replicated and live. A stopped container is only stored; Start applies it. Sending the current policy again is a no-op, so the PUT is safe to retry.

func (*Service) UpdateTags added in v0.1.7

func (s *Service) UpdateTags(ctx context.Context, sandboxID string, tags map[string]string) error

UpdateTags replaces sandboxes.tags_json for the given sandbox. Tags are the native key/value bag — facades use it for label-style metadata (Daytona labels, E2B metadata).

func (*Service) UpsertClusterSecretBlob added in v0.6.1

func (s *Service) UpsertClusterSecretBlob(ctx context.Context, blob secrets.SecretBlob, originatorNodeID string) error

UpsertClusterSecretBlob stores a peer-pushed sealed blob without re-sealing. Idempotent (store UPSERT). Used by POST /v1/cluster/internal/secrets.

Rejects blobs whose ref does not match sandbox_id/version, whose declared recipients omit this node, or whose sealed envelope recipients disagree / omit this node — so a compromised tenant token (if it ever reached the handler) cannot poison arbitrary peer rows.

func (*Service) UpsertCompatState added in v0.1.7

func (s *Service) UpsertCompatState(ctx context.Context, sandboxID, facade, stateJSON string) error

func (*Service) UpsertSnapshotAlias added in v0.1.7

func (s *Service) UpsertSnapshotAlias(ctx context.Context, alias models.SnapshotAlias) error

UpsertSnapshotAlias writes an alias onto a snapshot the caller may see. An existing row under the same alias must also point at a visible snapshot, or a user token could retarget (and so take over) another tenant's alias.

func (*Service) UseAuditSinkForTest added in v0.6.8

func (s *Service) UseAuditSinkForTest(sink SecretAuditSink)

UseAuditSinkForTest makes sink this Service's audit writer, for a package that drives the service over HTTP and must read the records it wrote (pkg/api's api-record tests). Call it before anything opens the writer.

func (*Service) ValidateSecretAuditSink added in v0.6.1

func (s *Service) ValidateSecretAuditSink() error

ValidateSecretAuditSink applies the boot policy after New has attempted to open the writer. Strict mode is the production default; the explicit false value remains useful to tests and emergency recovery, while metrics and drop accounting still make the degraded state visible. When an external witness is already installed, also runs ValidateSecretAuditWitness.

func (*Service) ValidateSecretAuditWitness added in v0.6.1

func (s *Service) ValidateSecretAuditWitness() error

ValidateSecretAuditWitness fails closed when a real external witness is installed and the local chain does not contain a witnessed ancestor head. No-op when only the noop witness is present (daemon enforces SB_SECRET_AUDIT_EXTERNAL_WITNESS ⇒ non-noop Witness separately).

func (*Service) VerifySandboxToken added in v0.6.12

func (s *Service) VerifySandboxToken(raw string) (sandboxtoken.Claims, error)

VerifySandboxToken checks a token's signature, claims and validity window with no store read. It does not check the scope against a request: the caller does that with Claims.Allows, because only it knows the request. The error is one of pkg/sandboxtoken's sentinels and never holds the token.

func (*Service) VerifySecretAuditChain added in v0.6.1

func (s *Service) VerifySecretAuditChain(ctx context.Context) (SecretAuditVerification, error)

VerifySecretAuditChain re-verifies every record of the local audit log against its hash chain, on demand. It is O(file) by design and runs one at a time per node (a second caller gets ErrSecretAuditBusy). Page reads no longer do this work; boot, retention, and this call are where the whole chain is proven.

func (*Service) VerifySecretAuditWitness added in v0.6.1

func (s *Service) VerifySecretAuditWitness() (ok bool, localHead, witnessedHead string, err error)

VerifySecretAuditWitness recomputes the local hash chain (never trusting secrets.tip) and checks that an external witness head is an ancestor of the verified tip. Local tip may be ahead of the last ship — that is success.

One pass, O(1) memory: the witnessed head is fetched first and the scan probes for it (and records the newest retention checkpoint's WitnessedThrough) instead of materializing every hash in the file.

func (*Service) WaitForTemplateRebuilds added in v0.6.1

func (s *Service) WaitForTemplateRebuilds(timeout time.Duration) bool

WaitForTemplateRebuilds blocks until every in-flight snapshot rebuild has finished, or until the deadline elapses. Returns false on timeout so a caller can report a stuck rebuild rather than hang.

func (*Service) WakeAwareL4PortTarget added in v0.2.4

func (s *Service) WakeAwareL4PortTarget(ctx context.Context, id string, port int) (string, error)

WakeAwareL4PortTarget resolves a raw TCP upstream, ensuring the sandbox is awake first. It is shared by raw TCP and TLS-SNI wake proxying.

func (*Service) WakeAwarePortTarget added in v0.2.4

func (s *Service) WakeAwarePortTarget(ctx context.Context, id string, port int) (PortEndpoint, error)

WakeAwarePortTarget resolves a sandbox's exposed-port upstream URL, ensuring the sandbox is awake first. Used by the loopback ingress proxy when Caddy forwards a wake-aware HTTP port route. The same sentinels EnsureSandboxAwakeForHTTP returns flow through here unchanged.

func (*Service) WakeAwareToolboxTarget added in v0.2.4

func (s *Service) WakeAwareToolboxTarget(ctx context.Context, id string) (ToolboxEndpoint, error)

WakeAwareToolboxTarget is the entry point every control-plane HTTP proxy (v1 toolbox/sessions, daytona toolbox, e2b runtime) calls in place of ToolboxTarget. It funnels every request through the wake helper so a stopped serverless sandbox with wake_armed=true is resurrected before the proxy attempts to dial the toolbox.

Behavior is exactly ToolboxTarget for: running sandboxes, stopped non-serverless sandboxes, and any case where cfg.EnableServerless is off (rollout-gate contract). The two new sentinels (ErrSandboxManuallyStopped, ErrWakeCircuitOpen) are surfaced upstream so apihttp can map them to 409 and 503+Retry-After:60 respectively.

func (*Service) WantsSecretRecipientFanout added in v0.6.1

func (s *Service) WantsSecretRecipientFanout(req models.CreateSandboxRequest) bool

WantsSecretRecipientFanout reports whether reserve/seal should use a multi-recipient set for this create request.

type SnapshotPushConfig added in v0.2.4

type SnapshotPushConfig struct {
	// Enabled gates the whole feature. When false the snapshot-create
	// path stays byte-identical to today (no push, no state transitions).
	Enabled bool
	// Host is the AOCR registry vhost the daemon pushes to
	// (e.g. `aocr.aerol.ai`). Falls back to ImageDistributionAOCRHost
	// at the wiring layer when MirrorPushHost is unset.
	Host string
	// ClusterID is the AOCR-scoped tenant the snapshot ends up under.
	// Used both as the registry-auth username (by convention; AOCR
	// validates the PAT, not the username) and as the path namespace
	// `cluster/<id>/snapshots/...` (see snapshotAOCRRef).
	ClusterID string
	// PATPath is the file path to the bearer token presented as the
	// Docker registry password. File-sourced so rotation is just a
	// file write; never logged.
	PATPath string
	// TagSuffix is an optional AOCR retention suffix appended at the end of the
	// pushed tag (`latest<arch><suffix>`), e.g. "--ttl-1h" or "--idle-30d".
	// AOCR's reaper parses `--(ttl|idle)-<dur>` off the END of the tag, so
	// `latest--arch-arm64--ttl-1h` expires 1h after push. Empty (the default
	// and prod behavior) pushes a plain `latest` tag, which the reaper keeps
	// latest-only forever. Because "suffixes are real tags" in AOCR, the
	// pull side (DestRefFor) must use the same suffix — both go through
	// snapshotAOCRRef, so they stay consistent. Used by ephemeral clusters
	// (e.g. integration tests) that want their snapshots auto-reaped.
	TagSuffix string
}

SnapshotPushConfig wires the optional background AOCR push to a specific registry endpoint. A zero value disables push; the snapshot path runs exactly as before. main() validates that PATPath, ClusterID, and Host are non-empty when Enabled=true.

Auth reuses the auto-import cluster PAT (same secret, same rotation surface). The PAT file is re-read on every call so an operator can rotate it without restarting sandboxd — there's no in-memory cache.

func (SnapshotPushConfig) Validate added in v0.2.4

func (c SnapshotPushConfig) Validate() error

Validate enforces the consistency rules the snapshot pusher relies on. Returns nil for a disabled config so callers can validate unconditionally.

type SnapshotPushDocker added in v0.2.4

type SnapshotPushDocker interface {
	PushImage(ctx context.Context, req docker.PushImageRequest) (string, error)
}

SnapshotPushDocker is the narrow Docker surface the pusher needs. Pulled out so tests can inject a fake without instantiating a real Docker client, and so the dependency surface of the pusher stays readable at a glance.

type SnapshotPushReconcileStats added in v0.2.4

type SnapshotPushReconcileStats struct {
	Scanned   int
	Succeeded int
	Failed    int
	Skipped   int
	// Retired / RetireFailed count registry copies of deleted snapshots
	// removed, and removals that failed and will be retried.
	Retired      int
	RetireFailed int
	// Coalesced is set when RunOnce found a sweep already running and
	// handed its request to that sweep instead of sweeping itself.
	Coalesced bool
}

SnapshotPushReconcileStats is what RunOnce reports back. Used by tests and by the daemon's sweep log.

type SnapshotPushReconciler added in v0.2.4

type SnapshotPushReconciler struct {
	// contains filtered or unexported fields
}

SnapshotPushReconciler pushes the rows the snapshot-create path flagged `push_state IN ('pending', 'error')` (and claims whose lease ran out) to AOCR. It is the node's one snapshot push worker: Run sweeps on a ticker and whenever Kick wakes it, RunOnce never sweeps alongside another RunOnce, and maxInFlight bounds the pushes in flight on the node — not per sweep — so a recovery storm cannot saturate the local engine or AOCR.

func NewSnapshotPushReconciler added in v0.2.4

func NewSnapshotPushReconciler(pusher *SnapshotPusher, store SnapshotPushStore, logger *slog.Logger, maxInFlight int) *SnapshotPushReconciler

NewSnapshotPushReconciler builds the reconciler. Returns nil if the pusher is nil (snapshot push disabled) so callers can `if r == nil { return }` cheaply. maxInFlight bounds concurrent pushes on the node; pass 0 for the safe default (2 — pushes are I/O-heavy).

func (*SnapshotPushReconciler) Kick added in v0.6.12

func (r *SnapshotPushReconciler) Kick()

Kick wakes the worker started by Run without blocking: the snapshot-create path calls it so a fresh row is pushed now rather than on the next tick. Safe on a nil or never-run reconciler (the wake-up waits for Run).

func (*SnapshotPushReconciler) Run added in v0.6.12

Run is the node's snapshot push worker: it sweeps every interval and on every Kick until ctx ends. report, when non-nil, receives every sweep's outcome. Start it once per reconciler.

func (*SnapshotPushReconciler) RunOnce added in v0.2.4

RunOnce sweeps the push queue and blocks until every push it started has returned. A call that arrives while another sweep runs does not sweep beside it: it asks that sweep to go round once more and returns at once (Coalesced), so the rows it came for are still picked up.

type SnapshotPushResult added in v0.2.4

type SnapshotPushResult struct {
	RegistryRef string
	// Digest is the manifest digest (`sha256:...`) the registry assigned
	// to the pushed tag, as reported by Docker's push stream `aux`
	// payload. Empty when the daemon did not surface one — older daemons
	// or registries that don't return a manifest digest will leave this
	// blank and the reconciler stores "" rather than fabricating a value.
	Digest string
}

SnapshotPushResult is what PushOnce returns on success. The destination ref is what the reconciler writes onto the snapshot row's ImageRegistryRef field — and what cluster placement on other nodes will see when they decide where to start a sandbox from this snapshot.

type SnapshotPushStore added in v0.2.4

type SnapshotPushStore interface {
	ListSnapshotsPendingPushPage(ctx context.Context, after store.SnapshotPushCursor, limit int) ([]*models.SandboxSnapshot, error)
	ClaimSnapshotPush(ctx context.Context, ownerRef, name string) (*models.SandboxSnapshot, time.Time, bool, error)
	CompleteSnapshotPush(ctx context.Context, ownerRef, name string, claimedAt time.Time, mode, registryRef, digest string) (bool, error)
	ReleaseSnapshotPush(ctx context.Context, ownerRef, name string, claimedAt time.Time, state, errMsg string) (bool, error)

	// The registry retirement ledger (internal/store/snapshot_retirement.go).
	ScheduleSnapshotRegistryRetirement(ctx context.Context, r store.SnapshotRegistryRetirement) error
	ListDueSnapshotRegistryRetirements(ctx context.Context, now time.Time, limit int) ([]store.SnapshotRegistryRetirement, error)
	SnapshotRegistryCopyInUse(ctx context.Context, registryRef, digest, ownerRef, name string) (inUse, pushPending bool, err error)
	DeferSnapshotRegistryRetirement(ctx context.Context, registryRef, digest, failure string) error
	DeleteSnapshotRegistryRetirement(ctx context.Context, registryRef, digest string) error
}

SnapshotPushStore is the narrow store surface the reconciler needs. Pulled out as an interface so the reconciler is testable without a real SQLite store and so future store backends can satisfy it without growing the reconciler's dependency surface.

A listed row is only a candidate: the reconciler owns a row from the claim that matches it to the completion or release fenced by that claim's stamp (see internal/store/push_queue.go).

type SnapshotPusher added in v0.2.4

type SnapshotPusher struct {
	// contains filtered or unexported fields
}

SnapshotPusher pushes a locally-committed snapshot image to AOCR. It holds no per-snapshot state; instantiate once at boot and reuse. Safe for concurrent use.

func NewSnapshotPusher added in v0.2.4

func NewSnapshotPusher(cfg SnapshotPushConfig, docker SnapshotPushDocker, logger *slog.Logger) (*SnapshotPusher, error)

NewSnapshotPusher builds a pusher. Returns nil if cfg.Enabled is false so callers can do `if pusher == nil { return }` on the snapshot-create path without an extra config check.

func (*SnapshotPusher) DestRefFor added in v0.2.4

func (p *SnapshotPusher) DestRefFor(ownerRef, snapshotName string) string

DestRefFor returns the AOCR destination ref this pusher would write for owner's snapshot of the given name. Used by callers that need the canonical ref without actually pushing — e.g. cross-node snapshot lookup, where a peer node took the snapshot and we want to pull from AOCR without asking the FSM whether the snapshot exists. Returns "" when the pusher is nil so callers can no-op cleanly when push is disabled.

func (*SnapshotPusher) PushOnce added in v0.2.4

PushOnce tags the snapshot's local image as the AOCR destination ref and pushes via the Docker daemon. Returns the destination ref on success — used by the reconciler to fill ImageRegistryRef and flip ImageDistributionMode to "aocr".

Idempotent: re-pushing the same snapshot to its `:latest` tag is safe; the registry treats the layer set as already-present and the daemon's push completes without error.

The PAT is read fresh from disk on every call so an operator can rotate it without restarting sandboxd. The PAT never appears in the returned error message — only the registry's response body does.

func (*SnapshotPusher) RetireManifest added in v0.6.12

func (p *SnapshotPusher) RetireManifest(ctx context.Context, registryRef, dgst string) error

RetireManifest removes one registry copy of a deleted snapshot: the manifest registryRef@digest, deleted BY DIGEST and never by tag. The tag is the snapshot's name, so it may already hold a newer snapshot of the same name (pushed with a new digest), which must survive. Credentials and host are this pusher's own, the ones the copy was pushed with. A manifest that is already gone counts as removed.

type TemplateArtifactManifest added in v0.4.1

type TemplateArtifactManifest struct {
	SchemaVersion     int    `json:"schema_version"`
	TemplateID        string `json:"template_id"`
	Image             string `json:"image"`
	RootfsSizeBytes   int64  `json:"rootfs_size_bytes"`
	SnapshotSizeBytes int64  `json:"snapshot_size_bytes"`
	SnapshotChecksum  string `json:"snapshot_checksum"`
	SnapshotVsockCID  uint32 `json:"snapshot_vsock_cid"`
	HasOverlay        bool   `json:"has_overlay"`
	MinSizeMiB        int    `json:"min_size_mib,omitempty"`
}

TemplateArtifactManifest is the JSON the puller in PR 6-B.2 reads to reconstruct the Template row on a fresh node. SchemaVersion is the forward-compat guard; everything else mirrors what the consumer needs to insert the row in status=ready without a second round-trip to the origin node.

type TemplateArtifactPullDocker added in v0.4.1

type TemplateArtifactPullDocker interface {
	PullImage(ctx context.Context, ref string, auth *models.RegistryAuth) error
	ExportImageTar(ctx context.Context, ref string) (io.ReadCloser, error)
	RemoveImage(ctx context.Context, ref string) error
}

TemplateArtifactPullDocker is the narrow Docker surface the puller needs. Mirrors TemplateArtifactPushDocker so test fakes look the same; pulled out so unit tests don't need a real *docker.Client.

func NewTemplateArtifactPullDockerAdapter added in v0.4.1

func NewTemplateArtifactPullDockerAdapter(c *docker.Client) TemplateArtifactPullDocker

NewTemplateArtifactPullDockerAdapter is the constructor used by cmd/sandboxd to satisfy the puller's docker dependency without exposing the full *docker.Client API.

type TemplateArtifactPuller added in v0.4.1

type TemplateArtifactPuller struct {
	// contains filtered or unexported fields
}

TemplateArtifactPuller is the consumer-side counterpart to TemplateArtifactPusher (template_push.go:60). It fetches a template's AOCR-pushed OCI image, walks the docker-save outer tar to find the single layer, walks that layer (which is the producer's original artifact tar) to extract manifest.json + rootfs.ext4 + snapshot.{memory,state}, validates the schema version + checksums, and atomically renames the staging dir into place under templatesDir.

Holds no per-template state; instantiate once at boot and reuse. The per-template single-flight latch lives on Service so the lazy EnsureTemplateLocal pattern (§3 of pr-review.md) can collapse concurrent first-creates against a missing-local template into one pull. Mirrors the EnsureLayer4Ready shape exactly.

func NewTemplateArtifactPuller added in v0.4.1

func NewTemplateArtifactPuller(d TemplateArtifactPullDocker, templatesDir string, logger *slog.Logger) (*TemplateArtifactPuller, error)

NewTemplateArtifactPuller mirrors NewTemplateArtifactPusher: returns nil when puller is disabled (templatesDir blank). Caller can then do `if puller == nil { return }` to keep the wire-up path tidy.

func (*TemplateArtifactPuller) PullOnce added in v0.4.1

func (p *TemplateArtifactPuller) PullOnce(ctx context.Context, tpl *models.Template) error

PullOnce fetches+extracts the template's artifacts. Idempotent: a row whose local files already exist is a fast no-op (no network call). Failure modes — all surface to the caller, none are silently swallowed:

  • registry_ref blank: nothing to pull, return descriptive error
  • docker pull failure (registry unreachable, auth, manifest unknown)
  • docker save failure (image went missing between pull and save)
  • tar walk failure / unexpected layout (missing manifest, missing files)
  • schema-version mismatch (refuse: a future producer can't roll back)
  • filesystem error mid-extract (out of space, permission denied)

On any failure, the .partial directory is removed before return so the next call starts clean. Row state is untouched — the puller is purely "make local files match registry_ref", not a state-machine driver.

type TemplateArtifactPushDocker added in v0.4.1

type TemplateArtifactPushDocker interface {
	ImportImage(ctx context.Context, req docker.ImportImageRequest) error
	PushImage(ctx context.Context, req docker.PushImageRequest) (string, error)
	RemoveImage(ctx context.Context, ref string) error
}

TemplateArtifactPushDocker is the narrow Docker surface the TemplateArtifactPusher needs. Mirrors SnapshotPushDocker so test fakes look the same; pulled out for the same reason — to avoid instantiating a real *docker.Client inside unit tests.

type TemplateArtifactPushReconciler added in v0.4.1

type TemplateArtifactPushReconciler struct {
	// contains filtered or unexported fields
}

TemplateArtifactPushReconciler walks rows the build success path flagged `push_state IN ('pending', 'error')` and re-tries them. Like SnapshotPushReconciler it is the node's one template push worker: Run sweeps on a ticker and on Kick, sweeps never overlap, and maxInFlight bounds the pushes in flight on the node, so a recovery storm doesn't saturate the local Docker daemon or AOCR.

func NewTemplateArtifactPushReconciler added in v0.4.1

func NewTemplateArtifactPushReconciler(pusher *TemplateArtifactPusher, store TemplateArtifactPushStore, logger *slog.Logger, maxInFlight int) *TemplateArtifactPushReconciler

NewTemplateArtifactPushReconciler builds the reconciler. Returns nil if the pusher is nil (push disabled) so callers can `if r == nil { return }` cheaply. maxInFlight clamps fan-out per tick; pass 0 for the safe default (2 — template pushes are even heavier than snapshot pushes because the rootfs.ext4 layer can run hundreds of MB).

func (*TemplateArtifactPushReconciler) Kick added in v0.6.12

Kick wakes the worker started by Run without blocking; the build path calls it once a template is marked for push.

func (*TemplateArtifactPushReconciler) Run added in v0.6.12

Run is the node's template push worker: a sweep every interval and on every Kick until ctx ends. report, when non-nil, receives each outcome.

func (*TemplateArtifactPushReconciler) RunOnce added in v0.4.1

RunOnce sweeps the template push queue and blocks until every push it started has returned. A call that arrives while another sweep runs asks that sweep to go round once more and returns at once (Coalesced) instead of sweeping beside it. Templates are few per node, so the queue is read in one go rather than paged.

type TemplateArtifactPushResult added in v0.4.1

type TemplateArtifactPushResult struct {
	RegistryRef string
	Digest      string
}

TemplateArtifactPushResult mirrors SnapshotPushResult. RegistryRef is what the reconciler writes to firecracker_templates.registry_ref; Digest is the OCI manifest digest reported by the daemon's push stream `aux` payload (may be empty for older daemons / registries).

type TemplateArtifactPushStats added in v0.4.1

type TemplateArtifactPushStats struct {
	Scanned   int
	Succeeded int
	Failed    int
	Skipped   int
	// Coalesced: a sweep was already running and took this request over.
	Coalesced bool
}

TemplateArtifactPushStats is what RunOnce reports back. Used by tests and by the metrics-collection wrapper in main.go (PR 6-E).

type TemplateArtifactPushStore added in v0.4.1

type TemplateArtifactPushStore interface {
	ListTemplatesPendingPush(ctx context.Context) ([]*models.Template, error)
	ClaimTemplatePush(ctx context.Context, id string) (*models.Template, time.Time, bool, error)
	CompleteTemplatePush(ctx context.Context, id string, claimedAt time.Time, registryRef, digest string) (bool, error)
	ReleaseTemplatePush(ctx context.Context, id string, claimedAt time.Time, state, errMsg string) (bool, error)
}

TemplateArtifactPushStore is the narrow store surface the template push reconciler needs. Pulled out as an interface so the reconciler is testable without a real SQLite store. Mirrors SnapshotPushStore.

Like the snapshot queue, a listed row is only a candidate: the reconciler owns a template from the claim that matches it to the completion or release fenced by that claim's stamp (internal/store/push_queue.go).

type TemplateArtifactPusher added in v0.4.1

type TemplateArtifactPusher struct {
	// contains filtered or unexported fields
}

TemplateArtifactPusher pushes the three artifact files (rootfs.ext4 + snapshot.memory + snapshot.state) plus a manifest.json to AOCR by (a) wrapping them in a tar, (b) docker-importing the tar as a local image under BuiltImageNamespace, and (c) pushing that image to `<host>/cluster/<id>/templates/<tid>:latest`. The local intermediate tag is removed on best-effort after the push so the built-image GC doesn't have to sweep it later.

Holds no per-template state; instantiate once at boot and reuse. Safe for concurrent use; the reconciler invokes PushOnce from goroutines under the bounded fanout semaphore.

func NewTemplateArtifactPusher added in v0.4.1

func NewTemplateArtifactPusher(cfg SnapshotPushConfig, docker TemplateArtifactPushDocker, templatesDir string, logger *slog.Logger) (*TemplateArtifactPusher, error)

NewTemplateArtifactPusher mirrors NewSnapshotPusher: returns nil if cfg.Enabled is false so callers can do `if pusher == nil { return }`. templatesDir is the absolute path that templateDir(id) was computed against in template.go so the pusher can re-derive the artifact paths from a Template row alone (the row's path columns are also checked for safety).

func (*TemplateArtifactPusher) DestRefFor added in v0.4.1

func (p *TemplateArtifactPusher) DestRefFor(templateID string) string

DestRefFor returns the AOCR destination ref this pusher would write for the given template ID. Used by the future consumer-side pull (PR 6-B.2) to compute the lookup ref without involving the pusher instance directly.

func (*TemplateArtifactPusher) PushOnce added in v0.4.1

PushOnce builds the artifact tar, imports it as a local image, and pushes it to AOCR. Returns the canonical ref + manifest digest on success. The PAT is read fresh on every call so an operator can rotate it without restarting sandboxd — same contract as the snapshot pusher.

Failure modes (caller maps each to push_state=error + push_error):

  • missing artifact file (rootfs.ext4 or snapshot.{memory,state})
  • read error mid-tar (e.g. disk fault)
  • docker import failure (daemon down, malformed tar)
  • docker push failure (registry down, auth rejected)

On any failure the local intermediate tag may or may not exist; RemoveImage is benign on 404/409 so the cleanup runs unconditionally.

type TemplateBuildRequest added in v0.4.1

type TemplateBuildRequest struct {
	ImageRef   string
	OutPath    string
	MinSizeMiB int
	Tag        string
}

TemplateBuildRequest is the small struct passed to TemplateBuilder. Mirrors pkg/oci.BuildRequest by name and shape so the production adapter is a one-field cast; declared locally so the service package does not import pkg/oci (which would pull internal/runtime/firecracker concerns into the service-layer dependency graph). Exported so the adapter in cmd/sandboxd can implement the interface.

type TemplateBuildResult added in v0.4.1

type TemplateBuildResult struct {
	RootfsPath string
	StagingDir string
	SizeBytes  int64
}

TemplateBuildResult is the subset of pkg/oci.Result the template service actually consumes; carried as a separate struct so the interface stays minimal.

type TemplateBuilder added in v0.4.1

type TemplateBuilder interface {
	Build(ctx context.Context, req TemplateBuildRequest) (*TemplateBuildResult, error)
}

TemplateBuilder is the seam the template service uses to invoke the OCI→ext4 pipeline. Production wires this to pkg/oci.Builder via a thin adapter in cmd/sandboxd/main.go; tests stub it to skip real subprocesses. The interface intentionally mirrors pkg/oci's surface — keeping the shapes identical means the adapter is trivial and the service's mental model matches the real builder.

type TemplateCIDAllocator added in v0.4.1

type TemplateCIDAllocator interface {
	AllocateForTemplate(ctx context.Context, templateID string) (uint32, error)
	ReleaseForTemplate(ctx context.Context, templateID string) error
}

TemplateCIDAllocator reserves the per-template host-side AF_VSOCK CID the snapshot is captured with and that every later clone reuses. The production impl wraps *internal/network/tap.Pool's Allocate/Release — the same primitive used for per-sandbox slot allocation — keyed by a synthetic "template:<id>" sandbox id so the pool's per-id PK gives us idempotent reservation. Declared in the service package so template.go does not have to import internal/network/tap.

type TemplateRotationConfig added in v0.4.1

type TemplateRotationConfig struct {
	Interval time.Duration
	MaxAge   time.Duration
}

TemplateRotationConfig pins the operator-tunable knobs the reconciler reads. Cached on the struct rather than re-read off cfg every tick so a config rewrite under a live reconciler doesn't change MaxAge mid-sweep. (Production has no config-rewrite path; this is defensive.)

func (TemplateRotationConfig) IsEnabled added in v0.4.1

func (c TemplateRotationConfig) IsEnabled() bool

IsEnabled returns true when both knobs are non-zero. Rotation is opt-in (Phase 6 PR-E); a daemon that sets Interval but leaves MaxAge zero would tick forever and find nothing — main.go warns at boot.

type TemplateRotationReconciler added in v0.4.1

type TemplateRotationReconciler struct {
	// contains filtered or unexported fields
}

TemplateRotationReconciler walks `ready` templates whose ready_at is older than MaxAge and re-kicks the snapshot rebuild via MarkSnapshotCorrupt(reason="rotation"). That re-uses the Phase 6 PR-A rebuild pipeline — same CAS, same gauge accounting, same recordTemplateUnhealthyGauge transitions — so rotation does not need its own rebuild plumbing.

Like the other reconcilers in this package, the type does not own a goroutine. RunOnce is the unit of work; the ticker lives in main.go. That keeps the type fully unit-testable and lets an operator trigger an immediate rotation sweep from a debug endpoint (future).

func NewTemplateRotationReconciler added in v0.4.1

func NewTemplateRotationReconciler(cfg TemplateRotationConfig, store TemplateRotationStore, marker rotationMarker, logger *slog.Logger) (*TemplateRotationReconciler, error)

NewTemplateRotationReconciler builds the reconciler. Returns nil if rotation is disabled so callers can `if r == nil { return }` cheaply. store/marker must both be non-nil when rotation is enabled.

func (*TemplateRotationReconciler) RunOnce added in v0.4.1

RunOnce processes a single rotation sweep. Each stale `ready` row is passed through MarkSnapshotCorrupt, whose CAS gates the transition — rows that another path already flipped to unhealthy are skipped without error. Unbounded fanout would be unsafe (every rebuild ties up a Firecracker VMM slot for the snapshot capture) so this method iterates serially; rotation is opt-in and not latency-sensitive, so "one rebuild kicked per tick" is fine.

Failure modes:

  • store.ListTemplatesReadyBefore failure: returned to the caller; the ticker wrapper logs and waits for the next tick.
  • MarkSnapshotCorrupt failure on an individual row: logged, stats count it as skipped, the sweep continues with the next row. One bad row should not abort the whole sweep.

type TemplateRotationStats added in v0.4.1

type TemplateRotationStats struct {
	Scanned int
	Marked  int
	Skipped int
}

TemplateRotationStats is what RunOnce reports. Used by tests and by the metrics-aware ticker wrapper in main.go.

type TemplateRotationStore added in v0.4.1

type TemplateRotationStore interface {
	// ListTemplatesReadyBefore returns templates with status='ready'
	// and ready_at older than cutoff. The reconciler is the only caller.
	ListTemplatesReadyBefore(ctx context.Context, cutoff time.Time) ([]*templateRotationCandidate, error)
}

TemplateRotationStore is the narrow store surface the rotation reconciler needs. Mirrors TemplateArtifactPushStore in shape so the reconciler is testable without a real SQLite store, and so a future refactor that splits store responsibilities along sub-interfaces doesn't have to revisit this file.

type TemplateRotationStoreAdapter added in v0.4.1

type TemplateRotationStoreAdapter struct {
	Store templateReadyBeforeQuerier
}

TemplateRotationStoreAdapter wraps the real *store.Store into the narrow TemplateRotationStore interface the reconciler uses. Lives in this package so the rotation reconciler has zero dependencies on internal/store at the type-level (the adapter is a one-method shim; the rest of the file stays unit-testable with a fake).

func (*TemplateRotationStoreAdapter) ListTemplatesReadyBefore added in v0.4.1

func (a *TemplateRotationStoreAdapter) ListTemplatesReadyBefore(ctx context.Context, cutoff time.Time) ([]*templateRotationCandidate, error)

ListTemplatesReadyBefore satisfies TemplateRotationStore by mapping the *models.Template rows the store returns into the minimal candidate shape the reconciler consumes.

type TemplateSnapshotRequest added in v0.4.1

type TemplateSnapshotRequest struct {
	TemplateID    string
	RootfsPath    string
	OutMemoryPath string
	OutStatePath  string
	GuestCID      uint32
	MemoryMB      int
	VCPU          int
}

TemplateSnapshotRequest is the input for the Phase 3 snapshot phase. RootfsPath is the absolute on-disk location of the rootfs.ext4 produced by the rootfs phase; the snapshotter mounts it as the boot drive. Out* paths are absolute file paths under the per-template dir where the snapshotter writes snapshot.memory and snapshot.state. GuestCID is the host-side AF_VSOCK CID the snapshotter will configure the transient template VMM with — re-used by every later clone load so the snapshot state's vsock device stays consistent. MemoryMB/VCPU are the transient VMM's resource budget; they also become the resumed clone's effective resources (Firecracker bakes them into the snapshot state).

type TemplateSnapshotResult added in v0.4.1

type TemplateSnapshotResult struct {
	MemorySizeBytes int64
	StateSizeBytes  int64
	Checksum        string
}

TemplateSnapshotResult is what the snapshot phase returns. Checksum is "sha256:<hex>|sha256:<hex>" (memory|state) so a later load can verify integrity in O(read) without re-decoding the format. The two size fields are reported separately for operator-facing metrics; the store persists their sum.

type TemplateSnapshotter added in v0.4.1

type TemplateSnapshotter interface {
	SnapshotTemplate(ctx context.Context, req TemplateSnapshotRequest) (*TemplateSnapshotResult, error)
}

TemplateSnapshotter is the seam the template service uses to capture the Phase 3 snapshot. Production wires this to *firecracker.Driver via an adapter in cmd/sandboxd/main.go; the driver owns the transient VMM lifecycle because it already owns the VMM/network seams. Tests stub it to skip the real Firecracker subprocess.

type ToolboxEndpoint

type ToolboxEndpoint struct {
	URL   string
	Token string
}

type ToolboxStatusRecorder added in v0.6.12

type ToolboxStatusRecorder struct {
	http.ResponseWriter
	// contains filtered or unexported fields
}

ToolboxStatusRecorder remembers the status a toolbox response wrote. It keeps WebSocket upgrades (Hijack, recorded as 101) and streaming (Flush) working for the reverse proxy.

func NewToolboxStatusRecorder added in v0.6.12

func NewToolboxStatusRecorder(w http.ResponseWriter) *ToolboxStatusRecorder

NewToolboxStatusRecorder wraps w.

func (*ToolboxStatusRecorder) Flush added in v0.6.12

func (r *ToolboxStatusRecorder) Flush()

func (*ToolboxStatusRecorder) Hijack added in v0.6.12

func (*ToolboxStatusRecorder) Status added in v0.6.12

func (r *ToolboxStatusRecorder) Status() int

Status is the written status, or 0 when nothing was written.

func (*ToolboxStatusRecorder) Unwrap added in v0.6.12

Unwrap lets http.ResponseController reach the underlying writer.

func (*ToolboxStatusRecorder) Write added in v0.6.12

func (r *ToolboxStatusRecorder) Write(b []byte) (int, error)

func (*ToolboxStatusRecorder) WriteHeader added in v0.6.12

func (r *ToolboxStatusRecorder) WriteHeader(code int)

type VolumeReclaimer added in v0.5.4

type VolumeReclaimer interface {
	Reclaim(ctx context.Context, backendKind, source string) error
}

VolumeReclaimer deletes the backend bytes of a deleted platform volume. The concrete implementation (pkg/volumes.Reclaimer) shells out to the AWS CLI / a transient NFS mount; the interface keeps the service offline-testable.

type WasmCheckpointPushResult added in v0.5.3

type WasmCheckpointPushResult struct {
	RegistryRef string
	Digest      string
}

WasmCheckpointPushResult is persisted on the sandbox row after a successful AOCR push.

type WasmCheckpointPusher added in v0.5.3

type WasmCheckpointPusher struct {
	// contains filtered or unexported fields
}

WasmCheckpointPusher uploads durable WASM checkpoints to AOCR via ORAS.

func NewWasmCheckpointPusher added in v0.5.3

func NewWasmCheckpointPusher(cfg SnapshotPushConfig, logger *slog.Logger) (*WasmCheckpointPusher, error)

NewWasmCheckpointPusher builds a pusher. Returns nil when cfg.Enabled is false.

func (*WasmCheckpointPusher) DeleteRef added in v0.5.3

func (p *WasmCheckpointPusher) DeleteRef(ctx context.Context, registryRef string) error

DeleteRef removes a tagged WASM checkpoint manifest from AOCR.

func (*WasmCheckpointPusher) DestRefTagged added in v0.5.3

func (p *WasmCheckpointPusher) DestRefTagged(sandboxID, tag string) string

DestRefTagged returns an AOCR ref with an explicit tag.

func (*WasmCheckpointPusher) PullOnce added in v0.5.3

func (p *WasmCheckpointPusher) PullOnce(ctx context.Context, registryRef, incarnationID, dstDir string) error

PullOnce downloads a durable checkpoint from AOCR into dstDir (§4.8 failover), refusing one another lifetime than incarnationID published.

func (*WasmCheckpointPusher) PushOnceTo added in v0.5.3

func (p *WasmCheckpointPusher) PushOnceTo(ctx context.Context, sandboxID, incarnationID, memSnapDir, dest string) (WasmCheckpointPushResult, error)

PushOnceTo uploads memSnapDir to an explicit AOCR ref, as an artifact bound to the sandbox lifetime incarnationID.

type WasmCheckpointStore added in v0.5.3

type WasmCheckpointStore interface {
	DestRefTagged(sandboxID, tag string) string
	PushOnceTo(ctx context.Context, sandboxID, incarnationID, memSnapDir, dest string) (WasmCheckpointPushResult, error)
	// PullOnce restores registryRef into dstDir, refusing a checkpoint that a
	// different lifetime than incarnationID published.
	PullOnce(ctx context.Context, registryRef, incarnationID, dstDir string) error
	DeleteRef(ctx context.Context, registryRef string) error
}

WasmCheckpointStore abstracts AOCR push/pull for durable WASM checkpoints. Production uses *WasmCheckpointPusher; tests inject fakes.

Every operation is bound to a sandbox LIFETIME (its incarnation), not just its id. A sandbox id outlives its lifetimes, and pushes run detached for minutes, so an id-keyed artifact let a dead lifetime's late push replace what the live one — or a failover owner restoring it — would read.

type WasmMigrateRequest added in v0.5.3

type WasmMigrateRequest struct {
	SandboxID    string `json:"sandbox_id"`
	TargetNodeID string `json:"target_node_id"`
}

WasmMigrateRequest is the operator body for POST /v1/cluster/wasm-migrate.

type WasmMigrateResponse added in v0.5.3

type WasmMigrateResponse struct {
	SandboxID       string `json:"sandbox_id"`
	SourceNodeID    string `json:"source_node_id"`
	TargetNodeID    string `json:"target_node_id"`
	CloneGeneration string `json:"clone_generation"`
	CheckpointPath  string `json:"checkpoint_path"`
}

WasmMigrateResponse summarizes a completed cross-node handoff.

type WasmModuleResolver added in v0.5.3

type WasmModuleResolver interface {
	Resolve(ctx context.Context, ref string) (*wasmmod.ResolvedModule, error)
}

WasmModuleResolver resolves a module reference to a local .wasm artifact.

type WasmWarmPoolNotifier added in v0.5.5

type WasmWarmPoolNotifier interface {
	NoteModule(digest, modulePath string)
}

WasmWarmPoolNotifier registers a resolved module as a warm-pool refill target. Production wiring passes *wasmpool.Pool; tests inject a recorder.

Source Files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL