backend

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: Apache-2.0 Imports: 39 Imported by: 0

Documentation

Overview

Package backend provides the interface and client for communicating with provisioning backends.

Fred supports multiple backends, each responsible for provisioning a specific type of resource (e.g., Kubernetes deployments, GPU allocations, VMs). The Router directs leases to the appropriate backend based on exact SKU UUID matching.

Backend Interface

Any backend must implement the Backend interface, which defines twelve operations:

  • Provision: Start async provisioning, backend calls callback URL when done
  • GetInfo: Retrieve connection details for a provisioned resource
  • GetProvision: Retrieve provision diagnostics (status, error, fail count)
  • GetLogs: Retrieve container logs for a provision
  • Deprovision: Clean up resources (must be idempotent)
  • ListProvisions: Return all provisioned resources (for reconciliation)
  • RefreshState: Synchronize in-memory state with actual resources
  • Restart: Restart containers without changing the manifest
  • Update: Deploy a new manifest, replacing containers
  • GetReleases: Return the release (deployment) history for a lease
  • Health: Report whether the backend can accept requests
  • Name: Return the backend's configured name

HTTPClient

The HTTPClient implements the Backend interface for HTTP-based backends. It includes:

  • Circuit breaker (sony/gobreaker) for fault tolerance
  • Configurable timeouts
  • Prometheus metrics for all operations

Router

The Router matches leases to backends using exact SKU UUIDs:

backends:
  - name: docker-1
    skus:
      - "a1b2c3d4-e5f6-7890-abcd-1234567890ab"
  - name: default
    default: true          # Fallback for unmatched SKUs

Callback Protocol

ProvisionRequest and RestoreRequest carry two distinct callback URLs. The callback_url is scoped to that exact operation and is used only for its asynchronous success or failure result. Backends preserve the complete URL, including its operation_id query, byte-for-byte. The separate lifecycle_callback_url carries a distinct typed lifecycle_id and is used for later restart/update completion, autonomous failure, or deprovision observations. Those lifecycle callbacks publish status only; they cannot settle an operation or mutate placement or chain state.

After async provisioning or restoration completes, backends call the exact operation callback URL:

POST {callback_url}
X-Fred-Signature: t=<unix-timestamp>,sha256=<hmac-sha256-hex>
Content-Type: application/json

{"lease_uuid": "...", "status": "success"|"failed", "error": "..."}

Subsequent lifecycle observations use the same request format but are sent to lifecycle_callback_url instead.

The HMAC is computed over the four-field canonical string "<timestamp>\n<METHOD>\n<canonical-URI>\n<hex(sha256(body))>" — binding the method and URI prevents cross-endpoint replay, and hashing the body makes the canonical string binary-safe. Callbacks older than 5 minutes are rejected, bounding same-endpoint replay to that freshness window; there is no nonce cache. Timestamps up to 1 minute in the future are accepted (clock skew tolerance). See internal/hmacauth for the reference implementation.

Index

Constants

View Source
const (
	// DefaultCallbackApplicationTimeout bounds Fred's synchronous processing of
	// one authenticated callback. The provider must return a terminal HTTP
	// response within this budget so the backend can safely advance its durable
	// per-lease callback queue.
	DefaultCallbackApplicationTimeout = 2 * time.Minute

	// DefaultCallbackDeliveryTimeout is the bundled backend sender's total
	// HTTP delivery retry budget. It remains strictly greater than
	// DefaultCallbackApplicationTimeout so a fresh first attempt normally gives
	// Fred time to serialize a retryable response after its application timeout.
	// Quick retries share whatever remains of this budget, so a later attempt may
	// reach the sender's deadline first.
	DefaultCallbackDeliveryTimeout = DefaultCallbackApplicationTimeout + 15*time.Second
)
View Source
const (
	DefaultMaxInfoBytes             int64 = 1 << 20 // 1 MiB — single lease info
	DefaultMaxProvisionBytes        int64 = 1 << 20 // 1 MiB — single provision record
	DefaultMaxProvisionsBytes       int64 = 8 << 20 // 8 MiB — list of all provisions
	DefaultMaxLookupProvisionsBytes int64 = 8 << 20 // 8 MiB — filtered provisions lookup; matches MaxProvisionsBytes because stack leases carry unbounded ServiceImages
	DefaultMaxLogsBytes             int64 = int64(MaxProjectedLogsResponseBytes)
	// MaxStoredReleaseHistoryBytes is the encoded per-lease history contract
	// enforced by the authoritative release journal. Keep this in the parent
	// backend package so both the HTTP boundary and the shared store use one
	// compile-time value without an import cycle.
	MaxStoredReleaseHistoryBytes = 32 << 20
	// MaxProjectedReleasesResponseBytes includes bounded projection headroom.
	// A legacy failed row may omit Reason on disk but gain ReasonUnknown at the
	// backend read boundary, so a valid response is not necessarily byte-smaller
	// than its stored history. Forty-eight MiB safely covers the worst valid
	// status-only history while retaining a finite response-body guard.
	MaxProjectedReleasesResponseBytes       = 48 << 20
	DefaultMaxReleasesBytes           int64 = MaxProjectedReleasesResponseBytes
	DefaultMaxStatsBytes              int64 = 1 << 20 // 1 MiB — load stats snapshot (small JSON)
	DefaultMaxRetentionsBytes         int64 = 1 << 20 // 1 MiB — bounds a single /retentions PAGE (not the whole list) since pagination

	// DefaultProvisionsPageLimit is the page size the client requests on
	// GET /provisions. The server coerces a larger value down to MaxPageLimit.
	DefaultProvisionsPageLimit = 1000

	// DefaultRetentionsPageLimit is the page size the client requests on
	// GET /retentions. The server coerces a larger value down to MaxPageLimit.
	DefaultRetentionsPageLimit = 1000
)

Default response body size limits (defense-in-depth against buggy/misrouted backends).

View Source
const (
	MaxInventoryItems       = 100_000
	MaxInventoryBytes int64 = 128 << 20
)

Inventory safety budgets apply to each complete endpoint walk, independently of backend page size. Exceeding either fails the whole observation; callers must never interpret a truncated inventory as evidence of absence.

View Source
const (
	// MaxLogContentBytes is the aggregate content budget for one log retrieval,
	// shared by live container reads and retained failed-attempt diagnostics.
	MaxLogContentBytes = 32 << 20

	// Failed-attempt records bound both entry count and key size before storage.
	MaxFailureDiagnosticEntries  = 4096
	MaxFailureDiagnosticKeyBytes = 256

	// AggregateLogLimitMessage also bounds the per-entry placeholder overhead
	// left by the live API after it exhausts the content budget.
	AggregateLogLimitMessage = "[log truncated: aggregate log size limit reached]"

	// JSON can expand each input byte to six bytes (control characters, HTML
	// escaping or invalid UTF-8). Include keys, per-entry markers, object framing
	// and the encoder's trailing newline, not just the unescaped content budget.
	MaxProjectedLogsResponseBytes = 6*(MaxLogContentBytes+maxProjectedLogEntries*(maxProjectedLogKeyBytes+len("\n")+len(AggregateLogLimitMessage))) +
		6*maxProjectedLogEntries + 3
)
View Source
const (
	MsgImagePullFailed        = "image pull failed"
	MsgRestartFailed          = "restart failed"
	MsgUpdateFailed           = "update failed"
	MsgRestoreFailed          = "restore failed"
	MsgVolumeCleanupExhausted = "volume cleanup exhausted"
	MsgCleanupFailed          = "cleanup failed"
	MsgBackendStorageLost     = "the backend storage holding this lease was irrecoverably lost"
	MsgVolumeDeletePending    = "an earlier deletion of this lease's volume is still finishing on the provider"
	// MsgVolumeDeletionInProgress is ReasonVolumeDeletionInProgress's message.
	MsgVolumeDeletionInProgress = "this lease's volume is still being deleted; the close completes when it finishes"
)

Curated human messages for the fixed (non-composed) reasons. Referenced at both the CallbackErr write site and the release-path call so the (reason, message) pair cannot drift. The composed restart/update messages use MsgRestartFailed/MsgUpdateFailed as their base + a runtime rollback suffix; container-exited/internal reuse leasesm.errMsg* consts.

View Source
const (
	// MsgContainerExitedDuringStartup / MsgContainerExitedDuringHealthCheck:
	// ReasonContainerExited.
	MsgContainerExitedDuringStartup     = "container exited during startup"
	MsgContainerExitedDuringHealthCheck = "container exited during health check"
	// MsgContainerUnhealthy / MsgHealthCheckDeadline: ReasonHealthCheckFailed.
	MsgContainerUnhealthy  = "container reported unhealthy"
	MsgHealthCheckDeadline = "container did not become healthy before the startup deadline"
	// MsgContainerStartRefused: ReasonContainerStartFailed.
	MsgContainerStartRefused = "container runtime refused to start the container"
	// MsgStartupUnverified / MsgStartupCanceled: ReasonInternal. Neither is an
	// observation of the tenant's workload.
	MsgStartupUnverified = "container startup could not be verified"
	MsgStartupCanceled   = "container startup verification canceled"
)

Curated messages for a failed startup verification (ENG-1125). Each one is paired with its reason where the failure is observed; only an observed exit is ContainerExited.

View Source
const CallbackLifecycleIDQueryParameter = "lifecycle_id"

CallbackLifecycleIDQueryParameter is the capability carried by an observation-only lifecycle callback URL. It is deliberately distinct from operation_id even though a paired URL derives both typed identities from the same canonical UUIDv4 value.

View Source
const CallbackOperationIDQueryParameter = "operation_id"

CallbackOperationIDQueryParameter is the capability carried only by an asynchronous provision or restore completion URL. A backend must not reuse that operation-scoped URL for later autonomous lifecycle observations.

View Source
const CodeAlreadyProvisioned = "already_provisioned"

CodeAlreadyProvisioned is the machine-readable code the docker-backend emits on /restore's 409 to distinguish "already provisioned" from the bare 409 meaning "invalid state". Defined here for the same reason as CodeDemoteExceedsTier: producer and consumer must agree on the exact string, and a shared constant makes drift a compile error rather than a silent misclassification.

View Source
const CodeCloseDeferred = "close_deferred"

CodeCloseDeferred reports retryable lifecycle contention, never completed cleanup.

View Source
const CodeDemoteExceedsTier = "demote_exceeds_tier"

CodeDemoteExceedsTier is the machine-readable error code the docker-backend emits (HTTP 422) and the HTTPClient parses to reconstruct ErrDemoteDataExceedsTier across the HTTP boundary. 422 is overloaded (a bare 422 with no code means ErrNotRetained), so producer and consumer must agree on this exact string; defining it once here makes drift a compile error rather than a silent misclassification. (ENG-438)

View Source
const CodeInsufficientResources = "insufficient_resources"

CodeInsufficientResources is the contract discriminator for a capacity refusal before asynchronous work starts. It establishes response-envelope conformance, not cryptographic authorship: backend responses are not HMAC authenticated, so deployments still trust the configured transport. A bare 503 is deliberately not equivalent because ordinary intermediaries may emit one after the backend accepted the request.

View Source
const CodeInvalidState = "invalid_state"

CodeInvalidState disambiguates a provision-state refusal from existing ownership.

View Source
const CodeLifecyclePending = "lifecycle_pending"

CodeLifecyclePending is a 503 observation of admitted lifecycle work. Unlike a capacity refusal it grants no settlement or no-dispatch authority.

View Source
const CodeMaintenanceExpired = "maintenance_expired"

CodeMaintenanceExpired is the code a backend sends with a 409 when a restart or update is older than the lease's retained maintenance history. The command was not executed and never will be.

View Source
const CodeOperationCompletionPending = "operation_completion_pending"

CodeOperationCompletionPending reports a valid earlier callback completion occupying the lease FIFO. It is an availability-success diagnostic only: callers must preserve the exact unresolved attempt, never settle a refusal.

View Source
const MaxLookupUUIDs = 100

MaxLookupUUIDs caps the number of lease UUIDs accepted by a single LookupProvisions call (and the matching server-side filter on GET /provisions?lease_uuid=...). Shared across the HTTP client, the docker-backend handler, and the Fred-level GetWorkloads handler so a future change to the cap stays consistent end-to-end.

View Source
const MaxOperationQuantity = 1024

MaxOperationQuantity is the maximum aggregate number of substrate instances an asynchronously accepted provision or restore operation may describe. It bounds all pre-admission allocations and recovery loops that are derived from tenant-controlled lease items.

View Source
const MaxPageLimit = 5000

MaxPageLimit is the server-side ceiling on a requested /provisions page size. A larger requested limit is coerced down to this value (AIP-158 "coerce down to the maximum") so a mis-set client limit cannot build an oversized page that trips the per-page byte guard and re-aborts the reconcile.

View Source
const ReleaseStatusActive = "active"

ReleaseStatusActive is the status of the release whose manifest a lease currently runs (BACKEND_GUIDE.md, GET /releases/{lease_uuid}).

Variables

View Source
var (
	// ErrInvalidOperationCallbackURL reports that an exact-completion URL mixes,
	// duplicates, or malforms callback authority.
	ErrInvalidOperationCallbackURL = errors.New("invalid operation callback URL")

	// ErrInvalidLifecycleCallbackURL reports that a separately supplied lifecycle
	// callback URL is not the exact lifecycle route derived from its completion
	// URL. Keeping this relationship strict prevents a backend request from
	// silently splitting settlement and observation traffic across unrelated
	// endpoints or carrying both kinds of authority at once.
	ErrInvalidLifecycleCallbackURL = errors.New("invalid lifecycle callback URL")
)
View Source
var (
	ErrUnknownSKU      = fmt.Errorf("%w: unknown SKU", ErrValidation)
	ErrInvalidManifest = fmt.Errorf("%w: invalid manifest", ErrValidation)
	ErrImageNotAllowed = fmt.Errorf("%w: image not allowed", ErrValidation)
)

Validation sub-category sentinels. These wrap ErrValidation so errors.Is(err, ErrValidation) still works, while allowing callers to classify the failure without string matching.

View Source
var (
	// ErrBackendStorageIdentityUnbound means the production client has no
	// durable expected identity and therefore refuses non-bootstrap requests.
	ErrBackendStorageIdentityUnbound = errors.New("backend storage identity is not bound")
	// ErrBackendStorageIdentityMissing means an upgraded backend response did
	// not carry the required identity header.
	ErrBackendStorageIdentityMissing = errors.New("backend response omitted storage identity")
	// ErrBackendStorageIdentityMismatch means a response or request path names
	// storage other than the identity pinned for the configured backend name.
	ErrBackendStorageIdentityMismatch = errors.New("backend storage identity mismatch")
	// ErrBackendUpgradeRequired proves an upgraded-only side-effect path was not
	// dispatched: an older backend returned its route-level 404 without an
	// identity header. Callers may refuse the exact write-ahead attempt.
	ErrBackendUpgradeRequired = errors.New("backend must be upgraded before side effects are enabled")
)
View Source
var ErrAlreadyProvisioned = errors.New("lease already provisioned")

ErrAlreadyProvisioned is returned when attempting to provision an already provisioned lease.

View Source
var ErrBackendFenced = errors.New("backend is fenced")

ErrBackendFenced means the operator fenced this backend: its client cannot reach the network, so the request was neither built nor sent.

View Source
var ErrCapacityRefused = fmt.Errorf("%w: backend refused the request", ErrInsufficientResources)

ErrCapacityRefused is returned only for a contract-conforming error envelope whose code is CodeInsufficientResources. It wraps ErrInsufficientResources so callers that only map the tenant-facing capacity result remain compatible, while lifecycle coordinators can clear an exact durable attempt under the configured backend transport's trust boundary.

View Source
var ErrCircuitOpen = errors.New("circuit breaker is open")

ErrCircuitOpen is returned when the circuit breaker is open.

View Source
var ErrCloseDeferred = fmt.Errorf("%w: close deferred", ErrInvalidState)

ErrCloseDeferred means close must be retried after current lifecycle work settles.

View Source
var ErrDemoteDataExceedsTier = errors.New("retained data exceeds the requested smaller tier")

ErrDemoteDataExceedsTier is returned when a restore requests a smaller disk tier (demote) than the retained volume's measured data footprint, so the data would not fit the new cap. A dedicated sentinel (NOT wrapping ErrValidation) so the API can map it to 422 distinctly. (ENG-438)

View Source
var ErrInsufficientResources = errors.New("insufficient resources")

ErrInsufficientResources is returned when there are not enough resources to fulfill a provision or restore request. On its own this sentinel does not prove whether the backend started work because legacy and intermediary 503 responses also map to it.

View Source
var ErrInvalidState = errors.New("invalid state for operation")

ErrInvalidState is returned when an operation is not valid for the current lease state.

View Source
var ErrMaintenanceExpired = errors.New("maintenance command is older than the lease's retained history")

ErrMaintenanceExpired is the backend refusal CodeMaintenanceExpired names.

View Source
var ErrMalformedErrorBody = errors.New("backend returned an unrecognized error response")

ErrMalformedErrorBody is returned when a backend answers with a client-error status whose body is NOT the declared JSON error envelope. fred cannot know why the request failed — or even whether the backend, rather than something between them, produced the response — so it authors its own message and forwards none of the body.

A well-formed envelope carrying an EMPTY "error" is a different case and is not this error: it came from the backend and its verdict stands, so the status-code sentinel is returned with no detail.

Deliberately NOT wrapping ErrValidation. ErrValidation is the reconciler's PERMANENT branch — it rejects a PENDING lease and CLOSES an ACTIVE one on-chain (handleProvisionResult) — and a body fred could not parse is no evidence that the tenant is at fault.

Scope, stated precisely (ENG-739): this is an operator-introduced intermediary being misread as a permanent tenant-side validation failure, NOT something a tenant can trigger. No tenant-reachable input produces a non-envelope 400 — docker-backend routes every 4xx through its ErrorResponse writer. The generic "a proxy 400s the whole hop" scenario is already contained by the ENG-356 snapshot gate, which defers every lease of a backend whose GET /provisions did not answer; the reachable variant needs a SELECTIVE failure (400 on POST /provision while GET /provisions still returns valid JSON — a body-inspecting WAF rule).

The Kubernetes apiserver makes the same call for the same shape: an admission-webhook response it cannot decode becomes a fixed ServiceUnavailable it authored, with the real error kept on the operator channel (ErrCallingWebhook.Reason), never returned to the caller.

View Source
var ErrNotProvisioned = errors.New("lease not provisioned")

ErrNotProvisioned is returned when a lease is not yet provisioned.

View Source
var ErrNotRetained = errors.New("no retained data for lease")

ErrNotRetained is returned when no retained data exists for the original lease (absent, expired, or owned by a different tenant). Distinct from ErrNotProvisioned (which concerns live provisions).

View Source
var ErrResponseTooLarge = errors.New("response body too large")

ErrResponseTooLarge is returned when a backend response body exceeds the configured size limit.

View Source
var ErrRestoreRefused = errors.New("backend refused the restore")

ErrRestoreRefused is returned when a /restore 4xx body IS the declared envelope but carries a "code" discriminator fred does not recognize for that status. fred relays the backend's authored message at the status the backend chose and states no lease-state fact of its own — an unrecognized code is evidence AGAINST "no retained data", not for it.

Deliberately NOT ErrMalformedErrorBody. The envelope was well-formed, so this is a business verdict fred could not classify, not an off-contract backend. "code" is an OPEN, add-only set (the same policy backend.Reason states), so a backend shipping a new discriminator before providerd learns it must not trip the circuit breaker for every other tenant on that backend, and must not land in fred_backend_malformed_error_body_total — whose runbook remedy ("fix the envelope, suspect an intermediary") would send the operator somewhere wrong.

View Source
var ErrValidation = errors.New("validation error")

ErrValidation is returned when a provision request fails pre-flight validation (e.g., unknown SKU, invalid manifest, disallowed image registry).

Functions

func DeprovisionLifecyclePending added in v0.14.0

func DeprovisionLifecyclePending(client Backend, leaseUUID string, err error) bool

DeprovisionLifecyclePending recognizes a deferred close observation from this exact HTTP client, lease and endpoint. It authorizes scheduling a retry, never completing the interrupted operation or claiming cleanup has run.

func Detail added in v0.13.0

func Detail(err error) (string, bool)

Detail returns the declared-envelope detail of err, if it carries one. Reports false for any other error, including a bare sentinel.

func IsBackendStorageIdentityMissingServerError added in v0.14.0

func IsBackendStorageIdentityMissingServerError(err error) bool

IsBackendStorageIdentityMissingServerError reports whether an identity-bound observation received an HTTP 5xx response with no storage-identity header at all. Observation-only health probes may treat that as transient endpoint unavailability: explicitly empty, duplicate, or malformed headers remain contradictions, and side-effect responses always return false because this classification cannot prove refusal.

func IsCanonicalLeaseUUID added in v0.14.0

func IsCanonicalLeaseUUID(value string) bool

IsCanonicalLeaseUUID reports whether value is the lowercase, hyphenated, non-nil UUID form used as the durable identity of a lease. Backend HTTP boundaries use the exact same predicate as the v2 callback outbox so an accepted mutating request cannot create an effect whose callback is later rejected by persistence.

func IsFenced added in v0.14.0

func IsFenced(b Backend) bool

IsFenced reports whether b is a fenced backend's client. It reads the client's own construction, the one place the fence lives, so routing, reconciliation, and metrics cannot disagree about it.

func IsReadCapacity added in v0.14.0

func IsReadCapacity(err error) bool

IsReadCapacity reports a complete, bounded backend read-capacity response. This observation only supports retry guidance; it grants no mutation authority.

func NewLogsHandler added in v0.14.0

func NewLogsHandler(next http.Handler, timeout, writeTimeout time.Duration) http.Handler

NewLogsHandler admits one materialized backend log response per server. Up to eight requests wait in FIFO order within their request deadline. Saturation returns the backend's string-code capacity envelope. A timeout remains a backend failure with no capacity code. Authentication middleware should wrap this handler.

The permit covers BOTH the worker and the buffered timeout response's final write. A timeout or a slow client cannot release it while either still owns memory. Keep this wrapper in place of (not inside) another TimeoutHandler. The final socket write has its own finite budget; a disabled write timeout uses the request timeout. ResponseWriter wrappers must preserve ResponseController support through Unwrap.

func NewPreparedTenantLogsHandler added in v0.14.0

func NewPreparedTenantLogsHandler(
	prepare func(http.ResponseWriter, *http.Request) http.Handler,
	timeout, writeTimeout time.Duration,
) http.Handler

NewPreparedTenantLogsHandler separates small request preparation (identity, authorization and routing) from materializing logs. prepare returns a handler bound to that authorized request, or nil after writing a bounded error. It must not fetch logs. At most nine requests prepare or wait; only one returned handler may own a log response. Both limits cover worker and socket lifetime. The preparation and materialization queue share the original request deadline.

func NewTenantLogsHandler added in v0.14.0

func NewTenantLogsHandler(next http.Handler, timeout, writeTimeout time.Duration) http.Handler

NewTenantLogsHandler applies the same response ownership to the provider's tenant route, whose error envelopes carry numeric HTTP status codes.

func NormalizeProvisionRequest

func NormalizeProvisionRequest(req *ProvisionRequest) error

NormalizeProvisionRequest applies boundary normalization to a provision request: any single-item request that lacks ServiceName is auto-tagged with defaultServiceName, transitioning legacy wire-format leases into the stack-shaped contract every downstream component now expects. Named items are passed through. Mixed-presence and multi-unnamed combinations are rejected — both were structurally invalid under the legacy contract too, so this only formalizes the rejection.

The normalization is in-place; callers should invoke this at the very entry of Provision/Update, before any docker work or pool allocation, so the rest of the path operates on uniformly stack-shaped state.

func ParsePageParams added in v0.7.0

func ParsePageParams(q url.Values) (limit int, continueToken string, err error)

ParsePageParams extracts keyset pagination params from a paginated list query (/provisions, /retentions). It returns a non-nil error (which handlers map to HTTP 400) when 'limit' is present but not a non-negative integer, or 'continue' is present but not a valid UUID. Absent params yield (0, "", nil) — the unpaginated passthrough.

func ProbeStorageIdentity added in v0.14.0

func ProbeStorageIdentity(ctx context.Context, policy ConnectionPolicy) (backendidentity.ID, error)

ProbeStorageIdentity asks one backend which storage it serves, with a single one-row inventory request. It reads only the identity response header, which a backend sets on every response once its storage identity verifies, so a backend whose inventory is failing still answers. The answer grants no authority: the offline retirement tool uses it only to refuse retiring a backend that still serves its pinned storage.

func ResolveLifecycleCallbackURL added in v0.14.0

func ResolveLifecycleCallbackURL(callbackURL, lifecycleCallbackURL string) (string, error)

ResolveLifecycleCallbackURL validates or derives the observation-only URL paired with a completion URL. A current completion URL must contain exactly one canonical UUIDv4 operation_id and no lifecycle_id; derivation replaces that one field with lifecycle_id while retaining all unrelated raw query fields byte for byte and in their original order. A legacy operationless URL remains tokenless. Duplicate identities, preexisting lifecycle authority, and non-canonical operation IDs are rejected.

The optional explicit form lets current Fred versions make the distinction visible on the wire. It must equal the derived URL exactly. Operationless derivation preserves callbacks already embedded in adopted v0.13 workloads; it does not authorize new tokenless provision or restore admission. A current request may omit lifecycle_callback_url when callback_url carries its typed operation identity.

func ResolveMaintenanceCallbackURLs added in v0.14.0

func ResolveMaintenanceCallbackURLs(
	callbackURL, lifecycleCallbackURL, requestedLifecycleURL string,
) (string, string, error)

ResolveMaintenanceCallbackURLs validates a trusted maintenance request against the callback authority already persisted by a backend. Restart, update, and autonomous reconciler replacement may move the callback route to a new base, but they must retain the same authority class and exact lifecycle UUID. A typed route can never be downgraded to tokenless (or vice versa), and a different typed ID is stale or misrouted authority.

requestedLifecycleURL may be empty for an autonomous backend operation; in that case the persisted route is retained. The returned operation/lifecycle pair uses the requested base and one shared identity, so it remains suitable for durable labels and later exact pair validation. Legacy tokenless pairs remain tokenless. When no route was persisted, the trusted requested route is adopted after full lifecycle validation.

func ValidateLifecycleCallbackURL added in v0.14.0

func ValidateLifecycleCallbackURL(callbackURL string) error

ValidateLifecycleCallbackURL validates an observation-only callback route. Current routes carry exactly one canonical UUIDv4 lifecycle_id and no operation_id. A legacy route with neither identity remains valid for the lifecycle of an adopted workload. Duplicate, mixed, or malformed authority is always rejected.

func ValidateOperationCallbackURL added in v0.14.0

func ValidateOperationCallbackURL(callbackURL string) error

ValidateOperationCallbackURL validates an exact requested-operation route. Current routes carry exactly one canonical UUIDv4 operation_id and no lifecycle_id. A legacy route with neither identity remains valid for stored evidence and exact replay lookup; new operation admission separately requires typed runtime authority. Duplicate, mixed, or malformed authority is rejected.

func ValidateOperationQuantities added in v0.14.0

func ValidateOperationQuantities(items []LeaseItem) (int, error)

ValidateOperationQuantities validates and sums lease-item quantities without integer overflow. Keeping the aggregate proof beside the wire type lets every backend and the durable operation journal enforce the same bound before any allocation, substrate loop, or asynchronous acceptance.

Types

type AuthenticatedEvidencePolicy added in v0.14.0

type AuthenticatedEvidencePolicy struct {
	// contains filtered or unexported fields
}

AuthenticatedEvidencePolicy is the authority to collect backend replies for an offline placement mutation. HMAC authenticates requests only; the reply and its storage-identity header require certificate-verified HTTPS. This capability captures the existing policy and cannot downgrade its transport.

func NewAuthenticatedEvidencePolicy added in v0.14.0

func NewAuthenticatedEvidencePolicy(policy ConnectionPolicy) (AuthenticatedEvidencePolicy, error)

func (AuthenticatedEvidencePolicy) NewIdentityBoundInventoryClient added in v0.14.0

func (policy AuthenticatedEvidencePolicy) NewIdentityBoundInventoryClient(resolver BackendStorageIdentityResolver) (BootstrapInventoryClient, error)

NewIdentityBoundInventoryClient additionally checks the stopped database's exact backend identity pin on every response.

func (AuthenticatedEvidencePolicy) NewInventoryClient added in v0.14.0

func (policy AuthenticatedEvidencePolicy) NewInventoryClient() (BootstrapInventoryClient, error)

NewInventoryClient exposes observations only; possession of authenticated evidence never provides backend provision/deprovision authority.

type Backend

type Backend interface {
	// Provision starts async provisioning of a resource.
	// The backend will call the callback URL when provisioning completes.
	Provision(ctx context.Context, req ProvisionRequest) error

	// GetInfo returns lease information including connection details.
	// Returns ErrNotProvisioned if the lease is not yet provisioned.
	GetInfo(ctx context.Context, leaseUUID string) (*LeaseInfo, error)

	// Deprovision releases resources for a lease. Must be idempotent.
	Deprovision(ctx context.Context, leaseUUID string) error

	// ListProvisions returns all currently provisioned resources.
	// Used by the reconciler for orphan detection.
	ListProvisions(ctx context.Context) ([]ProvisionInfo, error)

	// LookupProvisions returns provision info for the requested lease UUIDs.
	// Missing leases are simply absent from the returned slice (not an error).
	// Used by the /workloads endpoint to fetch metadata for a known set of leases
	// without scanning the full provisions corpus. Caller is responsible for
	// enforcing MaxLookupUUIDs; implementations may also enforce it defensively.
	//
	// Named LookupProvisions (not GetProvisions) to avoid visual ambiguity with
	// the existing singular GetProvision.
	LookupProvisions(ctx context.Context, leaseUUIDs []string) ([]ProvisionInfo, error)

	// Health checks if the backend is reachable and healthy.
	// Returns nil if healthy, error otherwise.
	Health(ctx context.Context) error

	// RefreshState synchronizes in-memory provision state with the
	// underlying infrastructure. Backends should query the real
	// container/VM state and update their internal tracking.
	// Called by the reconciler before ListProvisions to avoid stale reads.
	RefreshState(ctx context.Context) error

	// GetProvision returns status information for a single provision.
	// Returns ErrNotProvisioned if the lease is not found.
	GetProvision(ctx context.Context, leaseUUID string) (*ProvisionInfo, error)

	// GetLogs returns container logs for a provisioned lease.
	// The tail parameter limits the number of log lines per container.
	// Returns ErrNotProvisioned if the lease is not found.
	GetLogs(ctx context.Context, leaseUUID string, tail int) (map[string]string, error)

	// Restart restarts containers for a lease without changing the manifest.
	// Returns ErrNotProvisioned if lease not found, ErrInvalidState if not restartable.
	Restart(ctx context.Context, req RestartRequest) error

	// Update deploys a new manifest for a lease, replacing containers.
	// Returns ErrNotProvisioned if lease not found, ErrInvalidState if not updatable.
	Update(ctx context.Context, req UpdateRequest) error

	// Restore re-provisions a new lease from the retained volumes of a prior
	// soft-deleted lease (ENG-325). Returns ErrNotRetained if no retained data
	// exists for FromLeaseUUID, ErrInvalidState if the target lease is not in
	// a valid state for restore, ErrValidation for bad inputs.
	Restore(ctx context.Context, req RestoreRequest) error

	// ReconcileCustomDomain reapplies the per-LeaseItem custom_domain values
	// from chain onto the running provision. When any item's CustomDomain
	// differs from the in-memory provision state, the backend snapshots the
	// current values, updates them, and triggers a Restart() to re-emit
	// Traefik labels. On Restart failure the in-memory state is rolled back
	// so the next reconciler tick can retry.
	//
	// No-op when:
	//   - the lease is not provisioned by this backend (silent),
	//   - the provision is not currently active,
	//   - all items already match the incoming values.
	//
	// Idempotent: repeated calls with the same items list produce no
	// container churn beyond the first reconciling call.
	ReconcileCustomDomain(ctx context.Context, leaseUUID string, items []LeaseItem) error

	// GetReleases returns the release history for a lease.
	// Returns ErrNotProvisioned if lease not found.
	GetReleases(ctx context.Context, leaseUUID string) ([]ReleaseInfo, error)

	// GetLoadStats returns the backend's current resource-load snapshot,
	// used for least-loaded provision routing. Implementations that do not
	// track load may return a snapshot whose CPUAllocatedRatio is not ok.
	GetLoadStats(ctx context.Context) (*LoadStats, error)

	// ListRetentions returns the leases whose data this backend currently
	// retains (soft-deleted, awaiting restore or grace-reap). Used by the
	// reconciler to keep placement affinity for retained leases (ENG-333).
	// Backends without retention (e.g. k3s) return an empty slice.
	ListRetentions(ctx context.Context) ([]RetainedLease, error)

	// Name returns the backend's configured name.
	Name() string
}

Backend defines the interface for interacting with a provisioning backend. Any backend (Kubernetes, GPU, VM, etc.) must implement these operations.

type BackendEntry

type BackendEntry struct {
	Backend   Backend
	Match     MatchCriteria
	IsDefault bool
}

BackendEntry pairs a backend with its matching criteria.

type BackendHealth

type BackendHealth struct {
	Name    string `json:"name"`
	Healthy bool   `json:"healthy"`
	Error   string `json:"error,omitempty"`
	// Fenced means the operator fenced this backend, so it is unhealthy by
	// configuration and was not asked.
	Fenced bool `json:"fenced,omitempty"`
	// contains filtered or unexported fields
}

BackendHealth represents the health status of a single backend.

func (BackendHealth) ProbeDuration added in v0.14.0

func (health BackendHealth) ProbeDuration() time.Duration

ProbeDuration is the elapsed time of this exact client probe, including failed or canceled calls. It is diagnostic only, not a health verdict.

type BackendStorageIdentityResolver added in v0.14.0

type BackendStorageIdentityResolver interface {
	ExpectedBackendStorageIdentity(string) (backendidentity.ID, bool)
}

BackendStorageIdentityResolver supplies the durable identity pinned by the placement store. Implementations must be safe for concurrent calls.

type BootstrapInventoryClient added in v0.14.0

type BootstrapInventoryClient interface {
	Name() string
	ListProvisionsWithIdentity(context.Context) ([]ProvisionInfo, backendidentity.ID, error)
	ListRetentionsWithIdentity(context.Context) ([]RetainedLease, backendidentity.ID, error)
}

BootstrapInventoryClient is the complete read-only inventory capability required while a stopped/prepared placement database has not yet supplied a backend storage identity. Its method set deliberately excludes every backend mutation.

func NewBootstrapInventoryClient added in v0.14.0

func NewBootstrapInventoryClient(policy ConnectionPolicy, cfg HTTPClientOptions) (BootstrapInventoryClient, error)

NewBootstrapInventoryClient constructs the only unbound production client. It can collect identity-bearing inventory but cannot express a mutation.

type CallbackPayload

type CallbackPayload struct {
	// MaintenanceID identifies an exact restart/update completion. It is
	// authenticated in the body and matched to the provider's pending command;
	// ordinary lifecycle observations omit it.
	MaintenanceID string `json:"maintenance_id,omitempty"`
	// MaintenanceAdmittedAt echoes the provider's admission stamp of that
	// command (RFC 3339, UTC), so a stale completion of a reused key cannot
	// settle the provider's newer command. Omitted when the command was
	// unstamped.
	MaintenanceAdmittedAt string         `json:"maintenance_admitted_at,omitempty"`
	LeaseUUID             string         `json:"lease_uuid"`
	Status                CallbackStatus `json:"status"` // "success", "failed", or "deprovisioned"
	Error                 string         `json:"error,omitempty"`
	// BackendStorageID binds the observation to the backend storage lineage
	// that produced it. Upgraded senders persist this value with new outbox
	// entries and include it in the HMAC-covered body on every delivery.
	BackendStorageID string `json:"backend_storage_id,omitempty"`
	// Backend is optional sender metadata retained for bounded metrics on
	// callbacks that have no current operation. It is not lifecycle authority:
	// the exact HMAC-authenticated operation URL or durable lifecycle capability
	// selects the store-authoritative backend.
	Backend string `json:"backend,omitempty"`
	// OperationID is injected by fred's callback HTTP endpoint from the
	// HMAC-authenticated callback URL. Backends need only POST to the URL they
	// received; they do not interpret or echo the token in their JSON body.
	OperationID string `json:"operation_id,omitempty"`
	// LifecycleID is likewise injected from an authenticated lifecycle callback
	// URL. A provider overwrites either body field before application so backend
	// JSON can never mint callback authority.
	LifecycleID string `json:"lifecycle_id,omitempty"`
	// Retained is set true on a deprovisioned callback when the backend actually
	// soft-deleted (retained) the lease's volumes. Best-effort ground truth for
	// the optimistic push; the queryable retention status is the durable backstop.
	Retained bool `json:"retained,omitempty"`
}

CallbackPayload is sent by backends to fred's callback endpoint.

type CallbackStatus

type CallbackStatus string

CallbackStatus represents the status sent in a callback payload.

const (
	CallbackStatusSuccess       CallbackStatus = "success"
	CallbackStatusFailed        CallbackStatus = "failed"
	CallbackStatusDeprovisioned CallbackStatus = "deprovisioned"
)

Callback status constants.

type ConnectionConfig added in v0.14.0

type ConnectionConfig struct {
	Name              string
	BaseURL           string
	Secret            string
	Timeout           time.Duration
	TLSCAFile         string
	TLSSkipVerify     bool
	TLSClientCertFile string
	TLSClientKeyFile  string
}

ConnectionConfig is the configuration-time input for a backend connection. Configured CA/client credentials are loaded once; callers never supply a mutable tls.Config. With no CA file, Go's standard system-root lookup applies. Skip verification and plaintext are development options rejected by the provider's production configuration validation.

type ConnectionPolicy added in v0.14.0

type ConnectionPolicy struct {
	// contains filtered or unexported fields
}

ConnectionPolicy is an immutable, validated backend connection. Both the inventory-only and identity-bound factories require this same policy, so system-root trust cannot accidentally select a different protocol floor. A fenced backend's policy holds only its name, so every client built from it lacks the means to reach the network. Its zero value is invalid and cannot construct a usable client.

func NewConnectionPolicy added in v0.14.0

func NewConnectionPolicy(cfg ConnectionConfig) (ConnectionPolicy, error)

NewConnectionPolicy validates connection settings and captures privately owned configured TLS material. HTTPS always requires TLS 1.3, with either system roots or the configured private CA; clients cannot override it after construction.

func NewFencedConnectionPolicy added in v0.14.0

func NewFencedConnectionPolicy(name string) (ConnectionPolicy, error)

NewFencedConnectionPolicy is the policy of a fenced backend. It holds only the backend's name and loads no address, HMAC key, or TLS material, so a fenced backend's revoked certificate or key files may already be gone. Every client built from it refuses every operation locally.

func (ConnectionPolicy) Fenced added in v0.14.0

func (policy ConnectionPolicy) Fenced() bool

Fenced reports whether policy is a fenced backend's.

func (ConnectionPolicy) Format added in v0.14.0

func (policy ConnectionPolicy) Format(state fmt.State, _ rune)

Format and LogValue prevent reflective formatting from disclosing credentials.

func (ConnectionPolicy) LogValue added in v0.14.0

func (policy ConnectionPolicy) LogValue() slog.Value

type DeprovisionRefusal added in v0.14.0

type DeprovisionRefusal uint8

DeprovisionRefusal names why a client withheld a deprovision request. Its zero value proves nothing: the request may have been sent.

const (
	DeprovisionRefusalUnproven DeprovisionRefusal = iota
	// DeprovisionRefusedCircuitOpen: the circuit breaker refused admission.
	DeprovisionRefusedCircuitOpen
	// DeprovisionRefusedFenced: the backend is fenced, so its client has no
	// means to send anything until the operator lifts the fence.
	DeprovisionRefusedFenced
)

func DeprovisionRefusalOf added in v0.14.0

func DeprovisionRefusalOf(client Backend, leaseUUID string, err error) DeprovisionRefusal

DeprovisionRefusalOf recognizes a local refusal by this exact client for this exact lease. A sentinel error, remote response, unknown transport effect, or proof from another client or lease cannot establish that no request was sent.

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient implements Backend using HTTP calls to a backend service.

func NewIdentityBoundHTTPClient added in v0.14.0

func NewIdentityBoundHTTPClient(
	policy ConnectionPolicy,
	cfg HTTPClientOptions,
	resolver BackendStorageIdentityResolver,
) (*HTTPClient, error)

NewIdentityBoundHTTPClient creates the production client. It refuses every non-inventory request until the placement store has pinned this configured name to an immutable backend storage identity. Side effects use upgraded-only identity paths, so an old/reverted backend returns 404 before decoding or executing the request rather than merely ignoring a new query parameter. A strong HMAC secret and the shared TLS policy are required at construction.

func (*HTTPClient) Deprovision

func (c *HTTPClient) Deprovision(ctx context.Context, leaseUUID string) (err error)

Deprovision releases resources for a lease.

func (*HTTPClient) GetInfo

func (c *HTTPClient) GetInfo(ctx context.Context, leaseUUID string) (*LeaseInfo, error)

GetInfo retrieves lease information including connection details.

func (*HTTPClient) GetLoadStats added in v0.5.0

func (c *HTTPClient) GetLoadStats(ctx context.Context) (*LoadStats, error)

GetLoadStats retrieves the backend's current resource-load snapshot from GET /stats. Used by the router for least-loaded provision placement.

func (*HTTPClient) GetLogs

func (c *HTTPClient) GetLogs(ctx context.Context, leaseUUID string, tail int) (map[string]string, error)

GetLogs retrieves container logs for a provisioned lease.

func (*HTTPClient) GetProvision

func (c *HTTPClient) GetProvision(ctx context.Context, leaseUUID string) (*ProvisionInfo, error)

GetProvision retrieves status information for a single provision.

func (*HTTPClient) GetReleases

func (c *HTTPClient) GetReleases(ctx context.Context, leaseUUID string) ([]ReleaseInfo, error)

GetReleases retrieves release history for a lease.

func (*HTTPClient) Health

func (c *HTTPClient) Health(ctx context.Context) (err error)

Health checks if the backend is reachable and healthy. It sends a GET request to /health on the backend.

func (*HTTPClient) ListProvisions

func (c *HTTPClient) ListProvisions(ctx context.Context) (_ []ProvisionInfo, err error)

ListProvisions returns all provisioned resources from this backend. It walks the keyset-paginated /provisions endpoint, reassembling the complete set; it returns an error rather than a partial set (complete-or-error) so the reconciler never acts on incomplete data.

func (*HTTPClient) ListProvisionsWithIdentity added in v0.14.0

func (c *HTTPClient) ListProvisionsWithIdentity(
	ctx context.Context,
) ([]ProvisionInfo, backendidentity.ID, error)

ListProvisionsWithIdentity returns a complete provision inventory plus the one storage identity carried consistently by every page.

func (*HTTPClient) ListRetentions added in v0.5.0

func (c *HTTPClient) ListRetentions(ctx context.Context) (_ []RetainedLease, err error)

ListRetentions retrieves the leases whose data this backend currently retains (GET /retentions). Used by the reconciler for restore backend affinity. It walks the keyset-paginated endpoint, reassembling the complete set; it returns an error rather than a partial set (complete-or-error) so the reconciler never acts on incomplete retention data.

func (*HTTPClient) ListRetentionsWithIdentity added in v0.14.0

func (c *HTTPClient) ListRetentionsWithIdentity(
	ctx context.Context,
) ([]RetainedLease, backendidentity.ID, error)

ListRetentionsWithIdentity returns a complete retention inventory plus the one storage identity carried consistently by every page.

func (*HTTPClient) LookupProvisions

func (c *HTTPClient) LookupProvisions(ctx context.Context, uuids []string) ([]ProvisionInfo, error)

LookupProvisions returns provision info for the requested lease UUIDs. Missing leases are absent from the returned slice (not an error).

Wire path: GET {baseURL}/provisions?lease_uuid=...&lease_uuid=... The docker-backend handler shares the same route as the unfiltered ListProvisions path; the filter param toggles the subset behavior. Same wire type either way.

func (*HTTPClient) Name

func (c *HTTPClient) Name() string

Name returns the backend's configured name.

func (*HTTPClient) Provision

func (c *HTTPClient) Provision(ctx context.Context, req ProvisionRequest) error

Provision sends a provision request to the backend while preserving the legacy Backend surface. Placement settlement uses InvokeProvision so causal transport evidence is not flattened back into an arbitrary error tree.

func (*HTTPClient) ReconcileCustomDomain

func (c *HTTPClient) ReconcileCustomDomain(ctx context.Context, leaseUUID string, items []LeaseItem) (err error)

ReconcileCustomDomain forwards a reconciliation request to the remote backend. The reconciler calls this on every active lease each tick; the remote side decides whether any action is needed (idempotent on no-change).

func (*HTTPClient) RefreshState

func (c *HTTPClient) RefreshState(ctx context.Context) error

RefreshState is a no-op for remote backends (they refresh server-side).

func (*HTTPClient) Restart

func (c *HTTPClient) Restart(ctx context.Context, req RestartRequest) error

Restart preserves the legacy Backend surface. Placement consumes InvokeRestart so a verified refusal cannot be confused with an arbitrary wrapped error.

func (*HTTPClient) Restore added in v0.5.0

func (c *HTTPClient) Restore(ctx context.Context, req RestoreRequest) error

Restore preserves the legacy Backend surface. Placement consumes InvokeRestore so only transport-minted causal evidence reaches durable settlement.

func (*HTTPClient) Update

func (c *HTTPClient) Update(ctx context.Context, req UpdateRequest) error

Update preserves the legacy Backend surface. Placement consumes InvokeUpdate.

type HTTPClientOptions added in v0.14.0

type HTTPClientOptions struct {
	MaxIdleConns        int // Max idle connections across all hosts (default: 100)
	MaxIdleConnsPerHost int // Max idle connections per host (default: 10)

	// Circuit breaker settings
	CBMaxRequests   uint32        // Max requests in half-open state (default: 1)
	CBInterval      time.Duration // Interval to clear counts in closed state (default: 0, never clear)
	CBTimeout       time.Duration // Time to wait before transitioning from open to half-open (default: 60s)
	CBFailureThresh uint32        // Number of failures to trip the breaker (default: 5)

	// Response body size limits (0 = use default). Defense-in-depth caps to prevent
	// a buggy or misrouted backend from pressuring memory with unbounded responses.
	MaxInfoBytes             int64 // GetInfo response limit (default: 1 MiB)
	MaxProvisionBytes        int64 // GetProvision response limit (default: 1 MiB)
	MaxProvisionsBytes       int64 // ListProvisions response limit (default: 8 MiB)
	MaxLookupProvisionsBytes int64 // LookupProvisions response limit (default: 8 MiB)
	MaxLogsBytes             int64 // GetLogs encoded response limit (default: MaxProjectedLogsResponseBytes)
	MaxReleasesBytes         int64 // GetReleases response limit (default: 48 MiB projected response)
	MaxStatsBytes            int64 // GetLoadStats response limit (default: 1 MiB)
	MaxRetentionsBytes       int64 // /retentions per-page response limit (default: 1 MiB)
	ProvisionsPageLimit      int   // /provisions page size requested by the client (default: 1000)
	RetentionsPageLimit      int   // /retentions page size requested by the client (default: 1000)

	// Optional Prometheus metrics. When nil, metric recording is skipped.
	// This prevents binaries that don't use these metrics (e.g., docker-backend)
	// from registering phantom fred-level metrics via transitive imports.
	RequestDuration     *prometheus.HistogramVec // labels: backend, operation, status
	RequestsTotal       *prometheus.CounterVec   // labels: backend, operation, status
	CircuitBreakerState *prometheus.GaugeVec     // labels: backend
	// Fenced is set once at construction: 1 for a fenced backend's client,
	// 0 otherwise. It is the operator's signal to silence that backend's
	// availability alerts, and it cannot disagree with the client itself.
	Fenced *prometheus.GaugeVec // labels: backend
	// MalformedErrorBodyTotal counts client-error responses whose body was not
	// the declared JSON error envelope. A backend contributing to this is
	// off-contract (BACKEND_GUIDE.md) and its tenants are getting a generic
	// message in place of a real diagnostic.
	MalformedErrorBodyTotal *prometheus.CounterVec // labels: backend, operation
}

HTTPClientOptions tunes resource limits, circuit breaking, and telemetry. Connection authority is supplied separately as an opaque ConnectionPolicy.

type IdentityInventoryBackend added in v0.14.0

type IdentityInventoryBackend interface {
	ListProvisionsWithIdentity(context.Context) ([]ProvisionInfo, backendidentity.ID, error)
	ListRetentionsWithIdentity(context.Context) ([]RetainedLease, backendidentity.ID, error)
}

IdentityInventoryBackend exposes the physical storage identity observed on each complete inventory endpoint. Reconciliation requires this stronger interface before it can establish or reuse a placement admission baseline.

type LeaseInfo

type LeaseInfo struct {
	Host      string                  `json:"host,omitempty"`
	FQDN      string                  `json:"fqdn,omitempty"`
	Protocol  string                  `json:"protocol,omitempty"`
	Ports     map[string]PortBinding  `json:"ports,omitempty"`
	Instances []LeaseInstance         `json:"instances,omitempty"`
	Services  map[string]LeaseService `json:"services,omitempty"`
	Metadata  map[string]string       `json:"metadata,omitempty"`
}

LeaseInfo contains backend-specific information about a provisioned lease.

type LeaseInstance

type LeaseInstance struct {
	InstanceIndex int                    `json:"instance_index"`
	ContainerID   string                 `json:"container_id,omitempty"`
	Image         string                 `json:"image,omitempty"`
	Status        string                 `json:"status,omitempty"`
	FQDN          string                 `json:"fqdn,omitempty"`
	Ports         map[string]PortBinding `json:"ports,omitempty"`
}

LeaseInstance represents a single provisioned instance (container/pod).

type LeaseItem

type LeaseItem struct {
	SKU         string `json:"sku"`
	Quantity    int    `json:"quantity"`
	ServiceName string `json:"service_name,omitempty"`
	// CustomDomain is the optional FQDN the tenant has assigned to this item.
	// When non-empty (and the service has a routable HTTP port), the backend
	// emits a secondary Traefik router on the item's container(s) routing
	// Host(<CustomDomain>) to the same loadbalancer service. Validated on-chain
	// in MsgSetItemCustomDomain; Fred re-runs cheap defense-in-depth checks
	// before applying labels.
	CustomDomain string `json:"custom_domain,omitempty"`
}

LeaseItem represents a single SKU with its quantity in a lease.

type LeaseService

type LeaseService struct {
	FQDN      string          `json:"fqdn,omitempty"`
	Instances []LeaseInstance `json:"instances,omitempty"`
}

LeaseService groups instances belonging to a single service in a stack lease.

type LeaseStatusEvent

type LeaseStatusEvent struct {
	LeaseUUID string          `json:"lease_uuid"`
	Status    ProvisionStatus `json:"status"`
	Error     string          `json:"error,omitempty"`
	Timestamp time.Time       `json:"timestamp"`
}

LeaseStatusEvent is published when a lease's provisioning status changes. Used for real-time delivery (e.g., WebSocket) to notify tenants of status transitions.

type LifecycleGenerationKind added in v0.14.0

type LifecycleGenerationKind string

LifecycleGenerationKind classifies the callback authority a backend has durably paired with one live provision. The observation carries no URL or other bearer material; a typed observation exposes only the UUID that Fred originally supplied as both operation_id and lifecycle_id.

const (
	LifecycleGenerationUnknown  LifecycleGenerationKind = "unknown"
	LifecycleGenerationLegacy   LifecycleGenerationKind = "legacy"
	LifecycleGenerationTyped    LifecycleGenerationKind = "typed"
	LifecycleGenerationUnusable LifecycleGenerationKind = "unusable"
)

type LifecycleGenerationObservation added in v0.14.0

type LifecycleGenerationObservation struct {
	Kind LifecycleGenerationKind `json:"kind"`
	ID   string                  `json:"id,omitempty"`
}

LifecycleGenerationObservation is an internal backend observation. A nil *LifecycleGenerationObservation on ProvisionInfo is equivalent to Unknown and preserves compatibility with older and third-party backends. Its authority depends on the provenance documented on ProvisionInfo's field.

func ObserveLifecycleGeneration added in v0.14.0

func ObserveLifecycleGeneration(
	callbackURL, lifecycleCallbackURL string,
) LifecycleGenerationObservation

ObserveLifecycleGeneration classifies an exact callback pair already stored by a backend without returning either URL. Missing halves are Unknown so upgraded Fred remains compatible with records written by older backends. Present but malformed, mixed-authority, or non-matching pairs are Unusable and must quarantine lifecycle callbacks for that lease.

type ListProvisionsResponse

type ListProvisionsResponse struct {
	Provisions []ProvisionInfo `json:"provisions"`
	// Continue is the keyset cursor for the next page: the LeaseUUID of the last
	// item in this page when more remain, or "" when the set is exhausted.
	// Absent (omitempty) on the unpaginated path and from pre-pagination servers.
	Continue string `json:"continue,omitempty"`
}

ListProvisionsResponse is the response from the /provisions endpoint.

type ListRetentionsResponse added in v0.5.0

type ListRetentionsResponse struct {
	Retentions []RetainedLease `json:"retentions"`
	// Continue is the keyset cursor for the next page: the LeaseUUID of the last
	// item in this page when more remain, or "" when the set is exhausted.
	// Absent (omitempty) on the unpaginated path and from pre-pagination servers.
	Continue string `json:"continue,omitempty"`
}

ListRetentionsResponse is the response from the GET /retentions endpoint.

type LoadStats added in v0.5.0

type LoadStats struct {
	TotalCPUCores     float64 `json:"total_cpu_cores"`
	AllocatedCPUCores float64 `json:"allocated_cpu_cores"`
	ActiveContainers  int     `json:"active_containers"`
	// DiskWithheld: the backend currently admits no disk-bearing allocation,
	// because a footprint of unknown size remains on its disk. Routing prefers
	// any candidate that does not withhold disk. Absent (false) from a backend
	// that never withholds disk.
	DiskWithheld bool `json:"disk_withheld,omitempty"`
}

LoadStats is the backend load snapshot fred consumes for least-loaded provision routing. It is decoded from the backend's GET /stats response. Only the CPU-related fields and DiskWithheld are needed for routing — CPU is the binding resource on current configurations, so memory and disk sizes are intentionally ignored. Unknown JSON fields in the response are discarded by the decoder.

func (*LoadStats) CPUAllocatedRatio added in v0.5.0

func (s *LoadStats) CPUAllocatedRatio() (ratio float64, ok bool)

CPUAllocatedRatio returns the fraction of CPU allocated (allocated/total) and ok=true when that ratio is meaningful. ok is false when total capacity is unknown or non-positive (e.g., a backend that does not report CPU capacity, or a nil snapshot), signaling the caller to treat this backend as having no usable load signal.

type LogResponse added in v0.14.0

type LogResponse struct {
	// contains filtered or unexported fields
}

LogResponse owns a bounded snapshot of the projected live and failed-attempt entries. Its encoder writes small string fragments: encoding/json's ordinary Marshal and Encoder both allocate the entire escaped document before writing.

func NewLogResponse added in v0.14.0

func NewLogResponse(logs map[string]string) (LogResponse, error)

NewLogResponse takes an immutable snapshot without copying string contents.

func (LogResponse) WriteEnvelopeJSON added in v0.14.0

func (response LogResponse) WriteEnvelopeJSON(w io.Writer, metadata map[string]string) error

WriteEnvelopeJSON writes tenant response metadata followed by the same log map encoder. Metadata is small and separate from tenant-controlled log text.

func (LogResponse) WriteJSON added in v0.14.0

func (response LogResponse) WriteJSON(w io.Writer) error

WriteJSON writes the backend's log map. A nil map retains the JSON null shape.

type MaintenanceCallOutcome added in v0.14.0

type MaintenanceCallOutcome struct {
	// contains filtered or unexported fields
}

MaintenanceCallOutcome is a zero-invalid causal result for restart/update. Refusal is meaningful only when Refused reports true.

func ConservativeMaintenanceCallOutcome added in v0.14.0

func ConservativeMaintenanceCallOutcome(err error) MaintenanceCallOutcome

ConservativeMaintenanceCallOutcome adapts a legacy/custom backend. It can represent only accepted or ambiguous; it cannot mint a terminal refusal.

func InvokeRestart added in v0.14.0

func InvokeRestart(
	ctx context.Context,
	client Backend,
	request RestartRequest,
) MaintenanceCallOutcome

InvokeRestart is the restart counterpart to InvokeProvision.

func InvokeUpdate added in v0.14.0

func InvokeUpdate(
	ctx context.Context,
	client Backend,
	request UpdateRequest,
) MaintenanceCallOutcome

InvokeUpdate is the update counterpart to InvokeProvision.

func (MaintenanceCallOutcome) Accepted added in v0.14.0

func (outcome MaintenanceCallOutcome) Accepted() bool

func (MaintenanceCallOutcome) Ambiguous added in v0.14.0

func (outcome MaintenanceCallOutcome) Ambiguous() bool

func (MaintenanceCallOutcome) Err added in v0.14.0

func (outcome MaintenanceCallOutcome) Err() error

func (MaintenanceCallOutcome) NotDispatched added in v0.14.0

func (outcome MaintenanceCallOutcome) NotDispatched() bool

func (MaintenanceCallOutcome) Refusal added in v0.14.0

func (outcome MaintenanceCallOutcome) Refusal() MaintenanceRefusal

func (MaintenanceCallOutcome) RefusalDetail added in v0.14.0

func (outcome MaintenanceCallOutcome) RefusalDetail() string

RefusalDetail exposes only the diagnostic carried by a definitive validation refusal. It grants no settlement authority; ambiguous/custom-backend errors cannot acquire tenant-visible detail by wrapping a public sentinel.

func (MaintenanceCallOutcome) Refused added in v0.14.0

func (outcome MaintenanceCallOutcome) Refused() bool

func (MaintenanceCallOutcome) Valid added in v0.14.0

func (outcome MaintenanceCallOutcome) Valid() bool

type MaintenanceRefusal added in v0.14.0

type MaintenanceRefusal uint8

MaintenanceRefusal is the closed protocol verdict returned by a backend that positively refused a restart or update before mutation.

const (
	MaintenanceRefusalNone MaintenanceRefusal = iota
	MaintenanceRefusalNotProvisioned
	MaintenanceRefusalInvalidState
	MaintenanceRefusalValidation
	MaintenanceRefusalCapacity
	// MaintenanceRefusalExpired: the command is older than the lease's
	// retained maintenance history and was refused instead of executed.
	MaintenanceRefusalExpired
)

type MatchCriteria

type MatchCriteria struct {
	SKUs []string // Match if SKU is in this exact list
}

MatchCriteria defines how to match a lease to a backend.

type MockBackend

type MockBackend struct {
	// contains filtered or unexported fields
}

MockBackend is an in-memory backend for testing. It simulates provisioning with configurable delays.

func NewMockBackend

func NewMockBackend(cfg MockBackendConfig) *MockBackend

NewMockBackend creates a new mock backend for testing.

func (*MockBackend) Clear

func (m *MockBackend) Clear()

Clear removes all provisions (for testing).

func (*MockBackend) Deprovision

func (m *MockBackend) Deprovision(ctx context.Context, leaseUUID string) error

Deprovision removes a provision.

func (*MockBackend) GetInfo

func (m *MockBackend) GetInfo(ctx context.Context, leaseUUID string) (*LeaseInfo, error)

GetInfo returns mock lease information including connection details.

func (*MockBackend) GetLoadStats added in v0.5.0

func (m *MockBackend) GetLoadStats(_ context.Context) (*LoadStats, error)

GetLoadStats returns the configured load snapshot (a copy, so callers cannot mutate mock state), or (nil, nil) when unset.

func (*MockBackend) GetLogs

func (m *MockBackend) GetLogs(ctx context.Context, leaseUUID string, tail int) (map[string]string, error)

GetLogs returns empty logs for mock backend (Backend interface).

func (*MockBackend) GetProvision

func (m *MockBackend) GetProvision(ctx context.Context, leaseUUID string) (*ProvisionInfo, error)

GetProvision returns status information for a single provision (Backend interface).

func (*MockBackend) GetReleases

func (m *MockBackend) GetReleases(ctx context.Context, leaseUUID string) ([]ReleaseInfo, error)

GetReleases returns empty releases for mock backend.

func (*MockBackend) Health

func (m *MockBackend) Health(ctx context.Context) error

Health always reports healthy: the mock holds its state in memory and has no dependency that can be unreachable.

func (*MockBackend) ListProvisions

func (m *MockBackend) ListProvisions(ctx context.Context) ([]ProvisionInfo, error)

ListProvisions returns all provisions.

func (*MockBackend) ListProvisionsWithIdentity added in v0.14.0

func (m *MockBackend) ListProvisionsWithIdentity(
	ctx context.Context,
) ([]ProvisionInfo, backendidentity.ID, error)

func (*MockBackend) ListRetentions added in v0.5.0

func (m *MockBackend) ListRetentions(_ context.Context) ([]RetainedLease, error)

ListRetentions returns the configured retained leases (a copy), or an empty non-nil slice when unset.

func (*MockBackend) ListRetentionsWithIdentity added in v0.14.0

func (m *MockBackend) ListRetentionsWithIdentity(
	ctx context.Context,
) ([]RetainedLease, backendidentity.ID, error)

func (*MockBackend) LookupProvisions

func (m *MockBackend) LookupProvisions(ctx context.Context, uuids []string) ([]ProvisionInfo, error)

LookupProvisions returns provision info for the requested lease UUIDs. Missing leases are absent from the returned slice.

func (*MockBackend) Name

func (m *MockBackend) Name() string

Name returns the backend's name.

func (*MockBackend) Provision

func (m *MockBackend) Provision(ctx context.Context, req ProvisionRequest) error

Provision simulates starting a provision operation.

func (*MockBackend) ReconcileCustomDomain

func (m *MockBackend) ReconcileCustomDomain(ctx context.Context, leaseUUID string, items []LeaseItem) error

ReconcileCustomDomain is a no-op for mock backend.

func (*MockBackend) RefreshState

func (m *MockBackend) RefreshState(ctx context.Context) error

RefreshState is a no-op for mock backend.

func (*MockBackend) Restart

func (m *MockBackend) Restart(ctx context.Context, req RestartRequest) error

Restart is a no-op for mock backend.

func (*MockBackend) Restore added in v0.5.0

func (m *MockBackend) Restore(ctx context.Context, req RestoreRequest) error

Restore is a no-op for mock backend — returns ErrNotRetained unless the source lease exists in the provisions map (simulating "has retained data").

func (*MockBackend) SetCallbackFunc

func (m *MockBackend) SetCallbackFunc(fn func(CallbackPayload))

SetCallbackFunc sets the function to call when provisioning completes. This simulates the backend calling fred's callback endpoint.

func (*MockBackend) SetGetLoadStatsErr added in v0.13.0

func (m *MockBackend) SetGetLoadStatsErr(err error)

SetGetLoadStatsErr makes GetLoadStats fail with err (nil clears it).

Distinct from SetLoadStats(nil) at the API boundary — "could not be asked" versus "answered, with nothing usable" — but note that RouteForProvision currently collapses the two: either one leaves the candidate out of the load comparison, and round-robin is reached only when NO candidate has usable stats (router.go RouteForProvision). Use this to exercise the error path itself, not to produce a different routing outcome.

func (*MockBackend) SetLoadStats added in v0.5.0

func (m *MockBackend) SetLoadStats(stats *LoadStats)

SetLoadStats configures the snapshot returned by GetLoadStats. Pass nil to simulate a backend with no usable load signal.

func (*MockBackend) SetProvisionStatus

func (m *MockBackend) SetProvisionStatus(leaseUUID string, status ProvisionStatus)

SetProvisionStatus manually sets a provision's status (for testing).

func (*MockBackend) SetRetentions added in v0.5.0

func (m *MockBackend) SetRetentions(r []RetainedLease)

SetRetentions configures the slice returned by ListRetentions.

func (*MockBackend) Update

func (m *MockBackend) Update(ctx context.Context, req UpdateRequest) error

Update is a no-op for mock backend.

type MockBackendConfig

type MockBackendConfig struct {
	Name           string
	ProvisionDelay time.Duration // Simulated provisioning time
}

MockBackendConfig configures a mock backend.

type PortBinding

type PortBinding struct {
	HostIP   string `json:"host_ip"`
	HostPort string `json:"host_port"`
}

PortBinding represents a port mapping from container to host as reported by a backend.

type ProvisionCallOutcome added in v0.14.0

type ProvisionCallOutcome struct {
	// contains filtered or unexported fields
}

ProvisionCallOutcome is a zero-invalid, closed result for one provision transport call. Only the backend package can mint definitive refusal or no-dispatch evidence.

func ConservativeProvisionCallOutcome added in v0.14.0

func ConservativeProvisionCallOutcome(err error) ProvisionCallOutcome

ConservativeProvisionCallOutcome adapts a Backend implementation that does not expose causal transport evidence. Success is accepted; every error is ambiguous and can never authorize durable evidence deletion.

func InvokeProvision added in v0.14.0

func InvokeProvision(
	ctx context.Context,
	client Backend,
	request ProvisionRequest,
) ProvisionCallOutcome

InvokeProvision is the package-owned causal boundary for a provision call. Only the exact production HTTP transport can mint refusal or no-dispatch evidence. A decorator, custom backend, or legacy implementation receives the conservative adapter even if it exposes similarly named methods; Go method embedding therefore cannot forge causal authority.

func (ProvisionCallOutcome) Accepted added in v0.14.0

func (outcome ProvisionCallOutcome) Accepted() bool

func (ProvisionCallOutcome) Ambiguous added in v0.14.0

func (outcome ProvisionCallOutcome) Ambiguous() bool

func (ProvisionCallOutcome) Err added in v0.14.0

func (outcome ProvisionCallOutcome) Err() error

func (ProvisionCallOutcome) NotDispatched added in v0.14.0

func (outcome ProvisionCallOutcome) NotDispatched() bool

func (ProvisionCallOutcome) Refusal added in v0.14.0

func (outcome ProvisionCallOutcome) Refusal() ProvisionRefusal

func (ProvisionCallOutcome) Refused added in v0.14.0

func (outcome ProvisionCallOutcome) Refused() bool

func (ProvisionCallOutcome) Valid added in v0.14.0

func (outcome ProvisionCallOutcome) Valid() bool

type ProvisionInfo

type ProvisionInfo struct {
	LeaseUUID    string          `json:"lease_uuid"`
	ProviderUUID string          `json:"provider_uuid"`
	Status       ProvisionStatus `json:"status"` // see ProvisionStatus* constants
	CreatedAt    time.Time       `json:"created_at"`
	// FailCount is a lifetime diagnostic: how many failures this lease has
	// recorded, whoever caused them. It never decides whether a lease closes;
	// TerminalBudget does (ENG-799).
	FailCount int `json:"fail_count"`
	// LastError (the verbose operator failure detail) is intentionally NOT on
	// this wire type: it crosses to tenant-facing API responses, and verbose
	// exec output / host paths must never leak there (ENG-508). The curated,
	// tenant-safe (Reason, Message) pair replaces it. The operator-only verbose
	// detail lives on leasesm.ProvisionState.LastError and the diagnostics store
	// (DiagnosticEntry.Error), neither of which is tenant-serialized.
	Reason      Reason `json:"reason,omitempty"`
	Message     string `json:"message,omitempty"`
	BackendName string `json:"-"` // Set by the backend or reconciler; excluded from JSON serialization

	// RetainedUntil is the grace-window deadline (CreatedAt + RetentionMaxAge)
	// for a soft-deleted (Status=retained) lease. Zero, and omitted on the wire,
	// for live provisions or when age-based retention expiry is disabled.
	RetainedUntil time.Time `json:"retained_until,omitzero"`
	// Tenant is the owning tenant. It crosses the backend→providerd hop (an
	// HMAC-signed, trusted internal hop, like RestoreRequest.Tenant) so a complete
	// identity-bearing inventory can bind a live placement's runtime maintenance
	// principal, and so the closed-lease authz fallback (when the chain has no
	// record of the lease) can bind a retained record to its owner. It MUST NOT be
	// copied into tenant-facing API responses (LeaseStatusResponse/
	// LeaseProvisionResponse), which would leak one tenant's address to another.
	Tenant string `json:"tenant,omitempty"`

	// LifecycleGeneration reports only the class and, for current typed
	// callbacks, canonical UUID of the callback pair persisted by the backend.
	// Only an observation from complete identity-bearing ListProvisions inventory
	// is settlement evidence. A point lookup may repeat a historical diagnostics
	// observation for read-model continuity, but it is never settlement authority.
	// The field MUST NOT be copied into a tenant-facing response. Nil means that
	// the backend did not report this upgraded internal observation.
	LifecycleGeneration *LifecycleGenerationObservation `json:"lifecycle_generation,omitempty"`

	// TerminalBudget is the backend's consecutive-failure budget for this lease
	// (ENG-799). providerd closes an ACTIVE lease for repeated failure only when
	// Status is failed, Verdict is exactly exhausted and ConsecutiveFailures is
	// at least 1, read from complete ListProvisions inventory. Nil means the
	// backend does not report a budget (an older or third-party backend), and
	// providerd then never closes the lease for repeated failure. A point read
	// served from diagnostics omits it.
	TerminalBudget *TerminalBudgetObservation `json:"terminal_budget,omitempty"`

	// Partition is the retained record's optional sub-tenant grouping key
	// (retention partitioning). Unlike Tenant above, it MAY be copied into
	// tenant-facing retained responses: retained info is only ever surfaced to
	// its owning tenant (authorizeRetained / verifyLeaseAccess), and the value
	// is that tenant's own declaration — it discloses nothing cross-tenant.
	// It MUST NOT be added to the unauthenticated /workloads, /stats, or
	// /metrics surfaces (see WorkloadEntry).
	Partition string `json:"partition,omitempty"`

	// Workload metadata — populated by ListProvisions/GetProvision to describe what is running.
	Image         string            `json:"image,omitempty"`          // Docker image (non-stack leases)
	SKU           string            `json:"sku,omitempty"`            // SKU identifier (non-stack leases)
	Quantity      int               `json:"quantity"`                 // Total expected container count
	Items         []LeaseItem       `json:"items,omitempty"`          // Per-service items (stack leases)
	ServiceImages map[string]string `json:"service_images,omitempty"` // service name → image (stack leases)
}

ProvisionInfo describes a single provisioned resource.

func PaginateProvisions added in v0.5.0

func PaginateProvisions(all []ProvisionInfo, continueToken string, limit int) (page []ProvisionInfo, next string)

PaginateProvisions returns one keyset page of all, sorted by LeaseUUID ascending, containing the entries strictly greater than continueToken. It is a thin type-specialization of keysetPage; see that function for the full contract (limit semantics, cursor monotonicity, non-nil-empty, aliasing).

type ProvisionRefusal added in v0.14.0

type ProvisionRefusal uint8

ProvisionRefusal is the closed protocol verdict returned by a backend that positively refused one provision request before mutation.

const (
	ProvisionRefusalNone ProvisionRefusal = iota
	ProvisionRefusalValidation
	ProvisionRefusalCapacity
)

type ProvisionRequest

type ProvisionRequest struct {
	LeaseUUID            string      `json:"lease_uuid"`
	Tenant               string      `json:"tenant"`
	ProviderUUID         string      `json:"provider_uuid"`
	Items                []LeaseItem `json:"items"`
	CallbackURL          string      `json:"callback_url"`
	LifecycleCallbackURL string      `json:"lifecycle_callback_url,omitempty"`
	Payload              []byte      `json:"payload,omitempty"`
	PayloadHash          string      `json:"payload_hash,omitempty"`
}

ProvisionRequest contains the data needed to provision a resource.

func (ProvisionRequest) RoutingSKU

func (r ProvisionRequest) RoutingSKU() string

RoutingSKU returns the SKU of the first item for backend routing decisions.

Why this exists: A lease may contain multiple items with different SKUs (e.g., [{sku: "docker-micro", qty: 2}, {sku: "docker-large", qty: 1}]). However, all items in a single lease are guaranteed to belong to the same provider - this is enforced by the chain. Therefore, any SKU from the lease can be used to determine which backend should handle the request.

This method returns the first SKU purely for routing. It should NOT be used to determine resource allocation - use Items directly for that.

func (ProvisionRequest) TotalQuantity

func (r ProvisionRequest) TotalQuantity() int

TotalQuantity returns the sum of quantities across all items.

type ProvisionResponse

type ProvisionResponse struct {
	ProvisionID string `json:"provision_id"`
}

ProvisionResponse is returned by the backend after accepting a provision request.

type ProvisionStatus

type ProvisionStatus string

ProvisionStatus represents the status of a provisioned resource.

const (
	ProvisionStatusProvisioning   ProvisionStatus = "provisioning"
	ProvisionStatusReady          ProvisionStatus = "ready"
	ProvisionStatusFailing        ProvisionStatus = "failing"
	ProvisionStatusFailed         ProvisionStatus = "failed"
	ProvisionStatusUnknown        ProvisionStatus = "unknown"
	ProvisionStatusRestarting     ProvisionStatus = "restarting"
	ProvisionStatusUpdating       ProvisionStatus = "updating"
	ProvisionStatusDeprovisioning ProvisionStatus = "deprovisioning"
	// ProvisionStatusRetained is published to the tenant at lease close/expire
	// time to signal that managed volumes may have been kept under a
	// fred-retained- namespace if the backend has retain_on_close enabled.
	// This is a best-effort hint; the backend may not have retention configured.
	ProvisionStatusRetained ProvisionStatus = "retained"
)

Provision status constants.

ProvisionStatusFailing marks the window between a container's death being detected and the terminal Failed callback being emitted. A Deprovision request arriving in this window transitions the lease straight to Deprovisioning without ever reaching Failed — preventing a stale Failed callback under concurrent ownership transfer.

type Reason added in v0.13.0

type Reason string

Reason is a stable, machine-readable failure-category code surfaced to tenants alongside a human message (K8s condition Reason shape). It is AUTHORED at the failure source, never parsed from the human message (which may be dynamically composed). A defined type so a stray verbose string (e.g. err.Error()) is a compile error at an authoring site.

This is an OPEN, add-only set: consumers MUST tolerate an unrecognized value and fall back to the human message. Distinct from leasesm.ContainerExitInfo.Reason, which is the raw substrate termination reason (e.g. "OOMKilled").

const (
	ReasonContainerExited Reason = "ContainerExited"
	// ReasonHealthCheckFailed marks a container whose health check never passed
	// during startup verification: Docker reported it unhealthy, or it was still
	// not healthy at the startup deadline. It is never counted toward the
	// terminal failure budget (ENG-1125).
	ReasonHealthCheckFailed Reason = "HealthCheckFailed"
	// ReasonContainerStartFailed marks a provision container that the
	// container runtime refused to start: Docker answered its start request
	// with an error and the container never ran (for example, an entrypoint
	// that does not exist in the image). It is never counted toward the
	// terminal failure budget (ENG-1125).
	ReasonContainerStartFailed   Reason = "ContainerStartFailed"
	ReasonImagePullFailed        Reason = "ImagePullFailed"
	ReasonInternal               Reason = "Internal"
	ReasonRestartFailed          Reason = "RestartFailed"
	ReasonUpdateFailed           Reason = "UpdateFailed"
	ReasonRestoreFailed          Reason = "RestoreFailed"
	ReasonVolumeCleanupExhausted Reason = "VolumeCleanupExhausted"
	ReasonCleanupFailed          Reason = "CleanupFailed"
	// ReasonBackendStorageLost marks a lease whose backend an operator
	// retired as irrecoverably lost; it is authored by the placement store.
	ReasonBackendStorageLost Reason = "BackendStorageLost"
	// ReasonVolumeDeletePending marks a provision refused because an earlier
	// deletion of the lease's own volume is still finishing on the provider.
	// It is transient on the provider side; the tenant has nothing to retry.
	ReasonVolumeDeletePending Reason = "VolumeDeletePending"
	// ReasonVolumeDeletionInProgress marks a closing lease whose volume is
	// still being deleted by the provider. It is progress, not a failure: the
	// close completes when the deletion does.
	ReasonVolumeDeletionInProgress Reason = "VolumeDeletionInProgress"
	// ReasonUnknown is the read-boundary default for a FAILED lease with no
	// authored reason (a legacy pre-upgrade record or a future-unmapped
	// path). gRPC-UNKNOWN-equivalent: "failed, cause unclassified".
	ReasonUnknown Reason = "Unknown"
)

type ReconcileCustomDomainRequest

type ReconcileCustomDomainRequest struct {
	LeaseUUID string      `json:"lease_uuid"`
	Items     []LeaseItem `json:"items"`
}

ReconcileCustomDomainRequest is the wire format for POST /reconcile_custom_domain.

type ReleaseInfo

type ReleaseInfo struct {
	Version   int       `json:"version"`
	Image     string    `json:"image"`
	Status    string    `json:"status"`
	CreatedAt time.Time `json:"created_at"`
	// Error (verbose release-failure detail) is intentionally NOT on this
	// tenant-serialized wire type (ENG-508); the curated (Reason, Message) pair
	// replaces it. Operator-only verbose detail stays in the diagnostics store
	// and structured logs (the release store keeps only the curated pair).
	Reason   Reason `json:"reason,omitempty"`
	Message  string `json:"message,omitempty"`
	Manifest []byte `json:"manifest"`
}

ReleaseInfo describes a single release in the history.

type RestartRequest

type RestartRequest struct {
	LeaseUUID     string           `json:"lease_uuid"`
	MaintenanceID maintenanceid.ID `json:"maintenance_id"`
	CallbackURL   string           `json:"callback_url"`
	// AdmittedAt is the provider's per-lease admission stamp (RFC 3339, UTC).
	// Every replay of the command sends the same value. Omitted by older
	// providers.
	AdmittedAt time.Time `json:"admitted_at,omitzero"`
}

RestartRequest contains the data needed to restart a lease's containers.

type RestoreCallOutcome added in v0.14.0

type RestoreCallOutcome struct {
	// contains filtered or unexported fields
}

RestoreCallOutcome is the restore counterpart to ProvisionCallOutcome.

func ConservativeRestoreCallOutcome added in v0.14.0

func ConservativeRestoreCallOutcome(err error) RestoreCallOutcome

ConservativeRestoreCallOutcome adapts a legacy/custom backend without manufacturing definitive evidence from its arbitrary error tree.

func InvokeRestore added in v0.14.0

func InvokeRestore(
	ctx context.Context,
	client Backend,
	request RestoreRequest,
) RestoreCallOutcome

InvokeRestore is the restore counterpart to InvokeProvision.

func (RestoreCallOutcome) Accepted added in v0.14.0

func (outcome RestoreCallOutcome) Accepted() bool

func (RestoreCallOutcome) Ambiguous added in v0.14.0

func (outcome RestoreCallOutcome) Ambiguous() bool

func (RestoreCallOutcome) Err added in v0.14.0

func (outcome RestoreCallOutcome) Err() error

func (RestoreCallOutcome) NotDispatched added in v0.14.0

func (outcome RestoreCallOutcome) NotDispatched() bool

func (RestoreCallOutcome) Refusal added in v0.14.0

func (outcome RestoreCallOutcome) Refusal() RestoreRefusal

func (RestoreCallOutcome) Refused added in v0.14.0

func (outcome RestoreCallOutcome) Refused() bool

func (RestoreCallOutcome) Valid added in v0.14.0

func (outcome RestoreCallOutcome) Valid() bool

type RestoreRefusal added in v0.14.0

type RestoreRefusal uint8

RestoreRefusal is the closed protocol verdict returned by a backend that positively refused one restore request before mutation.

const (
	RestoreRefusalNone RestoreRefusal = iota
	RestoreRefusalNotRetained
	RestoreRefusalInvalidState
	RestoreRefusalCapacity
	RestoreRefusalDemoteDataExceedsTier
	RestoreRefusalValidation
)

type RestoreRequest added in v0.5.0

type RestoreRequest struct {
	LeaseUUID            string      `json:"lease_uuid"`      // NEW lease
	FromLeaseUUID        string      `json:"from_lease_uuid"` // original (retained) lease
	Tenant               string      `json:"tenant"`
	ProviderUUID         string      `json:"provider_uuid"` // when non-empty, cross-checked against the retained record
	Items                []LeaseItem `json:"items"`         // must match the retained set
	CallbackURL          string      `json:"callback_url"`
	LifecycleCallbackURL string      `json:"lifecycle_callback_url,omitempty"`
}

RestoreRequest contains the data needed to restore a soft-deleted lease's retained volumes into a NEW lease (ENG-325). FromLeaseUUID identifies the original (retained) lease whose data is adopted; LeaseUUID is the new lease the data is restored into. Items must match the retained set's shape (service-name → summed-quantity).

type RetainedLease added in v0.5.0

type RetainedLease struct {
	LeaseUUID    string `json:"lease_uuid"`
	ProviderUUID string `json:"provider_uuid,omitempty"`
	Tenant       string `json:"tenant,omitempty"`
}

RetainedLease identifies a lease whose data this backend currently retains (soft-deleted, awaiting restore or grace-reap). The reconciler consumes this to keep placement affinity for retained leases so a restore routes to the backend holding the source data (ENG-333). ProviderUUID and Tenant are optional for wire compatibility with older/third-party backends; when present they are trusted-internal inventory evidence, never tenant-facing response fields. Offline repair treats missing identity as routing-only and rejects contradictory nonempty identity.

func PaginateRetentions added in v0.7.0

func PaginateRetentions(all []RetainedLease, continueToken string, limit int) (page []RetainedLease, next string)

PaginateRetentions returns one keyset page of all, sorted by LeaseUUID ascending, containing the entries strictly greater than continueToken. It is the /retentions sibling of PaginateProvisions; both delegate to keysetPage, so they share identical cursor semantics (see keysetPage for the full contract).

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router routes requests to backends based on SKU matching.

func NewRouter

func NewRouter(cfg RouterConfig) (*Router, error)

NewRouter creates a new backend router.

func (*Router) Backends

func (r *Router) Backends() []Backend

Backends returns the constructor-validated unique backends for operations such as reconciliation and health checks.

func (*Router) Default

func (r *Router) Default() Backend

Default returns the default backend.

func (*Router) GetBackendByName

func (r *Router) GetBackendByName(name string) Backend

GetBackendByName returns a backend by its name. Returns nil if not found.

func (*Router) HasBackend added in v0.14.0

func (r *Router) HasBackend(name string) bool

HasBackend reports whether name belongs to this immutable router topology. It deliberately exposes no mutation-capable backend client, which makes it suitable for observational consumers such as bounded metric-label filters.

func (*Router) HealthCheck

func (r *Router) HealthCheck(ctx context.Context) ([]BackendHealth, bool)

HealthCheck checks the health of all configured backends. Returns a slice of health statuses and an overall healthy flag.

Probes run CONCURRENTLY, and the caller is expected to pass a deadline-bounded context. This sits on the /health and /readyz request path, where probing serially made the worst case the SUM of every backend's client timeout (30s each by default) rather than the MAX. One backend that accepts a connection and never answers was therefore enough to push the handler past the API server's own request timeout — whose response body is a 503, the exact status code /health exists never to send (ENG-522). Concurrency is what lets a bounded budget hold without starving the backends probed last.

Results stay in Backends() order regardless of completion order: each probe writes its own slot, so a caller diffing successive responses sees a stable sequence.

func (*Router) Route

func (r *Router) Route(sku string) Backend

Route returns the appropriate backend for the given SKU.

func (*Router) RouteAll

func (r *Router) RouteAll(sku string) []Backend

RouteAll returns all backends that match the given SKU. NewRouter rejects duplicate durable backend names, so every returned backend is already unique. If no backends match, returns nil.

func (*Router) RouteForProvision added in v0.5.0

func (r *Router) RouteForProvision(ctx context.Context, sku string, inFlightByBackend map[string]int) Backend

RouteForProvision selects the SKU-matching backend with the lowest observed allocated-CPU ratio for a new provision, preferring candidates that do not report DiskWithheld. Ties (equal ratio within cpuRatioEpsilon — the burst case, where concurrent provisions read the same /stats snapshot) are broken by the fewest in-flight provisions, then by the round-robin counter so a burst of identical-state provisions spreads across the tied backends. Falls back to round-robin when no SKU-matching backend exposes usable load stats. inFlightByBackend may be nil (treated as all-zero).

This is a deliberate live-query design — each candidate's /stats is fetched at decision time — NOT a passive-counter load balancer (Envoy/NGINX style). The binding CPU-allocation signal lives on the backends and fred holds no local per-SKU CPU weights, so the only way to learn load is to ask. A GetLoadStats error or timeout demotes that candidate to "no usable signal" (excluded from the comparison; the error is recorded by doGet as fred_backend_requests_total{operation="get_load_stats",status="error"}). A residual herd window remains when concurrent provisions read the same pre-update /stats snapshot and the in-flight tiebreak does not separate them; it is tolerated because the backend's 503 admission gate hard-caps any over-targeted backend and provision QPS is low. Tie rotation is intentionally approximate because the same counter also drives no-stats fallback routing.

A fenced backend never receives a new provision. A SKU that only fenced backends serve yields nil, so its lease waits for the fence to lift rather than landing on a backend the operator never assigned that SKU to.

func (*Router) RouteForProvisionAmong added in v0.14.0

func (r *Router) RouteForProvisionAmong(
	ctx context.Context,
	sku string,
	eligibleNames map[string]struct{},
	inFlightByBackend map[string]int,
) Backend

RouteForProvisionAmong selects a provision backend using the same SKU, default, and load-balancing rules as RouteForProvision, but treats eligibleNames as a hard routing boundary. An empty set, or a set containing neither a matching backend nor the default backend, yields nil.

When matching candidates expose no usable load stats, round-robin fallback is restricted to those eligible candidates. The default backend is considered only when no eligible backend matches the SKU, and only when it is itself eligible. Fenced backends are excluded exactly as in RouteForProvision. A SKU that only fenced backends serve yields nil, never the default; this is decided before eligibility, because a fenced backend never answers and so is never eligible.

type RouterConfig

type RouterConfig struct {
	Backends []BackendEntry

	// Optional Prometheus gauge for backend health status. When nil, HealthCheck
	// skips metric recording. This prevents binaries that don't use these metrics
	// from registering phantom fred-level gauges via transitive imports.
	BackendHealthy *prometheus.GaugeVec // labels: backend

	// AllocatedCPURatio, when non-nil, records the allocated-CPU ratio observed
	// for each backend at provision-routing time. Labels: backend.
	AllocatedCPURatio *prometheus.GaugeVec

	// RoutingFallback, when non-nil, counts provision-routing decisions that
	// fell back to round-robin because no candidate exposed usable load stats.
	RoutingFallback prometheus.Counter

	// HealthProbePanics, when non-nil, counts panics recovered inside a
	// per-backend health probe goroutine. Non-zero is always a bug.
	HealthProbePanics prometheus.Counter
}

RouterConfig configures the backend router.

type TerminalBudgetObservation added in v0.14.0

type TerminalBudgetObservation struct {
	Verdict             TerminalVerdict `json:"verdict"`
	ConsecutiveFailures int             `json:"consecutive_failures"`
}

TerminalBudgetObservation is the backend-authored state of one lease's consecutive-failure budget. The backend authors it where each failure happens and counts only failures its tenant's own workload caused; it is never derived from FailCount. ConsecutiveFailures is the recorded count: time alone never decays it, although the backend resets it at the next transition once the lease has stayed Ready long enough (BACKEND_GUIDE.md).

type TerminalVerdict added in v0.14.0

type TerminalVerdict string

TerminalVerdict is a backend's decision about a lease's consecutive-failure budget (ENG-799). It is a closed set: a consumer MUST treat any other value, including "", as no verdict, and no verdict never closes a lease.

const (
	// TerminalVerdictRetry means the lease may be re-provisioned.
	TerminalVerdictRetry TerminalVerdict = "retry"
	// TerminalVerdictExhausted means the lease is Failed and its tenant
	// workload has failed consecutively often enough, over a long enough
	// streak, that providerd closes the lease on-chain instead of
	// re-provisioning it. The backend records it at the counted failure;
	// time alone never produces it.
	TerminalVerdictExhausted TerminalVerdict = "exhausted"
)

type UpdateRequest

type UpdateRequest struct {
	LeaseUUID     string           `json:"lease_uuid"`
	MaintenanceID maintenanceid.ID `json:"maintenance_id"`
	CallbackURL   string           `json:"callback_url"`
	Payload       []byte           `json:"payload"`
	// AdmittedAt is the provider's per-lease admission stamp; see RestartRequest.
	AdmittedAt time.Time `json:"admitted_at,omitzero"`
}

UpdateRequest contains the data needed to update a lease to a new manifest.

type ValidationCode

type ValidationCode string

ValidationCode identifies the sub-category of a validation error in HTTP responses. Backends include this in 400 JSON bodies so the client can reconstruct the correct sentinel error across the HTTP boundary.

const (
	ValidationCodeUnknownSKU      ValidationCode = "unknown_sku"
	ValidationCodeInvalidManifest ValidationCode = "invalid_manifest"
	ValidationCodeImageNotAllowed ValidationCode = "image_not_allowed"
)

Validation code constants used in HTTP error responses.

func ClassifyValidationError

func ClassifyValidationError(err error) ValidationCode

ClassifyValidationError returns the ValidationCode for a validation error. Returns "" if the error does not match any known sub-category.

Directories

Path Synopsis
Package docker implements the Backend interface for Docker container provisioning.
Package docker implements the Backend interface for Docker container provisioning.
imageexec
Package imageexec admits immutable container images and binds their use to typed Docker and Compose execution sinks.
Package imageexec admits immutable container images and binds their use to typed Docker and Compose execution sinks.
imagefetch
Package imagefetch admits registry content before giving Docker any bytes.
Package imagefetch admits registry content before giving Docker any bytes.
tenantseccomp
Package tenantseccomp owns the seccomp profile fred applies to every container it creates for a tenant, and every Docker security option that names a seccomp profile.
Package tenantseccomp owns the seccomp profile fred applies to every container it creates for a tenant, and every Docker security option that names a seccomp profile.
Package k3s is the scaffold for Fred's K3s backend.
Package k3s is the scaffold for Fred's K3s backend.
Package shared provides backend-agnostic components that can be reused across different backend implementations (Docker, Kubernetes, Nomad, etc.).
Package shared provides backend-agnostic components that can be reused across different backend implementations (Docker, Kubernetes, Nomad, etc.).
completion
Package completion owns the bounded completion window of an admitted effect.
Package completion owns the bounded completion window of an admitted effect.
imagebudget
Package imagebudget keeps immutable image verification work and physical daemon allocation in separate dimensions across preparation and durable pins.
Package imagebudget keeps immutable image verification work and physical daemon allocation in separate dimensions across preparation and durable pins.
leasesm
Package leasesm owns the per-lease state machine and actor, plus the substrate-agnostic seams they consume.
Package leasesm owns the per-lease state machine and actor, plus the substrate-agnostic seams they consume.
leasesm/failurecause
Package failurecause attributes one lease workload failure to its cause, for the consecutive-failure terminal budget the lease state machine keeps (ENG-799).
Package failurecause attributes one lease workload failure to its cause, for the consecutive-failure terminal budget the lease state machine keeps (ENG-799).
manifest
Package manifest defines the tenant-facing container manifest schema (StackManifest, holding a map of named per-service Manifest entries), JSON parsing, and validation.
Package manifest defines the tenant-facing container manifest schema (StackManifest, holding a map of named per-service Manifest entries), JSON parsing, and validation.
substratemutation
Package substratemutation defines the causal protocol between a durable mutation journal and an attested physical-substrate executor.
Package substratemutation defines the causal protocol between a durable mutation journal and an attested physical-substrate executor.
workbarrier
Package workbarrier provides a reference-counted barrier whose zero signal is exposed as a real channel, so callers can wait-with-timeout using a plain select without spawning helper goroutines that would leak when the count never drops to zero.
Package workbarrier provides a reference-counted barrier whose zero signal is exposed as a real channel, so callers can wait-with-timeout using a plain select without spawning helper goroutines that would leak when the count never drops to zero.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL