api

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 28, 2026 License: Apache-2.0 Imports: 52 Imported by: 0

Documentation

Overview

Package api provides the HTTP API server for fred.

The API serves three main purposes:

  1. Tenant Access: Authenticated endpoints for tenants to retrieve connection details and upload deployment payloads for their leases.

  2. Backend Callbacks: Endpoint for backends to report provisioning results with HMAC-SHA256 authentication.

  3. Observability: Health check and Prometheus metrics endpoints.

Authentication

Tenant endpoints use ADR-036 signature-based authentication. Tenants create a bearer token containing their address, the lease UUID, a timestamp, their public key, and a signature over the message. The server validates:

  • The signature matches the message content
  • The public key derives to the tenant address
  • The timestamp is at most 30 seconds in the past and at most 10 seconds in the future (clock-skew tolerance)
  • Replay protection (TokenTracker): required for connection, restart, update, and restore — where a replayed token would re-leak sensitive data or re-run a mutating operation. Idempotent reads (status, provision, logs, releases, events) skip this check. The data upload endpoint skips it too and relies on its own idempotency (409 on a duplicate upload for the lease).

The token tracker uses fail-closed semantics: if the database is unavailable, requests are rejected with 503 Service Unavailable rather than proceeding without replay protection. Retries need a fresh token; deterministic signing with the same tenant, lease, and whole-second timestamp reproduces the same token. Wait for a later Unix timestamp before minting a retry token. Restart and update retries also retain the original command's Idempotency-Key.

Backend callbacks use HMAC-SHA256 authentication. Production selects a distinct per-backend key by the callback's HMAC-covered immutable storage identity; providerd maps it from backends[].hmac_secret and the corresponding backend maps the same value from callback_secret. A fleet-wide top-level callback_secret remains available only as a non-production compatibility mode.

Rate Limiting

Tenant and observability routes use:

  • Per-IP rate limiting (via RateLimiter)
  • Per-tenant rate limiting for authenticated endpoints (via TenantRateLimiter)

Both use token bucket algorithms with configurable RPS and burst sizes. POST /callbacks/provision instead has independent, fixed 100 RPS / 200 burst buckets for ingress IPs and verified backend storage identities. A valid HMAC may bypass an exhausted ingress bucket, but still spends its storage budget. This isolates backend completion from tenant traffic sharing the same NAT. The shared wire decoder bounds every callback to 1 MiB, 256 structural tokens, and 16 nesting levels before JSON decoding. Exact signed retries remain authenticated and budgeted; there is no callback replay cache.

Endpoints 

GET  /health                                - Health check with chain connectivity
GET  /metrics                               - Prometheus metrics
GET  /workloads?lease_uuid=<u>...           - Bulk workload metadata lookup (unauthenticated)
GET  /v1/leases/{lease_uuid}/connection     - Get connection details (authenticated)
GET  /v1/leases/{lease_uuid}/status         - Get provisioning status (authenticated)
GET  /v1/leases/{lease_uuid}/provision      - Get provision diagnostics (authenticated)
GET  /v1/leases/{lease_uuid}/logs           - Get container logs (authenticated)
GET  /v1/leases/{lease_uuid}/releases       - Get release history (authenticated)
GET  /v1/leases/{lease_uuid}/events         - Stream lease events via WebSocket (authenticated)
POST /v1/leases/{lease_uuid}/data           - Upload deployment payload (authenticated)
POST /v1/leases/{lease_uuid}/restart        - Restart a provisioned lease (authenticated)
POST /v1/leases/{lease_uuid}/update         - Update a provisioned lease (authenticated)
POST /v1/leases/{lease_uuid}/restore        - Restore retained data into a pending lease (authenticated)
POST /callbacks/provision                   - Backend provisioning callback (HMAC auth)

Index

Examples

Constants

View Source 
const (
	// MaxTokenAge is the maximum age of a valid authentication token.
	// Kept short (30 seconds) to limit replay attack window.
	MaxTokenAge = 30 * time.Second

	// MaxFutureClockSkew is the maximum allowed clock skew for tokens with future timestamps.
	// This is intentionally smaller than MaxTokenAge to limit pre-generated token attacks.
	// 10 seconds allows for reasonable clock drift without enabling abuse.
	MaxFutureClockSkew = 10 * time.Second
)
View Source 
const (

	// DefaultMaxSubscriptionsPerLease is the maximum number of concurrent
	// WebSocket subscriptions allowed per lease UUID.
	DefaultMaxSubscriptionsPerLease = 10

	// DefaultMaxTotalSubscriptions is the maximum number of concurrent
	// WebSocket subscriptions allowed globally across all leases.
	DefaultMaxTotalSubscriptions = 1000
)
View Source 
const (
	// CallbackSignatureHeader is the header name for HMAC signatures on callbacks.
	// Format: "t=<unix-timestamp>,sha256=<hex-encoded-hmac>"
	CallbackSignatureHeader = hmacauth.SignatureHeader

	// DefaultCallbackMaxAge is the default maximum age for callback timestamps.
	// Callbacks older than this are rejected, bounding same-endpoint replay to
	// this freshness window. The callback protocol has no nonce cache.
	DefaultCallbackMaxAge = 5 * time.Minute

	// MinCallbackSecretLength is the minimum required length for callback secrets.
	// It aliases the shared signing-boundary contract so senders and verifiers
	// cannot silently drift apart.
	MinCallbackSecretLength = hmacauth.MinSecretLength
)

Variables

View Source 
var (
	// ErrTokenAlreadyUsed indicates the token has already been used.
	ErrTokenAlreadyUsed = errors.New("token already used")

	// ErrInvalidReplayClaim indicates absent validation or an expired token.
	ErrInvalidReplayClaim = errors.New("invalid or expired replay claim")
)
View Source 
var ErrTooManySubscriptions = errors.New("too many subscriptions")

ErrTooManySubscriptions is returned when a subscription limit is reached.

Functions

func WSTokenPromoter 

func WSTokenPromoter(next http.Handler) http.Handler

WSTokenPromoter is middleware that promotes a WebSocket "token" query parameter to the Authorization header and strips it from r.URL. Upstream proxies have already seen the original URL, and r.RequestURI is unchanged; deployments must redact query tokens in access logs and traces separately.

Types

type AuthToken 

type AuthToken struct {
	Tenant    string `json:"tenant"`
	LeaseUUID string `json:"lease_uuid"`
	Timestamp int64  `json:"timestamp"`
	PubKey    string `json:"pub_key"`   // Base64-encoded public key
	Signature string `json:"signature"` // Base64-encoded signature
	// contains filtered or unexported fields
}

AuthToken represents the bearer token for tenant authentication.

func AuthTokenFromContext 

func AuthTokenFromContext(ctx context.Context) *AuthToken

AuthTokenFromContext retrieves the pre-validated AuthToken from request context. Returns nil when called outside the authenticated router middleware.

func ParseAuthToken 

func ParseAuthToken(encoded string) (*AuthToken, error)

ParseAuthToken parses a base64-encoded authentication token.

func (*AuthToken) Validate 

func (t *AuthToken) Validate(bech32Prefix string) error

Validate verifies the token's timestamp and ADR-036 signature. The bech32Prefix is used to verify the tenant address matches the public key. On success, t.Signature is updated to the low-S canonical form to ensure consistent replay tracking regardless of which signature variant was submitted.

type AuthenticatedRequest 

type AuthenticatedRequest struct {
	Token *AuthToken
	Lease *billingtypes.Lease
}

AuthenticatedRequest contains the result of a successful authentication.

type CallbackAuthenticator 

type CallbackAuthenticator struct {
	// contains filtered or unexported fields
}

CallbackAuthenticator verifies HMAC signatures on backend callbacks. Its timestamp limits same-endpoint replay to a bounded freshness window; method and URI binding prevent cross-endpoint replay. There is intentionally no nonce cache for backend protocol messages.

Performance note: The current implementation uses fmt.Sprintf to build the signed payload, which allocates an intermediate string. This is acceptable because callback payloads are small by design (~100 bytes: lease_uuid, status, error). The large Payload field (potentially megabytes) is in ProvisionRequest going TO backends, not in CallbackPayload coming back. If signing large data becomes necessary, consider writing to the HMAC incrementally to avoid copying the payload.

func NewCallbackAuthenticator 

func NewCallbackAuthenticator(
	secret string,
	proofVerifier hmacauth.CallbackProofVerifier,
) (*CallbackAuthenticator, error)

NewCallbackAuthenticator creates a new callback authenticator with the given secret. Uses DefaultCallbackMaxAge as its replay freshness bound. Returns an error if the secret is shorter than MinCallbackSecretLength bytes.

func (*CallbackAuthenticator) ComputeSignature 

func (a *CallbackAuthenticator) ComputeSignature(method, uri string, payload []byte) string

ComputeSignature computes the HMAC-SHA256 signature for a request shape with the current timestamp. method and uri must match what the verifier will see on the wire (typically req.Method and req.URL.RequestURI()). Returns the signature in the format "t=<timestamp>,sha256=<hex>".

Example

Example showing the signature format 

verifier, _ := hmacauth.NewCallbackProofBoundary()
auth, err := NewCallbackAuthenticator("my-secret-key-at-least-32-characters", verifier)
if err != nil {
	fmt.Println("Error:", err)
	return
}
payload := []byte(`{"lease_uuid":"abc-123","status":"success"}`)
signature := auth.ComputeSignature(testCallbackMethod, testCallbackURI, payload)
// Output format: t=<unix-timestamp>,sha256=<hex-encoded-hmac>
fmt.Println("Signature format:", strings.Split(signature, ",")[0][:2]) // "t="

Output:
Signature format: t=

func (*CallbackAuthenticator) VerifyCallbackEvidence added in v0.14.0 

func (a *CallbackAuthenticator) VerifyCallbackEvidence(
	r *http.Request,
) (hmacauth.VerifiedRequest, error)

VerifyCallbackEvidence authenticates the exact callback request envelope and returns opaque evidence rather than a caller-constructible DTO.

func (*CallbackAuthenticator) VerifySignature 

func (a *CallbackAuthenticator) VerifySignature(method, uri string, payload []byte, signature string) bool

VerifySignature verifies that the provided signature matches the request shape. method and uri must match what the sender used (typically r.Method and r.URL.RequestURI() on the inbound request). The signature should be in the format "t=<timestamp>,sha256=<hex>". Returns false if the signature is invalid, the timestamp is too old, or the timestamp is too far in the future.

func (*CallbackAuthenticator) VerifySignatureWithTime 

func (a *CallbackAuthenticator) VerifySignatureWithTime(method, uri string, payload []byte, signature string, now time.Time) bool

VerifySignatureWithTime verifies the signature against an explicit reference time. VerifySignature is the production entry point and delegates here with a.now(); tests call it directly to pin the clock and drive the replay window deterministically.

func (*CallbackAuthenticator) WithCanonicalPathPrefix 

func (a *CallbackAuthenticator) WithCanonicalPathPrefix(prefix string) *CallbackAuthenticator

WithCanonicalPathPrefix configures a static path prefix that is prepended to r.URL.RequestURI() before HMAC verification. Set this when fred is deployed behind a path-stripping reverse proxy (e.g., Traefik stripPrefix mapping /api/fred/* → /*) so the verifier's canonical URI matches what the signer used. Passing the empty string is a no-op and preserves the default direct- call behavior. Returns the receiver for chaining.

type CallbackKey added in v0.14.0 

type CallbackKey struct {
	Backend string
	Keys    hmacauth.VerifyKeys
	Fenced  bool
}

CallbackKey is one backend's callback authentication: its configured name, which labels metrics only after its key verified a callback, and the keys it may sign with (its current key and, during a rotation, its previous key). A fenced backend's keys authenticate nothing: they only let a refusal of its callbacks be counted as fenced rather than as a forgery.

type CallbackKeyringAuthenticator added in v0.14.0 

type CallbackKeyringAuthenticator struct {
	// contains filtered or unexported fields
}

CallbackKeyringAuthenticator verifies callbacks with the keys assigned to the immutable storage lineage named in the HMAC-covered payload. The map is copied at construction so callers cannot rotate authority behind an in-flight verification. Its zero value is invalid.

func NewCallbackKeyringAuthenticator added in v0.14.0 

func NewCallbackKeyringAuthenticator(
	keys map[backendidentity.ID]CallbackKey,
	proofVerifier hmacauth.CallbackProofVerifier,
) (*CallbackKeyringAuthenticator, error)

NewCallbackKeyringAuthenticator constructs the production callback verifier. Identities, backend names and every key must be unique, keys compared by HMAC equivalence: two lineages accepting a common key, even a previous key during a rotation, would silently restore fleet-wide callback authority.

func (*CallbackKeyringAuthenticator) VerifyCallbackEvidence added in v0.14.0 

func (a *CallbackKeyringAuthenticator) VerifyCallbackEvidence(
	r *http.Request,
) (hmacauth.VerifiedRequest, error)

VerifyCallbackEvidence authenticates with the immutable storage-lineage key selected by the signed body and binds that route into the returned proof. The backend storage ID is only an untrusted key selector until HMAC verification succeeds; callback application later binds the same signed ID to operation/lifecycle-owned placement authority.

func (*CallbackKeyringAuthenticator) WithCanonicalPathPrefix added in v0.14.0 

func (a *CallbackKeyringAuthenticator) WithCanonicalPathPrefix(
	prefix string,
) *CallbackKeyringAuthenticator

WithCanonicalPathPrefix applies the same reverse-proxy canonicalization contract as CallbackAuthenticator.

type CallbackPublisher 

type CallbackPublisher interface {
	PublishCallback(ctx context.Context, request hmacauth.VerifiedRequest) error
}

CallbackPublisher applies authenticated backend callbacks through the provisioner. Implementations return only after the callback has reached a terminal application result, so a backend can preserve per-lease ordering by waiting for the HTTP response before sending the next durable delivery.

type CallbackResponse 

type CallbackResponse struct {
	Status  string `json:"status"`
	Message string `json:"message,omitempty"`
}

CallbackResponse represents the response for backend callbacks. Used to provide debugging information to authenticated backends.

type ChainClient 

type ChainClient interface {
	GetLease(ctx context.Context, leaseUUID string) (*billingtypes.Lease, error)
	GetActiveLease(ctx context.Context, leaseUUID string) (*billingtypes.Lease, error)
	Ping(ctx context.Context) error
}

ChainClient defines the chain operations needed by handlers.

type CheckResult 

type CheckResult struct {
	Status  string `json:"status"`
	Message string `json:"message,omitempty"`
}

CheckResult represents the result of a single health check.

type ConnectionDetails 

type ConnectionDetails struct {
	Host      string                              `json:"host"`
	FQDN      string                              `json:"fqdn,omitempty"`
	Ports     map[string]PortMapping              `json:"ports,omitempty"`
	Instances []InstanceInfo                      `json:"instances,omitempty"`
	Services  map[string]ServiceConnectionDetails `json:"services,omitempty"`
	Protocol  string                              `json:"protocol,omitempty"`
	Metadata  map[string]string                   `json:"metadata,omitempty"`
}

ConnectionDetails contains the connection information for a lease. For multi-instance leases, the Instances array contains per-instance details. For stack (multi-service) leases, the Services map contains per-service details.

type ConnectionResponse 

type ConnectionResponse struct {
	LeaseUUID    string            `json:"lease_uuid"`
	Tenant       string            `json:"tenant"`
	ProviderUUID string            `json:"provider_uuid"`
	Connection   ConnectionDetails `json:"connection"`
}

ConnectionResponse represents the response for connection details.

type ErrorResponse 

type ErrorResponse struct {
	Error  string `json:"error"`
	Code   int    `json:"code"`
	Reason string `json:"reason,omitempty"`
}

ErrorResponse represents an error response.

type EventBroker 

type EventBroker struct {
	// contains filtered or unexported fields
}

EventBroker manages per-lease event client subscriptions with non-blocking fan-out. Slow clients drop events; they can re-fetch via REST.

func NewEventBroker 

func NewEventBroker() *EventBroker

NewEventBroker creates a new event broker with default subscription limits.

func NewEventBrokerWithLimits 

func NewEventBrokerWithLimits(maxPerLease, maxTotal int) *EventBroker

NewEventBrokerWithLimits creates a new event broker with the given subscription limits. maxPerLease limits concurrent subscriptions per lease UUID. maxTotal limits concurrent subscriptions globally across all leases.

func (*EventBroker) Close 

func (b *EventBroker) Close()

Close closes all subscriber channels and prevents new subscriptions. Safe to call multiple times.

func (*EventBroker) DispatchWithOrderedSettlement added in v0.14.0 

func (b *EventBroker) DispatchWithOrderedSettlement(
	start backend.LeaseStatusEvent,
	dispatch func() (accepted bool, err error),
) (bool, error)

DispatchWithOrderedSettlement extends DispatchWithOrderedStart across local settlement that must follow backend acceptance. accepted is independent of err: callers return accepted=true when the backend effect started even if a later durable write failed. The start event is then still truthful, while the gate remains active until settlement finishes and queued callbacks cannot be reordered ahead of it.

func (*EventBroker) DispatchWithOrderedStart added in v0.14.0 

func (b *EventBroker) DispatchWithOrderedStart(
	start backend.LeaseStatusEvent,
	dispatch func() error,
) error

DispatchWithOrderedStart calls dispatch while holding a nonblocking subscriber-order gate for start.LeaseUUID. If dispatch succeeds, start is delivered before every callback event that arrived during dispatch. If it fails synchronously, no start event is manufactured and any independently valid callback observations are released in their original order.

Publish never waits for dispatch: it only appends to the active gate under a short mutex. This matters for a remote backend that posts a callback and waits for Fred's HTTP response before returning its Restart/Update acceptance. Concurrent dispatches for the same lease are serialized; other leases are independent. The registry entry is reference-counted and removed after the last waiter, so tenant churn cannot grow it without bound.

func (*EventBroker) Publish 

func (b *EventBroker) Publish(event backend.LeaseStatusEvent)

Publish sends an event to all clients subscribed to the event's lease UUID. Non-blocking: if a client's channel is full, the event is dropped for that client.

func (*EventBroker) Subscribe 

func (b *EventBroker) Subscribe(leaseUUID string) (<-chan backend.LeaseStatusEvent, error)

Subscribe registers a client channel for events on the given lease UUID. The returned channel is buffered; the caller should read from it in a loop. Returns nil and ErrTooManySubscriptions if a subscription limit is reached. Returns nil and nil error if the broker has been closed.

func (*EventBroker) Unsubscribe 

func (b *EventBroker) Unsubscribe(leaseUUID string, ch <-chan backend.LeaseStatusEvent)

Unsubscribe removes a client channel. The channel is closed after removal.

type Handlers 

type Handlers struct {
	// contains filtered or unexported fields
}

Handlers contains HTTP request handlers.

func NewHandlers 

func NewHandlers(cfg HandlersConfig) *Handlers

NewHandlers creates a new Handlers instance.

func (*Handlers) AuthenticateLeaseRequest 

func (h *Handlers) AuthenticateLeaseRequest(r *http.Request, leaseUUID string, checkReplay bool, requireActive bool) (*AuthenticatedRequest, int, error)

AuthenticateLeaseRequest performs common authentication and authorization for lease endpoints. It extracts and validates the bearer token, optionally checks for replay attacks, queries the lease from chain, and verifies tenant and provider ownership.

Parameters:

  • r: the HTTP request
  • leaseUUID: the lease UUID from the URL path
  • checkReplay: whether to check for token replay (set false for idempotent/read-heavy endpoints like status)
  • requireActive: if true, only ACTIVE leases are accepted; if false, any state is allowed

Returns AuthenticatedRequest on success, or an error with the appropriate HTTP status code.

func (*Handlers) GetLeaseConnection 

func (h *Handlers) GetLeaseConnection(w http.ResponseWriter, r *http.Request)

GetLeaseConnection handles GET /v1/leases/{lease_uuid}/connection

func (*Handlers) GetLeaseLogs 

func (h *Handlers) GetLeaseLogs(w http.ResponseWriter, r *http.Request)

GetLeaseLogs handles GET /v1/leases/{lease_uuid}/logs

func (*Handlers) GetLeaseProvision 

func (h *Handlers) GetLeaseProvision(w http.ResponseWriter, r *http.Request)

GetLeaseProvision handles GET /v1/leases/{lease_uuid}/provision

Like GetLeaseStatus, authz is chain-primary with a retained-record fallback (ENG-329 #5), and provision discovery uses findProvision (placement fast-path, bounded fan-out fallback). Within the grace window a retained lease returns status=retained (with retained_until + items), not 404.

func (*Handlers) GetLeaseReleases 

func (h *Handlers) GetLeaseReleases(w http.ResponseWriter, r *http.Request)

GetLeaseReleases handles GET /v1/leases/{lease_uuid}/releases

func (*Handlers) GetLeaseStatus 

func (h *Handlers) GetLeaseStatus(w http.ResponseWriter, r *http.Request)

GetLeaseStatus handles GET /v1/leases/{lease_uuid}/status

Authz is chain-primary with a retained-record fallback (ENG-329 #5): the ADR-036-signed token is validated first, then the chain lease is queried (any state). If the chain still has the lease, tenant/provider ownership is verified against it (the existing path). If the chain has NO RECORD of the lease (x/billing never deletes leases, so only a lagging, reset or wrong chain endpoint answers not-found), the request is authorized iff the signed caller's tenant equals the retained record's Tenant, surfaced via the bounded fan-out GetProvision — mirroring restore's cross-tenant guard. A cross-tenant caller, or an absent retained record, is rejected.

func (*Handlers) GetWorkloads 

func (h *Handlers) GetWorkloads(w http.ResponseWriter, r *http.Request)

GetWorkloads handles GET /workloads?lease_uuid=<u1>&lease_uuid=<u2>...

Returns workload metadata for the requested lease UUIDs from any backend that knows about them. Caller is the manifest-admin SPA (cross-origin via CORS); the visible page of leases on the admin's leases pages drives the UUID list. The 1..MaxLookupUUIDs cap matches the admin's PAGE_SIZE (25) with headroom.

Per-backend errors are surfaced via the Warnings slice rather than failing the whole request, so a single broken backend doesn't blank the admin's image column. This is a deliberate divergence from reconciler.go's fetchAllProvisions, which aborts on any backend error to avoid mistaking a transient failure for "lease no longer exists" — that concern doesn't apply here since /workloads is read-only metadata for display.

func (*Handlers) HealthCheck 

func (h *Handlers) HealthCheck(w http.ResponseWriter, r *http.Request)

HealthCheck handles GET /health.

Always 200 while the process can answer at all. This is a liveness contract, deliberately: the endpoint is what load balancers poll, and no dependency probed here justifies pulling providerd out of rotation.

The reason is topology, not taste. providerd sits behind a single-server load-balancer pool with no fail-open floor, so an unhealthy verdict does not shed load onto a peer — there is no peer. It takes the whole provider down. And the backends' completion callbacks arrive on this same listener, so a backend-triggered 503 severs the exact channel a *recovering* backend needs to report that it finished. That is not a hypothetical: it is both ENG-522 (a one-minute chain blip on mainnet) and the 2026-08-17 dev outage (one backend of three stopped, 15 minutes of tenant-wide 503s, one orphaned workload). Accepting a callback touches neither the chain, nor a backend, nor any bbolt store, so there is no probe here whose failure means "cannot serve".

The dependency detail is not lost, it moves: the per-check map is still in the body, every non-backend check now exports fred_health_check_healthy, and backends keep fred_backend_healthy. Alert on those, not on this status code.

func (*Handlers) Readyz added in v0.13.0 

func (h *Handlers) Readyz(w http.ResponseWriter, r *http.Request)

Readyz handles GET /readyz.

Same body as /health, but the aggregate verdict reaches the status code: 503 when a local, process-owned resource is unusable. Remote impairment (chain, backends) stays 200 here too — "degraded" means providerd is still ready to serve, which is exactly what it is.

Nothing routes on this endpoint, by design. It exists so the "unhealthy" tier has a wire expression for uptime monitoring and blackbox probes, and so a human can curl one path for the deep verdict. Do NOT point a load balancer at it: the failure it reports is one a process supervisor can fix by restarting and a load balancer cannot fix at all, and pulling the single server would again take the callback route down with it.

func (*Handlers) RestartLease 

func (h *Handlers) RestartLease(w http.ResponseWriter, r *http.Request)

RestartLease handles POST /v1/leases/{lease_uuid}/restart

func (*Handlers) RestoreLease added in v0.5.0 

func (h *Handlers) RestoreLease(w http.ResponseWriter, r *http.Request)

RestoreLease handles POST /v1/leases/{lease_uuid}/restore (lease_uuid is the NEW, fresh lease; the body names the original retained lease).

func (*Handlers) StreamLeaseEvents 

func (h *Handlers) StreamLeaseEvents(w http.ResponseWriter, r *http.Request)

StreamLeaseEvents serves a WebSocket stream of lease status events. GET /v1/leases/{lease_uuid}/events

func (*Handlers) UpdateLease 

func (h *Handlers) UpdateLease(w http.ResponseWriter, r *http.Request)

UpdateLease handles POST /v1/leases/{lease_uuid}/update

type HandlersConfig 

type HandlersConfig struct {
	Client             ChainClient
	BackendRouter      *backend.Router
	TokenTracker       TokenTrackerInterface // optional but recommended for replay attack protection
	StatusChecker      StatusChecker         // optional but required for the /status endpoint
	PlacementLookup    PlacementLookup       // optional — used for routing reads to the correct backend
	RestoreService     RestoreService        // required by /restore; owns typed operation and placement capabilities
	MaintenanceService MaintenanceService    // required by restart/update; owns durable idempotent dispatch
	PayloadStoreHealth PayloadStoreHealth    // optional — health probe for the payload store's bbolt DB
	EventBroker        *EventBroker          // optional — if nil, the events endpoint will return 501
	// BillingPolicy lets /restore refuse a billing-only target before its
	// state check. Without one no target is billing-only here; the restore
	// service still refuses it.
	BillingPolicy *placement.BillingPolicy
	ProviderUUID  string
	Bech32Prefix  string
	// MaintenanceLegacyIdempotencyTenants lists tenants whose restart/update
	// requests may omit Idempotency-Key; each such request is keyed by its
	// single-use signed token. Empty keeps the header mandatory.
	MaintenanceLegacyIdempotencyTenants []string
}

HandlersConfig configures a Handlers instance.

type HealthResponse 

type HealthResponse struct {
	Status       string                  `json:"status"`
	ProviderUUID string                  `json:"provider_uuid"`
	Checks       map[string]*CheckResult `json:"checks"`
	Stats        *HealthStats            `json:"stats,omitempty"`
}

HealthResponse represents the health check response.

type HealthStats 

type HealthStats struct {
	InFlightProvisions int `json:"in_flight_provisions"`
}

HealthStats contains operational statistics for the health response.

type InstanceInfo 

type InstanceInfo struct {
	InstanceIndex int                    `json:"instance_index"`
	ContainerID   string                 `json:"container_id,omitempty"`
	Image         string                 `json:"image,omitempty"`
	Status        string                 `json:"status,omitempty"`
	FQDN          string                 `json:"fqdn,omitempty"`
	Ports         map[string]PortMapping `json:"ports,omitempty"`
}

InstanceInfo contains connection details for a single instance in a multi-instance lease.

type LeaseProvisionResponse 

type LeaseProvisionResponse struct {
	LeaseUUID    string `json:"lease_uuid"`
	Tenant       string `json:"tenant"`
	ProviderUUID string `json:"provider_uuid"`
	Status       string `json:"status"`
	// FailCount is a lifetime diagnostic; see LeaseStatusResponse.FailCount.
	FailCount int `json:"fail_count"`
	// TerminalBudget is the consecutive-failure budget; see
	// LeaseStatusResponse.TerminalBudget.
	TerminalBudget *backend.TerminalBudgetObservation `json:"terminal_budget,omitempty"`
	// Reason/Message are the curated, tenant-safe failure signal (ENG-508),
	// replacing the former verbose last_error. See LeaseStatusResponse.
	Reason  string `json:"reason,omitempty"`
	Message string `json:"message,omitempty"`
	// Retention fields (populated when status=retained, ENG-329). See
	// LeaseStatusResponse; the owning Tenant from ProvisionInfo is NOT surfaced,
	// but Partition IS (the owner's own sub-grouping key, shown only to the owner).
	RetainedUntil string              `json:"retained_until,omitempty"`
	Items         []backend.LeaseItem `json:"items,omitempty"`
	RestoreHint   string              `json:"restore_hint,omitempty"`
	Partition     string              `json:"partition,omitempty"`
}

LeaseProvisionResponse represents the response for provision diagnostics.

type LeaseReleasesResponse 

type LeaseReleasesResponse struct {
	LeaseUUID    string                `json:"lease_uuid"`
	Tenant       string                `json:"tenant"`
	ProviderUUID string                `json:"provider_uuid"`
	Releases     []backend.ReleaseInfo `json:"releases"`
}

LeaseReleasesResponse represents the response for release history.

type LeaseStatusResponse 

type LeaseStatusResponse struct {
	LeaseUUID           string `json:"lease_uuid"`
	Tenant              string `json:"tenant"`
	ProviderUUID        string `json:"provider_uuid"`
	State               string `json:"state"`
	RequiresPayload     bool   `json:"requires_payload"`
	MetaHashHex         string `json:"meta_hash_hex,omitempty"` // For debugging - shows the expected payload hash
	PayloadReceived     bool   `json:"payload_received"`
	ProvisioningStarted bool   `json:"provisioning_started"`
	ProvisionStatus     string `json:"provision_status,omitempty"`
	// FailCount is a lifetime diagnostic of every recorded failure, whoever
	// caused it. It never decides whether the lease is closed; TerminalBudget
	// does (ENG-799).
	FailCount int `json:"fail_count,omitempty"`
	// TerminalBudget is the backend's consecutive-failure budget: how many
	// consecutive failures of the tenant's own workload are recorded, and
	// whether the lease will be closed for them ("exhausted") or re-provisioned
	// ("retry"). Absent when the backend reports no budget.
	TerminalBudget *backend.TerminalBudgetObservation `json:"terminal_budget,omitempty"`
	// Reason/Message are the curated, tenant-safe failure signal (ENG-508): a
	// stable machine-readable category code and a short human message. They
	// REPLACE the former verbose last_error field, which leaked exec output and
	// host paths. A FAILED provision always surfaces a Reason (defaulting to
	// "Unknown"); Message may be empty.
	Reason  string `json:"reason,omitempty"`
	Message string `json:"message,omitempty"`
	// Retention fields (populated when provision_status=retained, ENG-329).
	// RetainedUntil is RFC3339; Items is the restore shape (service name + SKU +
	// quantity) the tenant uses to build a matching fresh PENDING lease;
	// RestoreHint is a short human-readable next-step. The owning Tenant from
	// ProvisionInfo is intentionally NOT surfaced here (cross-tenant leak guard);
	// Partition IS surfaced — it is the owning tenant's own sub-grouping key, shown
	// only to that tenant (this response is owner-authenticated), so no cross-tenant
	// disclosure. Empty (omitempty absent) for non-partitioned retained records.
	RetainedUntil string              `json:"retained_until,omitempty"`
	Items         []backend.LeaseItem `json:"items,omitempty"`
	RestoreHint   string              `json:"restore_hint,omitempty"`
	Partition     string              `json:"partition,omitempty"`
}

LeaseStatusResponse represents the response for lease status. Includes tenant and provider_uuid for consistency with ConnectionResponse.

type MaintenanceService added in v0.14.0 

type MaintenanceService interface {
	Execute(context.Context, maintenanceapp.Command) maintenanceapp.Result
}

MaintenanceService is the restart/update application boundary. It owns the durable idempotency journal, authority revalidation, backend dispatch, and update-payload settlement; handlers only authenticate and translate results.

type PayloadAuthToken 

type PayloadAuthToken struct {
	Tenant    string `json:"tenant"`
	LeaseUUID string `json:"lease_uuid"`
	MetaHash  string `json:"meta_hash"` // Hex-encoded SHA-256 hash
	Timestamp int64  `json:"timestamp"`
	PubKey    string `json:"pub_key"`   // Base64-encoded public key
	Signature string `json:"signature"` // Base64-encoded signature
}

PayloadAuthToken represents the bearer token for payload upload authentication. It includes meta_hash to bind the signature to a specific payload.

func ParsePayloadAuthToken 

func ParsePayloadAuthToken(encoded string) (*PayloadAuthToken, error)

ParsePayloadAuthToken parses a base64-encoded payload authentication token.

func PayloadAuthTokenFromContext 

func PayloadAuthTokenFromContext(ctx context.Context) *PayloadAuthToken

PayloadAuthTokenFromContext retrieves the pre-validated PayloadAuthToken from request context. Returns nil if no token was stored (e.g. rate limiting disabled).

func (*PayloadAuthToken) Validate 

func (t *PayloadAuthToken) Validate(bech32Prefix string) error

Validate verifies the token's timestamp and ADR-036 signature. The bech32Prefix is used to verify the tenant address matches the public key.

type PayloadHandler 

type PayloadHandler struct {
	// contains filtered or unexported fields
}

PayloadHandler handles payload upload requests.

func NewPayloadHandler 

func NewPayloadHandler(client ChainClient, publisher PayloadPublisher, providerUUID, bech32Prefix string, policy *placement.BillingPolicy) *PayloadHandler

NewPayloadHandler creates a new payload handler. The billing policy decides which leases accept a payload: a billing-only lease is never deployed, and without a policy no lease is accepted.

func (*PayloadHandler) HandlePayloadUpload 

func (h *PayloadHandler) HandlePayloadUpload(w http.ResponseWriter, r *http.Request)

HandlePayloadUpload handles POST /v1/leases/{lease_uuid}/data

type PayloadPublisher 

type PayloadPublisher interface {
	PublishPayload(event payload.Event) error
	StorePayload(leaseUUID string, payload []byte) bool
	DeletePayload(leaseUUID string) // Used for rollback on publish failure
}

PayloadPublisher publishes payload events to the provisioner.

type PayloadStoreHealth added in v0.13.0 

type PayloadStoreHealth interface {
	Healthy() error
}

PayloadStoreHealth reports whether the payload store's bbolt database is readable and carries the buckets the store expects. The store has always had a Healthy method; nothing called it, so a payloads.db that had been lost, truncated or opened against the wrong path was invisible to both the health endpoint and the metrics.

Note the probe is a read-only bbolt View: it proves the database opens and its buckets exist, not that a write would succeed. A full or read-only filesystem passes it and still fails /update.

Optional: a providerd without payload_store_db_path has no store to probe and omits the check entirely.

type PlacementBootstrap added in v0.14.0 

type PlacementBootstrap interface {
	InventoryReadiness() placement.InventoryReadiness
}

PlacementBootstrap reports whether this process has completed at least one authoritative placement projection since startup. It is intentionally a separate optional capability from PlacementLookup: ordinary read routing can remain available while new multi-backend side effects are still fail-closed.

type PlacementLookup 

type PlacementLookup interface {
	Lookup(leaseUUID string) placement.Placement
	Healthy() error
}

PlacementLookup provides lease→backend mapping for read-path routing.

type PortMapping 

type PortMapping struct {
	HostIP   string `json:"host_ip"`
	HostPort int    `json:"host_port"`
}

PortMapping represents a port binding from container to host.

type RateLimiter 

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter implements per-IP rate limiting using a token bucket algorithm.

func NewRateLimiter 

func NewRateLimiter(rps float64, burst int, trustedProxies *TrustedProxyConfig) *RateLimiter

NewRateLimiter creates a new rate limiter. rps is requests per second, burst is the maximum burst size. trustedProxies is optional - if nil or empty, X-Forwarded-For headers are ignored.

func (*RateLimiter) Middleware 

func (rl *RateLimiter) Middleware(next http.Handler) http.Handler

Middleware returns an HTTP middleware that enforces rate limiting.

type RestoreService added in v0.14.0 

type RestoreService interface {
	Execute(ctx context.Context, command restoreapp.Command) restoreapp.Result
}

RestoreService is the HTTP layer's sole restore dependency. The application service owns routing and all lifecycle mutations; handlers only authenticate, parse, invoke, and translate its typed outcome.

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server is the HTTP API server.

func NewServer 

func NewServer(cfg ServerConfig, deps ServerDeps) (*Server, error)

NewServer creates a new API server. Returns an error if token tracker initialization fails.

func (*Server) Shutdown 

func (s *Server) Shutdown(ctx context.Context) error

Shutdown gracefully shuts down the server.

func (*Server) Start 

func (s *Server) Start(ctx context.Context) error

Start begins serving HTTP requests and blocks until context is canceled or error. When the context is canceled, the server is gracefully shut down before returning.

func (*Server) StartBackground 

func (s *Server) StartBackground() (<-chan error, error)

StartBackground starts the server in the background and returns immediately once the server is listening. Returns an error channel that will receive any server errors. This is useful when you need to ensure the server is ready before proceeding with other startup tasks (e.g., reconciliation that triggers callbacks).

type ServerConfig 

type ServerConfig struct {
	Addr                        string
	ProviderUUID                string
	Bech32Prefix                string
	TLSCertFile                 string
	TLSKeyFile                  string
	RateLimitRPS                float64
	RateLimitBurst              int
	TenantRateLimitRPS          float64  // Per-tenant rate limit (requests per second), 0 = disabled
	TenantRateLimitBurst        int      // Per-tenant burst limit
	TrustedProxies              []string // CIDR blocks of trusted reverse proxies for X-Forwarded-For
	CORSOrigins                 []string // Allowed CORS origins (e.g. ["*"] for all). Empty or nil disables CORS middleware.
	ReadTimeout                 time.Duration
	WriteTimeout                time.Duration
	IdleTimeout                 time.Duration
	RequestTimeout              time.Duration // Timeout for individual request processing (default: 30s)
	CallbackApplicationTimeout  time.Duration // Timeout for terminal callback application (default: backend.DefaultCallbackApplicationTimeout)
	ShutdownTimeout             time.Duration // Timeout for graceful shutdown (default: 30s)
	MaxRequestBodySize          int64
	CallbackSecret              string                             // Non-production legacy single HMAC secret for isolated embeddings.
	CallbackKeys                map[backendidentity.ID]CallbackKey // Production callback keys indexed by immutable backend storage identity.
	CallbackCanonicalPathPrefix string                             // Path prefix prepended to inbound URIs before HMAC verification (proxy stripPrefix compensation)
	TokenTrackerDBPath          string                             // Path to token tracker database (enables replay protection)
	CallbackBaseURL             string                             // Base URL for backend callbacks (used by restart/update)
	// MaintenanceLegacyIdempotencyTenants may omit Idempotency-Key on
	// restart/update; see HandlersConfig.
	MaintenanceLegacyIdempotencyTenants []string
	// BillingPolicy is required: payload uploads and restores are refused for
	// a lease it classifies as billing-only, which Fred never deploys. It is
	// the same policy the provision manager binds into placement.
	BillingPolicy *placement.BillingPolicy
}

ServerConfig holds configuration for the API server.

type ServerDeps 

type ServerDeps struct {
	ChainClient           ChainClient
	BackendRouter         *backend.Router
	CallbackPublisher     CallbackPublisher
	PayloadPublisher      PayloadPublisher
	PayloadStoreHealth    PayloadStoreHealth // Optional — health probe for the payload store's bbolt DB.
	StatusChecker         StatusChecker
	PlacementLookup       PlacementLookup                // Required by providerd; nil is supported only by isolated/test API embeddings.
	MaintenanceService    MaintenanceService             // Required by providerd for durable restart/update execution.
	RestoreService        RestoreService                 // Required by /restore; missing service returns 503.
	EventBroker           *EventBroker                   // Optional — if nil, the events endpoint returns 501.
	CallbackProofVerifier hmacauth.CallbackProofVerifier // Required whenever callback authentication is configured.
}

ServerDeps holds the runtime dependencies for the API server. These are the collaborators injected into the server at startup.

type ServiceConnectionDetails 

type ServiceConnectionDetails struct {
	FQDN      string         `json:"fqdn,omitempty"`
	Instances []InstanceInfo `json:"instances"`
}

ServiceConnectionDetails contains connection details for a single service in a stack.

type StatusChecker 

type StatusChecker interface {
	HasPayload(leaseUUID string) (bool, error)
	IsInFlight(leaseUUID string) bool
	InFlightCount() int
}

StatusChecker provides status information about provisioning. Typically implemented by the provisioner.Manager.

type TenantRateLimiter 

type TenantRateLimiter struct {
	// contains filtered or unexported fields
}

TenantRateLimiter implements per-tenant rate limiting using a token bucket algorithm. This is used for authenticated endpoints where the tenant identity is known. Tokens are cryptographically validated before consuming from the bucket to prevent attackers from burning a victim's quota with forged tokens.

func NewTenantRateLimiter 

func NewTenantRateLimiter(rps float64, burst int, bech32Prefix string) *TenantRateLimiter

NewTenantRateLimiter creates a new per-tenant rate limiter. rps is requests per second, burst is the maximum burst size per tenant. bech32Prefix is used for cryptographic token validation before bucket consumption.

func (*TenantRateLimiter) Allow 

func (tl *TenantRateLimiter) Allow(tenant string) bool

Allow checks if a request from the tenant is allowed.

func (*TenantRateLimiter) AuthMiddleware 

func (tl *TenantRateLimiter) AuthMiddleware() func(http.Handler) http.Handler

AuthMiddleware returns middleware that validates AuthTokens and applies per-tenant rate limiting. Tokens are cryptographically validated BEFORE consuming from the bucket, preventing attackers from burning a victim's quota with forged tokens. The validated token is stored in request context so handlers skip re-validation.

func (*TenantRateLimiter) PayloadAuthMiddleware 

func (tl *TenantRateLimiter) PayloadAuthMiddleware() func(http.Handler) http.Handler

PayloadAuthMiddleware returns middleware that validates PayloadAuthTokens and applies per-tenant rate limiting. Same validate-before-consume pattern as AuthMiddleware. Increments PayloadUploadsTotal{invalid_auth} on rejection so the metric fires regardless of whether rate limiting is enabled (the handler fallback path also increments on its own rejections).

type TokenReplayClaim added in v0.14.0 

type TokenReplayClaim struct {
	// contains filtered or unexported fields
}

TokenReplayClaim binds the canonical signature to its signed validity window. Only successful cryptographic validation creates a usable claim; callers cannot choose a different cache key or shorten the replay lifetime.

type TokenTracker 

type TokenTracker struct {
	// contains filtered or unexported fields
}

TokenTracker tracks used authentication tokens to prevent replay attacks. It uses bbolt for persistence across restarts.

func NewTokenTracker 

func NewTokenTracker(cfg TokenTrackerConfig) (*TokenTracker, error)

NewTokenTracker creates a new token tracker with bbolt persistence.

func (*TokenTracker) Close 

func (t *TokenTracker) Close() error

Close shuts down the token tracker gracefully. Close is idempotent and safe to call multiple times.

func (*TokenTracker) Healthy 

func (t *TokenTracker) Healthy() error

Healthy checks if the bbolt database is accessible and the token bucket exists.

func (*TokenTracker) TryUse 

func (t *TokenTracker) TryUse(claim TokenReplayClaim) error

TryUse atomically consumes a validated token for its entire signed lifetime. bbolt serializes write transactions; retrying a closed database cannot repair it.

type TokenTrackerConfig 

type TokenTrackerConfig struct {
	DBPath          string        // Path to bbolt database file
	CleanupInterval time.Duration // How often to clean up expired entries
}

TokenTrackerConfig configures the token tracker.

type TokenTrackerInterface 

type TokenTrackerInterface interface {
	TryUse(claim TokenReplayClaim) error
	Healthy() error
	Close() error
}

TokenTrackerInterface defines the interface for token replay protection. This interface allows for testing with mock implementations.

type TrustedProxyConfig 

type TrustedProxyConfig struct {
	// contains filtered or unexported fields
}

TrustedProxyConfig holds parsed trusted proxy CIDR ranges.

func NewTrustedProxyConfig 

func NewTrustedProxyConfig(cidrs []string) *TrustedProxyConfig

NewTrustedProxyConfig parses CIDR strings into a trusted proxy configuration. Invalid CIDR strings are logged and skipped.

func (*TrustedProxyConfig) IsTrusted 

func (c *TrustedProxyConfig) IsTrusted(ipStr string) bool

IsTrusted returns true if the given IP address is within a trusted proxy range.

type WorkloadEntry 

type WorkloadEntry struct {
	Status      backend.ProvisionStatus `json:"status"`
	CreatedAt   time.Time               `json:"created_at"`
	BackendName string                  `json:"backend_name"`
	Items       []WorkloadItem          `json:"items"`
}

WorkloadEntry describes a single lease's workload for observability. LeaseUUID is intentionally absent — the map key in WorkloadLookupResponse carries it.

WorkloadEntry is served UNAUTHENTICATED (/workloads). Tenant and Partition (an aggregator's customer identifiers) must NEVER be added here — this endpoint is world-readable; the same invariant covers the backend's unauthenticated /stats and /metrics (no partition-valued metric labels).

type WorkloadItem 

type WorkloadItem struct {
	ServiceName string `json:"service_name,omitempty"`
	SKU         string `json:"sku"`
	Image       string `json:"image,omitempty"`
	Count       int    `json:"count"`
}

WorkloadItem describes a single SKU+image within a workload.

type WorkloadLookupResponse 

type WorkloadLookupResponse struct {
	Workloads map[string]WorkloadEntry `json:"workloads"`
	Warnings  []string                 `json:"warnings"`
}

WorkloadLookupResponse is the response from the GET /workloads endpoint. Workloads is a map keyed by lease_uuid so callers can join by UUID without building a client-side index. Unknown leases are absent from the map. Warnings is non-nil and may be empty (initialized as []string{} for stable JSON serialization as `[]` rather than `null`).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.