Documentation
¶
Overview ¶
Package serverlessoperator is the serverless operator core: it owns (tenant, partition) lease units, polls their endpoints' healthchecks, registers their workflows with the engine and delivers assigned tasks to the endpoints over signed HTTP or, for durable tasks, over a websocket relay.
The core reaches the engine through the operator host contract (pkg/operator.Host, Session and DurableChannel): one session per served tenant, opened by name and kind with the tenant's action union, its handler the registration that routes assigned actions to endpoints. The operator is a GRPC contract operator that leases itself in both modes (kind GRPC, leasing manager SELF): its rows are kept alive by the sessions it holds, never by the engine's claimer. The same core runs in either host as a deployment choice. Out of process, hatchet-serverless-operator gives it pkg/operator/hostgrpc, which speaks OperatorService with a per-tenant token; inside the engine, cmd/hatchet-engine gives it the dispatcher's in-process host, which upserts the rows itself and delivers by direct call. Nothing in the core distinguishes the two beyond Deps.Host.
Index ¶
Constants ¶
const ( DefaultOperatorName = "serverless" DefaultLinkName = "grpc" DefaultDefaultSlots int32 = 10000 DefaultDurableSlots int32 = 10000 DefaultEndpointPartitionCount int32 = 1 DefaultLeaseTTL = 15 * time.Second DefaultHeartbeatInterval = 5 * time.Second DefaultRebalanceInterval = 5 * time.Second DefaultShedHysteresis = 0.2 DefaultDrainTimeout = 60 * time.Second DefaultRoutingRefreshInterval = 10 * time.Second DefaultRoutingFullReloadInterval = 60 * time.Second DefaultHealthcheckTimeout = 10 * time.Second DefaultHealthcheckConcurrency = 256 DefaultHealthcheckTenantConcurrency = 32 DefaultHealthcheckApplyTimeout = 60 * time.Second DefaultMaxWorkflowsPerEndpoint = 200 DefaultMaxActionsPerEndpoint = 500 DefaultMaintenanceConcurrency = 8 DefaultLeaseMaxClaimPerTick int32 = 1024 DefaultWSMaxFrameBytes int64 = 4 * 1024 * 1024 DefaultWSPingInterval = 15 * time.Second DefaultHealthPort = 8080 // Relay resource limit defaults. DefaultWSMaxUpgradeHeaderBytes int64 = 64 * 1024 DefaultWSMaxQueuedBytes int64 = 16 * 1024 * 1024 DefaultWSMaxStreams = 16 DefaultWSMaxRunsPerStream = 1024 )
const ( MinEndpointPartitionCount int32 = 1 MaxEndpointPartitionCount int32 = 64 )
Bounds of Config.EndpointPartitionCount. Each partition is a lease unit, so the ceiling caps how many lease rows one tenant can spread over.
Variables ¶
var ErrFIPSOnlyMode = errors.New("the serverless operator cannot run under GODEBUG=fips140=only: the websocket upgrade handshake needs SHA-1, which that mode refuses; run with fips140=on or disable the operator")
ErrFIPSOnlyMode is returned when the process runs with GODEBUG=fips140=only. The durable relay dials endpoints over RFC 6455 websockets, whose upgrade handshake is specified over SHA-1, and Go refuses SHA-1 in that mode; fips140=on allows it.
Functions ¶
func CheckRuntimeSupport ¶
func CheckRuntimeSupport() error
CheckRuntimeSupport reports whether this process can run the operator at all. It runs once at startup so an unsupported mode fails the boot instead of the first durable delivery.
func Run ¶
Run starts the leaser, the maintenance loop and the health server and blocks until ctx is done or a loop fails, then shuts down gracefully: release every lease, drain deliveries up to DrainTimeout, close registrations, delete the process row.
func ValidateEndpointPartitionCount ¶
ValidateEndpointPartitionCount checks a configured partition count against the bounds; zero is the default and passes.
Types ¶
type Config ¶
type Config struct {
// OperatorName is the operator name the sessions register as.
OperatorName string
// LinkName labels metrics with how the core reaches the engine: grpc or engine.
LinkName string
DefaultSlots int32
DurableSlots int32
// EndpointPartitionCount is the endpoint_partition_count a tenant's v1_serverless_tenant row is
// created with, in MinEndpointPartitionCount to MaxEndpointPartitionCount. A tenant's endpoints
// hash over its partition count into lease units, so a count above 1 lets several processes
// serve one tenant. The row is created by the API server with the tenant's first endpoint and
// read whenever an endpoint is assigned a partition; a later change to this value applies only
// to tenants first seen after it, and existing endpoints keep their partition.
EndpointPartitionCount int32
// LeaseTTL is how long a process row stays live after a heartbeat.
LeaseTTL time.Duration
HeartbeatInterval time.Duration
RebalanceInterval time.Duration
// ShedHysteresis is the fraction above fair share a process tolerates before shedding.
ShedHysteresis float64
// DrainTimeout bounds how long in-flight deliveries are awaited when a unit is lost or the
// process stops.
DrainTimeout time.Duration
// RoutingRefreshInterval is the incremental (updated_at based) routing cache refresh
// cadence; RoutingFullReloadInterval is how often the cache is reconciled against the
// tenant's endpoint ids and versions (spread by up to ten percent per tenant), which is
// what drops hard-deleted endpoints.
RoutingRefreshInterval time.Duration
RoutingFullReloadInterval time.Duration
// HealthcheckTimeout bounds one healthcheck request; HealthcheckConcurrency caps
// healthchecks process-wide and HealthcheckTenantConcurrency caps them per tenant inside
// that limit, so one tenant's slow endpoints cannot hold every slot.
HealthcheckTimeout time.Duration
HealthcheckConcurrency int
HealthcheckTenantConcurrency int
// HealthcheckApplyTimeout bounds the application of one changed catalog: the workflow
// puts, the action delta and the registered_actions write.
HealthcheckApplyTimeout time.Duration
// MaxWorkflowsPerEndpoint and MaxActionsPerEndpoint cap what one healthcheck may
// advertise; a catalog over either cap is refused and the endpoint marked with the error.
MaxWorkflowsPerEndpoint int
MaxActionsPerEndpoint int
// MaintenanceConcurrency is how many tenants a maintenance pass refreshes at once.
MaintenanceConcurrency int
// LeaseMaxClaimPerTick caps how many lease units one rebalance tick claims; the process's
// share of the claimable units is taken up to this many, in statements of the leaser's
// claim batch.
LeaseMaxClaimPerTick int32
// WSMaxFrameBytes and WSPingInterval configure the durable websocket relay (a later
// phase); they are carried here so the binary's env binding is complete.
WSMaxFrameBytes int64
WSPingInterval time.Duration
// HealthPort serves /healthz, /readyz and /metrics. Zero disables the server.
HealthPort int
// WSMaxUpgradeHeaderBytes bounds an endpoint's websocket upgrade response head, which is
// parsed before WSMaxFrameBytes applies.
WSMaxUpgradeHeaderBytes int64
// WSMaxQueuedBytes bounds the encoded frames one relay retains for an endpoint that reads
// slower than the engine answers; crossing it closes the socket with backpressure.
WSMaxQueuedBytes int64
// WSMaxStreams caps the engine streams one socket may hold open at once through
// stream_open frames.
WSMaxStreams int
// WSMaxRunsPerStream caps the workflow runs one SubscribeToWorkflowRuns stream opened
// over a socket may await at once; a subscription past it closes the stream with
// ResourceExhausted.
WSMaxRunsPerStream int
}
Config holds the knobs shared by the out-of-process binary and the in-engine mode. Zero values are replaced by the defaults below; the binary binds SERVERLESS_OPERATOR_* onto it.
type Deps ¶
type Deps struct {
Repo repository.ServerlessRepository
// Host opens the engine session of each served tenant.
Host operator.Host
// WorkerName names the worker rows the sessions back; empty means the operator name.
// Only a host that can name workers accepts it (the in-process host does, the gRPC host
// refuses), so it is set by the engine wiring only.
WorkerName string
Encryption encryption.EncryptionService
Sender RequestSender
Logger *zerolog.Logger
Version string
Hostname string
Config Config
ProcessId uuid.UUID
}
Deps is everything Run needs. The binary and the in-engine mode build it differently: the host and the worker name are hosting facts the wiring decides.
type RequestSender ¶
type RequestSender interface {
Deliver(ctx context.Context, method, endpoint string, body []byte, headers http.Header) (*safeclient.DeliveryResult, error)
}
RequestSender is the outbound HTTP seam: safeclient.Sender in production, a fake in tests.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package contract holds the parts of the serverless endpoint contract that are not protobuf messages: header names, the upgrade signature payload, wire constants and the protojson encoding every message travels in.
|
Package contract holds the parts of the serverless endpoint contract that are not protobuf messages: header names, the upgrade signature payload, wire constants and the protojson encoding every message travels in. |
|
Package durable relays one task invocation between the engine and a serverless endpoint over an operator-dialed websocket.
|
Package durable relays one task invocation between the engine and a serverless endpoint over an operator-dialed websocket. |
|
internal
|
|
|
memrepo
Package memrepo is an in-memory repository.ServerlessRepository for unit tests of the serverless operator core and leaser.
|
Package memrepo is an in-memory repository.ServerlessRepository for unit tests of the serverless operator core and leaser. |
|
Package lease is the serverless operator's process membership and unit ownership: one heartbeat row per process, fair-share claiming of (tenant, partition) units with FOR UPDATE SKIP LOCKED, shedding above fair share, and a periodic sweep of expired process rows.
|
Package lease is the serverless operator's process membership and unit ownership: one heartbeat row per process, fair-share claiming of (tenant, partition) units with FOR UPDATE SKIP LOCKED, shedding above fair share, and a periodic sweep of expired process rows. |