Help us shape the future of Go! Take the 2026 Go Developer Survey

serverlessoperator

package
v0.113.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 21, 2026 License: MIT Imports: 43 Imported by: 0

Documentation

Overview

Package serverlessoperator is the serverless operator core: it owns (tenant, partition) lease units, polls their endpoints' healthchecks, registers their workflows with the engine and delivers assigned tasks to the endpoints over signed HTTP or, for durable tasks, over a websocket relay.

The core reaches the engine through the operator host contract (pkg/operator.Host, Session and DurableChannel): one session per served tenant, opened by name and kind with the tenant's action union, its handler the registration that routes assigned actions to endpoints. The operator is a GRPC contract operator that leases itself in both modes (kind GRPC, leasing manager SELF): its rows are kept alive by the sessions it holds, never by the engine's claimer. The same core runs in either host as a deployment choice. Out of process, hatchet-serverless-operator gives it pkg/operator/hostgrpc, which speaks OperatorService with a per-tenant token; inside the engine, cmd/hatchet-engine gives it the dispatcher's in-process host, which upserts the rows itself and delivers by direct call. Nothing in the core distinguishes the two beyond Deps.Host.

Index

Constants

View Source
const (
	DefaultOperatorName                       = "serverless"
	DefaultLinkName                           = "grpc"
	DefaultDefaultSlots                 int32 = 10000
	DefaultDurableSlots                 int32 = 10000
	DefaultEndpointPartitionCount       int32 = 1
	DefaultLeaseTTL                           = 15 * time.Second
	DefaultHeartbeatInterval                  = 5 * time.Second
	DefaultRebalanceInterval                  = 5 * time.Second
	DefaultShedHysteresis                     = 0.2
	DefaultDrainTimeout                       = 60 * time.Second
	DefaultRoutingRefreshInterval             = 10 * time.Second
	DefaultRoutingFullReloadInterval          = 60 * time.Second
	DefaultHealthcheckTimeout                 = 10 * time.Second
	DefaultHealthcheckConcurrency             = 256
	DefaultHealthcheckTenantConcurrency       = 32
	DefaultHealthcheckApplyTimeout            = 60 * time.Second
	DefaultMaxWorkflowsPerEndpoint            = 200
	DefaultMaxActionsPerEndpoint              = 500
	DefaultMaintenanceConcurrency             = 8
	DefaultLeaseMaxClaimPerTick         int32 = 1024
	DefaultWSMaxFrameBytes              int64 = 4 * 1024 * 1024
	DefaultWSPingInterval                     = 15 * time.Second
	DefaultHealthPort                         = 8080

	// Relay resource limit defaults.
	DefaultWSMaxUpgradeHeaderBytes int64 = 64 * 1024
	DefaultWSMaxQueuedBytes        int64 = 16 * 1024 * 1024
	DefaultWSMaxStreams                  = 16
	DefaultWSMaxRunsPerStream            = 1024
)
View Source
const (
	MinEndpointPartitionCount int32 = 1
	MaxEndpointPartitionCount int32 = 64
)

Bounds of Config.EndpointPartitionCount. Each partition is a lease unit, so the ceiling caps how many lease rows one tenant can spread over.

Variables

View Source
var ErrFIPSOnlyMode = errors.New("the serverless operator cannot run under GODEBUG=fips140=only: the websocket upgrade handshake needs SHA-1, which that mode refuses; run with fips140=on or disable the operator")

ErrFIPSOnlyMode is returned when the process runs with GODEBUG=fips140=only. The durable relay dials endpoints over RFC 6455 websockets, whose upgrade handshake is specified over SHA-1, and Go refuses SHA-1 in that mode; fips140=on allows it.

Functions

func CheckRuntimeSupport

func CheckRuntimeSupport() error

CheckRuntimeSupport reports whether this process can run the operator at all. It runs once at startup so an unsupported mode fails the boot instead of the first durable delivery.

func Run

func Run(ctx context.Context, deps Deps) error

Run starts the leaser, the maintenance loop and the health server and blocks until ctx is done or a loop fails, then shuts down gracefully: release every lease, drain deliveries up to DrainTimeout, close registrations, delete the process row.

func ValidateEndpointPartitionCount

func ValidateEndpointPartitionCount(n int32) error

ValidateEndpointPartitionCount checks a configured partition count against the bounds; zero is the default and passes.

Types

type Config

type Config struct {
	// OperatorName is the operator name the sessions register as.
	OperatorName string

	// LinkName labels metrics with how the core reaches the engine: grpc or engine.
	LinkName string

	DefaultSlots int32
	DurableSlots int32

	// EndpointPartitionCount is the endpoint_partition_count a tenant's v1_serverless_tenant row is
	// created with, in MinEndpointPartitionCount to MaxEndpointPartitionCount. A tenant's endpoints
	// hash over its partition count into lease units, so a count above 1 lets several processes
	// serve one tenant. The row is created by the API server with the tenant's first endpoint and
	// read whenever an endpoint is assigned a partition; a later change to this value applies only
	// to tenants first seen after it, and existing endpoints keep their partition.
	EndpointPartitionCount int32

	// LeaseTTL is how long a process row stays live after a heartbeat.
	LeaseTTL          time.Duration
	HeartbeatInterval time.Duration
	RebalanceInterval time.Duration

	// ShedHysteresis is the fraction above fair share a process tolerates before shedding.
	ShedHysteresis float64

	// DrainTimeout bounds how long in-flight deliveries are awaited when a unit is lost or the
	// process stops.
	DrainTimeout time.Duration

	// RoutingRefreshInterval is the incremental (updated_at based) routing cache refresh
	// cadence; RoutingFullReloadInterval is how often the cache is reconciled against the
	// tenant's endpoint ids and versions (spread by up to ten percent per tenant), which is
	// what drops hard-deleted endpoints.
	RoutingRefreshInterval    time.Duration
	RoutingFullReloadInterval time.Duration

	// HealthcheckTimeout bounds one healthcheck request; HealthcheckConcurrency caps
	// healthchecks process-wide and HealthcheckTenantConcurrency caps them per tenant inside
	// that limit, so one tenant's slow endpoints cannot hold every slot.
	HealthcheckTimeout           time.Duration
	HealthcheckConcurrency       int
	HealthcheckTenantConcurrency int

	// HealthcheckApplyTimeout bounds the application of one changed catalog: the workflow
	// puts, the action delta and the registered_actions write.
	HealthcheckApplyTimeout time.Duration

	// MaxWorkflowsPerEndpoint and MaxActionsPerEndpoint cap what one healthcheck may
	// advertise; a catalog over either cap is refused and the endpoint marked with the error.
	MaxWorkflowsPerEndpoint int
	MaxActionsPerEndpoint   int

	// MaintenanceConcurrency is how many tenants a maintenance pass refreshes at once.
	MaintenanceConcurrency int

	// LeaseMaxClaimPerTick caps how many lease units one rebalance tick claims; the process's
	// share of the claimable units is taken up to this many, in statements of the leaser's
	// claim batch.
	LeaseMaxClaimPerTick int32

	// WSMaxFrameBytes and WSPingInterval configure the durable websocket relay (a later
	// phase); they are carried here so the binary's env binding is complete.
	WSMaxFrameBytes int64
	WSPingInterval  time.Duration

	// HealthPort serves /healthz, /readyz and /metrics. Zero disables the server.
	HealthPort int

	// WSMaxUpgradeHeaderBytes bounds an endpoint's websocket upgrade response head, which is
	// parsed before WSMaxFrameBytes applies.
	WSMaxUpgradeHeaderBytes int64

	// WSMaxQueuedBytes bounds the encoded frames one relay retains for an endpoint that reads
	// slower than the engine answers; crossing it closes the socket with backpressure.
	WSMaxQueuedBytes int64

	// WSMaxStreams caps the engine streams one socket may hold open at once through
	// stream_open frames.
	WSMaxStreams int

	// WSMaxRunsPerStream caps the workflow runs one SubscribeToWorkflowRuns stream opened
	// over a socket may await at once; a subscription past it closes the stream with
	// ResourceExhausted.
	WSMaxRunsPerStream int
}

Config holds the knobs shared by the out-of-process binary and the in-engine mode. Zero values are replaced by the defaults below; the binary binds SERVERLESS_OPERATOR_* onto it.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns the plan's defaults.

type Deps

type Deps struct {
	Repo repository.ServerlessRepository

	// Host opens the engine session of each served tenant.
	Host operator.Host

	// WorkerName names the worker rows the sessions back; empty means the operator name.
	// Only a host that can name workers accepts it (the in-process host does, the gRPC host
	// refuses), so it is set by the engine wiring only.
	WorkerName string

	Encryption encryption.EncryptionService
	Sender     RequestSender
	Logger     *zerolog.Logger
	Version    string
	Hostname   string
	Config     Config
	ProcessId  uuid.UUID
}

Deps is everything Run needs. The binary and the in-engine mode build it differently: the host and the worker name are hosting facts the wiring decides.

type RequestSender

type RequestSender interface {
	Deliver(ctx context.Context, method, endpoint string, body []byte, headers http.Header) (*safeclient.DeliveryResult, error)
}

RequestSender is the outbound HTTP seam: safeclient.Sender in production, a fake in tests.

Directories

Path Synopsis
Package contract holds the parts of the serverless endpoint contract that are not protobuf messages: header names, the upgrade signature payload, wire constants and the protojson encoding every message travels in.
Package contract holds the parts of the serverless endpoint contract that are not protobuf messages: header names, the upgrade signature payload, wire constants and the protojson encoding every message travels in.
Package durable relays one task invocation between the engine and a serverless endpoint over an operator-dialed websocket.
Package durable relays one task invocation between the engine and a serverless endpoint over an operator-dialed websocket.
internal
memrepo
Package memrepo is an in-memory repository.ServerlessRepository for unit tests of the serverless operator core and leaser.
Package memrepo is an in-memory repository.ServerlessRepository for unit tests of the serverless operator core and leaser.
Package lease is the serverless operator's process membership and unit ownership: one heartbeat row per process, fair-share claiming of (tenant, partition) units with FOR UPDATE SKIP LOCKED, shedding above fair share, and a periodic sweep of expired process rows.
Package lease is the serverless operator's process membership and unit ownership: one heartbeat row per process, fair-share claiming of (tenant, partition) units with FOR UPDATE SKIP LOCKED, shedding above fair share, and a periodic sweep of expired process rows.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL