Help us shape the future of Go! Take the 2026 Go Developer Survey

extension

package
v1.9.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2026 License: MIT Imports: 44 Imported by: 0

Documentation

Overview

Package extension provides the Forge extension adapter for Dispatch.

It implements the forge.Extension interface to integrate Dispatch into a Forge application with automatic dependency discovery, route registration, and lifecycle management.

Configuration can be provided programmatically via Option functions or via YAML configuration files under "extensions.dispatch" or "dispatch" keys.

Index

Constants

View Source
const ExtensionDescription = "Durable execution engine for background jobs, workflows, and cron scheduling"

ExtensionDescription is the human-readable description.

View Source
const ExtensionName = "dispatch"

ExtensionName is the name registered with Forge.

View Source
const ExtensionVersion = "0.1.0"

ExtensionVersion is the semantic version.

View Source
const GroveExtensionName = "grove"

GroveExtensionName is the name the grove Forge extension registers under. Dispatch declares it as a dependency so a grove-backed store finds grove's database already in the container. It is spelled out here, not imported, so Dispatch does not pull in grove's extension module just to read a constant.

Variables

This section is empty.

Functions

This section is empty.

Types

type ArtifactCacheConfig

type ArtifactCacheConfig struct {
	// Dir is where staged artifacts are held on local disk.
	Dir string `default:"/var/lib/dispatch/cache" json:"dir" mapstructure:"dir" yaml:"dir"`

	// Budget caps the bytes the cache may hold. A job needing more
	// staging space than is free waits rather than filling the volume.
	//
	// With the resource model on this becomes the shared ledger's disk
	// capacity rather than a private ceiling inside the cache, so it is
	// the same number a job's disk requirement is admitted against.
	Budget int64 `json:"budget" mapstructure:"budget" yaml:"budget"`
}

ArtifactCacheConfig configures the worker-local staging cache.

type ArtifactConfig

type ArtifactConfig struct {
	// Enabled turns the artifact plane on. When false, no backend is
	// resolved even if a Trove instance is present in the container.
	Enabled bool `default:"false" json:"enabled" mapstructure:"enabled" yaml:"enabled"`

	// TroveStore is the name of a *trove.Trove registered in the DI
	// container. Empty resolves the default (unnamed) instance, which is
	// what a single-store Trove extension provides.
	TroveStore string `json:"trove_store" mapstructure:"trove_store" yaml:"trove_store"`

	// Bucket is where Dispatch writes the ephemeral artifacts it owns.
	Bucket string `default:"dispatch-artifacts" json:"bucket" mapstructure:"bucket" yaml:"bucket"`

	// EphemeralPrefix is the key prefix for Dispatch-owned objects.
	EphemeralPrefix string `default:"ephemeral" json:"ephemeral_prefix" mapstructure:"ephemeral_prefix" yaml:"ephemeral_prefix"`

	// Retention is how long an ephemeral artifact survives after every
	// owner reaches a terminal state.
	Retention time.Duration `default:"168h" json:"retention" mapstructure:"retention" yaml:"retention"`

	// PurgeGrace is how long a soft-deleted artifact's bytes survive
	// before the purge pass removes them. It is the window in which a
	// mistaken sweep can still be caught.
	PurgeGrace time.Duration `default:"24h" json:"purge_grace" mapstructure:"purge_grace" yaml:"purge_grace"`

	// Cache configures the worker-local staging cache.
	Cache ArtifactCacheConfig `json:"cache" mapstructure:"cache" yaml:"cache"`
}

ArtifactConfig configures the artifact plane — Dispatch's tracked object storage for job inputs and outputs.

The plane is entirely opt-in. With no backend resolved, Dispatch behaves exactly as it did before artifacts existed.

type Config

type Config struct {
	Security SecurityConfig `json:"security" yaml:"security" mapstructure:"security"`
	Durable  DurableConfig  `json:"durable" yaml:"durable" mapstructure:"durable"`
	// BasePath is the URL prefix for all dispatch API routes.
	BasePath string `default:"/dispatch" json:"base_path" mapstructure:"base_path" yaml:"base_path"`

	// DisableRoutes disables the registration of HTTP routes.
	// Useful when embedding Dispatch for background processing only.
	DisableRoutes bool `default:"false" json:"disable_routes" mapstructure:"disable_routes" yaml:"disable_routes"`

	// DisableMigrate disables auto-migration on start.
	DisableMigrate bool `default:"false" json:"disable_migrate" mapstructure:"disable_migrate" yaml:"disable_migrate"`

	// Dispatch is parsed from YAML (extensions.dispatch.dispatch.* or
	// dispatch.dispatch.*) but not currently applied anywhere: nothing in
	// this package reads e.config.Dispatch, and dispatch.Config's own
	// fields carry no yaml/mapstructure/json struct tags of their own, so
	// even a populated value here binds by Go field name at best. Setting
	// concurrency, queues, poll interval, or any other dispatch.Config
	// field under this key parses without error and has no effect.
	//
	// The dispatcher itself IS configurable through this extension —
	// just not through this field. WithConcurrency, WithQueues,
	// WithPollInterval, WithMaxPollInterval, WithHeartbeatInterval,
	// WithStaleJobThreshold, WithWorkerStoreCallTimeout, and the
	// WithCron* options (options.go) each translate one dispatch.Config
	// field into the matching dispatch.With* functional option; use
	// those from Go, not this field from YAML.
	Dispatch dispatch.Config `json:"dispatch" mapstructure:"dispatch" yaml:"dispatch"`

	// GroveDatabase is the name of a grove.DB registered in the DI container.
	// When set, the extension resolves this named database and auto-constructs
	// the appropriate store based on the driver type (pg/sqlite/mongo).
	// When empty and WithGroveDatabase was called, the default (unnamed) DB is used.
	GroveDatabase string `json:"grove_database" mapstructure:"grove_database" yaml:"grove_database"`

	// GroveKV is the name of a grove kv.Store registered in the DI container.
	// When set, the extension resolves this named KV store and auto-constructs
	// a Redis-backed store. When empty and WithGroveKV was called, the default
	// (unnamed) kv.Store is used.
	GroveKV string `json:"grove_kv" mapstructure:"grove_kv" yaml:"grove_kv"`

	// KeyPrefix namespaces every key the Redis-backed store writes, so
	// several dispatch instances can share one Redis without seeing each
	// other's jobs, cron locks or leadership. Only the grove KV path reads
	// it. Empty keeps the historical "dispatch:" keys.
	KeyPrefix string `json:"key_prefix" mapstructure:"key_prefix" yaml:"key_prefix"`

	// Artifacts configures the artifact plane.
	Artifacts ArtifactConfig `json:"artifacts" mapstructure:"artifacts" yaml:"artifacts"`

	// Resources configures the worker's resource model.
	Resources ResourceConfig `json:"resources" mapstructure:"resources" yaml:"resources"`

	// Execution configures which isolation rungs beyond the in-process
	// default are available to job definitions that declare a stronger
	// minimum via job.WithExecution.
	Execution ExecutionConfig `json:"execution" mapstructure:"execution" yaml:"execution"`

	// EnableDWP enables the Dispatch Wire Protocol for real-time
	// client communication (WebSocket, SSE, HTTP RPC).
	EnableDWP bool `default:"false" json:"enable_dwp" mapstructure:"enable_dwp" yaml:"enable_dwp"`

	// DWPBasePath is the URL prefix for DWP endpoints.
	// Default is "/dwp".
	DWPBasePath string `json:"dwp_base_path" mapstructure:"dwp_base_path" yaml:"dwp_base_path"`

	// RequireConfig requires config to be present in YAML files.
	// If true and no config is found, Register returns an error.
	RequireConfig bool `json:"-" yaml:"-"`
}

Config holds configuration for the Dispatch Forge extension. Fields can be set programmatically via Option functions or loaded from YAML configuration files (under "extensions.dispatch" or "dispatch" keys).

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a Config with sensible defaults.

type DeliveryConfig

type DeliveryConfig struct {
	AuditNamespace              durable.NamespaceConfig
	Namespaces                  []durable.NamespaceConfig
	Publisher                   delivery.Config
	Sinks                       map[durable.Destination]delivery.Sink
	AllowNamespaceAuditFallback bool
	ShutdownTimeout             time.Duration
}

DeliveryConfig is explicit host composition. Sinks must implement durable idempotent acceptance. Use durable/delivery/ecosystem for Chronicle and Relay.

type DurableConfig

type DurableConfig struct {
	Enabled       bool          `json:"enabled" yaml:"enabled" mapstructure:"enabled"`
	Namespace     string        `json:"namespace" yaml:"namespace" mapstructure:"namespace"`
	Queue         string        `json:"queue" yaml:"queue" mapstructure:"queue"`
	BuildID       string        `json:"build_id" yaml:"build_id" mapstructure:"build_id"`
	Owner         string        `json:"owner" yaml:"owner" mapstructure:"owner"`
	LeaseDuration time.Duration `json:"lease_duration" yaml:"lease_duration" mapstructure:"lease_duration"`
	PollInterval  time.Duration `json:"poll_interval" yaml:"poll_interval" mapstructure:"poll_interval"`
	StoreTimeout  time.Duration `json:"store_timeout" yaml:"store_timeout" mapstructure:"store_timeout"`
	Concurrency   int           `json:"concurrency" yaml:"concurrency" mapstructure:"concurrency"`
}

DurableConfig contains serializable worker routing and timing. Handlers are Go registrations supplied through WithDurableWorkflows.

type ExecutionConfig

type ExecutionConfig struct {
	// Subprocess configures the out-of-process rung (exec.LevelProcess) —
	// the handler runs in a re-exec'd child process instead of the
	// worker's own address space. A zero value (Enabled: false, the
	// default) registers nothing.
	Subprocess SubprocessConfig `json:"subprocess" mapstructure:"subprocess" yaml:"subprocess"`
}

ExecutionConfig configures which execution rungs beyond the always- present in-process default a deployment makes available.

A job definition declares the isolation it needs with job.WithExecution(exec.Isolate(...)); this block decides which rungs EXIST to satisfy that declaration — it never changes what any definition asks for, and it never lets a declaration that cannot be satisfied run anyway. engine.RegisterChecked already refuses a policy nothing configured here can satisfy (exec.ErrNoExecutor), unless the definition itself opted into exec.AllowDowngrade — that per-definition choice is deliberately not something this block can override, because a config-wide override would be exactly the silent downgrade RegisterChecked exists to prevent.

The whole block is additive and opt-in: a deployment that configures none of it registers no extra executor, and every job keeps running in-process exactly as it does today.

type ExtOption

type ExtOption func(*Extension)

ExtOption configures the Dispatch Forge extension.

func WithArtifactBackend

func WithArtifactBackend(b artifact.Backend) ExtOption

WithArtifactBackend supplies the object store backing the artifact plane explicitly, bypassing container discovery.

Use it outside Forge, or when the backend is not Trove.

func WithArtifactCacheBudget

func WithArtifactCacheBudget(bytes int64) ExtOption

WithArtifactCacheBudget caps the bytes the staging cache may hold.

With the resource model enabled this becomes the shared ledger's disk capacity, so it is the figure a job's declared disk requirement is admitted against rather than a private ceiling inside the cache.

func WithArtifactCacheDir

func WithArtifactCacheDir(dir string) ExtOption

WithArtifactCacheDir sets where staged artifacts are held on disk.

func WithArtifactStore

func WithArtifactStore(s artifact.Store) ExtOption

WithArtifactStore sets the store the artifact plane persists through. It defaults to the dispatcher's store when that store implements artifact.Store, which every bundled backend does.

func WithArtifacts

func WithArtifacts(troveStore string) ExtOption

WithArtifacts enables the artifact plane, resolving a Trove instance from the DI container. Pass a store name for multi-store Trove setups, or the empty string for the default instance.

func WithBackoff

func WithBackoff(b backoff.Strategy) ExtOption

WithBackoff sets the retry backoff strategy.

func WithBasePath

func WithBasePath(path string) ExtOption

WithBasePath sets the URL prefix for all dispatch routes.

func WithCPUOvercommit

func WithCPUOvercommit(factor float64) ExtOption

WithCPUOvercommit multiplies the detected core count (default 1.0).

CPU is compressible: exceeding it makes jobs slow, not dead. There is no memory equivalent, deliberately — overcommitting memory is how a box enters the OOM cascade this model prevents.

func WithConcurrency

func WithConcurrency(n int) ExtOption

WithConcurrency sets the maximum number of concurrent job processors.

func WithConfig

func WithConfig(cfg Config) ExtOption

WithConfig sets the extension configuration directly.

func WithCronLeaderTTL

func WithCronLeaderTTL(d time.Duration) ExtOption

WithCronLeaderTTL overrides the leader election TTL (default 60s). Renewal happens at half this interval.

func WithCronLockTTL

func WithCronLockTTL(d time.Duration) ExtOption

WithCronLockTTL overrides the per-entry distributed-lock TTL (default 30s).

func WithCronRefreshInterval

func WithCronRefreshInterval(d time.Duration) ExtOption

WithCronRefreshInterval overrides how often the cron leader re-lists entries from the store (default 30s). In-process registrations invalidate the cache immediately.

func WithCronStoreCallTimeout

func WithCronStoreCallTimeout(d time.Duration) ExtOption

WithCronStoreCallTimeout caps a single cron-scheduler store roundtrip (default 5s). Pass a negative duration to disable.

func WithCronTickInterval

func WithCronTickInterval(d time.Duration) ExtOption

WithCronTickInterval overrides the scheduler tick cadence (default 1s).

func WithDWP

func WithDWP(opts ...dwp.Option) ExtOption

WithDWP enables the Dispatch Wire Protocol (DWP) for real-time client communication over WebSocket, SSE, and HTTP RPC. Options configure authentication, codec, and server behaviour.

func WithDisableMigrate

func WithDisableMigrate() ExtOption

WithDisableMigrate disables auto-migration on start.

func WithDisableRoutes

func WithDisableRoutes() ExtOption

WithDisableRoutes disables the registration of HTTP routes.

func WithDurableDelivery

func WithDurableDelivery(config DeliveryConfig) ExtOption

func WithDurableHandlers

func WithDurableHandlers(workflows map[string]runtime.WorkflowFunc, activities map[string]runtime.ActivityFunc) ExtOption

WithDurableHandlers binds Go functions to YAML-configured worker routing. Registrations are captured when you create the option and copied per extension.

func WithDurableLifecycleHost

func WithDurableLifecycleHost(host operator.LifecycleHost) ExtOption

WithDurableLifecycleHost supplies trusted artifact and process resolvers. Configure it separately from authorization and the retained workflow resolver.

func WithDurableOperatorRuntime

func WithDurableOperatorRuntime(resolve func(namespace, build string) (*drt.Worker, error)) ExtOption

WithDurableOperatorRuntime resolves retained builds for commands and queries. Each returned worker must use this installation's store and the exact routing.

func WithDurableOperators

func WithDurableOperators(keys operator.CursorKeys, authorizer operator.Authorizer) ExtOption

WithDurableOperators enables namespace-aware inspection. Cursor keys belong to the host; store, installation and audit activation come from the extension.

func WithDurableWorkflows

func WithDurableWorkflows(options runtime.Options) ExtOption

WithDurableWorkflows captures immutable handler registrations immediately. Programmatic options replace YAML worker routing and timing as one unit.

func WithExplicitCapacity

func WithExplicitCapacity(sets ...resource.Set) ExtOption

WithExplicitCapacity overrides detection per key and is the only way to declare a custom resource — nothing detects "fpga".

Quantities are canonical units: cpu in millicores, memory and disk in bytes, gpu in milli-devices. Sets merge per key across calls.

func WithExtension

func WithExtension(x ext.Extension) ExtOption

WithExtension registers a dispatch extension (lifecycle hooks).

func WithGroveDatabase

func WithGroveDatabase(name string) ExtOption

WithGroveDatabase sets the name of the grove.DB to resolve from the DI container. The extension will auto-construct the appropriate store backend (postgres/sqlite/mongo) based on the grove driver type. Pass an empty string to use the default (unnamed) grove.DB.

func WithGroveKV

func WithGroveKV(name string) ExtOption

WithGroveKV sets the name of the grove kv.Store to resolve from the DI container. The extension will auto-construct a Redis-backed store from the resolved kv.Store. Pass an empty string to use the default (unnamed) kv.Store.

func WithHeartbeatInterval

func WithHeartbeatInterval(d time.Duration) ExtOption

WithHeartbeatInterval overrides the running-job heartbeat cadence (default 10s).

func WithKVKeyPrefix

func WithKVKeyPrefix(prefix string) ExtOption

WithKVKeyPrefix namespaces the Redis-backed store's keys (see Config.KeyPrefix). Pair it with WithGroveKV; the grove database path ignores it.

func WithLogger

func WithLogger(l log.Logger) ExtOption

WithLogger sets the structured logger for the dispatch engine.

func WithMaxPollInterval

func WithMaxPollInterval(d time.Duration) ExtOption

WithMaxPollInterval caps the worker fetcher's idle backoff (default 30s). Empty polls double the poll interval from PollInterval up to this value; new work or an in-process enqueue resets it.

func WithMemoryAuditForTesting

func WithMemoryAuditForTesting(config DeliveryConfig) ExtOption

WithMemoryAuditForTesting explicitly enables the memory store composition. It is not a production durability mode and still requires real local intents.

func WithMemoryFraction

func WithMemoryFraction(fraction float64) ExtOption

WithMemoryFraction sets the share of detected memory to advertise (default 0.8), leaving the remainder for the Go runtime, the page cache, and everything else sharing the box.

func WithMiddleware

func WithMiddleware(m mw.Middleware) ExtOption

WithMiddleware adds job middleware to the dispatch engine.

func WithOperatorSecurity

func WithOperatorSecurity(config SecurityConfig) ExtOption

WithOperatorSecurity sets the installation resource and lazy Forge/Warden adapters.

func WithPollInterval

func WithPollInterval(d time.Duration) ExtOption

WithPollInterval overrides the worker poll interval (default 1s). Increase to reduce constant driver-pool pressure when running against a single shared mongo / postgres node.

func WithQueues

func WithQueues(queues []string) ExtOption

WithQueues sets the queues the dispatcher will poll.

func WithRemoteSecurity

func WithRemoteSecurity(authenticator security.Authenticator, boundary security.Boundary) ExtOption

WithRemoteSecurity supplies a host-owned verified authenticator and authorizer. Custom authenticators must enforce their own cookie, origin and CSRF policy.

func WithRequireConfig

func WithRequireConfig(require bool) ExtOption

WithRequireConfig requires config to be present in YAML files. If true and no config is found, Register returns an error.

func WithResources

func WithResources() ExtOption

WithResources turns on capacity detection and resource-aware admission.

The extension then builds one resource.Manager over the detected capacity and hands the same instance to the staging cache and the worker pool: the cache holds a lease per cached entry and reclaims disk on demand, the pool admits every claimed job against what is actually free. Without this the pool dequeues unbounded and nothing changes.

Detection is cgroup-first, so a container with a two-core quota advertises two cores rather than the host's sixty-four.

func WithStaleJobThreshold

func WithStaleJobThreshold(d time.Duration) ExtOption

WithStaleJobThreshold overrides the no-heartbeat threshold and the matching reaper cadence (default 30s).

func WithStore

func WithStore(s dispatch.Storer) ExtOption

WithStore sets the persistence backend via a dispatcher option.

func WithWorkerCustomKeys

func WithWorkerCustomKeys(keys ...string) ExtOption

WithWorkerCustomKeys narrows the custom resource keys this worker advertises at dequeue. Empty advertises every custom key it has capacity for, which is usually what you want; this exists so a worker draining a device can stop attracting work for it.

Keys accumulate across calls and duplicates collapse, matching WithExplicitCapacity above. The slice is copied, so the caller keeps no handle on extension state.

func WithWorkerStoreCallTimeout

func WithWorkerStoreCallTimeout(d time.Duration) ExtOption

WithWorkerStoreCallTimeout caps a single worker store roundtrip (default 5s). Pass a negative duration to disable bounding (test-only).

type Extension

type Extension struct {
	*forge.BaseExtension
	// contains filtered or unexported fields
}

Extension adapts Dispatch as a Forge extension. It implements the forge.Extension interface so Dispatch can be mounted into any Forge app.

func New

func New(opts ...ExtOption) *Extension

New creates a Dispatch Forge extension with the given options.

func (*Extension) API

func (e *Extension) API() *api.API

API returns the API handler.

func (*Extension) ArtifactCache

func (e *Extension) ArtifactCache() *cache.Cache

ArtifactCache returns the staging cache, or nil when the plane is off.

func (*Extension) Artifacts

func (e *Extension) Artifacts() *artifact.Service

Artifacts returns the artifact service, or nil when the plane is off.

func (*Extension) AuditAcceptanceFailures

func (e *Extension) AuditAcceptanceFailures() uint64

AuditAcceptanceFailures exposes local security audit degradation without provider text. Unresolved command evidence is retained in LegacyAuditStore.

func (*Extension) BeginDurableDrain

func (e *Extension) BeginDurableDrain(ctx context.Context, request runtime.DrainRequest) (runtime.DrainHandle, error)

BeginDurableDrain keeps the operator boundary and publisher available while the explicitly bounded operation finishes. Stop by itself is a terminal stop.

func (*Extension) DWPServer

func (e *Extension) DWPServer() *dwp.Server

DWPServer returns the DWP server, or nil if DWP is not enabled.

func (*Extension) DeliveryStatus

func (e *Extension) DeliveryStatus(ctx context.Context, destination durable.Destination) (delivery.Status, error)

DeliveryStatus reports backlog separately from engine execution readiness. Sink outage does not fail Health while local acceptance remains available.

func (*Extension) Dependencies

func (e *Extension) Dependencies() []string

Dependencies implements forge.Extension. It adds the grove extension to whatever was set with SetDependencies, so Forge registers and starts grove first and the store resolution in Register can see its *grove.DB or kv.Store.

The dependency is declared unconditionally because the store is chosen inside Register (WithGroveDatabase, WithGroveKV, or auto-discovery of a default *grove.DB), after Forge has already ordered the extensions. That is safe without grove: Forge skips a declared dependency that is not a registered extension, so an app running Dispatch on its own store starts exactly as before.

func (*Extension) DurableReadiness

func (e *Extension) DurableReadiness(ctx context.Context) (runtime.WorkerReadiness, error)

DurableReadiness leaves database liveness and trusted host qualification separate.

func (*Extension) DurableStatus

func (e *Extension) DurableStatus() (runtime.WorkerStatus, error)

DurableStatus reports process readiness separately from storage Health.

func (*Extension) Engine

func (e *Extension) Engine() *engine.Engine

Engine returns the underlying dispatch engine. This is nil until Register is called.

func (*Extension) Handler

func (e *Extension) Handler() http.Handler

Handler returns the HTTP handler for all API routes. Convenience for standalone use outside Forge.

func (*Extension) Health

func (e *Extension) Health(ctx context.Context) error

Health implements forge.Extension.

func (*Extension) Register

func (e *Extension) Register(fapp forge.App) error

Register implements forge.Extension. It initializes the dispatcher, builds the engine, and optionally registers HTTP routes.

func (*Extension) RegisterContractContributor

func (e *Extension) RegisterContractContributor(d *dispatcher.Dispatcher, reg fc.Registry, wreg fc.WardenRegistry) error

RegisterContractContributor exposes the initialized engine to the dashboard.

func (*Extension) RegisterRoutes

func (e *Extension) RegisterRoutes(router forge.Router) error

RegisterRoutes registers all dispatch API routes into a Forge router. It returns every route forge refused; see api.API.RegisterRoutes.

func (*Extension) Resources

func (e *Extension) Resources() resource.Manager

Resources returns the shared admission ledger, or nil when the resource model is off. It is the same instance the staging cache and the worker pool were built with — which is the point of exposing it.

func (*Extension) Start

func (e *Extension) Start(ctx context.Context) error

Start begins job processing and runs auto-migration if enabled.

func (*Extension) Stop

func (e *Extension) Stop(ctx context.Context) error

Stop gracefully shuts down the dispatch engine.

func (*Extension) WaitDurableDrain

func (e *Extension) WaitDurableDrain(ctx context.Context, handle runtime.DrainHandle) (runtime.DrainResult, error)

WaitDurableDrain does not inherit cancellation from another observer.

type ResourceConfig

type ResourceConfig struct {
	// Enabled turns on capacity detection and resource-aware admission.
	Enabled bool `default:"false" json:"enabled" mapstructure:"enabled" yaml:"enabled"`

	// CPUOvercommit multiplies the detected core count. CPU is
	// compressible — exceeding it makes jobs slow rather than dead — so
	// values above 1.0 are a legitimate throughput trade. Zero means 1.0.
	//
	// There is deliberately no memory equivalent. Overcommitting memory
	// is how a box enters the OOM cascade this model exists to prevent.
	CPUOvercommit float64 `default:"1.0" json:"cpu_overcommit" mapstructure:"cpu_overcommit" yaml:"cpu_overcommit"`

	// MemoryFraction is the share of the detected memory limit to
	// advertise, leaving the rest for the Go runtime, the page cache, and
	// everything else on the box. Zero means 0.8.
	MemoryFraction float64 `default:"0.8" json:"memory_fraction" mapstructure:"memory_fraction" yaml:"memory_fraction"`

	// Explicit overrides detection per key, and is the ONLY way to
	// declare a custom resource: there is no detection for "fpga".
	// Quantities are canonical units — cpu in millicores, memory and disk
	// in bytes, gpu in milli-devices, custom keys in whatever integer the
	// declaring job means by them.
	Explicit resource.Set `json:"explicit" mapstructure:"explicit" yaml:"explicit"`

	// CustomKeys narrows the custom resource keys this worker advertises
	// at dequeue. Empty advertises every custom key in the detected
	// capacity, which is the honest default.
	CustomKeys []string `json:"custom_keys" mapstructure:"custom_keys" yaml:"custom_keys"`
}

ResourceConfig configures how this worker's capacity is derived and whether jobs are admitted against it at all.

Off by default, and off means the RUNTIME does nothing new: no manager is constructed, the pool offers no dequeue budget, every store backend skips its fit predicate, and the staging cache keeps the private disk budget it has always had. A deployment that does not set this behaves at runtime as it did before the resource model existed.

Two things happen regardless, and neither is under this flag. The resource columns are added by migration on every upgrade, because a job row has to be readable by every worker in a mixed-version fleet; on Postgres that includes an index build, which is why it is done CONCURRENTLY. And every enqueue writes those columns, at their zero values when nothing declares a requirement.

type RlimitsConfig

type RlimitsConfig struct {
	// AddressSpace caps RLIMIT_AS in bytes.
	AddressSpace int64 `json:"address_space" mapstructure:"address_space" yaml:"address_space"`
	// NoFile caps RLIMIT_NOFILE, the open file descriptor count.
	NoFile int64 `json:"nofile" mapstructure:"nofile" yaml:"nofile"`
	// NProc caps RLIMIT_NPROC, the number of processes the child's uid
	// may run.
	NProc int64 `json:"nproc" mapstructure:"nproc" yaml:"nproc"`
	// Core caps RLIMIT_CORE. Accepted for API symmetry; subprocess forces
	// the child's actual core limit to zero unconditionally regardless of
	// this value — see subprocess.Rlimits.Core.
	Core int64 `json:"core" mapstructure:"core" yaml:"core"`
	// FSize caps RLIMIT_FSIZE in bytes.
	FSize int64 `json:"fsize" mapstructure:"fsize" yaml:"fsize"`
}

RlimitsConfig configures the child process's POSIX resource limits. See subprocess.Rlimits for what each field does; the field names and units here mirror it exactly.

type SecurityConfig

type SecurityConfig struct {
	InstallationID string   `json:"installation_id" yaml:"installation_id" mapstructure:"installation_id"`
	PolicyTenant   string   `json:"policy_tenant" yaml:"policy_tenant" mapstructure:"policy_tenant"`
	Providers      []string `json:"providers" yaml:"providers" mapstructure:"providers"`
}

SecurityConfig identifies the host's installation and policy tenant. Request filters and identity claims cannot override it.

type SubprocessConfig

type SubprocessConfig struct {
	// Enabled registers the subprocess executor with the engine. Without
	// it, nothing else in this struct has any effect.
	Enabled bool `default:"false" json:"enabled" mapstructure:"enabled" yaml:"enabled"`

	// Binary overrides the path to the binary the executor re-execs for
	// every attempt. Empty resolves os.Executable() — the worker's own
	// binary — which is correct for every deployment that has not split
	// the sandboxed handlers into a separate build.
	Binary string `json:"binary" mapstructure:"binary" yaml:"binary"`

	// User and Group are the uid/gid the child process runs as
	// (subprocess.WithUser). Both must be set together, or neither: a
	// uid with no configured gid is rejected at startup rather than
	// silently running the child under the worker's own primary group.
	//
	// Zero means "not configured" rather than uid/gid 0 — this config
	// surface has no way to request running the child as root, which is
	// deliberate: it is never the isolation this rung exists to provide.
	User  int `json:"user" mapstructure:"user" yaml:"user"`
	Group int `json:"group" mapstructure:"group" yaml:"group"`

	// AllowSameUser is the single opt-out for running this rung unisolated
	// on the uid boundary (subprocess.WithAllowSameUser): it permits User
	// to name the worker's own uid, and it permits leaving User unset
	// entirely. Without it, either shape refuses at startup —
	// resolveExecutionOptions rejects it during Register, before this
	// worker ever starts processing jobs — rather than passing cleanly
	// and only then failing every attempt's launch, forever, once the
	// deployment is already running. That is a deliberate security
	// default (see WithAllowSameUser and checkLaunch, which enforces the
	// same rule again at Run() for callers that build subprocess.Executor
	// directly instead of through this config) that this config surface
	// passes through rather than working around: nothing here defaults it
	// to true, so a configuration mistake cannot silently defeat it.
	AllowSameUser bool `default:"false" json:"allow_same_user" mapstructure:"allow_same_user" yaml:"allow_same_user"`

	// ScratchDir is the root directory both the child process's working
	// directory (subprocess.WithScratchDir) and, when the artifact plane
	// is also enabled, the Runner's scratch OutputDir
	// (engine.WithScratchRoot) are created under. Empty means
	// os.TempDir() for both, their own independent defaults.
	//
	// Configuring this with the artifact plane OFF is not an error: the
	// child still gets a scratch working directory, but nothing commits
	// its outputs and PriorOutputs stays empty, exactly as
	// worker.Runner.WithArtifacts documents for a nil service — the
	// extension logs a warning at startup so that is a deliberate choice,
	// not a silent one.
	ScratchDir string `json:"scratch_dir" mapstructure:"scratch_dir" yaml:"scratch_dir"`

	// Rlimits configures POSIX resource limits applied to the child
	// (subprocess.WithRlimits). Fields are in bytes (AddressSpace, FSize,
	// Core) or counts (NoFile, NProc); zero leaves that limit at whatever
	// the worker itself runs with — except Core, which buildEnv forces
	// to zero unconditionally regardless of this value, so a configured
	// Core is always ignored (see RlimitsConfig.Core below). There is no
	// unit-suffixed string parsing here (no "16GB") — this repo takes no
	// new dependency to provide one, and every other byte-valued config
	// field (ArtifactCacheConfig.Budget, resource.Set) is already a plain
	// integer for the same reason.
	Rlimits RlimitsConfig `json:"rlimits" mapstructure:"rlimits" yaml:"rlimits"`

	// StrictRlimits makes a configured rlimit that did not actually take
	// effect a launch failure instead of a silently ignored warning (see
	// subprocess.WithStrictRlimits).
	StrictRlimits bool `default:"false" json:"strict_rlimits" mapstructure:"strict_rlimits" yaml:"strict_rlimits"`
}

SubprocessConfig configures exec/subprocess.Executor.

This rung refuses to launch outside Unix (exec/subprocess's checkLaunch and Available); the extension checks Available itself at startup, so enabling this on an unsupported platform fails registration once, loudly, instead of failing every job's first launch attempt at runtime.

Directories

Path Synopsis
Package contract serves the operator-wide Dispatch dashboard.
Package contract serves the operator-wide Dispatch dashboard.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL