workers

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package workers ports packages/chord/src/next-node/workers: Chord v2's workers component (slice 10 §3, §4), a worker process per address of a host kind, started on demand, stopped when idle, never restarted by itself, its plugins shipped from the server; `WorkerMain` is the worker's side. Pi's barrel `index.ts` is designed out: Go imports the package, and what the barrel exported is exported here (`CrossedError`, `ProcessLock`, `WorkerMain` and its types, `CreateWorkers` and its types).

A worker is a child process, not a goroutine

The observable contract of a worker is isolation, and a goroutine has none of it:

  • crash containment: a worker's crash (`os.Exit`, a signal, a panic, running out of memory) ends that worker only; the server sees its exit code or signal (`worker exited (code 7)`, `worker exited (SIGKILL)`) and fails the address. A panic in a goroutine ends the whole Go process.
  • kill: a worker that answers nothing (Pi's blocked event loop) is killed with everything it spawned (its process group, a job object on Windows). A goroutine cannot be killed.
  • lock semantics: one live worker per address across processes, held by the kernel for the holder's lifetime (workers_lock.go), released by its death, however it dies.
  • output: its stdout and stderr are its own, piped to the server, the stderr tail kept for the failure report.

A worker is therefore this program's own executable (or any program that calls `WorkerMain`), started with its IPC channel (a socketpair on Unix, a named pipe on Windows; workers_channel.go) beside its stdout and stderr pipes. `Entry` is the executable and `ExecArgv` its arguments (Node's `execArgv` and module). A test starts its own test binary again.

Async contract (AGENTS.md "TypeScript async/Promise parity")

  • Ordering: the control messages and the link's frames share one ordered stream, told to listeners in order by one goroutine; a write never blocks its sender. Per address, starts and stops run one after another (a chain of done channels); an event of a child is recorded on that child as it happens, in the order its source tells it, and a late event of an old child never changes a new one. A process's exit handler runs before anyone waiting on the exit resumes.
  • Cancellation: `context.Context` is `AbortSignal`. `Close` cancels starts waiting in the app's `Start`, for `started` or for the plugins; the caller's context ends only its own wait. A worker's context aborts at shutdown.
  • Shutdown draining: `Close` stops every child (`shutdown`, a 5 s grace, then the tree is killed), clears the timers and joins every goroutine the component owns. A worker runs its closers in reverse order, then closes its NodeHost, forcing its own exit after 3 s.
  • Unawaited Promises are owned tasks (the component's task group, the worker's reload loop), never naked goroutines.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func WorkerMain

func WorkerMain(fn func(start WorkerStart, ctx context.Context) (*WorkerResult, error), options WorkerMainOptions)

WorkerMain runs this process as a worker: fn once the start message arrived and the lock is taken. An error from fn (or a panic in it) sends `fatal` (redacted) and exits. It does not return.

Types

type AuthorizeFailedError

type AuthorizeFailedError struct {
	Address string
	Error   error
}

AuthorizeFailedError (`authorize_failed`): the app's `authorize` failed: the hold is refused (`forbidden`), the error stays private.

type BatchWorkersError

type BatchWorkersError struct{ state.BatchError }

BatchWorkersError is a `BatchError` of the state's batch.

type BusyWatchable

type BusyWatchable interface {
	Value() (delta.JsonValue, error)
	Watch(ctx context.Context) (chord.WatchHandle[delta.JsonValue], error)
}

BusyWatchable is a JSON watchable of numbers: a reference's member, or a source.

type CrossedCause

type CrossedCause struct {
	Kind  string
	Error CrossedError
}

CrossedCause is a failure cause as it crosses; Kind is the cause's kind.

type CrossedError

type CrossedError struct {
	Message string
	Code    string
	Reason  string
}

CrossedError is an error as it crosses: its message, and a ChordError's code and reason.

type CrossedFailure

type CrossedFailure struct {
	Plugin string
	Error  CrossedError
}

CrossedFailure is a plugin that failed to load, its error crossed.

type CrossedObject

type CrossedObject struct {
	Object string
	Kind   string
	Cause  *CrossedCause
}

CrossedObject is an object of a reload report as it crosses; Cause is set only for a parked one.

type CrossedReport

type CrossedReport struct {
	Plugins ReloadedPlugins
	Failed  []CrossedFailure
	Objects []CrossedObject
}

CrossedReport is a reload report as it crosses: its errors crossed.

type Exit

type Exit struct {
	Code   *int
	Signal *string
}

Exit is how a worker process ended: its exit code, or the signal that ended it; the other is nil (Node's null).

type ForkOptions

type ForkOptions struct {
	Entry    string
	ExecArgv []string
	Env      map[string]string
	// OnOutput is its output; without it, mirrored to this process's own.
	OnOutput func(stream string, chunk []byte)
}

ForkOptions is how a worker process is started: the bootstrap options of the workers component.

type FromWorker

type FromWorker struct {
	Kind   FromWorkerKind
	Idle   bool
	Report *CrossedReport
	Error  *CrossedError
}

FromWorker is what a worker sends the server: Idle with FromWorkerReady and FromWorkerIdle, Report with FromWorkerReady and a FromWorkerReloaded that applied, Error with FromWorkerFatal and a FromWorkerReloaded that failed.

type FromWorkerKind

type FromWorkerKind string

FromWorkerKind tags a FromWorker.

const (
	FromWorkerStarted  FromWorkerKind = "started"
	FromWorkerReady    FromWorkerKind = "ready"
	FromWorkerIdle     FromWorkerKind = "idle"
	FromWorkerFatal    FromWorkerKind = "fatal"
	FromWorkerReloaded FromWorkerKind = "reloaded"
)

type ProcessEvents

type ProcessEvents struct {
	Control func(message FromWorker)
	// LinkLost: its link was closed by this side (the heartbeat, a protocol error) while the child may live.
	LinkLost func()
	// LinkClosed: its link was closed by the child's side (it exits, or it closed its IPC channel and may live on).
	LinkClosed func()
	Exited     func(exit Exit)
}

ProcessEvents is what a worker process tells as it happens.

type ProcessLockHandle

type ProcessLockHandle struct {
	// contains filtered or unexported fields
}

ProcessLockHandle is a held lock: its listening socket and the connections of the waiters, kept open and never answered. Release frees it (the process's exit does too).

func ProcessLock

func ProcessLock(name string, ctx context.Context) (*ProcessLockHandle, error)

ProcessLock takes the lock name, waiting while another process holds it (until ctx aborts: it returns its cause). It returns what listening returns besides "in use", and for a stale socket file (macOS, the BSDs) an error naming it.

func (*ProcessLockHandle) Release

func (lock *ProcessLockHandle) Release() error

Release closes the lock's socket and its waiters' connections (they see them close) once it is gone; releasing again returns the first result.

type ReloadedPlugins

type ReloadedPlugins struct {
	Added   []string
	Changed []string
	Removed []string
}

ReloadedPlugins is the plugins of a reload report: names added, changed and removed.

type ToWorker

type ToWorker struct {
	Kind    ToWorkerKind
	Address contracts.Address
	Start   delta.JsonValue
	Scripts []plugins.PluginScript
	Lock    string
}

ToWorker is what the server sends a worker: Address, Start (JSON), Scripts and Lock only with ToWorkerStart; Scripts also with ToWorkerReload.

type ToWorkerKind

type ToWorkerKind string

ToWorkerKind tags a ToWorker.

const (
	ToWorkerStart    ToWorkerKind = "start"
	ToWorkerShutdown ToWorkerKind = "shutdown"
	ToWorkerReload   ToWorkerKind = "reload"
)

type WorkerExitedError

type WorkerExitedError struct {
	Address string
	Code    *int
	Signal  *string
}

WorkerExitedError (`worker_exited`): an exit nobody asked for.

type WorkerFailedError

type WorkerFailedError struct {
	Address    string
	Error      error
	StderrTail string
}

WorkerFailedError (`worker_failed`): Error is the full error (a start's throw, a crash, the child's `fatal`), for the app's logs, never public.

type WorkerLoad

type WorkerLoad struct {
	Kind string
	// Core is the app's own facets that are JavaScript: modules whose default export is the facet, loaded into the NodeHost
	// before the plugins (they import Pi's chord by its package name, `@earendil-works/chord/next`).
	Core []plugins.CoreFacet
	// GoCore are the app's own facets that are Go: they load on the worker's Go host, and the local types the NodeHost's
	// facets use are named by Bridged.
	GoCore []host.HostFacet
	// Bridged are the Go core facets whose local types the NodeHost's core facets and plugins use (plugins.BridgedFacet): the
	// NodeHost loads a proxy facet of each, in the same closure as the plugins. A BridgedFacet without a Host is the worker's.
	Bridged []plugins.BridgedFacet
	// HostKind is the kind of the worker's address, the one the server routes to this worker.
	HostKind *contracts.HostKind
	// Principal is who the NodeHost acts as, and how the worker's Go host stamps it; Peer is who the worker's Go host is for
	// the NodeHost (the server, whose holds it relays).
	Principal data.Principal
	Peer      data.Principal
}

WorkerLoad is what WorkerPlugins.Load takes besides the host.

The plugins of a worker run in a Node subprocess (docs/plan/chord-plugins-node-evaluator.md): Pi's worker host loads them in process, here the worker's Go host is the relay the server's holds pass through, and the host that serves the worker's address, the app's core facets and the plugins' is the NodeHost, the child of the worker's Go host. Load makes it, routes the worker's address kind to its link on the worker's Go host (`Route`, before `Load` of that host: the app's host has no address of its own, which the route would not reach) and loads the Go host.

type WorkerMainOptions

type WorkerMainOptions struct {
	// HostPackages is the host packages by specifier: the module files the NodeHost imports once and the plugins share.
	HostPackages plugins.HostPackages
	// Node is how the worker's Node subprocess is started (its output goes to this process's own unless OnOutput says
	// otherwise, as a worker's does).
	Node plugins.NodeOptions
	// Redact is what crosses as `fatal` for an error (never the original message unless this returns it).
	Redact  func(err any) CrossedError
	OnError func(err any)
}

WorkerMainOptions is what WorkerMain takes besides the app's function.

type WorkerPlugins

type WorkerPlugins struct {
	// contains filtered or unexported fields
}

WorkerPlugins is the worker's plugins: the shipped scripts of its kind, loaded on its host; the server's reloads applied to it.

func (*WorkerPlugins) Load

func (p *WorkerPlugins) Load(h host.Host, options WorkerLoad, ctx context.Context) (host.LoadReport, error)

Load loads the app's Go core facets (options.GoCore) on h (a host without an address, not loaded yet), routes the worker's address kind to the link of the NodeHost on h (`Route`, before `Load` of h), starts the NodeHost of the worker's address with the start message's scripts of Kind, the JavaScript Core and a proxy for each of options.Bridged (the Go core facets whose local types the NodeHost's facets use; a BridgedFacet without a Host is h's), and returns the NodeHost's report. The Go host is loaded first: the proxies ask for their Go objects as they activate, inside the NodeHost's own load. When the NodeHost ends before the worker is told to stop, the worker ends with it: it exits with its exit code, or sends `fatal` with the exception it died of. Its close is the worker's last: shutdown closes it after the app's closers, those registered after Load too, so the app's host closes the link to the server before the NodeHost ends and fails a reload in flight (Pi's host closer closes the link before its reload rejects).

type WorkerProcess

type WorkerProcess struct {
	Pid       int
	StartedAt int64
	// Link is the link to the child's host, made at the fork.
	Link link.Link
	// contains filtered or unexported fields
}

WorkerProcess is one started worker process.

func NewWorkerProcess

func NewWorkerProcess(options ForkOptions, events ProcessEvents) (*WorkerProcess, error)

NewWorkerProcess starts the worker process options describe and tells events as they happen.

func (*WorkerProcess) Exit

func (p *WorkerProcess) Exit() Exit

Exit is how the process ended; it is valid once Exited is closed.

func (*WorkerProcess) Exited

func (p *WorkerProcess) Exited() <-chan struct{}

Exited is closed when the process ended (every process it spawned is killed then); Exit is its result.

func (*WorkerProcess) Kill

func (p *WorkerProcess) Kill()

Kill kills the child and every process it spawned; it does nothing once the process ended.

func (*WorkerProcess) Send

func (p *WorkerProcess) Send(message ToWorker)

Send sends a control message; it is dropped when the child is gone (its exit tells).

func (*WorkerProcess) StderrTail

func (p *WorkerProcess) StderrTail() string

StderrTail is the last bytes it wrote to stderr; empty when it wrote none.

type WorkerResult

type WorkerResult struct {
	Busy BusyWatchable
	// Held is the number of holds the app's own code keeps on the NodeHost's objects (a reference it reads Busy through): the
	// NodeHost counts them as references like the server's, and they do not keep the worker busy.
	Held int
}

WorkerResult is what the app's function returns: extra keep-alive counts (busy while any is nonzero).

type WorkerStart

type WorkerStart struct {
	Address contracts.Address
	// Start is the start data the server's `start(address)` gave.
	Start delta.JsonValue
	// Parent is the link to the server: connect it before anything slow (`started` goes out then).
	Parent  link.Link
	Plugins *WorkerPlugins
	// OnClose registers a closer: closers run in reverse order at shutdown.
	OnClose func(closer func(ctx context.Context) error)
}

WorkerStart is what the app's function gets.

type WorkerState

type WorkerState struct {
	Status    WorkerStatus
	Pid       int
	StartedAt int64
	Error     CrossedError
	At        int64
}

WorkerState is one address's worker (R5): running or on its way, or failed (until `Retry` or a rollout) with a public error: a fixed message (`worker failed to start`, `worker exited (SIGSEGV)`) or the child's redacted `fatal`. Pid and StartedAt are set for a worker that lives, Error and At for a failed one.

type WorkerStatus

type WorkerStatus string

WorkerStatus is where a worker stands.

const (
	WorkerStarting WorkerStatus = "starting"
	WorkerRunning  WorkerStatus = "running"
	WorkerStopping WorkerStatus = "stopping"
	WorkerFailed   WorkerStatus = "failed"
)

type Workers

type Workers struct {
	// contains filtered or unexported fields
}

Workers is the workers component.

func CreateWorkers

func CreateWorkers(h host.Host, kind *contracts.HostKind, options WorkersOptions, ctx context.Context) (*Workers, error)

CreateWorkers creates the workers component on h: it routes kind itself (`host.Route`, before `Load`).

func (*Workers) Close

func (c *Workers) Close(ctx context.Context) error

Close cancels starts, stops every worker; new starts reject `released/closed`.

func (*Workers) Retry

func (c *Workers) Retry(address contracts.Address, _ context.Context) error

Retry clears address's failure and suspension: it starts again now if held (`RouteHandle.Retry`).

func (*Workers) State

func (c *Workers) State() state.SourceHandle

State is per address key its WorkerState; no entry when stopped or never started. A JSON watchable.

func (*Workers) Step

func (c *Workers) Step(name string) plugins.RolloutStep

Step is a rollout step shipping the kind's current set to every worker, one result per worker (§4).

func (*Workers) Stop

func (c *Workers) Stop(address contracts.Address, ctx context.Context) error

Stop suspends address while it has holders (`RouteHandle.Suspend`), then stops its worker.

type WorkersError

type WorkersError interface {
	// contains filtered or unexported methods
}

WorkersError is what the component reports (and the state's watch listeners that threw): one of WorkerFailedError, AuthorizeFailedError, WorkerExitedError and BatchWorkersError.

type WorkersOptions

type WorkersOptions struct {
	// Kind is the facet kind its workers load ("worker"), shipped by the plugins component.
	Kind string
	// Entry is the worker's executable (it calls WorkerMain), ExecArgv its arguments and Env its environment.
	Entry    string
	ExecArgv []string
	Env      map[string]string
	// Authorize is the route's `authorize`: whether principal may reach address.
	Authorize func(principal data.Principal, address contracts.Address) (bool, error)
	// Principal is who the worker at address is (stamped on its link).
	Principal func(address contracts.Address) (data.Principal, error)
	// Start is the worker's start data (JSON), before each start; abortable (`Close`), an error fails the start.
	Start func(address contracts.Address, ctx context.Context) (delta.JsonValue, error)
	// Lock is the lock one live worker of address holds (across processes).
	Lock func(address contracts.Address) (string, error)
	// IdleMs is how long a worker idles before it is stopped: 5 minutes by default.
	IdleMs  func() time.Duration
	Plugins *plugins.Plugins
	// OnOutput is a worker's output; without it, mirrored to this process's own.
	OnOutput func(address contracts.Address, stream string, chunk []byte)
	OnError  func(err WorkersError)
	// contains filtered or unexported fields
}

WorkersOptions is what CreateWorkers takes. Kind, Entry, ExecArgv and Env are bootstrap; the functions are called per use.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL