operator

package
v0.109.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 22, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotSupported is returned by a host or session for a capability the transport does
	// not offer: the gRPC host cannot open an already claimed operator row, and an in-process
	// host without an admin service cannot put workflows.
	ErrNotSupported = errors.New("operator: not supported by this host")

	// ErrSessionClosed is returned by session methods once Close was called.
	ErrSessionClosed = errors.New("operator: session closed")

	// ErrRequestInFlight is returned by DurableChannel.Send when an ack-bearing request (memo,
	// trigger_runs, wait_for, evict_invocation) is sent while a previous one is still waiting
	// for its ack. The engine keys pending acks by (task, invocation), so a second one would
	// clobber the first.
	ErrRequestInFlight = errors.New("operator: durable request already in flight for this invocation")

	// ErrChannelClosed is returned by DurableChannel.Send and Recv once Close was called.
	ErrChannelClosed = errors.New("operator: durable channel closed")

	// ErrSessionEnded is returned by DurableChannel.Recv when the engine tore the invocation
	// down without Close being called, which the operator reports as a retryable failure.
	ErrSessionEnded = errors.New("operator: engine durable session ended")
)

Functions

This section is empty.

Types

type ActionHandler added in v0.109.0

type ActionHandler interface {
	HandleAction(ctx context.Context, action *contracts.AssignedAction) error
}

ActionHandler receives assigned actions. It must not block for long: in process it runs on the dispatcher's delivery goroutine and its error requeues the task; over gRPC it runs on the session's deliver loop and its error is reported as a retryable failure.

type DAGStepTriggerRequest

type DAGStepTriggerRequest struct {
	ParentTaskExternalId uuid.UUID
	InvocationCount      int32
	WorkflowName         string
	// WorkflowVersionId pins triggering to the DAG's original version.
	WorkflowVersionId   uuid.UUID
	ActionId            string
	ChildIndex          int32
	Input               string
	AdditionalMetadata  []byte
	DagParentTaskRunIds []uuid.UUID
	IsSkipped           bool
	IsCancelled         bool
	DesiredWorkerLabels []*sqlcv1.GetDesiredLabelsRow

	// ParentReExecuted forces the step to re-run during a replay when any of its parents
	// re-executed this invocation.
	ParentReExecuted bool
}

type DAGStepTriggerResult

type DAGStepTriggerResult struct {
	NodeId                int64
	BranchId              int64
	WorkflowRunExternalId uuid.UUID

	IsSatisfied   bool
	ResultPayload []byte
	IsFailure     bool
	ErrorMessage  *string

	// ReExecuted is true when the step actually runs this invocation rather than being
	// satisfied from the log.
	ReExecuted bool
}

type DurableChannel added in v0.109.0

type DurableChannel interface {
	Send(ctx context.Context, req *v1.DurableTaskRequest) error
	Recv(ctx context.Context) (*v1.DurableTaskResponse, error)

	// ExpectEntry registers an entry of the invocation whose completion the operator awaits
	// and which no request on this channel acknowledged. A completion that already arrived is
	// delivered on the next Recv; one that arrives later is delivered as it comes. Registering
	// an entry twice, or after its completion was delivered, changes nothing.
	ExpectEntry(branchId, nodeId int64) error

	Close() error
}

DurableChannel is one durable invocation's pipe. Send stamps the invocation's task id and count on the request and admits one ack-bearing request at a time (ErrRequestInFlight); the slot is free again once the ack, or the error that replaces it, has arrived. register_worker is the host's and is refused. worker_status may be sent to report the entries the invocation is blocked on; a host whose transport reports them itself drops it. Responses arrive in engine order, with an entry completion never ahead of the ack that names its entry.

An entry the operator learns of outside the channel has no ack on it: the DAG operator's children are created by the engine-internal writer, which returns their refs directly. The operator registers such a ref with ExpectEntry, which stands in for the ack; without it the channel would hold the entry's completion for an ack that never comes.

type Host added in v0.109.0

type Host interface {
	Open(ctx context.Context, id Identity, opts OpenOpts) (Session, error)
}

Host opens sessions. One Host per process; it spans tenants. Implementations: internal/operator/hostinproc (inside the dispatcher process, over the engine's own session logic) and pkg/operator/hostgrpc (pkg/client over OperatorService with a TokenSource).

type Identity added in v0.109.0

type Identity struct {
	TenantId uuid.UUID

	// OperatorId is an existing operator row. The host takes the name, kind and leasing manager from
	// the row and points the row's worker_id at the session's worker, so the claimer keeps
	// recognising the assignment.
	OperatorId *uuid.UUID

	// Name, Kind and LeasingManager identify a row the host upserts. Kind defaults to GRPC, the only
	// kind a host registers by name, and LeasingManager to SELF, the only leasing manager the gRPC host can
	// register: a row it opened is kept alive by its stream, not by the claimer.
	Name           string
	Kind           sqlcv1.V1OperatorKind
	LeasingManager sqlcv1.V1OperatorLeasingManager
}

Identity names the operator a session registers as. Exactly one of OperatorId (an existing row, as claimed by the in-process claimer) or Name with Kind and LeasingManager (a row the host upserts by (tenant, name, kind)) is used. TenantId is always required: a Host spans tenants and a Session belongs to one.

Kind says what the operator is and LeasingManager who keeps it alive; they are separate axes. A contract operator is GRPC whichever host runs it. A SELF row keeps itself alive, through the Listen stream the gRPC host holds or the leaser an in-process operator runs, and is what the wire registers; for a DISPATCHER row the dispatcher claims the row through the claimer, which builds the operator from a factory and opens it by OperatorId.

type OpenOpts added in v0.109.0

type OpenOpts struct {
	// Handler receives the actions assigned to the session's worker. Required.
	Handler ActionHandler

	// Actions is the initial action set, linked before Open returns so the worker is never
	// live with an action set the caller did not ask for. Later changes go through
	// Session.AddActions and RemoveActions.
	Actions []string

	// SlotConfig maps slot type to max units. The engine defaults it to {"default": 100}.
	SlotConfig map[string]int32

	// Labels are worker labels for affinity assignment (string or int values).
	Labels map[string]interface{}

	// RuntimeInfo describes the process for the dashboard. The gRPC host reports its own
	// process and ignores it.
	RuntimeInfo *contracts.RuntimeInfo

	// ResumeWorkerId names a previous worker of the same operator to resume instead of
	// creating one. Not every host supports it (ErrNotSupported).
	ResumeWorkerId *uuid.UUID

	// WorkerName names the worker row; it defaults to the operator name. Not every host
	// supports it (ErrNotSupported).
	WorkerName string
}

OpenOpts describes the worker the session backs and the handler assigned actions go to.

type Operator

type Operator interface {
	ActionHandler

	Start(ctx context.Context, s Session) error
	Drain(ctx context.Context)
}

Operator is what a hosted operator implements. The host opens the session with the operator as its handler and calls Start once; the operator keeps the session for the rest of its life. Drain stops new work and waits for in-flight work, bounded by ctx; the host pauses the worker before Drain and closes the session after it.

type Registration added in v0.109.0

type Registration struct {
	TenantId   uuid.UUID
	OperatorId uuid.UUID
	WorkerId   uuid.UUID

	// Resumed reports whether WorkerId is the worker ResumeWorkerId named.
	Resumed bool
}

Registration is the identity the engine assigned to a session.

type Session added in v0.109.0

type Session interface {
	Registration() Registration

	// AddActions adds ids to the worker's action set; ids already in the set are ignored. A
	// host may apply the delta asynchronously; Flush waits for it.
	AddActions(ctx context.Context, ids []string) error

	// RemoveActions removes ids from the worker's action set; ids not in the set are ignored.
	RemoveActions(ctx context.Context, ids []string) error

	// Flush returns once every delta issued so far is committed by the engine and reports the
	// failure, if any. In-process deltas are synchronous and Flush returns at once.
	Flush(ctx context.Context) error

	// PutWorkflow registers or updates a workflow for the session's tenant and returns the
	// action ids its tasks derive, normalized the way the engine stores them. It does not
	// change the worker's action set.
	PutWorkflow(ctx context.Context, wf *v1.CreateWorkflowVersionRequest) ([]string, error)

	// SendStepActionEvent reports task progress (STARTED, COMPLETED, FAILED, CANCELLED). An
	// empty WorkerId is filled from the registration; another worker's id is refused.
	SendStepActionEvent(ctx context.Context, ev *contracts.StepActionEvent) error

	// OpenDurable opens one invocation's request and response pipe. The register_worker
	// handshake is done by the host; responses that arrive before it completes are held,
	// bounded, and delivered after the ack that names their entry. ctx bounds the handshake;
	// the invocation itself ends on the channel's Close.
	OpenDurable(ctx context.Context, taskExternalId uuid.UUID, invocation int32) (DurableChannel, error)

	// Pause stops the scheduler assigning to the worker and returns once the pause is
	// committed, so a caller that drains afterwards knows no further work will arrive.
	Pause(ctx context.Context) error

	// Close ends the session: the worker is paused if it is not already, the session is
	// released so nothing further is delivered, and the worker is deactivated, fenced on the
	// session id. A host's teardown order is Pause, the operator's Drain, then Close.
	Close(ctx context.Context) error

	// Done is closed once the session no longer serves its worker: after Close, or when the
	// host gave up on it. A host recovers from transient failures on its own (the gRPC host
	// reconnects, re-reads the tenant's token and restores the action set); it gives up on a
	// failure no retry can fix, such as a token source that no longer has a token for the
	// tenant. The caller that owns the session then closes it and opens another.
	Done() <-chan struct{}

	// Err reports why Done closed: nil after Close, the terminal failure when the host gave
	// up. It is nil while the session is live.
	Err() error
}

Session is one worker row's worth of traffic: its action set, its events, its durable invocations and its liveness, which the host maintains until Close. Methods are safe to call concurrently until Close.

type SharedOperator

type SharedOperator[T any] struct {
	// contains filtered or unexported fields
}

SharedOperator is the state an engine-internal operator shares: its config, the session the host opened for it, the engine-internal writer, and the bookkeeping for in-flight work. The session arrives with Start; the event senders, the action set and the durable channels go through it, so the operator's lifecycle is the host's.

func NewSharedOperator

func NewSharedOperator[T any](operator *sqlcv1.V1Operator, l *zerolog.Logger, taskEventWriter TaskEventWriter, t T) (*SharedOperator[T], error)

NewSharedOperator constructs the shared operator state from the operator row.

func (*SharedOperator[T]) CancelDAGChildren

func (s *SharedOperator[T]) CancelDAGChildren(ctx context.Context, taskExternalIds []uuid.UUID) error

func (*SharedOperator[T]) CancelTask

func (s *SharedOperator[T]) CancelTask(taskRunExternalId string) bool

func (*SharedOperator[T]) Config

func (s *SharedOperator[T]) Config() T

func (*SharedOperator[T]) Drain

func (s *SharedOperator[T]) Drain(ctx context.Context)

Drain stops accepting new tracked tasks and waits for the in-flight ones, or for ctx. The host pauses the worker before calling it, so nothing new arrives while it waits.

func (*SharedOperator[T]) Logger

func (s *SharedOperator[T]) Logger() *zerolog.Logger

func (*SharedOperator[T]) OpenDurable added in v0.109.0

func (s *SharedOperator[T]) OpenDurable(ctx context.Context, taskExternalId uuid.UUID, invocation int32) (DurableChannel, error)

OpenDurable opens one durable invocation's pipe through the session: the host does the register-worker handshake and holds what the engine sends before its ack, so the operator only ever reads invocation traffic, an entry never ahead of the ack that names it.

func (*SharedOperator[T]) OperatorId added in v0.109.0

func (s *SharedOperator[T]) OperatorId() uuid.UUID

func (*SharedOperator[T]) RecordTask

func (s *SharedOperator[T]) RecordTask() func()

RecordTask registers an in-flight task and returns a release function that the caller must invoke (typically via defer) when the task finishes. Drain blocks until every recorded task has been released.

If the operator is already draining, the returned release is a no-op and the task is not tracked: callers should avoid starting new work once Drain has begun, but in-flight work recorded before it is always awaited.

func (*SharedOperator[T]) RegisterCancellableContext

func (s *SharedOperator[T]) RegisterCancellableContext(ctx context.Context, taskRunExternalId string) (context.Context, func())

func (*SharedOperator[T]) SendCancelled

func (s *SharedOperator[T]) SendCancelled(action *contracts.AssignedAction) error

SendCancelled reports a cancelled task

func (*SharedOperator[T]) SendCancelledWithMessage

func (s *SharedOperator[T]) SendCancelledWithMessage(action *contracts.AssignedAction, msg string) error

SendCancelledWithMessage reports a cancelled task with a custom cancellation reason through the engine-internal writer rather than the generic step action event path, so the reason reaches the run's events verbatim. There is no such RPC: an operator hosted over gRPC reports a plain CANCELLED event instead.

func (*SharedOperator[T]) SendCompleted

func (s *SharedOperator[T]) SendCompleted(action *contracts.AssignedAction, output []byte) error

SendCompleted reports a successful result. output should be the task's JSON output.

func (*SharedOperator[T]) SendFailed

func (s *SharedOperator[T]) SendFailed(action *contracts.AssignedAction, errMsg string, shouldNotRetry bool) error

SendFailed reports a failure with the given error message. shouldNotRetry, when true, prevents the task from being retried.

func (*SharedOperator[T]) SendStarted

func (s *SharedOperator[T]) SendStarted(action *contracts.AssignedAction) error

SendStarted reports that the operator has started processing the assigned action.

func (*SharedOperator[T]) SendStartedAt added in v0.106.2

func (s *SharedOperator[T]) SendStartedAt(action *contracts.AssignedAction, at time.Time) error

func (*SharedOperator[T]) Session added in v0.109.0

func (s *SharedOperator[T]) Session() Session

Session is the session the host opened, or nil before Start.

func (*SharedOperator[T]) Start added in v0.109.0

func (s *SharedOperator[T]) Start(_ context.Context, session Session) error

Start records the session the host opened. Operators that embed SharedOperator call it from their own Start.

func (*SharedOperator[T]) TenantId

func (s *SharedOperator[T]) TenantId() uuid.UUID

func (*SharedOperator[T]) TriggerDAGStep

func (s *SharedOperator[T]) TriggerDAGStep(ctx context.Context, req *DAGStepTriggerRequest) (*DAGStepTriggerResult, error)

func (*SharedOperator[T]) UpdateWorkerActions

func (s *SharedOperator[T]) UpdateWorkerActions(ctx context.Context, actions []string) (bool, error)

UpdateWorkerActions makes actions the worker's action set: the difference from the set last advertised goes to the session as adds and removes, followed by a flush. It reports whether anything changed. A delta that fails leaves the advertised set as it was, so the next call repeats it; ids the engine already has are ignored by it.

The difference is what makes the call cheap to repeat: the engine would accept the whole set every time (adding an action the worker has is a no-op), but every send is a write, and the removes cannot be derived without the previous set. It is not safe for concurrent use: the operator calls it from one goroutine at a time (its Start, then the poller Start launches once that first call has returned), which is why lastActions needs no lock.

func (*SharedOperator[T]) WorkerId

func (s *SharedOperator[T]) WorkerId() uuid.UUID

WorkerId is the worker the session backs, or uuid.Nil before Start.

type TaskEventWriter

type TaskEventWriter interface {
	// CancelTaskWithReason reports a cancelled task with a custom cancellation reason. It is
	// the engine-internal writer behind SendCancelledWithMessage, distinct from the CANCELLED
	// step action event every host offers through Session.SendStepActionEvent, and it is not
	// on the gRPC surface: an out-of-process operator has no equivalent.
	CancelTaskWithReason(ctx context.Context, tenantId uuid.UUID, request *contracts.StepActionEvent) (*contracts.ActionEventResponse, error)

	TriggerDAGStep(ctx context.Context, tenantId uuid.UUID, req *DAGStepTriggerRequest) (*DAGStepTriggerResult, error)

	// CancelDAGChildren cancels already-triggered children when the orchestrator is cancelled.
	CancelDAGChildren(ctx context.Context, tenantId uuid.UUID, taskExternalIds []uuid.UUID) error
}

TaskEventWriter is the engine-internal surface for engine-internal operators (the DAG operator): calls that only exist inside the engine and are never available over gRPC. The dispatcher implements it. Everything an operator needs that both hosts offer (events, durable invocations, the action set) is on Session instead. Every call names its tenant explicitly: nothing here reads the tenant the gRPC auth middleware puts on a request context.

Directories

Path Synopsis
Package hostgrpc is the out-of-process operator host: it implements pkg/operator.Host over pkg/client/operatorclient, speaking OperatorService to the engine with a per-tenant API token from a TokenSource.
Package hostgrpc is the out-of-process operator host: it implements pkg/operator.Host over pkg/client/operatorclient, speaking OperatorService to the engine with a per-tenant API token from a TokenSource.
Package operatortest holds a sample contract operator for tests.
Package operatortest holds a sample contract operator for tests.
Package safeclient provides an SSRF-hardened HTTP client for delivering outbound requests to user-supplied endpoints.
Package safeclient provides an SSRF-hardened HTTP client for delivering outbound requests to user-supplied endpoints.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL