Documentation
¶
Index ¶
- Variables
- type ActionHandler
- type DAGStepTriggerRequest
- type DAGStepTriggerResult
- type DurableChannel
- type Host
- type Identity
- type OpenOpts
- type Operator
- type Registration
- type Session
- type SharedOperator
- func (s *SharedOperator[T]) CancelDAGChildren(ctx context.Context, taskExternalIds []uuid.UUID) error
- func (s *SharedOperator[T]) CancelTask(taskRunExternalId string) bool
- func (s *SharedOperator[T]) Config() T
- func (s *SharedOperator[T]) Drain(ctx context.Context)
- func (s *SharedOperator[T]) Logger() *zerolog.Logger
- func (s *SharedOperator[T]) OpenDurable(ctx context.Context, taskExternalId uuid.UUID, invocation int32) (DurableChannel, error)
- func (s *SharedOperator[T]) OperatorId() uuid.UUID
- func (s *SharedOperator[T]) RecordTask() func()
- func (s *SharedOperator[T]) RegisterCancellableContext(ctx context.Context, taskRunExternalId string) (context.Context, func())
- func (s *SharedOperator[T]) SendCancelled(action *contracts.AssignedAction) error
- func (s *SharedOperator[T]) SendCancelledWithMessage(action *contracts.AssignedAction, msg string) error
- func (s *SharedOperator[T]) SendCompleted(action *contracts.AssignedAction, output []byte) error
- func (s *SharedOperator[T]) SendFailed(action *contracts.AssignedAction, errMsg string, shouldNotRetry bool) error
- func (s *SharedOperator[T]) SendStarted(action *contracts.AssignedAction) error
- func (s *SharedOperator[T]) SendStartedAt(action *contracts.AssignedAction, at time.Time) error
- func (s *SharedOperator[T]) Session() Session
- func (s *SharedOperator[T]) Start(_ context.Context, session Session) error
- func (s *SharedOperator[T]) TenantId() uuid.UUID
- func (s *SharedOperator[T]) TriggerDAGStep(ctx context.Context, req *DAGStepTriggerRequest) (*DAGStepTriggerResult, error)
- func (s *SharedOperator[T]) UpdateWorkerActions(ctx context.Context, actions []string) (bool, error)
- func (s *SharedOperator[T]) WorkerId() uuid.UUID
- type TaskEventWriter
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotSupported is returned by a host or session for a capability the transport does // not offer: the gRPC host cannot open an already claimed operator row, and an in-process // host without an admin service cannot put workflows. ErrNotSupported = errors.New("operator: not supported by this host") // ErrSessionClosed is returned by session methods once Close was called. ErrSessionClosed = errors.New("operator: session closed") // ErrRequestInFlight is returned by DurableChannel.Send when an ack-bearing request (memo, // trigger_runs, wait_for, evict_invocation) is sent while a previous one is still waiting // for its ack. The engine keys pending acks by (task, invocation), so a second one would // clobber the first. ErrRequestInFlight = errors.New("operator: durable request already in flight for this invocation") // ErrChannelClosed is returned by DurableChannel.Send and Recv once Close was called. ErrChannelClosed = errors.New("operator: durable channel closed") // ErrSessionEnded is returned by DurableChannel.Recv when the engine tore the invocation // down without Close being called, which the operator reports as a retryable failure. ErrSessionEnded = errors.New("operator: engine durable session ended") )
Functions ¶
This section is empty.
Types ¶
type ActionHandler ¶ added in v0.109.0
type ActionHandler interface {
HandleAction(ctx context.Context, action *contracts.AssignedAction) error
}
ActionHandler receives assigned actions. It must not block for long: in process it runs on the dispatcher's delivery goroutine and its error requeues the task; over gRPC it runs on the session's deliver loop and its error is reported as a retryable failure.
type DAGStepTriggerRequest ¶
type DAGStepTriggerRequest struct {
ParentTaskExternalId uuid.UUID
InvocationCount int32
WorkflowName string
// WorkflowVersionId pins triggering to the DAG's original version.
WorkflowVersionId uuid.UUID
ActionId string
ChildIndex int32
Input string
AdditionalMetadata []byte
DagParentTaskRunIds []uuid.UUID
IsSkipped bool
IsCancelled bool
DesiredWorkerLabels []*sqlcv1.GetDesiredLabelsRow
// ParentReExecuted forces the step to re-run during a replay when any of its parents
// re-executed this invocation.
ParentReExecuted bool
}
type DAGStepTriggerResult ¶
type DurableChannel ¶ added in v0.109.0
type DurableChannel interface {
Send(ctx context.Context, req *v1.DurableTaskRequest) error
Recv(ctx context.Context) (*v1.DurableTaskResponse, error)
// ExpectEntry registers an entry of the invocation whose completion the operator awaits
// and which no request on this channel acknowledged. A completion that already arrived is
// delivered on the next Recv; one that arrives later is delivered as it comes. Registering
// an entry twice, or after its completion was delivered, changes nothing.
ExpectEntry(branchId, nodeId int64) error
Close() error
}
DurableChannel is one durable invocation's pipe. Send stamps the invocation's task id and count on the request and admits one ack-bearing request at a time (ErrRequestInFlight); the slot is free again once the ack, or the error that replaces it, has arrived. register_worker is the host's and is refused. worker_status may be sent to report the entries the invocation is blocked on; a host whose transport reports them itself drops it. Responses arrive in engine order, with an entry completion never ahead of the ack that names its entry.
An entry the operator learns of outside the channel has no ack on it: the DAG operator's children are created by the engine-internal writer, which returns their refs directly. The operator registers such a ref with ExpectEntry, which stands in for the ack; without it the channel would hold the entry's completion for an ack that never comes.
type Host ¶ added in v0.109.0
Host opens sessions. One Host per process; it spans tenants. Implementations: internal/operator/hostinproc (inside the dispatcher process, over the engine's own session logic) and pkg/operator/hostgrpc (pkg/client over OperatorService with a TokenSource).
type Identity ¶ added in v0.109.0
type Identity struct {
TenantId uuid.UUID
// OperatorId is an existing operator row. The host takes the name, kind and leasing manager from
// the row and points the row's worker_id at the session's worker, so the claimer keeps
// recognising the assignment.
OperatorId *uuid.UUID
// Name, Kind and LeasingManager identify a row the host upserts. Kind defaults to GRPC, the only
// kind a host registers by name, and LeasingManager to SELF, the only leasing manager the gRPC host can
// register: a row it opened is kept alive by its stream, not by the claimer.
Name string
Kind sqlcv1.V1OperatorKind
LeasingManager sqlcv1.V1OperatorLeasingManager
}
Identity names the operator a session registers as. Exactly one of OperatorId (an existing row, as claimed by the in-process claimer) or Name with Kind and LeasingManager (a row the host upserts by (tenant, name, kind)) is used. TenantId is always required: a Host spans tenants and a Session belongs to one.
Kind says what the operator is and LeasingManager who keeps it alive; they are separate axes. A contract operator is GRPC whichever host runs it. A SELF row keeps itself alive, through the Listen stream the gRPC host holds or the leaser an in-process operator runs, and is what the wire registers; for a DISPATCHER row the dispatcher claims the row through the claimer, which builds the operator from a factory and opens it by OperatorId.
type OpenOpts ¶ added in v0.109.0
type OpenOpts struct {
// Handler receives the actions assigned to the session's worker. Required.
Handler ActionHandler
// Actions is the initial action set, linked before Open returns so the worker is never
// live with an action set the caller did not ask for. Later changes go through
// Session.AddActions and RemoveActions.
Actions []string
// SlotConfig maps slot type to max units. The engine defaults it to {"default": 100}.
SlotConfig map[string]int32
// Labels are worker labels for affinity assignment (string or int values).
Labels map[string]interface{}
// RuntimeInfo describes the process for the dashboard. The gRPC host reports its own
// process and ignores it.
RuntimeInfo *contracts.RuntimeInfo
// ResumeWorkerId names a previous worker of the same operator to resume instead of
// creating one. Not every host supports it (ErrNotSupported).
ResumeWorkerId *uuid.UUID
// WorkerName names the worker row; it defaults to the operator name. Not every host
// supports it (ErrNotSupported).
WorkerName string
}
OpenOpts describes the worker the session backs and the handler assigned actions go to.
type Operator ¶
type Operator interface {
ActionHandler
Start(ctx context.Context, s Session) error
Drain(ctx context.Context)
}
Operator is what a hosted operator implements. The host opens the session with the operator as its handler and calls Start once; the operator keeps the session for the rest of its life. Drain stops new work and waits for in-flight work, bounded by ctx; the host pauses the worker before Drain and closes the session after it.
type Registration ¶ added in v0.109.0
type Registration struct {
TenantId uuid.UUID
OperatorId uuid.UUID
WorkerId uuid.UUID
// Resumed reports whether WorkerId is the worker ResumeWorkerId named.
Resumed bool
}
Registration is the identity the engine assigned to a session.
type Session ¶ added in v0.109.0
type Session interface {
Registration() Registration
// AddActions adds ids to the worker's action set; ids already in the set are ignored. A
// host may apply the delta asynchronously; Flush waits for it.
AddActions(ctx context.Context, ids []string) error
// RemoveActions removes ids from the worker's action set; ids not in the set are ignored.
RemoveActions(ctx context.Context, ids []string) error
// Flush returns once every delta issued so far is committed by the engine and reports the
// failure, if any. In-process deltas are synchronous and Flush returns at once.
Flush(ctx context.Context) error
// PutWorkflow registers or updates a workflow for the session's tenant and returns the
// action ids its tasks derive, normalized the way the engine stores them. It does not
// change the worker's action set.
PutWorkflow(ctx context.Context, wf *v1.CreateWorkflowVersionRequest) ([]string, error)
// SendStepActionEvent reports task progress (STARTED, COMPLETED, FAILED, CANCELLED). An
// empty WorkerId is filled from the registration; another worker's id is refused.
SendStepActionEvent(ctx context.Context, ev *contracts.StepActionEvent) error
// OpenDurable opens one invocation's request and response pipe. The register_worker
// handshake is done by the host; responses that arrive before it completes are held,
// bounded, and delivered after the ack that names their entry. ctx bounds the handshake;
// the invocation itself ends on the channel's Close.
OpenDurable(ctx context.Context, taskExternalId uuid.UUID, invocation int32) (DurableChannel, error)
// Pause stops the scheduler assigning to the worker and returns once the pause is
// committed, so a caller that drains afterwards knows no further work will arrive.
Pause(ctx context.Context) error
// Close ends the session: the worker is paused if it is not already, the session is
// released so nothing further is delivered, and the worker is deactivated, fenced on the
// session id. A host's teardown order is Pause, the operator's Drain, then Close.
Close(ctx context.Context) error
// Done is closed once the session no longer serves its worker: after Close, or when the
// host gave up on it. A host recovers from transient failures on its own (the gRPC host
// reconnects, re-reads the tenant's token and restores the action set); it gives up on a
// failure no retry can fix, such as a token source that no longer has a token for the
// tenant. The caller that owns the session then closes it and opens another.
Done() <-chan struct{}
// Err reports why Done closed: nil after Close, the terminal failure when the host gave
// up. It is nil while the session is live.
Err() error
}
Session is one worker row's worth of traffic: its action set, its events, its durable invocations and its liveness, which the host maintains until Close. Methods are safe to call concurrently until Close.
type SharedOperator ¶
type SharedOperator[T any] struct { // contains filtered or unexported fields }
SharedOperator is the state an engine-internal operator shares: its config, the session the host opened for it, the engine-internal writer, and the bookkeeping for in-flight work. The session arrives with Start; the event senders, the action set and the durable channels go through it, so the operator's lifecycle is the host's.
func NewSharedOperator ¶
func NewSharedOperator[T any](operator *sqlcv1.V1Operator, l *zerolog.Logger, taskEventWriter TaskEventWriter, t T) (*SharedOperator[T], error)
NewSharedOperator constructs the shared operator state from the operator row.
func (*SharedOperator[T]) CancelDAGChildren ¶
func (*SharedOperator[T]) CancelTask ¶
func (s *SharedOperator[T]) CancelTask(taskRunExternalId string) bool
func (*SharedOperator[T]) Config ¶
func (s *SharedOperator[T]) Config() T
func (*SharedOperator[T]) Drain ¶
func (s *SharedOperator[T]) Drain(ctx context.Context)
Drain stops accepting new tracked tasks and waits for the in-flight ones, or for ctx. The host pauses the worker before calling it, so nothing new arrives while it waits.
func (*SharedOperator[T]) Logger ¶
func (s *SharedOperator[T]) Logger() *zerolog.Logger
func (*SharedOperator[T]) OpenDurable ¶ added in v0.109.0
func (s *SharedOperator[T]) OpenDurable(ctx context.Context, taskExternalId uuid.UUID, invocation int32) (DurableChannel, error)
OpenDurable opens one durable invocation's pipe through the session: the host does the register-worker handshake and holds what the engine sends before its ack, so the operator only ever reads invocation traffic, an entry never ahead of the ack that names it.
func (*SharedOperator[T]) OperatorId ¶ added in v0.109.0
func (s *SharedOperator[T]) OperatorId() uuid.UUID
func (*SharedOperator[T]) RecordTask ¶
func (s *SharedOperator[T]) RecordTask() func()
RecordTask registers an in-flight task and returns a release function that the caller must invoke (typically via defer) when the task finishes. Drain blocks until every recorded task has been released.
If the operator is already draining, the returned release is a no-op and the task is not tracked: callers should avoid starting new work once Drain has begun, but in-flight work recorded before it is always awaited.
func (*SharedOperator[T]) RegisterCancellableContext ¶
func (*SharedOperator[T]) SendCancelled ¶
func (s *SharedOperator[T]) SendCancelled(action *contracts.AssignedAction) error
SendCancelled reports a cancelled task
func (*SharedOperator[T]) SendCancelledWithMessage ¶
func (s *SharedOperator[T]) SendCancelledWithMessage(action *contracts.AssignedAction, msg string) error
SendCancelledWithMessage reports a cancelled task with a custom cancellation reason through the engine-internal writer rather than the generic step action event path, so the reason reaches the run's events verbatim. There is no such RPC: an operator hosted over gRPC reports a plain CANCELLED event instead.
func (*SharedOperator[T]) SendCompleted ¶
func (s *SharedOperator[T]) SendCompleted(action *contracts.AssignedAction, output []byte) error
SendCompleted reports a successful result. output should be the task's JSON output.
func (*SharedOperator[T]) SendFailed ¶
func (s *SharedOperator[T]) SendFailed(action *contracts.AssignedAction, errMsg string, shouldNotRetry bool) error
SendFailed reports a failure with the given error message. shouldNotRetry, when true, prevents the task from being retried.
func (*SharedOperator[T]) SendStarted ¶
func (s *SharedOperator[T]) SendStarted(action *contracts.AssignedAction) error
SendStarted reports that the operator has started processing the assigned action.
func (*SharedOperator[T]) SendStartedAt ¶ added in v0.106.2
func (s *SharedOperator[T]) SendStartedAt(action *contracts.AssignedAction, at time.Time) error
func (*SharedOperator[T]) Session ¶ added in v0.109.0
func (s *SharedOperator[T]) Session() Session
Session is the session the host opened, or nil before Start.
func (*SharedOperator[T]) Start ¶ added in v0.109.0
func (s *SharedOperator[T]) Start(_ context.Context, session Session) error
Start records the session the host opened. Operators that embed SharedOperator call it from their own Start.
func (*SharedOperator[T]) TenantId ¶
func (s *SharedOperator[T]) TenantId() uuid.UUID
func (*SharedOperator[T]) TriggerDAGStep ¶
func (s *SharedOperator[T]) TriggerDAGStep(ctx context.Context, req *DAGStepTriggerRequest) (*DAGStepTriggerResult, error)
func (*SharedOperator[T]) UpdateWorkerActions ¶
func (s *SharedOperator[T]) UpdateWorkerActions(ctx context.Context, actions []string) (bool, error)
UpdateWorkerActions makes actions the worker's action set: the difference from the set last advertised goes to the session as adds and removes, followed by a flush. It reports whether anything changed. A delta that fails leaves the advertised set as it was, so the next call repeats it; ids the engine already has are ignored by it.
The difference is what makes the call cheap to repeat: the engine would accept the whole set every time (adding an action the worker has is a no-op), but every send is a write, and the removes cannot be derived without the previous set. It is not safe for concurrent use: the operator calls it from one goroutine at a time (its Start, then the poller Start launches once that first call has returned), which is why lastActions needs no lock.
func (*SharedOperator[T]) WorkerId ¶
func (s *SharedOperator[T]) WorkerId() uuid.UUID
WorkerId is the worker the session backs, or uuid.Nil before Start.
type TaskEventWriter ¶
type TaskEventWriter interface {
// CancelTaskWithReason reports a cancelled task with a custom cancellation reason. It is
// the engine-internal writer behind SendCancelledWithMessage, distinct from the CANCELLED
// step action event every host offers through Session.SendStepActionEvent, and it is not
// on the gRPC surface: an out-of-process operator has no equivalent.
CancelTaskWithReason(ctx context.Context, tenantId uuid.UUID, request *contracts.StepActionEvent) (*contracts.ActionEventResponse, error)
TriggerDAGStep(ctx context.Context, tenantId uuid.UUID, req *DAGStepTriggerRequest) (*DAGStepTriggerResult, error)
// CancelDAGChildren cancels already-triggered children when the orchestrator is cancelled.
CancelDAGChildren(ctx context.Context, tenantId uuid.UUID, taskExternalIds []uuid.UUID) error
}
TaskEventWriter is the engine-internal surface for engine-internal operators (the DAG operator): calls that only exist inside the engine and are never available over gRPC. The dispatcher implements it. Everything an operator needs that both hosts offer (events, durable invocations, the action set) is on Session instead. Every call names its tenant explicitly: nothing here reads the tenant the gRPC auth middleware puts on a request context.
Directories
¶
| Path | Synopsis |
|---|---|
|
e2e
Package e2e holds tests that run the DAG operator against an in-process engine.
|
Package e2e holds tests that run the DAG operator against an in-process engine. |
|
Package hostgrpc is the out-of-process operator host: it implements pkg/operator.Host over pkg/client/operatorclient, speaking OperatorService to the engine with a per-tenant API token from a TokenSource.
|
Package hostgrpc is the out-of-process operator host: it implements pkg/operator.Host over pkg/client/operatorclient, speaking OperatorService to the engine with a per-tenant API token from a TokenSource. |
|
Package operatortest holds a sample contract operator for tests.
|
Package operatortest holds a sample contract operator for tests. |
|
Package safeclient provides an SSRF-hardened HTTP client for delivering outbound requests to user-supplied endpoints.
|
Package safeclient provides an SSRF-hardened HTTP client for delivering outbound requests to user-supplied endpoints. |