intake

package
v2.25.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: GPL-3.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrLocalExecutionAlreadyExists = errors.New("local execution already exists")
	ErrProcAcquisitionFailed       = errors.New("failed to acquire process handle")
)

Functions

func CheckOutputSources

func CheckOutputSources(dag *ir.DAG, outputs map[string]map[string]string, sources map[string]OutputSource) error

CheckOutputSources makes sure that every entry of sources is for a step in outputs. A step in outputs may have no entry. Steps are given by name or ID. The caller checks the run IDs.

func CheckRunStepSources

func CheckRunStepSources(dag *ir.DAG, steps []string, fromRuns RunStepSources) error

CheckRunStepSources makes sure that every step of fromRuns exists, is not selected to run, and can be reused from its run: the run is finished and its node for the step succeeded, or carries outputs of an earlier run. The error names the step and the run.

func MarkSeedFailed added in v2.17.2

func MarkSeedFailed(ctx context.Context, repo *persis.DAGRunRepository, status *ir.DAGRunStatus, cause error)

MarkSeedFailed records cause on a seeded run that never left the queued state, so a launch failure does not leave the run queued forever.

func SeedRun added in v2.17.2

SeedRun creates the queued attempt described by req and returns it with the status it recorded. The attempt is removed again if any later step fails.

func SelectedStepNodes added in v2.17.2

func SelectedStepNodes(dag *ir.DAG, steps []string, source *ir.DAGRunStatus, fromRuns RunStepSources, outputs map[string]map[string]string, sources map[string]OutputSource) ([]runtime.NodeData, []ir.ReusedInput, error)

SelectedStepNodes returns seed nodes for a run of dag that executes only steps, each given by step name or ID, and the provenance of the outputs that the skipped steps carry. Every other step is recorded as skipped; when source is set, it carries the outputs source recorded for it. fromRuns gives a step another run to carry from; it takes precedence over source for that step, and the work directory still comes from source only. outputs sets outputs of skipped steps, keyed by step name or ID and then by output name, and takes precedence over outputs carried from a run. sources names the run each step of outputs was captured from; a step of outputs with no entry has no recorded provenance.

func SelectionSkipsHandlers

func SelectionSkipsHandlers(dag *ir.DAG, steps []string) (bool, error)

SelectionSkipsHandlers reports whether a run of dag that executes only steps, each given by step name or ID, must not call the end-of-run handlers of the DAG. It does when no selected step is a graph end step: a step that no other step depends on, by depends, by an inferred dependency, or as the target of a router (which the DAG definition records as a dependency).

func WebhookHeaders

func WebhookHeaders(forward []string, header func(name string) []string) (string, error)

WebhookHeaders encodes the request headers a webhook's run receives: those named in forward, under lowercase names, each with the values header gives for it. Names the run must never see, such as Authorization, are left out.

func WebhookRuntimeParams

func WebhookRuntimeParams(payload, headers string) string

WebhookRuntimeParams are the params a webhook's run receives: the request body as WEBHOOK_PAYLOAD and the forwarded headers, encoded by WebhookHeaders, as WEBHOOK_HEADERS.

Types

type LocalAttemptBuilder

type LocalAttemptBuilder func(context.Context) (dagrun.Attempt, error)

LocalAttemptBuilder creates or resolves the attempt that a local execution will own.

type LocalPreparation

type LocalPreparation struct {
	Attempt dagrun.Attempt
	Proc    proc.ProcHandle
}

LocalPreparation is the successfully prepared local execution ownership.

func PrepareLocalExecution

func PrepareLocalExecution(ctx context.Context, req LocalRequest) (*LocalPreparation, error)

PrepareLocalExecution creates or resolves the execution attempt, acquires the local process heartbeat, and records a failed status if heartbeat acquisition fails after an attempt was prepared.

type LocalProcRepository added in v2.14.0

type LocalProcRepository interface {
	WithLock(ctx context.Context, groupName string, fn func() error) error
	Acquire(ctx context.Context, groupName string, meta proc.ProcMeta) (proc.ProcHandle, error)
}

LocalProcRepository is the process-repository surface needed to claim local execution ownership.

type LocalRequest

type LocalRequest struct {
	ProcRepository LocalProcRepository
	DAG            *ir.DAG
	DAGRunID       string
	DefinitionID   string

	Root         ir.DAGRunRef
	Parent       ir.DAGRunRef
	TriggerType  ir.TriggerType
	TriggerActor string

	ScheduleTime string
	ProfileName  string

	LogBaseDir      string
	ArtifactBaseDir string

	BuildAttempt LocalAttemptBuilder
}

LocalRequest describes local DAG-run intake before execution starts.

type OutputSource

type OutputSource struct {
	RunID string
	// CapturedAt is an RFC 3339 timestamp, or empty when unknown.
	CapturedAt string
}

OutputSource names the run that supplied outputs of a skipped step were captured from. The run may no longer exist.

type QueueRequest

type QueueRequest struct {
	DAGRunRepository *persis.DAGRunRepository
	QueueStore       queue.QueueStore
	DAG              *ir.DAG
	DAGRunID         string

	QueueName string

	LogBaseDir      string
	ArtifactBaseDir string

	Root         ir.DAGRunRef
	Parent       ir.DAGRunRef
	TriggerType  ir.TriggerType
	TriggerActor string
	ParallelItem string
	ScheduleTime string
	ProfileName  string
	DefinitionID string
	NoReuse      bool

	AttemptOptions persis.DAGRunCreateAttemptOptions

	// ProceedOnStatusCloseErr preserves legacy CLI enqueue behavior: publish
	// the queue item after best-effort close so readers can see the queued status.
	ProceedOnStatusCloseErr bool

	Now func() time.Time
}

QueueRequest describes a DAG-run intake operation that persists a queued attempt before publishing the queue item.

type QueuedRun

type QueuedRun struct {
	DAGRun      ir.DAGRunRef
	Attempt     dagrun.Attempt
	Status      ir.DAGRunStatus
	QueueName   string
	LogFile     string
	ArtifactDir string
	// StatusCloseErr is set only when ProceedOnStatusCloseErr allowed intake
	// to continue after the status attempt close failed.
	StatusCloseErr error
}

QueuedRun is the result of successful DAG-run queue intake.

func EnqueueRun

func EnqueueRun(ctx context.Context, req QueueRequest) (*QueuedRun, error)

EnqueueRun creates a queued DAG-run attempt, writes the queued status, closes the attempt, then publishes the queue item. If any post-create step fails, the created attempt is rolled back.

type RunStepSourceError

type RunStepSourceError struct {
	Message string
}

RunStepSourceError reports a run named for a step that cannot be read. Its message names the step and the run.

func (*RunStepSourceError) Error

func (e *RunStepSourceError) Error() string

type RunStepSources

type RunStepSources map[string]*ir.DAGRunStatus

RunStepSources names, by step, the finished run whose recorded outputs the step carries, in place of the main source run. Steps are given by name or ID.

func LoadRunStepSources

func LoadRunStepSources(ctx context.Context, repo *persis.DAGRunRepository, dagName string, refs map[string]string, check func(*ir.DAGRunStatus) error) (RunStepSources, error)

LoadRunStepSources reads the run that each step of refs carries its outputs from. refs maps a step name or ID to a run ID. A run that does not exist for the DAG (it was removed, or it belongs to another DAG) gives a *RunStepSourceError that names the step and the run. A run ID that refs repeats is read once. check, when set, accepts or rejects each run read; its error is returned as it is.

type SeedRequest added in v2.17.2

type SeedRequest struct {
	DAGRunRepository *persis.DAGRunRepository
	DAG              *ir.DAG
	DAGRunID         string
	Nodes            []runtime.NodeData
	// ReusedInputs records where the carried outputs of skipped nodes come from.
	ReusedInputs []ir.ReusedInput

	// SkipHandlers records that the run does not call the end-of-run handlers
	// of the DAG. See SelectionSkipsHandlers.
	SkipHandlers bool

	// Source is the run whose recorded outputs the skipped nodes carry. When
	// set, its work directory is copied into the new run and the carried
	// output variables are rewritten to point at the copy.
	Source *ir.DAGRunStatus

	Params       string
	TriggerType  ir.TriggerType
	TriggerActor string
	ProfileName  string
	DefinitionID string
	NoReuse      bool
	// Ephemeral stores the run apart from the history and marks it so that it
	// leaves no trace.
	Ephemeral bool

	LogBaseDir      string
	ArtifactBaseDir string
}

SeedRequest describes a new DAG-run whose first attempt is persisted as queued with fixed node states, to be executed through the retry path.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL