Documentation
¶
Index ¶
- Variables
- func CheckOutputSources(dag *ir.DAG, outputs map[string]map[string]string, ...) error
- func CheckRunStepSources(dag *ir.DAG, steps []string, fromRuns RunStepSources) error
- func MarkSeedFailed(ctx context.Context, repo *persis.DAGRunRepository, status *ir.DAGRunStatus, ...)
- func SeedRun(ctx context.Context, req SeedRequest) (dagrun.Attempt, *ir.DAGRunStatus, error)
- func SelectedStepNodes(dag *ir.DAG, steps []string, source *ir.DAGRunStatus, fromRuns RunStepSources, ...) ([]runtime.NodeData, []ir.ReusedInput, error)
- func SelectionSkipsHandlers(dag *ir.DAG, steps []string) (bool, error)
- func WebhookHeaders(forward []string, header func(name string) []string) (string, error)
- func WebhookRuntimeParams(payload, headers string) string
- type LocalAttemptBuilder
- type LocalPreparation
- type LocalProcRepository
- type LocalRequest
- type OutputSource
- type QueueRequest
- type QueuedRun
- type RunStepSourceError
- type RunStepSources
- type SeedRequest
Constants ¶
This section is empty.
Variables ¶
var ( ErrLocalExecutionAlreadyExists = errors.New("local execution already exists") ErrProcAcquisitionFailed = errors.New("failed to acquire process handle") )
Functions ¶
func CheckOutputSources ¶
func CheckOutputSources(dag *ir.DAG, outputs map[string]map[string]string, sources map[string]OutputSource) error
CheckOutputSources makes sure that every entry of sources is for a step in outputs. A step in outputs may have no entry. Steps are given by name or ID. The caller checks the run IDs.
func CheckRunStepSources ¶
func CheckRunStepSources(dag *ir.DAG, steps []string, fromRuns RunStepSources) error
CheckRunStepSources makes sure that every step of fromRuns exists, is not selected to run, and can be reused from its run: the run is finished and its node for the step succeeded, or carries outputs of an earlier run. The error names the step and the run.
func MarkSeedFailed ¶ added in v2.17.2
func MarkSeedFailed(ctx context.Context, repo *persis.DAGRunRepository, status *ir.DAGRunStatus, cause error)
MarkSeedFailed records cause on a seeded run that never left the queued state, so a launch failure does not leave the run queued forever.
func SeedRun ¶ added in v2.17.2
func SeedRun(ctx context.Context, req SeedRequest) (dagrun.Attempt, *ir.DAGRunStatus, error)
SeedRun creates the queued attempt described by req and returns it with the status it recorded. The attempt is removed again if any later step fails.
func SelectedStepNodes ¶ added in v2.17.2
func SelectedStepNodes(dag *ir.DAG, steps []string, source *ir.DAGRunStatus, fromRuns RunStepSources, outputs map[string]map[string]string, sources map[string]OutputSource) ([]runtime.NodeData, []ir.ReusedInput, error)
SelectedStepNodes returns seed nodes for a run of dag that executes only steps, each given by step name or ID, and the provenance of the outputs that the skipped steps carry. Every other step is recorded as skipped; when source is set, it carries the outputs source recorded for it. fromRuns gives a step another run to carry from; it takes precedence over source for that step, and the work directory still comes from source only. outputs sets outputs of skipped steps, keyed by step name or ID and then by output name, and takes precedence over outputs carried from a run. sources names the run each step of outputs was captured from; a step of outputs with no entry has no recorded provenance.
func SelectionSkipsHandlers ¶
SelectionSkipsHandlers reports whether a run of dag that executes only steps, each given by step name or ID, must not call the end-of-run handlers of the DAG. It does when no selected step is a graph end step: a step that no other step depends on, by depends, by an inferred dependency, or as the target of a router (which the DAG definition records as a dependency).
func WebhookHeaders ¶
WebhookHeaders encodes the request headers a webhook's run receives: those named in forward, under lowercase names, each with the values header gives for it. Names the run must never see, such as Authorization, are left out.
func WebhookRuntimeParams ¶
WebhookRuntimeParams are the params a webhook's run receives: the request body as WEBHOOK_PAYLOAD and the forwarded headers, encoded by WebhookHeaders, as WEBHOOK_HEADERS.
Types ¶
type LocalAttemptBuilder ¶
LocalAttemptBuilder creates or resolves the attempt that a local execution will own.
type LocalPreparation ¶
type LocalPreparation struct {
Attempt dagrun.Attempt
Proc proc.ProcHandle
}
LocalPreparation is the successfully prepared local execution ownership.
func PrepareLocalExecution ¶
func PrepareLocalExecution(ctx context.Context, req LocalRequest) (*LocalPreparation, error)
PrepareLocalExecution creates or resolves the execution attempt, acquires the local process heartbeat, and records a failed status if heartbeat acquisition fails after an attempt was prepared.
type LocalProcRepository ¶ added in v2.14.0
type LocalProcRepository interface {
WithLock(ctx context.Context, groupName string, fn func() error) error
Acquire(ctx context.Context, groupName string, meta proc.ProcMeta) (proc.ProcHandle, error)
}
LocalProcRepository is the process-repository surface needed to claim local execution ownership.
type LocalRequest ¶
type LocalRequest struct {
ProcRepository LocalProcRepository
DAG *ir.DAG
DAGRunID string
DefinitionID string
Root ir.DAGRunRef
Parent ir.DAGRunRef
TriggerType ir.TriggerType
TriggerActor string
ScheduleTime string
ProfileName string
LogBaseDir string
ArtifactBaseDir string
BuildAttempt LocalAttemptBuilder
}
LocalRequest describes local DAG-run intake before execution starts.
type OutputSource ¶
type OutputSource struct {
RunID string
// CapturedAt is an RFC 3339 timestamp, or empty when unknown.
CapturedAt string
}
OutputSource names the run that supplied outputs of a skipped step were captured from. The run may no longer exist.
type QueueRequest ¶
type QueueRequest struct {
DAGRunRepository *persis.DAGRunRepository
QueueStore queue.QueueStore
DAG *ir.DAG
DAGRunID string
QueueName string
LogBaseDir string
ArtifactBaseDir string
Root ir.DAGRunRef
Parent ir.DAGRunRef
TriggerType ir.TriggerType
TriggerActor string
ParallelItem string
ScheduleTime string
ProfileName string
DefinitionID string
NoReuse bool
AttemptOptions persis.DAGRunCreateAttemptOptions
// ProceedOnStatusCloseErr preserves legacy CLI enqueue behavior: publish
// the queue item after best-effort close so readers can see the queued status.
ProceedOnStatusCloseErr bool
Now func() time.Time
}
QueueRequest describes a DAG-run intake operation that persists a queued attempt before publishing the queue item.
type QueuedRun ¶
type QueuedRun struct {
DAGRun ir.DAGRunRef
Attempt dagrun.Attempt
Status ir.DAGRunStatus
QueueName string
LogFile string
ArtifactDir string
// StatusCloseErr is set only when ProceedOnStatusCloseErr allowed intake
// to continue after the status attempt close failed.
StatusCloseErr error
}
QueuedRun is the result of successful DAG-run queue intake.
func EnqueueRun ¶
func EnqueueRun(ctx context.Context, req QueueRequest) (*QueuedRun, error)
EnqueueRun creates a queued DAG-run attempt, writes the queued status, closes the attempt, then publishes the queue item. If any post-create step fails, the created attempt is rolled back.
type RunStepSourceError ¶
type RunStepSourceError struct {
Message string
}
RunStepSourceError reports a run named for a step that cannot be read. Its message names the step and the run.
func (*RunStepSourceError) Error ¶
func (e *RunStepSourceError) Error() string
type RunStepSources ¶
type RunStepSources map[string]*ir.DAGRunStatus
RunStepSources names, by step, the finished run whose recorded outputs the step carries, in place of the main source run. Steps are given by name or ID.
func LoadRunStepSources ¶
func LoadRunStepSources(ctx context.Context, repo *persis.DAGRunRepository, dagName string, refs map[string]string, check func(*ir.DAGRunStatus) error) (RunStepSources, error)
LoadRunStepSources reads the run that each step of refs carries its outputs from. refs maps a step name or ID to a run ID. A run that does not exist for the DAG (it was removed, or it belongs to another DAG) gives a *RunStepSourceError that names the step and the run. A run ID that refs repeats is read once. check, when set, accepts or rejects each run read; its error is returned as it is.
type SeedRequest ¶ added in v2.17.2
type SeedRequest struct {
DAGRunRepository *persis.DAGRunRepository
DAG *ir.DAG
DAGRunID string
Nodes []runtime.NodeData
// ReusedInputs records where the carried outputs of skipped nodes come from.
ReusedInputs []ir.ReusedInput
// SkipHandlers records that the run does not call the end-of-run handlers
// of the DAG. See SelectionSkipsHandlers.
SkipHandlers bool
// Source is the run whose recorded outputs the skipped nodes carry. When
// set, its work directory is copied into the new run and the carried
// output variables are rewritten to point at the copy.
Source *ir.DAGRunStatus
Params string
TriggerType ir.TriggerType
TriggerActor string
ProfileName string
DefinitionID string
NoReuse bool
// Ephemeral stores the run apart from the history and marks it so that it
// leaves no trace.
Ephemeral bool
LogBaseDir string
ArtifactBaseDir string
}
SeedRequest describes a new DAG-run whose first attempt is persisted as queued with fixed node states, to be executed through the retry path.