workflow

package
v0.42.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package workflow (application) translates the YAML file into the domain.

The separation exists because the rule is explicit: once published, the database is the source of truth, not the file. The YAML is publishing INPUT -- it comes in here, becomes domain, and the domain is what persists. Changing the file format must not touch the graph's invariants.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Parse

func Parse(path string, conteudo []byte) (dominio.Workflow, error)

Parse reads the YAML and returns the workflow already validated.

`caminho` serves two purposes: deriving the slug when the file carries no `name`, and naming the file in error messages -- a graph error without the file's name is useless when there are dozens of them.

func Resolve

func Resolve(all []dominio.Workflow) ([]dominio.Workflow, error)

Resolve expands every `uses:` across a set of workflows.

It works on the WHOLE set and not on one file, because a `uses:` names a sibling. That is also why the child must be in the same publish, and not merely published already: expansion that reads the database would make `brevis validate` -- which touches no database, on purpose -- answer a different question from `brevis publish`, and the file that passed CI would be the file that failed the deploy.

The child's steps take the parent step's place, prefixed with its id, and the parent node disappears. What arrives at the runner is one flat graph, which is the whole point: a step that TRIGGERED a child run and waited would hold a slot from the same per-process semaphore the child needs, and hang under load.

func UnknownFields

func UnknownFields(path string, content []byte) []string

UnknownFields lists what the decoder threw away, so a typo becomes a line somebody reads instead of a field that quietly does nothing.

IT WARNS AND DOES NOT REFUSE, which is the whole design. This decoder has never been strict, so any workflow anybody has may carry a stray key today; turning that into an error would fail files that have worked for a year in order to catch a typo in one of them. The engine behaves exactly as it did.

It is the near half of the gap `host:` exposed. An engine that does not know a field DROPS it -- so a step meant for the machine with the licence on it runs in the engine's own container instead, with nothing said. Nothing here fixes that for an OLD engine, and nothing can; it is why the silence was worth breaking on this one.

Separate from Parse rather than inside it: Parse's signature is what three call sites and the whole suite agree on, and a warning is not an error.

func YAML

func YAML(w dominio.Workflow) ([]byte, error)

YAML is a published workflow as the file that publishes it, indented as the examples are.

Types

type ConcurrencySpec

type ConcurrencySpec struct {
	Limit int    `yaml:"limit,omitempty"`
	Group string `yaml:"group,omitempty"`
}

ConcurrencySpec is `concurrency:`, in either shape:

concurrency: 1
concurrency:
  group: jaffle_models
  limit: 1

The bare number limits runs of THIS workflow, as it always has. The mapping names a group: every workflow that names it counts against one limit, which is how a daily and a */15 that build the same models stop overlapping (#32).

func (ConcurrencySpec) MarshalYAML

func (c ConcurrencySpec) MarshalYAML() (any, error)

MarshalYAML writes the short form when it says everything: `concurrency: 3` for a limit alone, the mapping when there is a group.

func (*ConcurrencySpec) UnmarshalYAML

func (c *ConcurrencySpec) UnmarshalYAML(value *yaml.Node) error

UnmarshalYAML accepts a scalar or a mapping, and refuses a key it does not know.

Refuses, where the rest of the file only warns: KnownFields does not reach inside a custom unmarshaler (TestACustomUnmarshalerStillHidesItsFields), and a `grop:` dropped here would leave a workflow with no group, overlapping its partner in silence -- the bug this field exists to fix, back with no sign.

type Dependency

type Dependency struct {
	Step  string `yaml:"step,omitempty"`
	Label string `yaml:"label,omitempty"`
}

Dependency is one entry of `depends_on`. It accepts both shapes:

depends_on: [extract]
depends_on:
  - {step: determine_load_type, label: changed existing data}

The bare form stays the normal one -- most dependencies have nothing to say and a label on every arrow is noise. The object exists for the branch, where two arrows leaving the same step with no labels is a diagram that requires opening the source to read.

func (Dependency) MarshalYAML

func (d Dependency) MarshalYAML() (any, error)

MarshalYAML writes a dependency with no label as its step's name.

func (*Dependency) UnmarshalYAML

func (d *Dependency) UnmarshalYAML(value *yaml.Node) error

UnmarshalYAML accepts a scalar or a mapping.

The scalar branch is what keeps every workflow ever written working: a list of strings decodes exactly as it did, and nothing in a published file has to change.

type OnErrorSpec

type OnErrorSpec struct {
	Type string `yaml:"type,omitempty"`
	When string `yaml:"when,omitempty"`
}

OnErrorSpec is a step's alert declaration as written in the file.

There is no `webhook`, `url` or `channel` field, and there will not be. The destination is a credential; the installation owns it. This says WHETHER and HOW.

type ParamSpec

type ParamSpec struct {
	Name        string   `yaml:"name,omitempty"`
	Type        string   `yaml:"type,omitempty"`
	Default     string   `yaml:"default,omitempty"`
	Description string   `yaml:"description,omitempty"`
	Enum        []string `yaml:"enum,omitempty"`
	Pattern     string   `yaml:"pattern,omitempty"`
	Title       string   `yaml:"title,omitempty"`
	Group       string   `yaml:"group,omitempty"`
	Warn        string   `yaml:"warn,omitempty"`
	Uses        string   `yaml:"uses,omitempty"`
}

ParamSpec is a run parameter as written in the file.

params:
  - name: load_full
    type: boolean
    default: "false"
  - name: start_date
    type: string
    pattern: '^\d{4}-\d{2}-\d{2}$'

`title`, `group`, `warn` and `uses` are how the form presents it, and do not change what a value may be.

type ResourceSpec

type ResourceSpec struct {
	CPU    string `yaml:"cpu,omitempty"`
	Memory string `yaml:"memory,omitempty"`
	Limits struct {
		CPU    string `yaml:"cpu,omitempty"`
		Memory string `yaml:"memory,omitempty"`
	} `yaml:"limits,omitempty"`
}

ResourceSpec is the CPU and memory request in Kubernetes' format.

`limits` kept apart from `requests` because the difference between them is the difference between "how much I reserve" and "when I get killed": a dbt that blows the limit dies with OOMKilled, one that merely exceeds the request keeps running.

type Spec

type Spec struct {
	Name        string       `yaml:"name,omitempty"`
	Description string       `yaml:"description,omitempty"`
	Schedule    string       `yaml:"schedule,omitempty"`
	Type        string       `yaml:"type,omitempty"`
	Tags        []string     `yaml:"tags,omitempty"`
	Image       string       `yaml:"image,omitempty"`
	Resources   ResourceSpec `yaml:"resources,omitempty"`
	Params      []ParamSpec  `yaml:"params,omitempty"`

	// Env and Secrets apply to every step; a step overrides them name by name.
	Env     map[string]string `yaml:"env,omitempty"`
	Secrets map[string]string `yaml:"secrets,omitempty"`
	// Concurrency is Kestra's `concurrency.limit`, under the name most
	// orchestrators use. See ConcurrencySpec for the two shapes.
	Concurrency ConcurrencySpec `yaml:"concurrency,omitempty"`

	// Trigger is why this runs other than the clock. See TriggerSpec.
	Trigger TriggerSpec `yaml:"trigger,omitempty"`

	Steps []StepSpec `yaml:"steps,omitempty"`
}

Spec mirrors the YAML, and nothing more. Loose fields here, invariants in the domain.

Every key is `omitempty`: reading ignores it, and writing a Spec back (SpecOf, for the console's Definition tab) leaves out what was not said, as a file somebody wrote would.

func SpecOf

func SpecOf(w dominio.Workflow) Spec

SpecOf is a published workflow written back as the file: the inverse of Parse, for the console's Definition tab. The database keeps the domain and not the file, so this is not the file the author wrote -- no comments, keys in the format's order -- but publishing it publishes the same workflow, which TestEveryExampleRoundTripsThroughItsYAML holds.

type StepSpec

type StepSpec struct {
	ID        string         `yaml:"id,omitempty"`
	Run       string         `yaml:"run,omitempty"`
	Action    string         `yaml:"action,omitempty"`
	With      map[string]any `yaml:"with,omitempty"`
	DependsOn []Dependency   `yaml:"depends_on,omitempty"`

	// Image e Resources sobrescrevem os do workflow. Ausentes = herda.
	Image     string       `yaml:"image,omitempty"`
	Host      string       `yaml:"host,omitempty"`
	Resources ResourceSpec `yaml:"resources,omitempty"`

	// Env are variables with a literal value in the file.
	//
	//	env:
	//	  BREVIS_LOG_LEVEL: info
	Env map[string]string `yaml:"env,omitempty"`

	// PersistContext lists the keys this step may read and write through
	// sdk/persist. Declaring it is what makes the engine point the step at the
	// installation's store; a step that declares nothing gets no access at all.
	//
	// It is a LIST and not a boolean because the alternative is a flat global
	// namespace where any step touches any key: two workflows that both know a
	// good name for `station_codes` would overwrite each other and both keep
	// running. Naming them is also what lets the reader of the file see that
	// this step depends on whatever wrote them.
	PersistContext []string `yaml:"persist_context,omitempty"`

	// Secrets are variables whose value is NOT in the file: the key is the
	// variable's name, the value is where to find it.
	//
	//	secrets:
	//	  GABRIEL_SESSION_COOKIE: gabriel-session/cookie
	Secrets map[string]string `yaml:"secrets,omitempty"`

	// Runtime and Tools say what this step runs in, when the engine cannot
	// work it out from `run:` and `image:`.
	//
	//	runtime: python
	//	tools: [dbt]
	//
	// Both optional, and the engine INFERS both when they are absent -- which
	// is the normal case. Declaring one is for when the inference is wrong or
	// blind: a wrapper script, a bare binary path, an image whose name says
	// nothing.
	//
	// An id outside the vocabulary is refused at publish, naming what is
	// valid. The alternative is a chip that renders blank on a screen three
	// days later, with nothing to trace it to.
	Runtime string   `yaml:"runtime,omitempty"`
	Tools   []string `yaml:"tools,omitempty"`

	// Shell: a pointer, to tell "did not declare" from "declared false". Without
	// the pointer, every step without the key would become `shell: false` and
	// images that do have a shell -- most of them -- would start receiving an
	// argv, breaking any command with a pipe or a variable.
	Shell *bool `yaml:"shell,omitempty"`

	// When is the trigger rule. Empty is `all_success`, which is what every
	// workflow written before this existed means.
	When string `yaml:"when,omitempty"`

	// Marker is a step that does nothing and exists to be a point in the
	// graph -- a `start`, an `end`, a join. See dominio.Node.Marker.
	Marker bool `yaml:"marker,omitempty"`

	// UnlessEmpty names a context key that decides whether this step runs.
	// A key, not an expression. See dominio.Node.UnlessEmpty.
	UnlessEmpty string `yaml:"unless_empty,omitempty"`

	// ForEach names a context key holding a list. The step runs once per
	// element. See dominio.Node.ForEach.
	ForEach string `yaml:"for_each,omitempty"`

	// Group draws this step inside a named box on the graph. Visual only.
	// See dominio.Node.Group.
	Group string `yaml:"group,omitempty"`

	// Uses names another workflow whose steps take this one's place, expanded
	// at publish. See dominio.Node.Uses.
	Uses string `yaml:"uses,omitempty"`

	// OnError announces this step's failures.
	//
	//	on_error:
	//	  type: SLACK
	//	  when: attempt     # optional; the default is give_up
	//
	// A pointer for the same reason Shell is: absent has to be different from
	// declared-and-empty, and a zero OnError would look like a step asking to
	// be announced to nowhere.
	OnError *OnErrorSpec `yaml:"on_error,omitempty"`
}

StepSpec is a step as written in the file.

type TriggerSpec

type TriggerSpec struct {
	OnLanded []string      `yaml:"on_landed,omitempty"`
	Debounce time.Duration `yaml:"debounce,omitempty"`
}

TriggerSpec is `trigger:` in the YAML.

trigger:
  on_landed: [bigquery://acme-prod/bronze/orders]
  debounce: 5m

A time.Duration straight through, as the gateway's `idle_timeout` and `max_conn_age` are. yaml.v3 reads `5m` as five minutes and REFUSES a bare `5` -- measured, because the opposite would have been quiet and expensive: a window read as five nanoseconds collapses nothing and looks exactly like the feature not working. TestADebounceNeedsItsUnit pins it.

func (TriggerSpec) MarshalYAML

func (t TriggerSpec) MarshalYAML() (any, error)

MarshalYAML writes the debounce in its unit: yaml.v3 would write the nanoseconds, which the reader refuses (TestADebounceNeedsItsUnit).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL