Documentation
¶
Overview ¶
Package workflow (application) translates the YAML file into the domain.
The separation exists because the rule is explicit: once published, the database is the source of truth, not the file. The YAML is publishing INPUT -- it comes in here, becomes domain, and the domain is what persists. Changing the file format must not touch the graph's invariants.
Index ¶
- func Parse(path string, conteudo []byte) (dominio.Workflow, error)
- func Resolve(all []dominio.Workflow) ([]dominio.Workflow, error)
- func UnknownFields(path string, content []byte) []string
- func YAML(w dominio.Workflow) ([]byte, error)
- type ConcurrencySpec
- type Dependency
- type OnErrorSpec
- type ParamSpec
- type ResourceSpec
- type Spec
- type StepSpec
- type TriggerSpec
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Parse ¶
Parse reads the YAML and returns the workflow already validated.
`caminho` serves two purposes: deriving the slug when the file carries no `name`, and naming the file in error messages -- a graph error without the file's name is useless when there are dozens of them.
func Resolve ¶
Resolve expands every `uses:` across a set of workflows.
It works on the WHOLE set and not on one file, because a `uses:` names a sibling. That is also why the child must be in the same publish, and not merely published already: expansion that reads the database would make `brevis validate` -- which touches no database, on purpose -- answer a different question from `brevis publish`, and the file that passed CI would be the file that failed the deploy.
The child's steps take the parent step's place, prefixed with its id, and the parent node disappears. What arrives at the runner is one flat graph, which is the whole point: a step that TRIGGERED a child run and waited would hold a slot from the same per-process semaphore the child needs, and hang under load.
func UnknownFields ¶
UnknownFields lists what the decoder threw away, so a typo becomes a line somebody reads instead of a field that quietly does nothing.
IT WARNS AND DOES NOT REFUSE, which is the whole design. This decoder has never been strict, so any workflow anybody has may carry a stray key today; turning that into an error would fail files that have worked for a year in order to catch a typo in one of them. The engine behaves exactly as it did.
It is the near half of the gap `host:` exposed. An engine that does not know a field DROPS it -- so a step meant for the machine with the licence on it runs in the engine's own container instead, with nothing said. Nothing here fixes that for an OLD engine, and nothing can; it is why the silence was worth breaking on this one.
Separate from Parse rather than inside it: Parse's signature is what three call sites and the whole suite agree on, and a warning is not an error.
Types ¶
type ConcurrencySpec ¶
type ConcurrencySpec struct {
Limit int `yaml:"limit,omitempty"`
Group string `yaml:"group,omitempty"`
}
ConcurrencySpec is `concurrency:`, in either shape:
concurrency: 1 concurrency: group: jaffle_models limit: 1
The bare number limits runs of THIS workflow, as it always has. The mapping names a group: every workflow that names it counts against one limit, which is how a daily and a */15 that build the same models stop overlapping (#32).
func (ConcurrencySpec) MarshalYAML ¶
func (c ConcurrencySpec) MarshalYAML() (any, error)
MarshalYAML writes the short form when it says everything: `concurrency: 3` for a limit alone, the mapping when there is a group.
func (*ConcurrencySpec) UnmarshalYAML ¶
func (c *ConcurrencySpec) UnmarshalYAML(value *yaml.Node) error
UnmarshalYAML accepts a scalar or a mapping, and refuses a key it does not know.
Refuses, where the rest of the file only warns: KnownFields does not reach inside a custom unmarshaler (TestACustomUnmarshalerStillHidesItsFields), and a `grop:` dropped here would leave a workflow with no group, overlapping its partner in silence -- the bug this field exists to fix, back with no sign.
type Dependency ¶
type Dependency struct {
Step string `yaml:"step,omitempty"`
Label string `yaml:"label,omitempty"`
}
Dependency is one entry of `depends_on`. It accepts both shapes:
depends_on: [extract]
depends_on:
- {step: determine_load_type, label: changed existing data}
The bare form stays the normal one -- most dependencies have nothing to say and a label on every arrow is noise. The object exists for the branch, where two arrows leaving the same step with no labels is a diagram that requires opening the source to read.
func (Dependency) MarshalYAML ¶
func (d Dependency) MarshalYAML() (any, error)
MarshalYAML writes a dependency with no label as its step's name.
func (*Dependency) UnmarshalYAML ¶
func (d *Dependency) UnmarshalYAML(value *yaml.Node) error
UnmarshalYAML accepts a scalar or a mapping.
The scalar branch is what keeps every workflow ever written working: a list of strings decodes exactly as it did, and nothing in a published file has to change.
type OnErrorSpec ¶
OnErrorSpec is a step's alert declaration as written in the file.
There is no `webhook`, `url` or `channel` field, and there will not be. The destination is a credential; the installation owns it. This says WHETHER and HOW.
type ParamSpec ¶
type ParamSpec struct {
Name string `yaml:"name,omitempty"`
Type string `yaml:"type,omitempty"`
Default string `yaml:"default,omitempty"`
Description string `yaml:"description,omitempty"`
Enum []string `yaml:"enum,omitempty"`
Pattern string `yaml:"pattern,omitempty"`
Title string `yaml:"title,omitempty"`
Group string `yaml:"group,omitempty"`
Warn string `yaml:"warn,omitempty"`
Uses string `yaml:"uses,omitempty"`
}
ParamSpec is a run parameter as written in the file.
params:
- name: load_full
type: boolean
default: "false"
- name: start_date
type: string
pattern: '^\d{4}-\d{2}-\d{2}$'
`title`, `group`, `warn` and `uses` are how the form presents it, and do not change what a value may be.
type ResourceSpec ¶
type ResourceSpec struct {
CPU string `yaml:"cpu,omitempty"`
Memory string `yaml:"memory,omitempty"`
Limits struct {
CPU string `yaml:"cpu,omitempty"`
Memory string `yaml:"memory,omitempty"`
} `yaml:"limits,omitempty"`
}
ResourceSpec is the CPU and memory request in Kubernetes' format.
`limits` kept apart from `requests` because the difference between them is the difference between "how much I reserve" and "when I get killed": a dbt that blows the limit dies with OOMKilled, one that merely exceeds the request keeps running.
type Spec ¶
type Spec struct {
Name string `yaml:"name,omitempty"`
Description string `yaml:"description,omitempty"`
Schedule string `yaml:"schedule,omitempty"`
Type string `yaml:"type,omitempty"`
Tags []string `yaml:"tags,omitempty"`
Image string `yaml:"image,omitempty"`
Resources ResourceSpec `yaml:"resources,omitempty"`
Params []ParamSpec `yaml:"params,omitempty"`
// Env and Secrets apply to every step; a step overrides them name by name.
Env map[string]string `yaml:"env,omitempty"`
Secrets map[string]string `yaml:"secrets,omitempty"`
// Concurrency is Kestra's `concurrency.limit`, under the name most
// orchestrators use. See ConcurrencySpec for the two shapes.
Concurrency ConcurrencySpec `yaml:"concurrency,omitempty"`
// Trigger is why this runs other than the clock. See TriggerSpec.
Trigger TriggerSpec `yaml:"trigger,omitempty"`
Steps []StepSpec `yaml:"steps,omitempty"`
}
Spec mirrors the YAML, and nothing more. Loose fields here, invariants in the domain.
Every key is `omitempty`: reading ignores it, and writing a Spec back (SpecOf, for the console's Definition tab) leaves out what was not said, as a file somebody wrote would.
func SpecOf ¶
SpecOf is a published workflow written back as the file: the inverse of Parse, for the console's Definition tab. The database keeps the domain and not the file, so this is not the file the author wrote -- no comments, keys in the format's order -- but publishing it publishes the same workflow, which TestEveryExampleRoundTripsThroughItsYAML holds.
type StepSpec ¶
type StepSpec struct {
ID string `yaml:"id,omitempty"`
Run string `yaml:"run,omitempty"`
Action string `yaml:"action,omitempty"`
With map[string]any `yaml:"with,omitempty"`
DependsOn []Dependency `yaml:"depends_on,omitempty"`
// Image e Resources sobrescrevem os do workflow. Ausentes = herda.
Image string `yaml:"image,omitempty"`
Host string `yaml:"host,omitempty"`
Resources ResourceSpec `yaml:"resources,omitempty"`
// Env are variables with a literal value in the file.
//
// env:
// BREVIS_LOG_LEVEL: info
Env map[string]string `yaml:"env,omitempty"`
// PersistContext lists the keys this step may read and write through
// sdk/persist. Declaring it is what makes the engine point the step at the
// installation's store; a step that declares nothing gets no access at all.
//
// It is a LIST and not a boolean because the alternative is a flat global
// namespace where any step touches any key: two workflows that both know a
// good name for `station_codes` would overwrite each other and both keep
// running. Naming them is also what lets the reader of the file see that
// this step depends on whatever wrote them.
PersistContext []string `yaml:"persist_context,omitempty"`
// Secrets are variables whose value is NOT in the file: the key is the
// variable's name, the value is where to find it.
//
// secrets:
// GABRIEL_SESSION_COOKIE: gabriel-session/cookie
Secrets map[string]string `yaml:"secrets,omitempty"`
// Runtime and Tools say what this step runs in, when the engine cannot
// work it out from `run:` and `image:`.
//
// runtime: python
// tools: [dbt]
//
// Both optional, and the engine INFERS both when they are absent -- which
// is the normal case. Declaring one is for when the inference is wrong or
// blind: a wrapper script, a bare binary path, an image whose name says
// nothing.
//
// An id outside the vocabulary is refused at publish, naming what is
// valid. The alternative is a chip that renders blank on a screen three
// days later, with nothing to trace it to.
Runtime string `yaml:"runtime,omitempty"`
Tools []string `yaml:"tools,omitempty"`
// Shell: a pointer, to tell "did not declare" from "declared false". Without
// the pointer, every step without the key would become `shell: false` and
// images that do have a shell -- most of them -- would start receiving an
// argv, breaking any command with a pipe or a variable.
Shell *bool `yaml:"shell,omitempty"`
// When is the trigger rule. Empty is `all_success`, which is what every
// workflow written before this existed means.
When string `yaml:"when,omitempty"`
// Marker is a step that does nothing and exists to be a point in the
// graph -- a `start`, an `end`, a join. See dominio.Node.Marker.
Marker bool `yaml:"marker,omitempty"`
// UnlessEmpty names a context key that decides whether this step runs.
// A key, not an expression. See dominio.Node.UnlessEmpty.
UnlessEmpty string `yaml:"unless_empty,omitempty"`
// ForEach names a context key holding a list. The step runs once per
// element. See dominio.Node.ForEach.
ForEach string `yaml:"for_each,omitempty"`
// Group draws this step inside a named box on the graph. Visual only.
// See dominio.Node.Group.
Group string `yaml:"group,omitempty"`
// Uses names another workflow whose steps take this one's place, expanded
// at publish. See dominio.Node.Uses.
Uses string `yaml:"uses,omitempty"`
// OnError announces this step's failures.
//
// on_error:
// type: SLACK
// when: attempt # optional; the default is give_up
//
// A pointer for the same reason Shell is: absent has to be different from
// declared-and-empty, and a zero OnError would look like a step asking to
// be announced to nowhere.
OnError *OnErrorSpec `yaml:"on_error,omitempty"`
}
StepSpec is a step as written in the file.
type TriggerSpec ¶
type TriggerSpec struct {
OnLanded []string `yaml:"on_landed,omitempty"`
Debounce time.Duration `yaml:"debounce,omitempty"`
}
TriggerSpec is `trigger:` in the YAML.
trigger: on_landed: [bigquery://acme-prod/bronze/orders] debounce: 5m
A time.Duration straight through, as the gateway's `idle_timeout` and `max_conn_age` are. yaml.v3 reads `5m` as five minutes and REFUSES a bare `5` -- measured, because the opposite would have been quiet and expensive: a window read as five nanoseconds collapses nothing and looks exactly like the feature not working. TestADebounceNeedsItsUnit pins it.
func (TriggerSpec) MarshalYAML ¶
func (t TriggerSpec) MarshalYAML() (any, error)
MarshalYAML writes the debounce in its unit: yaml.v3 would write the nanoseconds, which the reader refuses (TestADebounceNeedsItsUnit).