Documentation
¶
Overview ¶
Package workflow (application) translates the YAML file into the domain.
The separation exists because §22 of the plan is explicit: once published, the database is the source of truth, not the file. The YAML is publishing INPUT -- it comes in here, becomes domain, and the domain is what persists. Changing the file format must not touch the graph's invariants.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type ParamSpec ¶
type ParamSpec struct {
Name string `yaml:"name"`
Type string `yaml:"type"`
Default string `yaml:"default"`
Description string `yaml:"description"`
Enum []string `yaml:"enum"`
Pattern string `yaml:"pattern"`
}
ParamSpec is a run parameter as written in the file.
params:
- name: load_full
type: boolean
default: "false"
- name: start_date
type: string
pattern: '^\d{4}-\d{2}-\d{2}$'
type ResourceSpec ¶
type ResourceSpec struct {
CPU string `yaml:"cpu"`
Memory string `yaml:"memory"`
Limits struct {
CPU string `yaml:"cpu"`
Memory string `yaml:"memory"`
} `yaml:"limits"`
}
ResourceSpec is the CPU and memory request in Kubernetes' format.
`limits` kept apart from `requests` because the difference between them is the difference between "how much I reserve" and "when I get killed": a dbt that blows the limit dies with OOMKilled, one that merely exceeds the request keeps running.
type Spec ¶
type Spec struct {
Name string `yaml:"name"`
Schedule string `yaml:"schedule"`
Type string `yaml:"type"`
Tags []string `yaml:"tags"`
Image string `yaml:"image"`
Resources ResourceSpec `yaml:"resources"`
Params []ParamSpec `yaml:"params"`
// Env and Secrets apply to every step; a step overrides them name by name.
Env map[string]string `yaml:"env"`
Secrets map[string]string `yaml:"secrets"`
// Concurrency is Kestra's `concurrency.limit`, under the name most
// orchestrators use.
Concurrency int `yaml:"concurrency"`
Steps []StepSpec `yaml:"steps"`
}
Spec mirrors the YAML, and nothing more. Loose fields here, invariants in the domain.
type StepSpec ¶
type StepSpec struct {
ID string `yaml:"id"`
Run string `yaml:"run"`
Action string `yaml:"action"`
With map[string]any `yaml:"with"`
DependsOn []string `yaml:"depends_on"`
// Image e Resources sobrescrevem os do workflow. Ausentes = herda.
Image string `yaml:"image"`
Resources ResourceSpec `yaml:"resources"`
// Env are variables with a literal value in the file.
//
// env:
// BREVIS_LOG_LEVEL: info
Env map[string]string `yaml:"env"`
// Secrets are variables whose value is NOT in the file: the key is the
// variable's name, the value is where to find it.
//
// secrets:
// GABRIEL_SESSION_COOKIE: gabriel-session/cookie
Secrets map[string]string `yaml:"secrets"`
// Runtime and Tools say what this step runs in, when the engine cannot
// work it out from `run:` and `image:`.
//
// runtime: python
// tools: [dbt]
//
// Both optional, and the engine INFERS both when they are absent -- which
// is the normal case. Declaring one is for when the inference is wrong or
// blind: a wrapper script, a bare binary path, an image whose name says
// nothing.
//
// An id outside the vocabulary is refused at publish, naming what is
// valid. The alternative is a chip that renders blank on a screen three
// days later, with nothing to trace it to.
Runtime string `yaml:"runtime"`
Tools []string `yaml:"tools"`
// Shell: a pointer, to tell "did not declare" from "declared false". Without
// the pointer, every step without the key would become `shell: false` and
// images that do have a shell -- most of them -- would start receiving an
// argv, breaking any command with a pipe or a variable.
Shell *bool `yaml:"shell"`
}
StepSpec is a step as written in the file.