Documentation
¶
Overview ¶
Package kubernetes executa cada passo de um workflow como um POD proprio.
A dinamica e a da secao 2 do plano, e e o que separa este motor de um worker monolitico: o pod sobe com a imagem EXATA do passo, roda um comando, reporta e morre. Um passo de dbt sobe a imagem de dbt com 1Gi; o fetcher em Go ao lado sobe uma imagem de 10 MB com 32Mi. Numa imagem unica os dois pagariam o maior dos dois — em bytes de pull, em memoria reservada e em superficie.
O cliente e escrito sobre a stdlib, sem client-go. A biblioteca oficial traz centenas de dependencias e dezenas de MB para o que aqui sao quatro chamadas REST: criar pod, ler status, ler log, apagar pod. A mesma escolha ja foi feita para o React (bundle vendorizado) e para o CSS (Tailwind standalone): o custo de uma dependencia grande so se paga quando se usa uma fracao grande dela.
Index ¶
- func NomeDoPod(t execution.TaskExec) string
- type API
- type Cliente
- func (c *Cliente) ApagarPod(ctx context.Context, nome string) error
- func (c *Cliente) CriarPod(ctx context.Context, p Pod) (Pod, error)
- func (c *Cliente) LerPod(ctx context.Context, nome string) (Pod, error)
- func (c *Cliente) Logs(ctx context.Context, nome string, seguir bool) (io.ReadCloser, error)
- func (c *Cliente) Namespace() string
- type Condicao
- type Container
- type ErrForaDoCluster
- type Executor
- type FonteEnv
- type FontePVC
- type FonteVar
- type Metadata
- type MontagemDeVolume
- type Opcoes
- type Pod
- type PodSpec
- type PodStatus
- type Recursos
- type RefChave
- type RefLocal
- type StatusContainer
- type Toleracao
- type Var
- type Volume
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NomeDoPod ¶
NomeDoPod produces a valid and STABLE name for the same attempt.
Estavel importa: se o processo morrer entre criar o pod e registrar isso, a tentativa seguinte encontra o pod existente (409 AlreadyExists) em vez de subir um segundo pod rodando o mesmo dbt em paralelo com o primeiro.
O sufixo de hash resolve a colisao que o corte de 63 caracteres criaria entre dois nodes de nome longo e prefixo comum.
Types ¶
type API ¶
type API interface {
CriarPod(ctx context.Context, p Pod) (Pod, error)
LerPod(ctx context.Context, nome string) (Pod, error)
Logs(ctx context.Context, nome string, seguir bool) (io.ReadCloser, error)
ApagarPod(ctx context.Context, nome string) error
}
API is what the executor needs from the server. An interface in the consumer: it is what makes it possible to test the pod's whole lifecycle against a fake server.
type Cliente ¶
type Cliente struct {
// contains filtered or unexported fields
}
Cliente fala com o servidor de API.
type Condicao ¶
type Condicao struct {
Type string `json:"type"`
Status string `json:"status"`
Reason string `json:"reason,omitempty"`
Message string `json:"message,omitempty"`
}
Condicao carries PodScheduled, where the scheduler explains why it did not fit.
type Container ¶
type Container struct {
Name string `json:"name"`
Image string `json:"image"`
Command []string `json:"command,omitempty"`
Args []string `json:"args,omitempty"`
Env []Var `json:"env,omitempty"`
EnvFrom []FonteEnv `json:"envFrom,omitempty"`
Resources *Recursos `json:"resources,omitempty"`
WorkingDir string `json:"workingDir,omitempty"`
VolumeMounts []MontagemDeVolume `json:"volumeMounts,omitempty"`
}
type ErrForaDoCluster ¶
type ErrForaDoCluster struct{ Motivo string }
ErrForaDoCluster e devolvido quando nao ha service account montada.
func (ErrForaDoCluster) Error ¶
func (e ErrForaDoCluster) Error() string
type Executor ¶
type Executor struct {
// Status polling interval. Polling rather than watching is deliberate: a
// watch needs reconnection, resync and handling of missed events to gain
// seconds on a task that lasts minutes.
Intervalo time.Duration
// contains filtered or unexported fields
}
Executor runs each step as a pod.
The cycle is always the same: create the pod, wait for it to leave Pending, follow the log while it runs, read the exit code and delete it. No state lives here beyond the in-flight pods -- if the process restarts, the pods keep running and the dispatcher finds them again by their deterministic name.
func NewExecutor ¶
func (*Executor) Execute ¶
func (e *Executor) Execute(ctx context.Context, t execution.TaskExec) (<-chan execution.Event, error)
Execute creates the pod and returns the event channel. The channel closes when the pod finishes -- the same shape as the local executor, so the runner cannot tell them apart.
type FonteVar ¶
type FonteVar struct {
SecretKeyRef *RefChave `json:"secretKeyRef,omitempty"`
}
FonteVar points a variable at a key of a Secret. The value never passes through the engine: the kubelet reads it when starting the container.
type MontagemDeVolume ¶
type Opcoes ¶
type Opcoes struct {
Namespace string
ServiceAccount string
PullSecrets []string
NodeSelector map[string]string
Tolerations []Toleracao
EnvFromSecrets []string
EnvFromConfigMaps []string
// CredencialPVC and CredencialPath mount a volume where the SDK keeps the
// credential it rotates between runs.
//
// With both set, EVERY step pod gets the volume and a BREVIS_CREDENTIAL_DIR
// env pointing at the mount. Without them nothing changes -- which is how
// the feature stays a shortcut rather than a requirement.
//
// The credential on the volume is encrypted; the key is an ordinary Secret,
// arriving through EnvFromSecrets. The engine neither sees it nor needs
// it.
CredencialPVC string
CredencialPath string
// SecretsPermitidos are the Secrets a YAML may name in `secrets:`.
//
// It exists because `secrets:` inverts who chooses. EnvFromSecrets comes
// from the scheduler's environment: the INSTALLATION decides. `secrets:` is
// in the file, and the file is written by somebody else -- without this
// list, a workflow could mount any Secret in the namespace, including
// Brevis's own database secret, and run an arbitrary command holding it.
//
// Empty denies everything. Denying by default costs one variable in the
// installation; allowing by default costs the opposite, and the opposite is
// irreversible.
//
// The final division is this: the installation says WHICH secrets exist for
// workflows, the YAML says WHICH step receives each one.
SecretsPermitidos []string
Labels map[string]string
Shell []string
// EsperaParaIniciar is how long a pod may go without starting before the
// step gives up. It exists because `Pending` is not an error to Kubernetes:
// a pod that fits on no node sits there forever, and without this limit the
// step waits along with it -- no log, no failure, no retry. It happened in
// dev with a CPU request larger than the pool's free capacity.
EsperaParaIniciar time.Duration
// ManterPodEmFalha leaves the pod around for inspection when a step fails.
// A successful one is always deleted: thousands of Completed pods clutter
// the namespace and say nothing Brevis's own history does not say better.
ManterPodEmFalha bool
}
Opcoes parameterises how pods are created. These are the INSTALLATION's decisions -- credentials, node pool, service account -- not the workflow author's: a pipeline YAML must not get to pick the service account it runs as.
type Pod ¶
type Pod struct {
APIVersion string `json:"apiVersion,omitempty"`
Kind string `json:"kind,omitempty"`
Metadata Metadata `json:"metadata"`
Spec PodSpec `json:"spec,omitempty"`
// A pointer because `omitempty` does not omit an empty struct: without it
// every created pod would send `"status":{}` to the server -- harmless, but
// noise in an object people read to debug.
Status *PodStatus `json:"status,omitempty"`
}
Pod is the subset of the object this engine uses. Writing the structs by hand instead of importing client-go's keeps the dependency tree small and makes it visible exactly what is sent to the API server.
func MontarPod ¶
MontarPod translates a task into the object that goes to the API server.
A pure function: it takes a task and options and returns the object. That is what makes it possible to test the whole spec -- image, command, resources, labels -- with no cluster at all.
func (Pod) Motivo ¶
Motivo e o `reason` do status (DeadlineExceeded, OOMKilled, Evicted) — a diferenca entre "o codigo falhou" e "o cluster matou o processo".
func (Pod) MotivoDeEspera ¶
MotivoDeEspera explains why the container has not run yet.
It is the most useful piece of information when a step "does nothing": ImagePullBackOff and CreateContainerConfigError are configuration problems that, without this, would show up only as a pod sitting still until the timeout.
type PodSpec ¶
type PodSpec struct {
RestartPolicy string `json:"restartPolicy,omitempty"`
ServiceAccountName string `json:"serviceAccountName,omitempty"`
ImagePullSecrets []RefLocal `json:"imagePullSecrets,omitempty"`
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
Tolerations []Toleracao `json:"tolerations,omitempty"`
ActiveDeadlineSeconds *int64 `json:"activeDeadlineSeconds,omitempty"`
Volumes []Volume `json:"volumes,omitempty"`
Containers []Container `json:"containers"`
}
type PodStatus ¶
type PodStatus struct {
Phase string `json:"phase,omitempty"`
Conditions []Condicao `json:"conditions,omitempty"`
Reason string `json:"reason,omitempty"`
Message string `json:"message,omitempty"`
ContainerStatuses []StatusContainer `json:"containerStatuses,omitempty"`
}
type StatusContainer ¶
type StatusContainer struct {
Name string `json:"name"`
State struct {
Waiting *struct {
Reason string `json:"reason"`
Message string `json:"message"`
} `json:"waiting,omitempty"`
Running *struct {
StartedAt string `json:"startedAt"`
} `json:"running,omitempty"`
Terminated *struct {
ExitCode int `json:"exitCode"`
Reason string `json:"reason"`
Message string `json:"message"`
} `json:"terminated,omitempty"`
} `json:"state"`
}
type Volume ¶
type Volume struct {
Name string `json:"name"`
PVC *FontePVC `json:"persistentVolumeClaim,omitempty"`
}
Volume is a PersistentVolumeClaim mounted into the pod.
PVC only, and not the union of everything Kubernetes accepts: the engine mounts a volume for one purpose -- keeping a rotated credential between runs -- and a field that exists for one purpose should not accept ten shapes.