Documentation
¶
Index ¶
- Variables
- func DefaultCRDClientBuilder(c *rest.Config) (apiextensionsclient.Interface, error)
- func EffectiveTaskTypes(taskType string, taskTypes []string) []string
- func IsJobFailed(job *batchv1.Job) bool
- func IsJobSucceeded(job *batchv1.Job) bool
- func NewResourceEventHandlerConfig(ctx context.Context, handler EventHandlerConfig) cache.ResourceEventHandler
- func TaskTypeInSet(taskType string, effectiveTypes []string) bool
- type AgentConfiguration
- type AgentConfigurations
- type CRDClientBuilder
- type ConfigResolver
- type CronScheduler
- type EventHandlerConfig
- type JobWatcher
- type K8sConnector
- type ResultPublisher
- type TaskStore
- type ZombieReason
- type ZombieSweeper
Constants ¶
This section is empty.
Variables ¶
var ErrConfigNotFound = stderrors.New("config not found")
ErrConfigNotFound is returned by ConfigResolver.Resolve when no Config in the store has a matching Spec.Assignee.
Functions ¶
func DefaultCRDClientBuilder ¶
func DefaultCRDClientBuilder(c *rest.Config) (apiextensionsclient.Interface, error)
DefaultCRDClientBuilder wraps apiextensionsclient.NewForConfig to satisfy CRDClientBuilder.
func EffectiveTaskTypes ¶
EffectiveTaskTypes computes the union of a singular taskType and a taskTypes list. The singular taskType is first in the result when non-empty. Duplicates are removed while preserving order. The result is the set an agent accepts.
func IsJobFailed ¶ added in v0.4.7
IsJobFailed reports whether the Job carries a true JobFailed condition.
Conditions, not the pod counters, are the source of truth: an activeDeadlineSeconds kill sets this condition while leaving .status.failed and .status.active at zero. Any caller deciding whether a Job is still running must use this (see spawner.IsJobActive) — reading the counters alone reports a deadline-killed Job as active forever.
func IsJobSucceeded ¶ added in v0.4.7
IsJobSucceeded reports whether the Job carries a true JobComplete condition.
func NewResourceEventHandlerConfig ¶
func NewResourceEventHandlerConfig( ctx context.Context, handler EventHandlerConfig, ) cache.ResourceEventHandler
NewResourceEventHandlerConfig adapts an EventHandlerConfig to the cache.ResourceEventHandler the informer expects.
func TaskTypeInSet ¶
TaskTypeInSet reports whether taskType is in the effectiveTypes set. Empty taskType never matches — strict semantics, no bypass for legacy tasks.
Types ¶
type AgentConfiguration ¶
type AgentConfiguration struct {
// Assignee is the task frontmatter assignee value that routes to this agent.
Assignee string
// TaskType is the singular task_type value from ConfigSpec.TaskType.
// Deprecated in favour of TaskTypes; stays functional.
TaskType string
// TaskTypes is the list of task_type values from ConfigSpec.TaskTypes.
// Nil when the CRD only sets the singular TaskType field.
TaskTypes []string
// Image is the container image base name (without tag). Tag is appended at runtime from branch.
Image string
// Env holds per-agent environment variables (e.g. API keys, config).
// These are merged with shared env vars (TASK_CONTENT, TASK_ID, KAFKA_BROKERS, BRANCH)
// when spawning the K8s Job.
Env map[string]string
// VolumeClaim is the name of an existing PVC to mount into the container.
// Empty means no volume mount.
VolumeClaim string
// VolumeMountPath is the container path where the PVC is mounted.
// Required when VolumeClaim is set.
VolumeMountPath string
// SecretName is the name of a K8s Secret to mount as envFrom on the container.
// Empty means no secret is mounted.
SecretName string
// Resources declares optional resource requests and limits for the agent container.
// Nil means "do not set, keep the k8s builder default".
Resources *agentv1.AgentResources
// MaxConcurrentJobs caps how many Jobs this agent may run at once.
// 0 means unlimited.
MaxConcurrentJobs int
// PriorityClassName is the Kubernetes PriorityClass name to stamp onto spawned Job PodTemplates.
PriorityClassName string
// ImagePullSecret is the name of the K8s Secret used for image pulls.
// Empty uses the cluster default (typically "docker").
ImagePullSecret string
// Trigger declares the per-agent phase and status conditions under which the executor spawns a Job.
Trigger *agentv1.Trigger
// ZombieJobTimeoutSeconds mirrors ConfigSpec.ZombieJobTimeoutSeconds. The
// spawner stamps this value onto Job.Spec.ActiveDeadlineSeconds; the sweeper
// uses it as the elapsed-time threshold. nil means "use the default
// DefaultZombieJobTimeoutSeconds from the CRD types package".
ZombieJobTimeoutSeconds *int32
}
AgentConfiguration defines the container image and environment for one agent type.
func (AgentConfiguration) EffectiveZombieJobTimeoutSeconds ¶
func (a AgentConfiguration) EffectiveZombieJobTimeoutSeconds() int32
EffectiveZombieJobTimeoutSeconds returns the effective deadline in seconds: the configured value when non-nil, else agentv1.DefaultZombieJobTimeoutSeconds.
type AgentConfigurations ¶
type AgentConfigurations []AgentConfiguration
AgentConfigurations is a list of agent configurations.
func (AgentConfigurations) FindByAssignee ¶
func (a AgentConfigurations) FindByAssignee(assignee string) (AgentConfiguration, bool)
FindByAssignee returns the configuration for the given assignee name. Returns the config and true if found, zero value and false otherwise.
func (AgentConfigurations) TaggedConfigurations ¶
func (a AgentConfigurations) TaggedConfigurations(branch string) AgentConfigurations
TaggedConfigurations returns a new AgentConfigurations with the branch appended to each untagged image as a tag (e.g. "registry/image" + ":" + "dev" → "registry/image:dev"). Images that already carry a tag or digest are left as-is (see appendBranchTag), so semver-pinned images are not double-tagged.
type CRDClientBuilder ¶
type CRDClientBuilder func(*rest.Config) (apiextensionsclient.Interface, error)
CRDClientBuilder constructs the apiextensions clientset from a rest.Config. Injected to allow fake clientsets in tests.
type ConfigResolver ¶
type ConfigResolver interface {
Resolve(ctx context.Context, assignee string) (AgentConfiguration, error)
}
ConfigResolver looks up the AgentConfiguration for an assignee by iterating the in-memory Config store and converting the matching entry.
func NewConfigResolver ¶
func NewConfigResolver( provider k8s.Provider[agentv1.Config], branch base.Branch, vaultName string, ) ConfigResolver
NewConfigResolver returns a ConfigResolver backed by the given typed store. The branch is captured here and appended as the image tag at resolution time. Resolutions are scoped to the given vault: each task's plain assignee is matched against the composed {assignee}-{vaultName} Config assignee, so per-vault Config CRs apply per executor instance. When no composed Config exists, Resolve falls back to the plain assignee, so the singular shared-topic executor (v0.6.x Config CRs without a vault suffix) keeps resolving on the v0.7+ line.
type CronScheduler ¶
CronScheduler runs a cron expression on a schedule.
type EventHandlerConfig ¶
type EventHandlerConfig k8s.EventHandler[agentv1.Config]
EventHandlerConfig is the typed in-memory event handler / store for Config resources. Backed by github.com/bborbe/k8s generics.
func NewEventHandlerConfig ¶
func NewEventHandlerConfig() EventHandlerConfig
NewEventHandlerConfig returns an empty EventHandlerConfig.
type JobWatcher ¶
type JobWatcher interface {
// Run starts the Job and Pod informers and blocks until ctx is cancelled.
Run(ctx context.Context) error
// HandleJob processes a single Job (invoked by the informer event handlers
// and by unit tests directly, avoiding the need for a fake informer).
HandleJob(ctx context.Context, job *batchv1.Job)
// HandlePod processes a single Pod (invoked by the Pod informer event handler
// and by unit tests directly).
HandlePod(ctx context.Context, pod *corev1.Pod)
// PodLister returns the Pod lister backed by the shared informer cache, for
// use by the deadline sweeper. The returned lister is safe for concurrent
// read access.
PodLister() corev1listers.PodLister
}
JobWatcher watches batch/v1 Jobs and their Pods in the executor's namespace and publishes synthetic failure results for terminal-state objects that belong to spawned tasks.
func NewJobWatcher ¶
func NewJobWatcher( kubeClient kubernetes.Interface, namespace libk8s.Namespace, taskStore *TaskStore, publisher ResultPublisher, ) JobWatcher
NewJobWatcher creates a JobWatcher.
type K8sConnector ¶
type K8sConnector interface {
SetupCustomResourceDefinition(ctx context.Context) error
Listen(
ctx context.Context,
namespace libk8s.Namespace,
handler cache.ResourceEventHandler,
) error
}
K8sConnector installs the Config CRD and starts an informer.
func NewK8sConnector ¶
func NewK8sConnector(config *rest.Config, crdBuilder CRDClientBuilder) K8sConnector
NewK8sConnector returns a K8sConnector using the given rest config and builder. Production callers pass DefaultCRDClientBuilder as the builder.
type ResultPublisher ¶
type ResultPublisher interface {
// PublishSpawnNotification publishes current_job, job_started_at, and
// spawn_notification without touching any other frontmatter keys.
PublishSpawnNotification(ctx context.Context, task lib.Task, jobName string) error
// PublishFailure publishes a zombie failure: clears current_job and atomically
// bumps trigger_count by 1 via a paired IncrementFrontmatterCommand. Leaves
// phase, status, and assignee untouched so the existing trigger_count retry
// cap (applyTriggerCap in task/controller/pkg/result/result_writer.go) handles
// eventual operator-inbox escalation. Idempotent per current_job via a TTL'd
// LRU; concurrent classifications for the same job emit one event.
PublishFailure(ctx context.Context, task lib.Task, jobName string, reason string) error
// PublishClearCurrentJob clears current_job in the vault task frontmatter
// (writes "" only — no body section, no trigger_count bump, no status/phase
// touch). Used on the job-success path, where the agent already published its
// real result and the task is about to be evicted from the TaskStore: without
// this clear, a succeeded job leaves current_job/job_started_at set forever
// (the zombie sweeper is blind to it because the task left Snapshot()). The
// write is idempotent (empty → empty is a no-op visually) and deduped per
// jobName by the same TTL'd LRU as PublishFailure, so the informer's repeated
// terminal re-deliveries do not spam Kafka.
PublishClearCurrentJob(ctx context.Context, task lib.Task, jobName string) error
// PublishIncrementTriggerCount sends an IncrementFrontmatterCommand that atomically
// increments trigger_count by 1. Must complete before SpawnJob is called.
PublishIncrementTriggerCount(ctx context.Context, task lib.Task) error
// PublishSetTriggerScope sends an UpdateFrontmatterCommand that records a
// trigger_scope and sets trigger_count to count, in one atomic write. Called
// instead of PublishIncrementTriggerCount in the two cases where the scope
// itself has to be written:
//
// - the scope CHANGED (count 1): a re-dispatch against a new phase or a new
// ref starts from a fresh budget, counting the spawn about to happen.
// - the scope was ABSENT (count = existing trigger_count + 1): the task
// predates the field and adopts the current scope WITHOUT a budget reset.
// Absent must not be treated as changed — every task in flight today has no
// trigger_scope, so resetting on absence would hand each of them a free
// budget on the first event after deploy, including one already at cap.
//
// This is an UpdateFrontmatterCommand rather than a result publish on purpose.
// trigger_count is controller-owned: the result writer's ownership guard
// (MergeFrontmatter in task/controller) discards any incoming counter value, so
// a reset routed through a result payload would be silently dropped. The atomic
// frontmatter commands take a different path (buildUpdateModifyFn), which applies
// Updates directly to the on-disk frontmatter — an explicit, intentional write
// rather than a stale spawn-time snapshot.
PublishSetTriggerScope(ctx context.Context, task lib.Task, scope string, count int) error
// PublishTypeMismatchFailure publishes a synthetic failure when the task's task_type
// is not in the agent's effective type set. Clears assignee and current_job so the
// task surfaces in the operator inbox via assignee=="" filter. Does not bump
// trigger_count or retry_count.
PublishTypeMismatchFailure(ctx context.Context, task lib.Task, reason string) error
// PublishRaw publishes a raw payload for testing error paths.
PublishRaw(ctx context.Context, operation base.CommandOperation, payload interface{}) error
}
ResultPublisher publishes agent-task-v1-request commands to Kafka so the controller writes them to the vault task file.
func NewResultPublisher ¶
func NewResultPublisher( syncProducer libkafka.SyncProducer, topicPrefix base.TopicPrefix, currentDateTime libtime.CurrentDateTimeGetter, ) ResultPublisher
NewResultPublisher creates a ResultPublisher.
type TaskStore ¶
type TaskStore struct {
// contains filtered or unexported fields
}
TaskStore is a thread-safe map from TaskIdentifier to Task. It is populated when the executor spawns a Job and consumed by the job informer when publishing synthetic failure results.
func (*TaskStore) Delete ¶
func (s *TaskStore) Delete(id lib.TaskIdentifier)
Delete removes the task for the given identifier (called on job termination).
type ZombieReason ¶
type ZombieReason string
ZombieReason is the closed set of machine-readable reason strings emitted in the ## Failure body section. Operators grep on these values to triage. OOM kills use the dedicated pod_oom_killed value so operators can grep for OOM specifically; other non-zero pod terminations fall back to the generic pod_crash_no_stdout. Adding a new value requires updating this list and the documentation; renaming or removing a value is a breaking change to the on-disk task body contract.
const ( ZombieReasonImagePullBackOff ZombieReason = "image_pull_backoff" ZombieReasonPodEvicted ZombieReason = "pod_evicted" ZombieReasonDeadlineExceeded ZombieReason = "deadline_exceeded" ZombieReasonPodNotScheduled ZombieReason = "pod_not_scheduled" ZombieReasonPodCrashNoStdout ZombieReason = "pod_crash_no_stdout" ZombieReasonExecutorWatchLost ZombieReason = "executor_watch_lost" ZombieReasonTypeMismatch ZombieReason = "type_mismatch" ZombieReasonPodOOMKilled ZombieReason = "pod_oom_killed" )
func JobFailureReason ¶
func JobFailureReason(job *batchv1.Job) ZombieReason
JobFailureReason maps a failed Job's conditions to a ZombieReason. Returns ZombieReasonDeadlineExceeded when a Failed condition has Reason "DeadlineExceeded" (the kubelet killed the pod for running past activeDeadlineSeconds). Returns the generic pod-failure reason ZombieReasonPodCrashNoStdout for "BackoffLimitExceeded" (the Job exhausted its retry backoff limit — a distinct reason from a deadline kill; HandleJob resolves the pod's terminated reason + exit code and surfaces them in the published reason) and for any other Failed condition (the pod terminated non-zero and no AgentResult was observed; the Job-condition informer only fires AFTER terminal state, so absence of an AgentResult is implicit at this point).
func (ZombieReason) String ¶
func (r ZombieReason) String() string
String returns the reason as a string (for use with PublishFailure).
type ZombieSweeper ¶
type ZombieSweeper interface {
// Run blocks until ctx is cancelled. Each tick (interval sourced from the
// first non-nil ConfigSpec.ZombieSweeperIntervalSeconds across the resolver's
// configs, else DefaultZombieSweeperIntervalSeconds) it calls SweepOnce.
Run(ctx context.Context) error
// SweepOnce performs a single sweep pass. Exposed for unit tests so they
// do not have to manage tickers. Returns an error only on context
// cancellation paths; per-task classification errors are logged.
SweepOnce(ctx context.Context) error
}
ZombieSweeper is a background goroutine that periodically classifies stuck tasks as zombies and emits failure events. It is the safety net for the informer-driven paths in JobWatcher (which handle the cases k8s notifies us about). The sweeper handles: pods unschedulable beyond a grace window, executor restart losing watch on a Job, and any deadline path the informer misses (Job-condition deferred indefinitely, informer cache drift).
func NewZombieSweeper ¶
func NewZombieSweeper( jobWatcher JobWatcher, namespace libk8s.Namespace, taskStore *TaskStore, publisher ResultPublisher, configProvider EventHandlerConfig, currentDateTime libtime.CurrentDateTimeGetter, ) ZombieSweeper
NewZombieSweeper creates a ZombieSweeper. The JobWatcher is held by reference (not its lister) because the lister is only populated after JobWatcher.Run completes its cache sync; service.Run starts all components concurrently, so extracting the lister at wiring time would capture nil. SweepOnce resolves the lister lazily on every tick and skips the tick if it is not yet ready.