pkg

package
v0.11.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 6, 2026 License: BSD-2-Clause Imports: 33 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrConfigNotFound = stderrors.New("config not found")

ErrConfigNotFound is returned by ConfigResolver.Resolve when no Config in the store has a matching Spec.Assignee.

Functions

func DefaultCRDClientBuilder

func DefaultCRDClientBuilder(c *rest.Config) (apiextensionsclient.Interface, error)

DefaultCRDClientBuilder wraps apiextensionsclient.NewForConfig to satisfy CRDClientBuilder.

func EffectiveTaskTypes

func EffectiveTaskTypes(taskType string, taskTypes []string) []string

EffectiveTaskTypes computes the union of a singular taskType and a taskTypes list. The singular taskType is first in the result when non-empty. Duplicates are removed while preserving order. The result is the set an agent accepts.

func IsJobFailed added in v0.4.7

func IsJobFailed(job *batchv1.Job) bool

IsJobFailed reports whether the Job carries a true JobFailed condition.

Conditions, not the pod counters, are the source of truth: an activeDeadlineSeconds kill sets this condition while leaving .status.failed and .status.active at zero. Any caller deciding whether a Job is still running must use this (see spawner.IsJobActive) — reading the counters alone reports a deadline-killed Job as active forever.

func IsJobSucceeded added in v0.4.7

func IsJobSucceeded(job *batchv1.Job) bool

IsJobSucceeded reports whether the Job carries a true JobComplete condition.

func NewResourceEventHandlerConfig

func NewResourceEventHandlerConfig(
	ctx context.Context,
	handler EventHandlerConfig,
) cache.ResourceEventHandler

NewResourceEventHandlerConfig adapts an EventHandlerConfig to the cache.ResourceEventHandler the informer expects.

func ReadGitRestGatewaySecret added in v0.11.0

func ReadGitRestGatewaySecret(
	ctx context.Context,
	kubeClient kubernetes.Interface,
	namespace libk8s.Namespace,
	secretName string,
) (string, error)

ReadGitRestGatewaySecret returns the git-rest gateway secret value from the named K8s Secret, or "" when secretName is empty (gateway auth disabled). The value is read once at startup and held in memory only — it is never logged, never embedded in the image, and never in env (spec 005 Security; the env var GITREST_GATEWAY_SECRET references the secret by NAME, matching the JobKafkaClientCertSecret pattern). Returns an error when the Secret is missing or lacks the gateway-secret data key, so a misconfigured deployment fails loudly at startup instead of silently sending no auth header.

func TaskTypeInSet

func TaskTypeInSet(taskType string, effectiveTypes []string) bool

TaskTypeInSet reports whether taskType is in the effectiveTypes set. Empty taskType never matches — strict semantics, no bypass for legacy tasks.

Types

type AgentConfiguration

type AgentConfiguration struct {
	// Assignee is the task frontmatter assignee value that routes to this agent.
	Assignee string
	// TaskType is the singular task_type value from ConfigSpec.TaskType.
	// Deprecated in favour of TaskTypes; stays functional.
	TaskType string
	// TaskTypes is the list of task_type values from ConfigSpec.TaskTypes.
	// Nil when the CRD only sets the singular TaskType field.
	TaskTypes []string
	// Image is the container image base name (without tag). Tag is appended at runtime from branch.
	Image string
	// Env holds per-agent environment variables (e.g. API keys, config).
	// These are merged with shared env vars (TASK_CONTENT, TASK_ID, KAFKA_BROKERS, BRANCH)
	// when spawning the K8s Job.
	Env map[string]string
	// VolumeClaim is the name of an existing PVC to mount into the container.
	// Empty means no volume mount.
	VolumeClaim string
	// VolumeMountPath is the container path where the PVC is mounted.
	// Required when VolumeClaim is set.
	VolumeMountPath string
	// SecretName is the name of a K8s Secret to mount as envFrom on the container.
	// Empty means no secret is mounted.
	SecretName string
	// Resources declares optional resource requests and limits for the agent container.
	// Nil means "do not set, keep the k8s builder default".
	Resources *agentv1.AgentResources
	// MaxConcurrentJobs caps how many Jobs this agent may run at once.
	// 0 means unlimited.
	MaxConcurrentJobs int
	// PriorityClassName is the Kubernetes PriorityClass name to stamp onto spawned Job PodTemplates.
	PriorityClassName string
	// ImagePullSecret is the name of the K8s Secret used for image pulls.
	// Empty uses the cluster default (typically "docker").
	ImagePullSecret string
	// Trigger declares the per-agent phase and status conditions under which the executor spawns a Job.
	Trigger *agentv1.Trigger
	// ZombieJobTimeoutSeconds mirrors ConfigSpec.ZombieJobTimeoutSeconds. The
	// spawner stamps this value onto Job.Spec.ActiveDeadlineSeconds; the sweeper
	// uses it as the elapsed-time threshold. nil means "use the default
	// DefaultZombieJobTimeoutSeconds from the CRD types package".
	ZombieJobTimeoutSeconds *int32
}

AgentConfiguration defines the container image and environment for one agent type.

func (AgentConfiguration) EffectiveZombieJobTimeoutSeconds

func (a AgentConfiguration) EffectiveZombieJobTimeoutSeconds() int32

EffectiveZombieJobTimeoutSeconds returns the effective deadline in seconds: the configured value when non-nil, else agentv1.DefaultZombieJobTimeoutSeconds.

type AgentConfigurations

type AgentConfigurations []AgentConfiguration

AgentConfigurations is a list of agent configurations.

func (AgentConfigurations) FindByAssignee

func (a AgentConfigurations) FindByAssignee(assignee string) (AgentConfiguration, bool)

FindByAssignee returns the configuration for the given assignee name. Returns the config and true if found, zero value and false otherwise.

func (AgentConfigurations) TaggedConfigurations

func (a AgentConfigurations) TaggedConfigurations(branch string) AgentConfigurations

TaggedConfigurations returns a new AgentConfigurations with the branch appended to each untagged image as a tag (e.g. "registry/image" + ":" + "dev" → "registry/image:dev"). Images that already carry a tag or digest are left as-is (see appendBranchTag), so semver-pinned images are not double-tagged.

type CRDClientBuilder

type CRDClientBuilder func(*rest.Config) (apiextensionsclient.Interface, error)

CRDClientBuilder constructs the apiextensions clientset from a rest.Config. Injected to allow fake clientsets in tests.

type ConfigResolver

type ConfigResolver interface {
	Resolve(ctx context.Context, assignee string) (AgentConfiguration, error)
}

ConfigResolver looks up the AgentConfiguration for an assignee by iterating the in-memory Config store and converting the matching entry.

func NewConfigResolver

func NewConfigResolver(
	provider k8s.Provider[agentv1.Config],
	branch base.Branch,
	vaultName string,
) ConfigResolver

NewConfigResolver returns a ConfigResolver backed by the given typed store. The branch is captured here and appended as the image tag at resolution time. Resolutions are scoped to the given vault: each task's plain assignee is matched against the composed {assignee}-{vaultName} Config assignee, so per-vault Config CRs apply per executor instance. When no composed Config exists, Resolve falls back to the plain assignee, so the singular shared-topic executor (v0.6.x Config CRs without a vault suffix) keeps resolving on the v0.7+ line.

type CronScheduler

type CronScheduler interface {
	Run(ctx context.Context) error
}

CronScheduler runs a cron expression on a schedule.

type EventHandlerConfig

type EventHandlerConfig k8s.EventHandler[agentv1.Config]

EventHandlerConfig is the typed in-memory event handler / store for Config resources. Backed by github.com/bborbe/k8s generics.

func NewEventHandlerConfig

func NewEventHandlerConfig() EventHandlerConfig

NewEventHandlerConfig returns an empty EventHandlerConfig.

type JobWatcher

type JobWatcher interface {
	// Run starts the Job and Pod informers and blocks until ctx is cancelled.
	Run(ctx context.Context) error
	// HandleJob processes a single Job (invoked by the informer event handlers
	// and by unit tests directly, avoiding the need for a fake informer).
	HandleJob(ctx context.Context, job *batchv1.Job)
	// HandlePod processes a single Pod (invoked by the Pod informer event handler
	// and by unit tests directly).
	HandlePod(ctx context.Context, pod *corev1.Pod)
	// PodLister returns the Pod lister backed by the shared informer cache, for
	// use by the deadline sweeper. The returned lister is safe for concurrent
	// read access.
	PodLister() corev1listers.PodLister
}

JobWatcher watches batch/v1 Jobs and their Pods in the executor's namespace and publishes synthetic failure results for terminal-state objects that belong to spawned tasks.

func NewJobWatcher

func NewJobWatcher(
	kubeClient kubernetes.Interface,
	namespace libk8s.Namespace,
	taskStore *TaskStore,
	publisher ResultPublisher,
) JobWatcher

NewJobWatcher creates a JobWatcher.

type K8sConnector

type K8sConnector interface {
	SetupCustomResourceDefinition(ctx context.Context) error
	Listen(
		ctx context.Context,
		namespace libk8s.Namespace,
		handler cache.ResourceEventHandler,
	) error
}

K8sConnector installs the Config CRD and starts an informer.

func NewK8sConnector

func NewK8sConnector(config *rest.Config, crdBuilder CRDClientBuilder) K8sConnector

NewK8sConnector returns a K8sConnector using the given rest config and builder. Production callers pass DefaultCRDClientBuilder as the builder.

type ResultPublisher

type ResultPublisher interface {
	// PublishSpawnNotification publishes current_job, job_started_at, and
	// spawn_notification without touching any other frontmatter keys.
	PublishSpawnNotification(ctx context.Context, task lib.Task, jobName string) error
	// PublishFailure publishes a zombie failure: clears current_job and atomically
	// bumps trigger_count by 1 via a paired IncrementFrontmatterCommand. Leaves
	// phase, status, and assignee untouched so the existing trigger_count retry
	// cap (applyTriggerCap in task/controller/pkg/result/result_writer.go) handles
	// eventual operator-inbox escalation. Idempotent per current_job via a TTL'd
	// LRU; concurrent classifications for the same job emit one event.
	PublishFailure(ctx context.Context, task lib.Task, jobName string, reason string) error
	// PublishClearCurrentJob clears current_job in the vault task frontmatter
	// (writes "" only — no body section, no trigger_count bump, no status/phase
	// touch). Used on the job-success path, where the agent already published its
	// real result and the task is about to be evicted from the TaskStore: without
	// this clear, a succeeded job leaves current_job/job_started_at set forever
	// (the zombie sweeper is blind to it because the task left Snapshot()). The
	// write is idempotent (empty → empty is a no-op visually) and deduped per
	// jobName by the same TTL'd LRU as PublishFailure, so the informer's repeated
	// terminal re-deliveries do not spam Kafka.
	PublishClearCurrentJob(ctx context.Context, task lib.Task, jobName string) error
	// PublishIncrementTriggerCount sends an IncrementFrontmatterCommand that atomically
	// increments trigger_count by 1. Must complete before SpawnJob is called.
	PublishIncrementTriggerCount(ctx context.Context, task lib.Task) error
	// PublishSetTriggerScope sends an UpdateFrontmatterCommand that records a
	// trigger_scope and sets trigger_count to count, in one atomic write. Called
	// instead of PublishIncrementTriggerCount in the two cases where the scope
	// itself has to be written:
	//
	//   - the scope CHANGED (count 1): a re-dispatch against a new phase or a new
	//     ref starts from a fresh budget, counting the spawn about to happen.
	//   - the scope was ABSENT (count = existing trigger_count + 1): the task
	//     predates the field and adopts the current scope WITHOUT a budget reset.
	//     Absent must not be treated as changed — every task in flight today has no
	//     trigger_scope, so resetting on absence would hand each of them a free
	//     budget on the first event after deploy, including one already at cap.
	//
	// This is an UpdateFrontmatterCommand rather than a result publish on purpose.
	// trigger_count is controller-owned: the result writer's ownership guard
	// (MergeFrontmatter in task/controller) discards any incoming counter value, so
	// a reset routed through a result payload would be silently dropped. The atomic
	// frontmatter commands take a different path (buildUpdateModifyFn), which applies
	// Updates directly to the on-disk frontmatter — an explicit, intentional write
	// rather than a stale spawn-time snapshot.
	PublishSetTriggerScope(ctx context.Context, task lib.Task, scope string, count int) error
	// PublishTypeMismatchFailure publishes a synthetic failure when the task's task_type
	// is not in the agent's effective type set. Clears assignee and current_job so the
	// task surfaces in the operator inbox via assignee=="" filter. Does not bump
	// trigger_count or retry_count.
	PublishTypeMismatchFailure(ctx context.Context, task lib.Task, reason string) error
	// PublishRaw publishes a raw payload for testing error paths.
	PublishRaw(ctx context.Context, operation base.CommandOperation, payload interface{}) error
}

ResultPublisher publishes agent-task-v1-request commands to Kafka so the controller writes them to the vault task file.

func NewResultPublisher

func NewResultPublisher(
	syncProducer libkafka.SyncProducer,
	topicPrefix base.TopicPrefix,
	currentDateTime libtime.CurrentDateTimeGetter,
) ResultPublisher

NewResultPublisher creates a ResultPublisher.

type TaskStore

type TaskStore struct {
	// contains filtered or unexported fields
}

TaskStore is a thread-safe map from TaskIdentifier to Task. It is populated when the executor spawns a Job and consumed by the job informer when publishing synthetic failure results.

func NewTaskStore

func NewTaskStore() *TaskStore

NewTaskStore creates an empty TaskStore.

func (*TaskStore) Delete

func (s *TaskStore) Delete(id lib.TaskIdentifier)

Delete removes the task for the given identifier (called on job termination).

func (*TaskStore) Load

func (s *TaskStore) Load(id lib.TaskIdentifier) (lib.Task, bool)

Load retrieves the task for the given identifier.

func (*TaskStore) Snapshot

func (s *TaskStore) Snapshot() map[lib.TaskIdentifier]lib.Task

Snapshot returns a shallow copy of the current task map for read-only iteration. Safe to call concurrently with Store/Delete; the returned map is owned by the caller.

func (*TaskStore) Store

func (s *TaskStore) Store(id lib.TaskIdentifier, task lib.Task)

Store saves the task for the given identifier (called on job spawn).

type ZombieReason

type ZombieReason string

ZombieReason is the closed set of machine-readable reason strings emitted in the ## Failure body section. Operators grep on these values to triage. OOM kills use the dedicated pod_oom_killed value so operators can grep for OOM specifically; other non-zero pod terminations fall back to the generic pod_crash_no_stdout. Adding a new value requires updating this list and the documentation; renaming or removing a value is a breaking change to the on-disk task body contract.

const (
	ZombieReasonImagePullBackOff  ZombieReason = "image_pull_backoff"
	ZombieReasonPodEvicted        ZombieReason = "pod_evicted"
	ZombieReasonDeadlineExceeded  ZombieReason = "deadline_exceeded"
	ZombieReasonPodNotScheduled   ZombieReason = "pod_not_scheduled"
	ZombieReasonPodCrashNoStdout  ZombieReason = "pod_crash_no_stdout"
	ZombieReasonExecutorWatchLost ZombieReason = "executor_watch_lost"
	ZombieReasonTypeMismatch      ZombieReason = "type_mismatch"
	ZombieReasonPodOOMKilled      ZombieReason = "pod_oom_killed"
)

func JobFailureReason

func JobFailureReason(job *batchv1.Job) ZombieReason

JobFailureReason maps a failed Job's conditions to a ZombieReason. Returns ZombieReasonDeadlineExceeded when a Failed condition has Reason "DeadlineExceeded" (the kubelet killed the pod for running past activeDeadlineSeconds). Returns the generic pod-failure reason ZombieReasonPodCrashNoStdout for "BackoffLimitExceeded" (the Job exhausted its retry backoff limit — a distinct reason from a deadline kill; HandleJob resolves the pod's terminated reason + exit code and surfaces them in the published reason) and for any other Failed condition (the pod terminated non-zero and no AgentResult was observed; the Job-condition informer only fires AFTER terminal state, so absence of an AgentResult is implicit at this point).

func (ZombieReason) String

func (r ZombieReason) String() string

String returns the reason as a string (for use with PublishFailure).

type ZombieSweeper

type ZombieSweeper interface {
	// Run blocks until ctx is cancelled. Each tick (interval sourced from the
	// first non-nil ConfigSpec.ZombieSweeperIntervalSeconds across the resolver's
	// configs, else DefaultZombieSweeperIntervalSeconds) it calls SweepOnce.
	Run(ctx context.Context) error
	// SweepOnce performs a single sweep pass. Exposed for unit tests so they
	// do not have to manage tickers. Returns an error only on context
	// cancellation paths; per-task classification errors are logged.
	SweepOnce(ctx context.Context) error
}

ZombieSweeper is a background goroutine that periodically classifies stuck tasks as zombies and emits failure events. It is the safety net for the informer-driven paths in JobWatcher (which handle the cases k8s notifies us about). The sweeper handles: pods unschedulable beyond a grace window, executor restart losing watch on a Job, and any deadline path the informer misses (Job-condition deferred indefinitely, informer cache drift).

func NewZombieSweeper

func NewZombieSweeper(
	jobWatcher JobWatcher,
	namespace libk8s.Namespace,
	taskStore *TaskStore,
	publisher ResultPublisher,
	configProvider EventHandlerConfig,
	currentDateTime libtime.CurrentDateTimeGetter,
) ZombieSweeper

NewZombieSweeper creates a ZombieSweeper. The JobWatcher is held by reference (not its lister) because the lister is only populated after JobWatcher.Run completes its cache sync; service.Run starts all components concurrently, so extracting the lister at wiring time would capture nil. SweepOnce resolves the lister lazily on every tick and skips the tick if it is not yet ready.

Directories

Path Synopsis
mocks
Code generated by counterfeiter.
Code generated by counterfeiter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL