nomad

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 31, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package nomad compares HCL job definitions against a live Nomad cluster and reports any diffs it finds.

Index

Constants

View Source
const MaxPlanDiffObjectDepth = 200

MaxPlanDiffObjectDepth caps how deep nomad-gitops recurses into a Nomad plan-diff's Objects tree (classification, redaction; internal/server/render.go mirrors this cap for text rendering). A legitimate job spec never nests anywhere near this deep — real ObjectDiff trees bottom out within a handful of levels (job/group/task/config). Depth beyond this cap can only come from a deliberately crafted HCL job spec, and traversal stops there rather than recursing without bound: unbounded recursion risks a stack-overflow crash, which is not a recoverable panic and would take down the whole process, not just the one job's check.

View Source
const RedactedValue = "[REDACTED]"

RedactedValue replaces potentially sensitive values in plan diffs when secret redaction is enabled (--redact-secrets, on by default).

Variables

This section is empty.

Functions

func RedactJobDiff

func RedactJobDiff(d *nomadapi.JobDiff) int

RedactJobDiff replaces potentially sensitive field values throughout d with RedactedValue, in place, and annotates each redacted field. The diff structure (field names, added/deleted/edited types, nesting) is preserved so the rendered output still reads like a plan diff. Returns the number of fields redacted.

func ValidUpdatePolicy

func ValidUpdatePolicy(s string) bool

ValidUpdatePolicy reports whether s is a recognised policy value.

Types

type ApplyAction

type ApplyAction string

ApplyAction is the disposition of a detected diff: whether it will be applied and, if not, the reason.

const (
	// ApplyActionQueued means an update was enqueued and will be applied.
	ApplyActionQueued ApplyAction = "queued"
	// ApplyActionPolicyBlocked means the effective update policy disallows it.
	ApplyActionPolicyBlocked ApplyAction = "blocked_by_policy"
	// ApplyActionPreExisting means the drift pre-dated the scope change that
	// brought it into scope — the job's opt-in, or a policy widening.
	ApplyActionPreExisting ApplyAction = "blocked_preexisting_drift"
	// ApplyActionCreationBlocked means first-time registration is disabled.
	ApplyActionCreationBlocked ApplyAction = "blocked_creation_disabled"
	// ApplyActionMetaOnly means the diff is confined to our own meta keys.
	ApplyActionMetaOnly ApplyAction = "skipped_meta_only"
	// ApplyActionObservationOnly means a job is running in Nomad with no HCL,
	// and deregistration is disabled (or the job is not deregister-eligible):
	// it is left running, observation-only.
	ApplyActionObservationOnly ApplyAction = "observation_only"
	// ApplyActionDeregisterQueued means an orphaned job will be deregistered.
	ApplyActionDeregisterQueued ApplyAction = "queued_deregister"
	// ApplyActionDeregisterGrace means an orphaned job is deregister-eligible
	// but its grace period has not yet elapsed.
	ApplyActionDeregisterGrace ApplyAction = "deregister_pending_grace"
	// ApplyActionNoChange means the only diff is autoscaler-owned churn.
	ApplyActionNoChange ApplyAction = "no_actionable_change"
	// ApplyActionKnownFailed means the flap-loop guard is holding the apply:
	// the HCL spec matches a recent Nomad job version whose deployment failed,
	// so re-applying it would re-enter a known failure. Released when Git moves
	// to a spec that has not failed.
	ApplyActionKnownFailed ApplyAction = "blocked_known_failed"
)

func (ApplyAction) Describe

func (a ApplyAction) Describe() string

Describe returns a human-readable explanation for display.

type DiffClass

type DiffClass int

DiffClass categorises a plan diff for update-policy decisions.

const (
	// DiffClassNone means the diff contains no changes that Git owns —
	// either it is empty, or everything in it is autoscaler-owned
	// Count/Scaling churn. Nothing to apply.
	DiffClassNone DiffClass = iota
	// DiffClassImageOnly means every Git-owned change is a Docker image
	// reference (the "image" field inside a task's Config object), possibly
	// alongside nomad-gitops's own managed-meta keys.
	DiffClassImageOnly
	// DiffClassManagedMetaOnly means every Git-owned change is to one of
	// nomad-gitops's own managed-prefix meta keys (e.g. gitops_managed,
	// gitops_update_policy). These are not applied on their own by default:
	// re-registering a running job purely to push our keys onto it is
	// disruptive and unnecessary, since the HCL is already the source of
	// truth for them. They ride along the next real update.
	DiffClassManagedMetaOnly
	// DiffClassOther means the diff contains at least one Git-owned change
	// that is not an image reference or a managed-meta key.
	DiffClassOther
)

func (DiffClass) String

func (c DiffClass) String() string

type DiffType

type DiffType string

DiffType describes the relationship between a job in HCL and in Nomad.

const (
	// DiffTypeModified means the job exists in both HCL and Nomad but the
	// definitions differ (Nomad plan shows changes).
	DiffTypeModified DiffType = "modified"

	// DiffTypeMissingFromNomad means the job is defined in HCL but not
	// currently registered in Nomad.
	DiffTypeMissingFromNomad DiffType = "missing_from_nomad"

	// DiffTypeMissingFromHCL means the job is running in Nomad but there is
	// no corresponding HCL file in the repo.
	DiffTypeMissingFromHCL DiffType = "missing_from_hcl"
)

type Differ

type Differ struct {
	// contains filtered or unexported fields
}

Differ runs periodic diff checks and stores the latest results.

func NewDiffer

func NewDiffer(cfg *config.Config) (*Differ, error)

NewDiffer creates a Differ backed by a real Nomad API client, registering metrics into the default Prometheus registry.

func NewDifferWithRegistry

func NewDifferWithRegistry(cfg *config.Config, reg prometheus.Registerer) (*Differ, error)

NewDifferWithRegistry is like NewDiffer but registers metrics into reg rather than the default registry, so more than one real-client Differ can be built in a single process (e.g. tests exercising token resolution, or embedding) without a duplicate-registration panic.

func NewWithClient

func NewWithClient(cfg *config.Config, jobs NomadJobsClient) *Differ

NewWithClient creates a Differ with a custom jobs client, intended for tests.

func NewWithClientAndRegistry

func NewWithClientAndRegistry(cfg *config.Config, jobs NomadJobsClient, reg prometheus.Registerer) *Differ

NewWithClientAndRegistry creates a Differ with a custom jobs client and Prometheus registry. Use this in tests that need to inspect metric values.

func (*Differ) Check

func (d *Differ) Check(hclFiles map[string]string, commit string) error

Check compares the given HCL files (path → content) against the live Nomad cluster and stores the results. commit is recorded for informational purposes.

func (*Differ) Diffs

func (d *Differ) Diffs() ([]JobDiff, time.Time, string)

Diffs returns a snapshot of the latest diffs, the time they were computed, and the git commit they were computed against.

func (*Differ) ForceCheck

func (d *Differ) ForceCheck(hclFiles map[string]string, commit string) error

ForceCheck runs a diff check unconditionally because the Nomad state has exceeded the configured maximum staleness. Increments the staleness counter and delegates to Check.

func (*Differ) Ready

func (d *Differ) Ready() bool

Ready reports whether at least one diff check has completed successfully. Before the first check finishes, callers cannot distinguish "no drift" from "haven't checked yet", so they should treat the Differ as unavailable.

func (*Differ) RunApplier

func (d *Differ) RunApplier(ctx context.Context)

RunApplier drains the update queue until ctx is cancelled. It wakes on every enqueue and on a fallback ticker (--apply-interval). Detection and application are deliberately decoupled: a slow or failing apply never delays the next diff check.

func (*Differ) RunTokenRefresher

func (d *Differ) RunTokenRefresher(ctx context.Context)

RunTokenRefresher keeps the Nomad token current: in login mode it re-exchanges the workload-identity JWT before the ACL token expires; in file mode it re-reads the token file. It is a no-op for a static token or no token, and blocks until ctx is cancelled.

func (*Differ) SelectedJobs

func (d *Differ) SelectedJobs() ([]SelectedJob, time.Time, string)

SelectedJobs returns a snapshot of the jobs that matched the configured selection criteria during the last check, together with the reason each matched. The second and third return values are the same last-check time and commit as Diffs().

func (*Differ) SetHistorySource

func (d *Differ) SetHistorySource(h HistorySource)

SetHistorySource wires the git-history accessor used to detect pre-existing drift. Called once at startup after the watcher exists.

func (*Differ) Updates

func (d *Differ) Updates() []JobUpdate

Updates returns a snapshot of the update queue for the JSON API.

type HistorySource

type HistorySource interface {
	// FileAtParentOf returns the content of path at the first parent of the
	// named commit. ok is false when the commit is unknown, has no parent
	// (root commit), or the file is absent there. The lookup is keyed off the
	// commit being evaluated — not the repo's current HEAD — so the decision
	// stays consistent with the HCL snapshot passed to Check even if the
	// watcher pulls a newer commit concurrently.
	FileAtParentOf(commit, path string) (content string, ok bool)
}

HistorySource provides read-only access to prior git state, used to decide whether drift pre-dates a job entering management scope. *gitwatch.Watcher satisfies it. When nil, pre-existing-drift detection is disabled and drift reconciles normally.

type JobDiff

type JobDiff struct {
	JobID    string   `json:"job_id"`
	HCLFile  string   `json:"hcl_file,omitempty"` // empty for MissingFromHCL
	DiffType DiffType `json:"diff_type"`
	Detail   string   `json:"detail"`

	// ApplyAction records what nomad-gitops will do about this diff and,
	// when it will not apply it, why. Lets the API and web console explain
	// non-application without log scraping.
	ApplyAction ApplyAction `json:"apply_action,omitempty"`

	// ApplyDetail is an optional, more specific human-readable explanation
	// that refines ApplyAction with the actual values involved — for example,
	// which update policy blocked the change, where that policy came from, and
	// what would need to change. Empty when ApplyAction.Describe() already says
	// everything there is to say.
	ApplyDetail string `json:"apply_detail,omitempty"`

	// PlanDiff holds the structured diff from the Nomad plan API.
	// Only populated for DiffTypeModified entries.
	PlanDiff *nomadapi.JobDiff `json:"-"`
}

JobDiff describes a single divergence between the git repo and Nomad.

type JobUpdate

type JobUpdate struct {
	// UpdateID is <job_id>/<git_commit_short> — deliberately derived from
	// stable inputs so the same intent re-detected after a restart or a
	// failure is recognisably the same update.
	UpdateID string `json:"update_id"`

	JobID string `json:"job_id"`

	// HCLFile is the repo path that is the source of truth for this job.
	HCLFile string `json:"hcl_file,omitempty"`

	// GitCommit is the commit hash that triggered this update.
	GitCommit string `json:"git_commit"`

	Operation JobUpdateOperation `json:"operation"`
	Status    JobUpdateStatus    `json:"status"`

	// Policy is the effective update policy that allowed this update.
	Policy UpdatePolicy `json:"policy"`

	// NomadJobModifyIndex is the job's ModifyIndex at detection time, used
	// as the CAS token on Jobs.Register (EnforceIndex). Zero means the job
	// did not exist in Nomad at detection time.
	NomadJobModifyIndex uint64 `json:"nomad_job_modify_index"`

	// NomadRaftIndex is the cluster Raft index at detection time, recorded
	// for auditability.
	NomadRaftIndex uint64 `json:"nomad_raft_index"`

	DetectedAt string `json:"detected_at"`          // RFC3339
	AppliedAt  string `json:"applied_at,omitempty"` // RFC3339; empty until applied
	Error      string `json:"error,omitempty"`

	// Revert-only. RevertToVersion is the stable job version to roll back to.
	// RevertFromVersion is the failed version the job must still be at for the
	// revert to land (the enforcePriorVersion CAS guard).
	RevertToVersion   uint64 `json:"revert_to_version,omitempty"`
	RevertFromVersion uint64 `json:"revert_from_version,omitempty"`
	// contains filtered or unexported fields
}

JobUpdate represents a single intended change to a Nomad job, derived from a detected diff between Git and the cluster. A JobDiff is an observation; a JobUpdate is an intended transition.

type JobUpdateOperation

type JobUpdateOperation string

JobUpdateOperation is the kind of change a JobUpdate applies.

const (
	JobUpdateOperationRegister   JobUpdateOperation = "REGISTER"
	JobUpdateOperationDeregister JobUpdateOperation = "DEREGISTER"
	// JobUpdateOperationRevert rolls a job back to a prior stable version after
	// a failed deployment (active rollback, for jobs without auto_revert).
	JobUpdateOperationRevert JobUpdateOperation = "REVERT"
)

type JobUpdateStatus

type JobUpdateStatus string

JobUpdateStatus is the lifecycle state of a JobUpdate.

const (
	JobUpdateStatusPending    JobUpdateStatus = "PENDING"
	JobUpdateStatusInProgress JobUpdateStatus = "IN_PROGRESS"
	JobUpdateStatusSucceeded  JobUpdateStatus = "SUCCEEDED"
	JobUpdateStatusFailed     JobUpdateStatus = "FAILED"
	JobUpdateStatusSuperseded JobUpdateStatus = "SUPERSEDED"
)

type NomadJobsClient

type NomadJobsClient interface {
	ParseHCL(jobHCL string, canonicalize bool) (*nomadapi.Job, error)
	Plan(job *nomadapi.Job, diff bool, q *nomadapi.WriteOptions) (*nomadapi.JobPlanResponse, *nomadapi.WriteMeta, error)
	Info(jobID string, q *nomadapi.QueryOptions) (*nomadapi.Job, *nomadapi.QueryMeta, error)
	List(q *nomadapi.QueryOptions) ([]*nomadapi.JobListStub, *nomadapi.QueryMeta, error)
	RegisterOpts(job *nomadapi.Job, opts *nomadapi.RegisterOptions, q *nomadapi.WriteOptions) (*nomadapi.JobRegisterResponse, *nomadapi.WriteMeta, error)
	Deregister(jobID string, purge bool, q *nomadapi.WriteOptions) (string, *nomadapi.WriteMeta, error)
	// Versions returns a job's retained version history (most recent first).
	Versions(jobID string, diffs bool, q *nomadapi.QueryOptions) ([]*nomadapi.Job, []*nomadapi.JobDiff, *nomadapi.QueryMeta, error)
	// Deployments returns a job's deployments (most recent first).
	Deployments(jobID string, all bool, q *nomadapi.QueryOptions) ([]*nomadapi.Deployment, *nomadapi.QueryMeta, error)
	// LatestDeployment returns the job's most recent deployment, or nil.
	LatestDeployment(jobID string, q *nomadapi.QueryOptions) (*nomadapi.Deployment, *nomadapi.QueryMeta, error)
	// Revert rolls a job back to a prior version. enforcePriorVersion, when
	// non-nil, is a CAS guard: the revert only lands if the job is still at that
	// version.
	Revert(jobID string, version uint64, enforcePriorVersion *uint64, q *nomadapi.WriteOptions, consulToken, vaultToken string) (*nomadapi.JobRegisterResponse, *nomadapi.WriteMeta, error)
	// TagVersion attaches a durable name to a job version so it survives GC.
	TagVersion(jobID string, version uint64, name, description string, q *nomadapi.WriteOptions) (*nomadapi.WriteMeta, error)
}

NomadJobsClient is the subset of the Nomad API jobs client we use. The concrete *nomadapi.Jobs satisfies this interface; tests inject a mock.

type SelectedJob

type SelectedJob struct {
	JobID  string          `json:"job_id"`
	Reason SelectionReason `json:"selection_reason"`
}

SelectedJob records a job that matched the configured selection criteria and the reason it was included.

type SelectionReason

type SelectionReason string

SelectionReason describes why a job was included in the watched set.

const (
	// SelectionReasonGlob means the job was selected by the job-selector-glob pattern.
	SelectionReasonGlob SelectionReason = "glob"
	// SelectionReasonMeta means the job was selected by the managed-meta-prefix key.
	SelectionReasonMeta SelectionReason = "meta"
	// SelectionReasonBoth means the job matched both the glob and the meta key.
	SelectionReasonBoth SelectionReason = "both"
)

type UpdatePolicy

type UpdatePolicy string

UpdatePolicy controls how much detected drift may be applied to a job automatically. It is declared per job in HCL meta (<prefix>_update_policy) and falls back to --default-update-policy.

const (
	// UpdatePolicyFull applies any detected drift.
	UpdatePolicyFull UpdatePolicy = "full"
	// UpdatePolicyImageOnly applies drift only when the entire plan diff is
	// confined to Docker image references.
	UpdatePolicyImageOnly UpdatePolicy = "image-only"
	// UpdatePolicyNone detects and surfaces drift but never applies it.
	UpdatePolicyNone UpdatePolicy = "none"
)

type UpdateQueue

type UpdateQueue struct {
	// contains filtered or unexported fields
}

UpdateQueue is the in-memory queue between detection and application. Restart loses it by design: the next diff cycle recreates any update whose drift still exists, and CAS plus re-planning make a re-apply harmless. See docs/design/gitops-job-updates.md ("Restart safety and recovery").

func NewUpdateQueue

func NewUpdateQueue() *UpdateQueue

NewUpdateQueue returns an empty queue.

func (*UpdateQueue) Complete

func (q *UpdateQueue) Complete(updateID string, status JobUpdateStatus, appliedIndex uint64, errMsg string)

Complete records the outcome of an apply attempt.

func (*UpdateQueue) Enqueue

func (q *UpdateQueue) Enqueue(u JobUpdate) (superseded int)

Enqueue records an intended update. Rules, keyed on any existing entry with the same UpdateID (same job, same commit):

  • PENDING: refreshed in place (CAS token, job pointer) rather than duplicated — it has not started, so mutating it is safe.
  • IN_PROGRESS: left strictly untouched and no new entry is added. The applier reads the update's fields (CAS token, job pointer, preserveCounts) without holding the queue lock, so mutating an in-flight update would race it and could make the apply use a different token or job than it started with. If that apply fails, the next diff cycle re-enqueues against the by-then terminal record.
  • terminal: dropped and replaced with a fresh PENDING entry — the same intent is being retried after a failure or a cluster-side change.

A PENDING update for the same job with a *different* UpdateID (a newer commit arrived before the old one applied) is marked SUPERSEDED; the most recent intended state wins. An IN_PROGRESS update for a different UpdateID is also left alone for the same race reason.

Returns the number of updates marked SUPERSEDED by this enqueue.

func (*UpdateQueue) NextPending

func (q *UpdateQueue) NextPending() *JobUpdate

NextPending returns the oldest PENDING update, marking it IN_PROGRESS, or nil when nothing is waiting.

func (*UpdateQueue) PendingCount

func (q *UpdateQueue) PendingCount() int

PendingCount returns the number of PENDING updates.

func (*UpdateQueue) Snapshot

func (q *UpdateQueue) Snapshot() []JobUpdate

Snapshot returns a copy of all queue entries, newest last. The internal job pointer is not exposed.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL