Documentation
¶
Overview ¶
Package nomad compares HCL job definitions against a live Nomad cluster and reports any diffs it finds.
Index ¶
- Constants
- func RedactJobDiff(d *nomadapi.JobDiff) int
- func ValidUpdatePolicy(s string) bool
- type ApplyAction
- type DiffClass
- type DiffType
- type Differ
- func NewDiffer(cfg *config.Config) (*Differ, error)
- func NewDifferWithRegistry(cfg *config.Config, reg prometheus.Registerer) (*Differ, error)
- func NewWithClient(cfg *config.Config, jobs NomadJobsClient) *Differ
- func NewWithClientAndRegistry(cfg *config.Config, jobs NomadJobsClient, reg prometheus.Registerer) *Differ
- func (d *Differ) Check(hclFiles map[string]string, commit string) error
- func (d *Differ) Diffs() ([]JobDiff, time.Time, string)
- func (d *Differ) ForceCheck(hclFiles map[string]string, commit string) error
- func (d *Differ) Ready() bool
- func (d *Differ) RunApplier(ctx context.Context)
- func (d *Differ) RunTokenRefresher(ctx context.Context)
- func (d *Differ) SelectedJobs() ([]SelectedJob, time.Time, string)
- func (d *Differ) SetHistorySource(h HistorySource)
- func (d *Differ) Updates() []JobUpdate
- type HistorySource
- type JobDiff
- type JobUpdate
- type JobUpdateOperation
- type JobUpdateStatus
- type NomadJobsClient
- type SelectedJob
- type SelectionReason
- type UpdatePolicy
- type UpdateQueue
- func (q *UpdateQueue) Complete(updateID string, status JobUpdateStatus, appliedIndex uint64, errMsg string)
- func (q *UpdateQueue) Enqueue(u JobUpdate) (superseded int)
- func (q *UpdateQueue) NextPending() *JobUpdate
- func (q *UpdateQueue) PendingCount() int
- func (q *UpdateQueue) Snapshot() []JobUpdate
Constants ¶
const MaxPlanDiffObjectDepth = 200
MaxPlanDiffObjectDepth caps how deep nomad-gitops recurses into a Nomad plan-diff's Objects tree (classification, redaction; internal/server/render.go mirrors this cap for text rendering). A legitimate job spec never nests anywhere near this deep — real ObjectDiff trees bottom out within a handful of levels (job/group/task/config). Depth beyond this cap can only come from a deliberately crafted HCL job spec, and traversal stops there rather than recursing without bound: unbounded recursion risks a stack-overflow crash, which is not a recoverable panic and would take down the whole process, not just the one job's check.
const RedactedValue = "[REDACTED]"
RedactedValue replaces potentially sensitive values in plan diffs when secret redaction is enabled (--redact-secrets, on by default).
Variables ¶
This section is empty.
Functions ¶
func RedactJobDiff ¶
RedactJobDiff replaces potentially sensitive field values throughout d with RedactedValue, in place, and annotates each redacted field. The diff structure (field names, added/deleted/edited types, nesting) is preserved so the rendered output still reads like a plan diff. Returns the number of fields redacted.
func ValidUpdatePolicy ¶
ValidUpdatePolicy reports whether s is a recognised policy value.
Types ¶
type ApplyAction ¶
type ApplyAction string
ApplyAction is the disposition of a detected diff: whether it will be applied and, if not, the reason.
const ( // ApplyActionQueued means an update was enqueued and will be applied. ApplyActionQueued ApplyAction = "queued" // ApplyActionPolicyBlocked means the effective update policy disallows it. ApplyActionPolicyBlocked ApplyAction = "blocked_by_policy" // ApplyActionPreExisting means the drift pre-dated the scope change that // brought it into scope — the job's opt-in, or a policy widening. ApplyActionPreExisting ApplyAction = "blocked_preexisting_drift" // ApplyActionCreationBlocked means first-time registration is disabled. ApplyActionCreationBlocked ApplyAction = "blocked_creation_disabled" // ApplyActionMetaOnly means the diff is confined to our own meta keys. ApplyActionMetaOnly ApplyAction = "skipped_meta_only" // ApplyActionObservationOnly means a job is running in Nomad with no HCL, // and deregistration is disabled (or the job is not deregister-eligible): // it is left running, observation-only. ApplyActionObservationOnly ApplyAction = "observation_only" // ApplyActionDeregisterQueued means an orphaned job will be deregistered. ApplyActionDeregisterQueued ApplyAction = "queued_deregister" // ApplyActionDeregisterGrace means an orphaned job is deregister-eligible // but its grace period has not yet elapsed. ApplyActionDeregisterGrace ApplyAction = "deregister_pending_grace" // ApplyActionNoChange means the only diff is autoscaler-owned churn. ApplyActionNoChange ApplyAction = "no_actionable_change" // ApplyActionKnownFailed means the flap-loop guard is holding the apply: // the HCL spec matches a recent Nomad job version whose deployment failed, // so re-applying it would re-enter a known failure. Released when Git moves // to a spec that has not failed. ApplyActionKnownFailed ApplyAction = "blocked_known_failed" )
func (ApplyAction) Describe ¶
func (a ApplyAction) Describe() string
Describe returns a human-readable explanation for display.
type DiffClass ¶
type DiffClass int
DiffClass categorises a plan diff for update-policy decisions.
const ( // DiffClassNone means the diff contains no changes that Git owns — // either it is empty, or everything in it is autoscaler-owned // Count/Scaling churn. Nothing to apply. DiffClassNone DiffClass = iota // DiffClassImageOnly means every Git-owned change is a Docker image // reference (the "image" field inside a task's Config object), possibly // alongside nomad-gitops's own managed-meta keys. DiffClassImageOnly // DiffClassManagedMetaOnly means every Git-owned change is to one of // nomad-gitops's own managed-prefix meta keys (e.g. gitops_managed, // gitops_update_policy). These are not applied on their own by default: // re-registering a running job purely to push our keys onto it is // disruptive and unnecessary, since the HCL is already the source of // truth for them. They ride along the next real update. DiffClassManagedMetaOnly // DiffClassOther means the diff contains at least one Git-owned change // that is not an image reference or a managed-meta key. DiffClassOther )
type DiffType ¶
type DiffType string
DiffType describes the relationship between a job in HCL and in Nomad.
const ( // DiffTypeModified means the job exists in both HCL and Nomad but the // definitions differ (Nomad plan shows changes). DiffTypeModified DiffType = "modified" // DiffTypeMissingFromNomad means the job is defined in HCL but not // currently registered in Nomad. DiffTypeMissingFromNomad DiffType = "missing_from_nomad" // DiffTypeMissingFromHCL means the job is running in Nomad but there is // no corresponding HCL file in the repo. DiffTypeMissingFromHCL DiffType = "missing_from_hcl" )
type Differ ¶
type Differ struct {
// contains filtered or unexported fields
}
Differ runs periodic diff checks and stores the latest results.
func NewDiffer ¶
NewDiffer creates a Differ backed by a real Nomad API client, registering metrics into the default Prometheus registry.
func NewDifferWithRegistry ¶
func NewDifferWithRegistry(cfg *config.Config, reg prometheus.Registerer) (*Differ, error)
NewDifferWithRegistry is like NewDiffer but registers metrics into reg rather than the default registry, so more than one real-client Differ can be built in a single process (e.g. tests exercising token resolution, or embedding) without a duplicate-registration panic.
func NewWithClient ¶
func NewWithClient(cfg *config.Config, jobs NomadJobsClient) *Differ
NewWithClient creates a Differ with a custom jobs client, intended for tests.
func NewWithClientAndRegistry ¶
func NewWithClientAndRegistry(cfg *config.Config, jobs NomadJobsClient, reg prometheus.Registerer) *Differ
NewWithClientAndRegistry creates a Differ with a custom jobs client and Prometheus registry. Use this in tests that need to inspect metric values.
func (*Differ) Check ¶
Check compares the given HCL files (path → content) against the live Nomad cluster and stores the results. commit is recorded for informational purposes.
func (*Differ) Diffs ¶
Diffs returns a snapshot of the latest diffs, the time they were computed, and the git commit they were computed against.
func (*Differ) ForceCheck ¶
ForceCheck runs a diff check unconditionally because the Nomad state has exceeded the configured maximum staleness. Increments the staleness counter and delegates to Check.
func (*Differ) Ready ¶
Ready reports whether at least one diff check has completed successfully. Before the first check finishes, callers cannot distinguish "no drift" from "haven't checked yet", so they should treat the Differ as unavailable.
func (*Differ) RunApplier ¶
RunApplier drains the update queue until ctx is cancelled. It wakes on every enqueue and on a fallback ticker (--apply-interval). Detection and application are deliberately decoupled: a slow or failing apply never delays the next diff check.
func (*Differ) RunTokenRefresher ¶
RunTokenRefresher keeps the Nomad token current: in login mode it re-exchanges the workload-identity JWT before the ACL token expires; in file mode it re-reads the token file. It is a no-op for a static token or no token, and blocks until ctx is cancelled.
func (*Differ) SelectedJobs ¶
func (d *Differ) SelectedJobs() ([]SelectedJob, time.Time, string)
SelectedJobs returns a snapshot of the jobs that matched the configured selection criteria during the last check, together with the reason each matched. The second and third return values are the same last-check time and commit as Diffs().
func (*Differ) SetHistorySource ¶
func (d *Differ) SetHistorySource(h HistorySource)
SetHistorySource wires the git-history accessor used to detect pre-existing drift. Called once at startup after the watcher exists.
type HistorySource ¶
type HistorySource interface {
// FileAtParentOf returns the content of path at the first parent of the
// named commit. ok is false when the commit is unknown, has no parent
// (root commit), or the file is absent there. The lookup is keyed off the
// commit being evaluated — not the repo's current HEAD — so the decision
// stays consistent with the HCL snapshot passed to Check even if the
// watcher pulls a newer commit concurrently.
FileAtParentOf(commit, path string) (content string, ok bool)
}
HistorySource provides read-only access to prior git state, used to decide whether drift pre-dates a job entering management scope. *gitwatch.Watcher satisfies it. When nil, pre-existing-drift detection is disabled and drift reconciles normally.
type JobDiff ¶
type JobDiff struct {
JobID string `json:"job_id"`
HCLFile string `json:"hcl_file,omitempty"` // empty for MissingFromHCL
DiffType DiffType `json:"diff_type"`
Detail string `json:"detail"`
// ApplyAction records what nomad-gitops will do about this diff and,
// when it will not apply it, why. Lets the API and web console explain
// non-application without log scraping.
ApplyAction ApplyAction `json:"apply_action,omitempty"`
// ApplyDetail is an optional, more specific human-readable explanation
// that refines ApplyAction with the actual values involved — for example,
// which update policy blocked the change, where that policy came from, and
// what would need to change. Empty when ApplyAction.Describe() already says
// everything there is to say.
ApplyDetail string `json:"apply_detail,omitempty"`
// PlanDiff holds the structured diff from the Nomad plan API.
// Only populated for DiffTypeModified entries.
PlanDiff *nomadapi.JobDiff `json:"-"`
}
JobDiff describes a single divergence between the git repo and Nomad.
type JobUpdate ¶
type JobUpdate struct {
// UpdateID is <job_id>/<git_commit_short> — deliberately derived from
// stable inputs so the same intent re-detected after a restart or a
// failure is recognisably the same update.
UpdateID string `json:"update_id"`
JobID string `json:"job_id"`
// HCLFile is the repo path that is the source of truth for this job.
HCLFile string `json:"hcl_file,omitempty"`
// GitCommit is the commit hash that triggered this update.
GitCommit string `json:"git_commit"`
Operation JobUpdateOperation `json:"operation"`
Status JobUpdateStatus `json:"status"`
// Policy is the effective update policy that allowed this update.
Policy UpdatePolicy `json:"policy"`
// NomadJobModifyIndex is the job's ModifyIndex at detection time, used
// as the CAS token on Jobs.Register (EnforceIndex). Zero means the job
// did not exist in Nomad at detection time.
NomadJobModifyIndex uint64 `json:"nomad_job_modify_index"`
// NomadRaftIndex is the cluster Raft index at detection time, recorded
// for auditability.
NomadRaftIndex uint64 `json:"nomad_raft_index"`
DetectedAt string `json:"detected_at"` // RFC3339
AppliedAt string `json:"applied_at,omitempty"` // RFC3339; empty until applied
Error string `json:"error,omitempty"`
// Revert-only. RevertToVersion is the stable job version to roll back to.
// RevertFromVersion is the failed version the job must still be at for the
// revert to land (the enforcePriorVersion CAS guard).
RevertToVersion uint64 `json:"revert_to_version,omitempty"`
RevertFromVersion uint64 `json:"revert_from_version,omitempty"`
// contains filtered or unexported fields
}
JobUpdate represents a single intended change to a Nomad job, derived from a detected diff between Git and the cluster. A JobDiff is an observation; a JobUpdate is an intended transition.
type JobUpdateOperation ¶
type JobUpdateOperation string
JobUpdateOperation is the kind of change a JobUpdate applies.
const ( JobUpdateOperationRegister JobUpdateOperation = "REGISTER" JobUpdateOperationDeregister JobUpdateOperation = "DEREGISTER" // JobUpdateOperationRevert rolls a job back to a prior stable version after // a failed deployment (active rollback, for jobs without auto_revert). JobUpdateOperationRevert JobUpdateOperation = "REVERT" )
type JobUpdateStatus ¶
type JobUpdateStatus string
JobUpdateStatus is the lifecycle state of a JobUpdate.
const ( JobUpdateStatusPending JobUpdateStatus = "PENDING" JobUpdateStatusInProgress JobUpdateStatus = "IN_PROGRESS" JobUpdateStatusSucceeded JobUpdateStatus = "SUCCEEDED" JobUpdateStatusFailed JobUpdateStatus = "FAILED" JobUpdateStatusSuperseded JobUpdateStatus = "SUPERSEDED" )
type NomadJobsClient ¶
type NomadJobsClient interface {
ParseHCL(jobHCL string, canonicalize bool) (*nomadapi.Job, error)
Plan(job *nomadapi.Job, diff bool, q *nomadapi.WriteOptions) (*nomadapi.JobPlanResponse, *nomadapi.WriteMeta, error)
Info(jobID string, q *nomadapi.QueryOptions) (*nomadapi.Job, *nomadapi.QueryMeta, error)
List(q *nomadapi.QueryOptions) ([]*nomadapi.JobListStub, *nomadapi.QueryMeta, error)
RegisterOpts(job *nomadapi.Job, opts *nomadapi.RegisterOptions, q *nomadapi.WriteOptions) (*nomadapi.JobRegisterResponse, *nomadapi.WriteMeta, error)
Deregister(jobID string, purge bool, q *nomadapi.WriteOptions) (string, *nomadapi.WriteMeta, error)
// Versions returns a job's retained version history (most recent first).
Versions(jobID string, diffs bool, q *nomadapi.QueryOptions) ([]*nomadapi.Job, []*nomadapi.JobDiff, *nomadapi.QueryMeta, error)
// Deployments returns a job's deployments (most recent first).
Deployments(jobID string, all bool, q *nomadapi.QueryOptions) ([]*nomadapi.Deployment, *nomadapi.QueryMeta, error)
// LatestDeployment returns the job's most recent deployment, or nil.
LatestDeployment(jobID string, q *nomadapi.QueryOptions) (*nomadapi.Deployment, *nomadapi.QueryMeta, error)
// Revert rolls a job back to a prior version. enforcePriorVersion, when
// non-nil, is a CAS guard: the revert only lands if the job is still at that
// version.
Revert(jobID string, version uint64, enforcePriorVersion *uint64, q *nomadapi.WriteOptions, consulToken, vaultToken string) (*nomadapi.JobRegisterResponse, *nomadapi.WriteMeta, error)
// TagVersion attaches a durable name to a job version so it survives GC.
TagVersion(jobID string, version uint64, name, description string, q *nomadapi.WriteOptions) (*nomadapi.WriteMeta, error)
}
NomadJobsClient is the subset of the Nomad API jobs client we use. The concrete *nomadapi.Jobs satisfies this interface; tests inject a mock.
type SelectedJob ¶
type SelectedJob struct {
JobID string `json:"job_id"`
Reason SelectionReason `json:"selection_reason"`
}
SelectedJob records a job that matched the configured selection criteria and the reason it was included.
type SelectionReason ¶
type SelectionReason string
SelectionReason describes why a job was included in the watched set.
const ( // SelectionReasonGlob means the job was selected by the job-selector-glob pattern. SelectionReasonGlob SelectionReason = "glob" // SelectionReasonMeta means the job was selected by the managed-meta-prefix key. SelectionReasonMeta SelectionReason = "meta" // SelectionReasonBoth means the job matched both the glob and the meta key. SelectionReasonBoth SelectionReason = "both" )
type UpdatePolicy ¶
type UpdatePolicy string
UpdatePolicy controls how much detected drift may be applied to a job automatically. It is declared per job in HCL meta (<prefix>_update_policy) and falls back to --default-update-policy.
const ( // UpdatePolicyFull applies any detected drift. UpdatePolicyFull UpdatePolicy = "full" // UpdatePolicyImageOnly applies drift only when the entire plan diff is // confined to Docker image references. UpdatePolicyImageOnly UpdatePolicy = "image-only" // UpdatePolicyNone detects and surfaces drift but never applies it. UpdatePolicyNone UpdatePolicy = "none" )
type UpdateQueue ¶
type UpdateQueue struct {
// contains filtered or unexported fields
}
UpdateQueue is the in-memory queue between detection and application. Restart loses it by design: the next diff cycle recreates any update whose drift still exists, and CAS plus re-planning make a re-apply harmless. See docs/design/gitops-job-updates.md ("Restart safety and recovery").
func (*UpdateQueue) Complete ¶
func (q *UpdateQueue) Complete(updateID string, status JobUpdateStatus, appliedIndex uint64, errMsg string)
Complete records the outcome of an apply attempt.
func (*UpdateQueue) Enqueue ¶
func (q *UpdateQueue) Enqueue(u JobUpdate) (superseded int)
Enqueue records an intended update. Rules, keyed on any existing entry with the same UpdateID (same job, same commit):
- PENDING: refreshed in place (CAS token, job pointer) rather than duplicated — it has not started, so mutating it is safe.
- IN_PROGRESS: left strictly untouched and no new entry is added. The applier reads the update's fields (CAS token, job pointer, preserveCounts) without holding the queue lock, so mutating an in-flight update would race it and could make the apply use a different token or job than it started with. If that apply fails, the next diff cycle re-enqueues against the by-then terminal record.
- terminal: dropped and replaced with a fresh PENDING entry — the same intent is being retried after a failure or a cluster-side change.
A PENDING update for the same job with a *different* UpdateID (a newer commit arrived before the old one applied) is marked SUPERSEDED; the most recent intended state wins. An IN_PROGRESS update for a different UpdateID is also left alone for the same race reason.
Returns the number of updates marked SUPERSEDED by this enqueue.
func (*UpdateQueue) NextPending ¶
func (q *UpdateQueue) NextPending() *JobUpdate
NextPending returns the oldest PENDING update, marking it IN_PROGRESS, or nil when nothing is waiting.
func (*UpdateQueue) PendingCount ¶
func (q *UpdateQueue) PendingCount() int
PendingCount returns the number of PENDING updates.
func (*UpdateQueue) Snapshot ¶
func (q *UpdateQueue) Snapshot() []JobUpdate
Snapshot returns a copy of all queue entries, newest last. The internal job pointer is not exposed.