Documentation
¶
Overview ¶
Package gate implements the grafana-alertcheck coverage-and-classification gate.
Index ¶
- Constants
- func CheckBudget(t map[string]RuleTimings, measured map[string]time.Duration, concurrency int) error
- func CheckGrafanaVersion(version string) error
- func CheckStartupHandoff(t map[string]RuleTimings, measured map[string]time.Duration, first []Poll, ...) error
- func ParsePromDuration(s string) (time.Duration, error)
- func ReadLog(path string) (Header, []Poll, *time.Time, error)
- func ReadPidFile(path string) (int, error)
- func RunDaemonChild(ctx context.Context, cfg DaemonChildConfig) error
- func StartupSummary(from, to time.Time, global GlobalTimings) string
- func StopRecorder(ctx context.Context, cfg StopConfig) (*os.File, error)
- func VerifyNormalInstancesVisible(rules []StateRule) error
- func Watch(ctx context.Context, cfg WatchConfig) error
- type Clock
- type Config
- type CoverageResult
- type DaemonChildConfig
- type Definition
- func DefinitionsFromDatasource(rules []StateRule, dsUID, dsName string) []Definition
- func ListAllDefinitions(ctx context.Context, src Source) ([]Definition, error)
- func ParseDefinitions(body []byte) ([]Definition, error)
- func Resolve(defs []Definition, names []string, folder string) (resolved []Definition, notes []string, err error)
- func SelectByLabels(defs []Definition, include, exclude []LabelMatcher) ([]Definition, error)
- type GlobalThresholds
- type GlobalTimings
- type Header
- type Instance
- type LabelMatcher
- type LoggedRule
- type Observation
- type Outcome
- type Policy
- type Poll
- type PreexistingPolicy
- type RecordType
- type Reducer
- type Result
- type RetryExhaustedError
- type RuleKind
- type RuleRef
- type RuleSource
- type RuleThresholds
- type RuleTimings
- type RuleVerdict
- type Scheduler
- type Source
- type State
- type StateRule
- type StopConfig
- type SystemClock
- type Termination
- type TerminationKind
- type TransportError
- type UnobservableReason
- type Violation
- type WatchConfig
- type Writer
Constants ¶
const DaemonChildFlag = "--daemon-child"
DaemonChildFlag is the hidden flag the parent passes when it re-execs itself as the detached recorder. It is deliberately absent from the CLI's usage text: an operator never types it, and a child started by hand against a log no parent prepared fails immediately on the header read.
const LogSchemaVersion = 1
LogSchemaVersion is the version stamped into every log header. A log with any other value is a read error, never a best-effort read: the log is the gate's only evidence, and misreading a stale shape is a fail-open.
const ReadyFDFlag = "--ready-fd"
ReadyFDFlag names the inherited descriptor the child reports readiness on. "Ready" is a POSITIVE byte from the child (header read, flock taken, in its poll loop), never a timer heuristic — a timer can't tell a healthy child from one about to die on a slow runner.
const SkewHardLimit = 60 * time.Second
SkewHardLimit is the largest runner↔Grafana clock skew a run tolerates. Exported so the CLI reports it verbatim next to a measured skew.
Variables ¶
This section is empty.
Functions ¶
func CheckBudget ¶
func CheckBudget(t map[string]RuleTimings, measured map[string]time.Duration, concurrency int) error
CheckBudget proves at start that a fully resolved schedule can be served. t and measured are keyed by UID, and measured must carry every UID in t (a rule never measured cannot have its budget proved). It fails on any of three conditions:
- utilization — the long-run request rate exceeds the concurrency;
- a single rule's request cannot fit inside its own cadence;
- the burst bound — the slowest measured request is slower than the fleet's tightest cadence, which can open a mid-run gap beyond that rule's maxGap.
The message names only the two operator controls: concurrency and the alert list.
func CheckGrafanaVersion ¶
CheckGrafanaVersion enforces the supported range. An unparseable or out-of-range version is a hard error naming both what was found and what is supported: the response schemas this gate parses are only verified against that range, and trusting an unverified one is how a deprecation turns into a silent misread.
func CheckStartupHandoff ¶ added in v0.1.7
func CheckStartupHandoff(t map[string]RuleTimings, measured map[string]time.Duration, first []Poll, readyAt, windowOpen time.Time, concurrency int) error
CheckStartupHandoff proves the first poll after the measurement pass arrives before each rule's maxGap — a transient the steady-state budget cannot see. It simulates the poller's first cycles and fails on any first poll past it. windowOpen is the earliest instant the classification window can open.
func ParsePromDuration ¶
ParsePromDuration parses a Grafana/Prometheus-style duration ("1h30m", "1d", "1w"). Unlike time.ParseDuration, it accepts "d" and "w". "" and "0" are 0.
func ReadLog ¶
ReadLog reads the whole log once and returns its header, its polls in recorded order, and the sentinel time when one is present (nil when the recording never finished — check turns that into unobservable, never a pass).
Call this only after the writer has exited. Reading a log a writer can still append to can only produce a shorter window than the one that was recorded.
The parse rules are deliberately the crudest possible: the header must be line 1 with a matching schema version, and ANY unparseable line — including the last, or one after a sentinel — is an error, full stop. A truncated log is evidence something killed the recorder, which must not pass.
func ReadPidFile ¶
ReadPidFile is the other side of that contract: the pid of the recorder check must stop before it may read the log.
func RunDaemonChild ¶
func RunDaemonChild(ctx context.Context, cfg DaemonChildConfig) error
RunDaemonChild is the detached recorder. The CLI dispatches to it when it sees DaemonChildFlag; nothing else ever calls it.
It re-reads the log the parent wrote, restores the transition-marker state from the polls already in it, reopens the log for appending, takes the flock the parent released, and then polls until it is signalled or reaches Until.
func StartupSummary ¶
func StartupSummary(from, to time.Time, global GlobalTimings) string
StartupSummary formats the pre-run print an operator sees before the wait: the total planned run time and the rule (with its `for` value) that set transitionGrace. from/to are the requested classification window.
func StopRecorder ¶ added in v0.1.2
StopRecorder signals the recorder holding cfg.Log and waits for it to go. It returns the log under a shared flock — the caller must keep it open across any read and close it when done — even when no writer was found.
The recorder holds the log exclusively; readers and cleanup hold it shared. Shared is refused only by an exclusive holder, so contention always means the recorder: a pid can be reused, the lock cannot.
func VerifyNormalInstancesVisible ¶
VerifyNormalInstancesVisible checks, on a first observation, that the state endpoint really returns normal instances: if it ever stops, the reduction's "keep the non-normal" becomes "keep everything the API sent", a silent fail-open in the transition markers. Counts are summed over every totals key whose lowercased name is "normal" or "inactive" — never a literal key, since the vocabulary is mixed and its case has drifted.
func Watch ¶
func Watch(ctx context.Context, cfg WatchConfig) error
Watch is the record step's parent process, returning only once the window is genuinely being recorded (version gate, resolve, one observation per rule, budget check, header with ReadyAt, then detach and await the child's readiness). The first-observation wait is what surfaces auth, name-resolution and parse failures before deploy.sh runs, rather than ten minutes later.
Types ¶
type Clock ¶
Clock is the seam that lets tests advance time without sleeping — the only two operations the gate ever needs from a clock.
type Config ¶
type Config struct {
// URL and Token are the connection details, read from the environment by
// the CLI and never registered as flags. Token never enters the pure layer,
// an error string, or a Result.
URL, Token string
// Alerts is REQUIRED in single-step mode and must be EMPTY in log mode:
// with a log, the header IS the alert set, and there is nothing to compare
// a second list against.
Alerts []string
Folder string
// IncludeLabels selects by labels instead of names; ExcludeLabels drops
// matching rules. Both are refused in log mode. ExcludeAlerts is an
// enumerated list subtracted from either selection.
IncludeLabels, ExcludeLabels []LabelMatcher
ExcludeAlerts []string
States []State
Preexisting PreexistingPolicy
MinObserved int
AllowPaused bool
NodataIsUnobservable bool
// NoFailFast disables early exit, so the loop always runs to
// to+transitionGrace for a full-window coverage proof. The CLI sets it
// with --fail-fast=false; the gate never exits 0 early either way.
NoFailFast bool
// From is the moment the deploy finished and To is the end of the work.
// In recorder mode an absent From is a hard error; in single-step mode it
// falls back to the start of this step, and a From before the
// first-observation pass completes is a declared blind interval.
From, To time.Time
// Log is the path of a recording made by watch; "" selects single-step
// mode. PidFile defaults to <Log>.pid, the convention watch's parent
// writes and the only way check can reach the recorder it must stop before
// it may read the log.
Log string
PidFile string
// There is deliberately NO cadence override anywhere. In log mode the
// cadence comes from the header — the cadence the recording actually used
// — and in single-step mode the same process records and classifies, so
// half the evaluation interval is the only cadence there is.
Concurrency int
Clock Clock
// Notes is where the shell prints what an operator has to see while the
// run is in progress: the planned run time, the grace and its source, the
// countdown, the blind-interval warning. nil discards them. The library
// renders no table — the CLI owns presentation.
Notes io.Writer
}
Config is check's whole input. It is the CLI's view of a run, and it is deliberately wider than Policy: Policy is the narrowed, pure-layer subset that reaches decide (classify.go), and the token is the field that must never cross that line.
type CoverageResult ¶
type CoverageResult struct {
Proved bool
LargestGap time.Duration
LargestGapAt time.Time
Unobservable bool
Reason UnobservableReason
Notes []string
// BlindFor is the worst staleness (GrafanaNow - LastEvaluation) that
// tripped check 6; zero when check 6 never fired.
BlindFor time.Duration
}
CoverageResult is proveCoverage's whole answer for one rule. No interval list: proved-or-not plus the largest gap and where is everything a human reads on exit 2, and everything the rendered table needs.
type DaemonChildConfig ¶
type DaemonChildConfig struct {
URL, Token string // from the inherited environment, never from argv
Out string
Until time.Time
Concurrency int
Clock Clock
// ReadyFD is the inherited descriptor to report readiness on (ReadyFDFlag).
// Zero means nobody is waiting — a hand-started child — and the report is
// then skipped rather than written to stdin.
ReadyFD int
}
DaemonChildConfig is the detached recorder's whole input, and it is deliberately tiny. The rule set and every cadence come from the header the parent already wrote — one source of truth, no parent/child drift, and it exercises ReadLog's header path — and the connection details come from the inherited environment. Only the run facts the header does not carry travel in argv.
type Definition ¶
type Definition struct {
// Key is the map key across both source kinds (identity.go). UID stays the
// API-given uid and is empty for datasource-managed rules.
Key string
UID, Title, Folder, FolderUID, Group string
// DatasourceUID, DatasourceName and File are populated for
// KindDatasourceManaged only; File is the Prometheus rule group's file.
DatasourceUID, DatasourceName, File string
For time.Duration
IntervalSeconds int
NoDataState string
ExecErrState string
IsPaused bool
Kind RuleKind
// PauseObservable is true only for Grafana-managed rules, whose state
// endpoint reports isPaused. A datasource-managed rule has no pause signal.
PauseObservable bool
Labels map[string]string
}
Definition is one rule from the ruler endpoint (/api/ruler/grafana/api/v1/rules). IntervalSeconds, NoDataState and ExecErrState live inside the grafana_alert block and are only populated for KindGrafanaManaged — a datasource-managed rule has no such block by definition. Labels is the rule's own label set, optional (the 13.1 fleet capture has three unlabeled rules) and the only input to label selection. relativeTimeRange and keep_firing_for are deliberately not parsed: nothing in the gate reads them.
func DefinitionsFromDatasource ¶ added in v0.1.10
func DefinitionsFromDatasource(rules []StateRule, dsUID, dsName string) []Definition
DefinitionsFromDatasource converts datasource rule states into Definitions. Its input is already alerting-only (ParseDatasourceRules drops recording rules). A datasource-managed rule has no pause signal and no uid, so PauseObservable is false and UID stays empty.
func ListAllDefinitions ¶ added in v0.1.10
func ListAllDefinitions(ctx context.Context, src Source) ([]Definition, error)
ListAllDefinitions is the CLI's entry point for `list`: every Grafana-managed and datasource-managed definition, discovered from scratch.
func ParseDefinitions ¶
func ParseDefinitions(body []byte) ([]Definition, error)
ParseDefinitions strictly parses a ruler-endpoint response body (map[namespace][]group) into its rule definitions.
func Resolve ¶
func Resolve(defs []Definition, names []string, folder string) (resolved []Definition, notes []string, err error)
Resolve turns the operator-supplied alert names into resolved Definitions. Order is load-bearing:
- Trim each name.
- Discard empty lines.
- Resolve each name to a rule (this is what resolveOne does).
- Collapse the result by key — two names hitting the same rule is a note, never an error (almost always a copy mistake, and a message costs the user less than a failure).
The caller-visible consequence: len(resolved) is the count *after* the collapse, and MinObserved must default from that length, never from len(names) — using the input line count would make one rule named twice turn an achievable default into an unsatisfiable one.
func SelectByLabels ¶ added in v0.1.4
func SelectByLabels(defs []Definition, include, exclude []LabelMatcher) ([]Definition, error)
SelectByLabels resolves a label selection: a rule is selected when it carries every include pair, and dropped when it carries any exclude pair. A missing label never triggers an exclude (unlike a Prometheus != matcher); it only fails an include. Zero matches and a matched unsupported kind are errors, never a smaller watch set.
type GlobalThresholds ¶
type GlobalThresholds struct {
TransitionGrace time.Duration
// GraceSource names, and already carries the `for` value of, the rule that
// set TransitionGrace — an operator has to see both. "none" when no rule
// contributed (TransitionGrace is then 0).
GraceSource string
DrainTimeout time.Duration
}
GlobalThresholds is the run-wide half of the same information: transitionGrace and drainTimeout apply once, across every non-skipped watched rule, not per rule (globalTimings).
type GlobalTimings ¶
type GlobalTimings struct {
// contains filtered or unexported fields
}
GlobalTimings groups the values that apply to the whole run rather than to one rule: transitionGrace and drainTimeout are each derived once, across every non-skipped watched rule, not per rule.
func DeriveTimings ¶
func DeriveTimings(defs []Definition) (rules map[string]RuleTimings, global GlobalTimings)
DeriveTimings computes every resolved rule's ruleTimings (keyed by UID) plus the shared globalTimings. Every rule polls at half its own evaluation interval: the cadence is fixed so no state that lasts a full evaluation interval can fall between polls, and there is deliberately no override that could widen maxGap and hide one.
func DeriveTimingsFromLog ¶
func DeriveTimingsFromLog(h Header, defs []Definition) (rules map[string]RuleTimings, global GlobalTimings, err error)
DeriveTimingsFromLog is DeriveTimings' log-mode counterpart: pollEvery comes from the header (the cadence actually used), not the definitions — re-deriving it here would compare recorded gaps against thresholds derived from a different cadence, an exit 2 on a clean window when the recording was slower or a silently passing recorder gap when it was faster. evalStaleAfter still comes from defs (2 × intervalSeconds).
Three header shapes are hard errors rather than a best-effort derivation, because each would silently widen a threshold: a rule with no matching definition, a non-positive recorded cadence, and a UID listed twice (last-one-wins would widen maxGap on a corrupt log).
It checks only the header-to-defs direction. A definition absent from the header is Check's log-identity validation to judge, not this function's.
type Header ¶
type Header struct {
SchemaVersion int `json:"schema_version"`
URL string `json:"url"` // the log's identity
GrafanaVersion string `json:"grafana_version"`
StartedAt time.Time `json:"started_at"` // the record start
// ReadyAt is when the first-observation pass completed — the earliest
// instant a `from` can honestly name, since before it some rules were
// never observed. Zero on older logs and single-step synthesis; readyAt()
// then falls back to StartedAt.
ReadyAt time.Time `json:"ready_at,omitzero"`
Rules []LoggedRule `json:"rules"` // THE alert set
}
Header is the log's first line: what was recorded, from where, and when the recording started. It carries no States field — recording is deliberately unfiltered, so the same log can be re-classified under different --states without re-recording.
func ReadLogHeader ¶
ReadLogHeader reads ONLY line 1 — the one read safe while a writer may still hold the log. The header is written once by watch's parent before any child appends a byte, so line 1 is immutable. It lets check fail closed EARLY on a wrong URL or unresolvable rule; it is advisory only, and the authoritative identity read is still ReadLog after the writer exits.
type Instance ¶
type Instance struct {
Labels map[string]string `json:"labels"`
State State `json:"state"`
Reason string `json:"reason,omitempty"`
ActiveAt time.Time `json:"active_at"`
Value string `json:"value,omitempty"`
}
Instance is one entry of a rule's alerts[]. State is always canonical; Reason is the opaque suffix of a "State (Reason)" composite ("" when the API gave a bare state). Reason is reporting-only except for the MissingSeries routing done downstream in the log markers.
The json tags are for the JSONL log's abnormal-instance list only — parsing an API response never goes through them, because parseInstance decodes field by field through req/opt to keep the presence checks explicit.
type LabelMatcher ¶ added in v0.1.4
type LabelMatcher struct {
Key, Value string
}
LabelMatcher is one exact-match label requirement.
type LoggedRule ¶
type LoggedRule struct {
// Key is the rule's identity across both source kinds; uid stays empty for
// datasource-managed rules. SourceKind, DatasourceUID/Name and File are
// additive (schema 1) and let a later check re-resolve a ds rule without
// discovery.
Key string `json:"key,omitempty"`
UID string `json:"uid"`
Title string `json:"title"`
Folder string `json:"folder"`
Group string `json:"group"`
SourceKind string `json:"source_kind,omitempty"`
DatasourceUID string `json:"datasource_uid,omitempty"`
DatasourceName string `json:"datasource_name,omitempty"`
File string `json:"file,omitempty"`
// ForSeconds, IntervalSeconds, NoDataState and ExecErrState are purely
// forensic: a resolve-time snapshot that makes the uploaded artifact
// self-describing to a human reading it after the runner is gone. check
// never converts them back into a Definition — it always re-resolves
// definitions from the ruler API.
ForSeconds float64 `json:"for_seconds"`
IntervalSeconds int `json:"interval_seconds"`
// IsPaused is load-bearing (beside PollEverySeconds): the pause state at
// record start, the only moment `skipped` can honestly mean. decide reads
// it via Header.pausedAtStart, never a ruler read taken after the window.
IsPaused bool `json:"is_paused"`
NoDataState string `json:"no_data_state"`
ExecErrState string `json:"exec_err_state"`
// PollEverySeconds is the cadence this recording ACTUALLY used. Load-bearing:
// check derives maxGap from it, never from the definitions — the two must
// not be allowed to diverge.
PollEverySeconds float64 `json:"poll_every_seconds"`
}
LoggedRule is the per-rule identity written into the header. Together with the header URL it IS the log's identity, which check validates, and it supplies the alert set in check mode.
type Observation ¶
type Observation struct {
Rules []StateRule // may be empty — an authoritative 2xx saying the rule is absent
GrafanaNow time.Time // the response's Date header
Skew time.Duration // serverDate - (t_send+t_headers)/2, signed
SkewBound time.Duration // (t_headers-t_send)/2 — RTT/2 to the response headers
Latency time.Duration // t_send through the full body read — see requestResult.Latency
}
Observation is one successful poll of the state endpoint for a single rule.
type Outcome ¶
type Outcome string
Outcome is the verdict of one instance's timeline, and (after decide takes the worst across instances) of the rule. It is a published JSON output: the fail values stay distinct even though v1 maps them all to exit 1, so a later version can split them without breaking the interface.
const ( OutcomeHealthy Outcome = "healthy" OutcomeNewFailure Outcome = "new_failure" OutcomeRecovered Outcome = "recovered" OutcomeStillFailing Outcome = "still_failing" OutcomeUnstable Outcome = "unstable" OutcomePaused Outcome = "paused" OutcomeNotVerified Outcome = "not_verified" // OutcomeNotCounted is synthetic: decide uses it for a --min-observed // deficit row that no resolved rule explains. It is not a verdict on an // alert, so it is deliberately not named after one. OutcomeNotCounted Outcome = "not_counted" )
type Policy ¶
type Policy struct {
States []State
Preexisting PreexistingPolicy
MinObserved int
AllowPaused, NodataIsUnobservable bool
From, To time.Time
}
Policy is decide's narrowed, pure-layer view of a Config: the classification knobs and the window, nothing else. No URL, no token, no I/O handles — those never reach the pure layer.
type Poll ¶
type Poll struct {
// RuleKey is the map key across both source kinds; RuleUID is kept for
// compatibility and is empty for datasource-managed rules. pollKey reads
// RuleKey when present, else RuleUID, so a v1 log written before rule_key
// stays readable.
RuleKey string `json:"rule_key,omitempty"`
RuleUID string `json:"rule_uid,omitempty"`
GrafanaNow time.Time `json:"grafana_now"` // the response's Date header
// SkewMS, SkewBoundMS and LatencyMS are milliseconds for JSONL
// compactness ONLY. The pure layer never touches raw ms: it reads
// Skew(), SkewBound() and Latency() below, which convert at the
// (de)serialization boundary.
SkewMS int64 `json:"skew_ms"`
SkewBoundMS int64 `json:"skew_bound_ms"`
LatencyMS int64 `json:"latency_ms"`
// KeepFiringForMS is the rule's recovery period as reported by this
// response; 0/absent means no instance can be Recovering.
KeepFiringForMS int64 `json:"keep_firing_for_ms,omitempty"`
// Found false means an authoritative 2xx in which this rule was absent —
// never a transport failure, which the transport retries and never turns
// into a Poll. The coverage proof turns it into unobservable.
Found bool `json:"found"`
// State, Health and LastError are the raw rule-level strings, reporting
// only and never classified.
State string `json:"state,omitempty"`
Health string `json:"health,omitempty"`
LastError string `json:"last_error,omitempty"`
// omitzero, not omitempty: a not-found poll (and a paused rule) has no
// evaluation time, and writing "0001-01-01T00:00:00Z" into an artifact
// humans and jq read invites reading it as a real timestamp.
LastEvaluation time.Time `json:"last_evaluation,omitzero"`
IsPaused bool `json:"is_paused"`
Histogram map[string]int `json:"histogram,omitempty"` // written, never analysed
// Reasons counts this poll's non-empty instance reasons, e.g.
// {"NoData":1091,"Error":14}; nil when none. Reporting-only, and the only
// place composite states stay visible (they are canonical normal, dropped
// from Abnormal). Keys are raw reason strings and can be comma-joined
// composites ("KeepLast, MissingSeries"), so consumers must test membership
// via reasonNames and never index a literal key.
Reasons map[string]int `json:"reasons,omitempty"`
// Abnormal holds the instances whose CANONICAL state is not normal.
// "Normal (NoData)" and "Normal (Error)" are canonical normal and are
// deliberately not retained here.
Abnormal []Instance `json:"abnormal,omitempty"`
// Cleared and Vanished are instance keys that left the abnormal set,
// resolved against the SAME response — a clear and a discontinuity are not
// the same fact.
Cleared []string `json:"cleared,omitempty"`
Vanished []string `json:"vanished,omitempty"`
}
Poll is one reduced observation of one rule — the log's heartbeat and the only input the pure coverage and classification layers ever see.
func (Poll) KeepFiringFor ¶ added in v0.1.9
KeepFiringFor is the rule's recovery period as reported by this poll.
type PreexistingPolicy ¶
type PreexistingPolicy string
PreexistingPolicy governs only the ONE ambiguous case in the outcome table: an instance that was already bad when the window opened. A new_failure or unstable instance is a fail under every policy, so this type only ever changes how `recovered` and `still_failing` are judged (isViolation below).
const ( // PreexistingFailUnlessRecovered is the default: a preexisting instance // that clears and stays clear is a pass (`recovered`); one that never // clears is still a fail (`still_failing`). PreexistingFailUnlessRecovered PreexistingPolicy = "fail-unless-recovered" // PreexistingFail makes ANY preexisting instance a fail, even one that // recovers — for a user who wants no benefit of the doubt for a // condition this release did not cause. PreexistingFail PreexistingPolicy = "fail" // PreexistingIgnore disregards a preexisting instance entirely, whether // it recovers or stays bad for the whole window: only a genuinely NEW // bad episode (new_failure or unstable) can fail the rule. PreexistingIgnore PreexistingPolicy = "ignore" )
type RecordType ¶
type RecordType string
RecordType tags each JSONL line. There are exactly three, and a poll record IS the heartbeat — there is deliberately no separate heartbeat type.
const ( RecordHeader RecordType = "header" RecordPoll RecordType = "poll" RecordStopped RecordType = "stopped" )
type Reducer ¶
type Reducer struct {
// contains filtered or unexported fields
}
Reducer turns each Observation into the single Poll record that goes into the log. It holds the previous poll's abnormal instance keys per rule, which is all the state the transition markers need.
A Reducer is safe for concurrent use: watch polls a fleet of rules concurrently and every one of those goroutines reduces through the same instance, because the per-rule marker state has to live in one place. The lock is per-Reducer rather than per-rule — Reduce only touches maps and slices, so it never blocks on I/O while holding it.
func NewReducer ¶
func NewReducer() *Reducer
func (*Reducer) Reduce ¶
func (r *Reducer) Reduce(uid string, obs Observation) Poll
Reduce selects the rule identified by uid out of obs and reduces it to a Poll. Selection is BY UID, never by title: a filtered response can carry several rules sharing one title, and picking the first would silently watch the wrong rule.
The reduction keeps the rule-level fields, the raw totals histogram, the reason counts, and only the instances whose canonical state is not normal. That makes per-poll size independent of NORMAL cardinality — not of cardinality outright: a rule with 449 firing instances still stores all 449.
type Result ¶
type Result struct {
From, To time.Time
GrafanaVersion string
ClockSkew time.Duration // the largest |skew| across every poll decide was given, not only the ones a rule's window actually used
// ClockSkewBound is the skew BOUND (RTT/2) of that SAME poll — not
// the largest bound seen overall, which would pair a wide bound from an
// unrelated slow request with the worst skew and misstate how tightly
// that skew is actually known. SkewHardLimit is a separate, fixed input
// validation threshold (source.go) and is not an error bound on this
// value; the CLI prints both, but must not conflate them.
ClockSkewBound time.Duration
Coverage map[string]CoverageResult
// Thresholds carries one RuleThresholds per rule Coverage also covers —
// every non-skipped rule, keyed by UID. A skipped rule has neither: it was
// never scheduled, so it has no maxGap/healthGrace/evalStaleAfter to
// report.
Thresholds map[string]RuleThresholds
Global GlobalThresholds
Verdicts []RuleVerdict
Violations []Violation
// Caveats are run-level, kind-level policy notes (e.g. datasource-managed
// pause is unobservable). They are not per-rule details; the CLI prints them
// once so they never bloat the table.
Caveats []string `json:"caveats,omitempty"`
// TerminatedEarly is set only when fail-fast stopped before the window
// closed; the coverage proof is then over [from, At]. To and Global below
// still report the requested values. Published JSON output.
TerminatedEarly *Termination `json:"terminated_early,omitempty"`
}
Result is decide's whole answer: everything the human table and the JSON output need. Coverage carries one CoverageResult per non-skipped rule — there is deliberately no separate Interval type anywhere in the project.
func Check ¶
Check is the I/O shell: HTTP, signals, the pidfile, file reads, the countdown print. Every correctness question it touches is answered elsewhere (proveCoverage, decide — both pure), which is the most important seam in the project. A pass is exactly len(Violations) == 0 && err == nil; every error path leaves err non-nil.
type RetryExhaustedError ¶
RetryExhaustedError is the hard, terminal failure retryTransport returns once it gives up. It deliberately omits Unwrap into *TransportError so errors.AsType can never re-classify it as retryable; Cause stays a plain field for logging only.
func (*RetryExhaustedError) Error ¶
func (e *RetryExhaustedError) Error() string
type RuleKind ¶
type RuleKind int
RuleKind classifies a ruler-endpoint rule by shape, not by name. Resolve rejects KindDatasourceManaged and KindRecording, but only for rules a user actually named — ParseDefinitions itself never rejects.
type RuleRef ¶ added in v0.1.10
type RuleRef struct {
Key string
Kind RuleKind
DatasourceUID string // "" = Grafana-managed
UID string // Grafana-managed only
Group, Name, File, Title string
}
RuleRef is everything a poll needs to find one rule again: the identity key, the source kind, and the exact filters each source API expects.
type RuleSource ¶ added in v0.1.10
type RuleSource struct{ UID, Name string }
RuleSource is one discovered datasource that can serve alerting rules.
type RuleThresholds ¶
type RuleThresholds struct {
MaxGap time.Duration
HealthGrace time.Duration
EvalStaleAfter time.Duration
}
RuleThresholds is one non-skipped rule's resolved coverage thresholds, carried on Result so the CLI's table can print the numbers that answer "why" on exit 2 without decide exposing the unexported ruleTimings type itself.
type RuleTimings ¶
type RuleTimings struct {
// contains filtered or unexported fields
}
RuleTimings groups the per-rule thresholds derived from a rule's poll cadence and its own evaluation interval. title is carried for operator-facing messages only — it is never compared or applied.
type RuleVerdict ¶
type RuleVerdict struct {
Alert, RuleUID string
RuleKey string `json:"rule_key,omitempty"`
// SourceKind is "grafana" or "datasource" (sourceKind), so a reader can see
// which classification semantics apply without prose.
SourceKind string `json:"source_kind,omitempty"`
Outcome Outcome
BadFor time.Duration // total wall-clock time any instance was bad inside the window, overlaps merged
PollEvery time.Duration
Note string
}
RuleVerdict is one rule's worst-of outcome, present for every resolved rule (passes included) so the table shows every alert asked for.
type Scheduler ¶
type Scheduler struct {
// contains filtered or unexported fields
}
Scheduler drives one per-rule schedule, never a global cycle: a rule at intervalSeconds=10 alongside twenty at 300 keeps its own 5s cadence without forcing the same cadence onto the other twenty.
func NewScheduler ¶
NewScheduler builds a Scheduler over per-rule cadences, staggering each rule's initial next-due time across [0, pollEvery) so a phase-aligned fleet (which would void CheckBudget's burst bound) never arises by construction. It takes cadences, not ruleTimings: a scheduler only decides when to poll and must not be handed coverage thresholds it never applies.
func NewSchedulerFromPolls ¶ added in v0.1.7
NewSchedulerFromPolls continues an existing recording instead of starting a fresh phase: each rule is due one cadence after its last recorded observation, so the parent/child handoff keeps the spread the first-observation pass created and adds no stagger. An overdue rule keeps its recorded due time and is due immediately. Polls outside every are ignored; a rule with no recorded observation falls back to the fresh stagger.
func (*Scheduler) Due ¶
Due returns the due UIDs, earliest-due-first. Ties break by tightest cadence first: the burst bound assumes a newly-due tight rule waits at most one in-flight request, which only holds if a simultaneous batch serves the tightest rule first. A map-order tie-break would silently void that.
func (*Scheduler) Mark ¶
Mark records that uid was just polled at now, scheduling its next poll one cadence later. It fails on an unknown uid rather than silently treating the missing cadence as zero: a zero cadence would schedule an immediate re-due (next = now.Add(0)) and insert a bogus next-due entry, hiding a caller that is polling a rule the scheduler never owns.
type Source ¶
type Source interface {
Version(ctx context.Context) (string, error)
GrafanaDefinitions(ctx context.Context) ([]Definition, error)
DiscoverRuleSources(ctx context.Context) ([]RuleSource, error)
// DatasourceDefinitions names empty = fetch every rule; names non-empty =
// one request with repeated rule_name[] params.
DatasourceDefinitions(ctx context.Context, src RuleSource, names []string) ([]Definition, error)
RuleState(ctx context.Context, ref RuleRef) (Observation, error)
}
Source is everything the gate reads from Grafana. httpSource is the one production implementation; the tests use a scripted fake (source_fake_test.go) instead of real HTTP.
func NewHTTPSource ¶
NewHTTPSource builds the production Source. token is never logged and never enters an error string — it is used only to set the Authorization header.
type State ¶
type State string
State is the canonical instance state. It is distinct from the raw, unnormalized vocabularies the API uses at the rule level and at the instance level — see normalizeInstanceState.
type StateRule ¶
type StateRule struct {
// Key is the map key across both source kinds (identity.go); UID is empty
// for datasource-managed rules.
Key string
UID, Title, Folder, Group string
// DatasourceUID and File are set for datasource-managed rules only; Type is
// the Prometheus rule type ("alerting"/"recording"); Query and For carry the
// Prometheus definition.
DatasourceUID string
File string
Type, Query string
For time.Duration
Labels map[string]string
Interval time.Duration
// KeepFiringFor is the rule's recovery period in seconds; 0/absent means
// no instance can be Recovering. Carried onto the Poll.
KeepFiringFor time.Duration
// State and Health are raw, lowercase, and reporting-only — never
// classified. State in particular is never normalized.
State, Health string
LastError string
LastEvaluation time.Time
IsPaused bool
Instances []Instance
Totals map[string]int
}
StateRule is one rule from the state endpoint (/api/prometheus/grafana/api/v1/rules), fully and strictly parsed.
func ParseDatasourceRules ¶ added in v0.1.10
ParseDatasourceRules parses a Prometheus/vmalert rules response (/api/prometheus/{uid}/api/v1/rules) into StateRules. It shares parseInstance with the Grafana parser but uses the datasource vocabulary: lowercase instance states, health "err" (not "error"), and a zero lastEvaluation is allowed (liveness treats zero as maximally stale). A missing or unparseable required field is an error, never a zero value.
Recording rules are dropped here, at the one place a response is parsed: they have no instances or state to observe, and keeping them would let a recording rule that shares a datasource/group/name/file shadow the alerting rule in state selection.
func ParseState ¶
ParseState strictly parses a state-endpoint response body into its rules. A missing or unparseable required field (health, state, lastEvaluation on each rule; interval on each group) is an error, never a zero value.
type StopConfig ¶ added in v0.1.2
type StopConfig struct {
// Log is the JSONL path whose flock proves a writer exists right now.
// PidFile defaults to <Log>.pid.
Log string
PidFile string
Clock Clock
Notes io.Writer
// Timeout bounds each wait (SIGTERM, then SIGKILL when Cleanup is set).
// Zero means recorderStopTimeout; a test overrides it to drive the SIGKILL
// path quickly.
Timeout time.Duration
// Cleanup is the `stop` command's semantics rather than check's: SIGKILL a
// recorder that ignores SIGTERM, and remove the pidfile once no writer
// holds the log. check leaves it false — a writer that will not exit means
// the log cannot be trusted, which is a could-not-check, never a silent
// kill.
Cleanup bool
}
StopConfig locates the recorder and selects the stop semantics.
type SystemClock ¶
type SystemClock struct{}
SystemClock is the production Clock: the real wall clock.
func (SystemClock) Now ¶
func (SystemClock) Now() time.Time
type Termination ¶ added in v0.1.2
type Termination struct {
Kind TerminationKind `json:"kind"`
Alert string `json:"alert,omitempty"`
RuleKey string `json:"rule_key,omitempty"`
RuleUID string `json:"rule_uid,omitempty"`
Outcome Outcome `json:"outcome,omitempty"`
Reason UnobservableReason `json:"reason,omitempty"`
At time.Time `json:"at"`
}
Termination is why a fail-fast run stopped before the window closed. At is the runner-domain detection time.
type TerminationKind ¶ added in v0.1.2
type TerminationKind string
TerminationKind is the kind of terminal verdict. It reaches the JSON output.
const ( // TerminationViolation is a post-`from` bad onset (new_failure or unstable). TerminationViolation TerminationKind = "violation" // TerminationNotVerified is an inability that has already happened. TerminationNotVerified TerminationKind = "not_verified" )
type TransportError ¶
type TransportError struct {
Err error
Status int // 0 when the failure never got a status (network/transport failure)
}
TransportError marks a failure worth retrying: a 5xx/429 response, a network failure, or a body that failed to parse. Not a 4xx (wrong auth, missing resource), not a deleted rule (an authoritative 2xx) and not a clock problem (a hard error — see doRequest).
func (*TransportError) Error ¶
func (e *TransportError) Error() string
func (*TransportError) Unwrap ¶
func (e *TransportError) Unwrap() error
type UnobservableReason ¶
type UnobservableReason string
UnobservableReason names why proveCoverage could not prove a rule's window. It reaches the JSON output, so it is a published vocabulary like Outcome; prose belongs in Notes.
const ( ReasonNoSentinel UnobservableReason = "no_sentinel" ReasonSentinelEarly UnobservableReason = "sentinel_early" ReasonFromBeforeRecord UnobservableReason = "from_before_record" ReasonHeartbeatGap UnobservableReason = "heartbeat_gap" ReasonHealthError UnobservableReason = "health_error" ReasonStaleEvaluation UnobservableReason = "stale_evaluation" ReasonFutureEvaluation UnobservableReason = "future_evaluation" ReasonPausedInWindow UnobservableReason = "paused_in_window" ReasonRuleAbsent UnobservableReason = "rule_absent" // ReasonDrainTimeout is set by check.go's drain wait, never by // proveCoverage: the wait is I/O and must not be added to this pure // function — that would put HTTP inside the pure layer and destroy the // seam this design depends on. ReasonDrainTimeout UnobservableReason = "drain_timeout" )
const ReasonNodata UnobservableReason = "nodata"
ReasonNodata is decide's own unobservable reason: proveCoverage never sets it — health=nodata is a note there, never fatal — because escalating it needs Policy.NodataIsUnobservable, which only decide (the Policy-holding seam) has.
type Violation ¶
type Violation struct {
Alert, RuleUID string
// RuleKey is the identity across both source kinds; RuleUID is empty for a
// datasource-managed rule.
RuleKey string `json:"rule_key,omitempty"`
Outcome Outcome
State State
Health string // raw, reporting-only, like Poll.Health
LastError string
// FirstSeen is the episode's onset in the runner domain (translated by the
// poll's own skew), or `from` when preexisting — never a raw Grafana time.
FirstSeen time.Time
// ClearedAt is zero unless the episode closed via a genuine Cleared event.
ClearedAt time.Time
InstanceLabels map[string]string
// Note explains a Violation with no instance behind it — decide's synthetic
// MinObserved shortfall — and must not double as LastError (reporting-only
// rule state from a real poll).
Note string
}
Violation is one instance whose timeline outcome counts against the run, after the preexisting policy has been applied (isViolation below).
type WatchConfig ¶
type WatchConfig struct {
// URL and Token are the connection details. The CLI reads both from the
// environment and never from a flag; Token is never logged and never
// enters an error string.
URL, Token string
// Alerts are the operator-supplied names, one per line, in any of the forms
// Resolve accepts. Empty lines are discarded by Resolve.
Alerts []string
Folder string
// IncludeLabels selects by labels instead of names; ExcludeLabels drops
// matching rules. The two modes cannot be combined. ExcludeAlerts is an
// enumerated list subtracted from either selection.
IncludeLabels, ExcludeLabels []LabelMatcher
ExcludeAlerts []string
// Out is the JSONL log path. PidFile and DaemonLog default to
// <Out>.pid and <Out>.daemon.log — the same convention check uses to find
// the recorder it must stop, so nothing has to be wired by hand.
Out string
PidFile string
DaemonLog string
// Until is an optional hard stop for the child. Zero means "record until
// signalled", which is the normal case: check sends SIGTERM when its
// collection loop ends.
Until time.Time
Concurrency int
Clock Clock
// Notes is where the parent prints what an operator has to see before the
// deploy step runs: resolve notes, the cadence per rule, the rules it will
// not wait for. nil discards them. The library prints nothing else — the
// CLI owns presentation.
Notes io.Writer
}
WatchConfig is the record step's whole input.
It has no To field (and must never gain one): watch writes the sentinel with its OWN stop time and makes no `to` comparison — only check knows `to`, and the recorder exits before the grace it would have to wait for.
It has no States field either: recording is unfiltered, so the same raw log can be re-classified under different --states without re-recording.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer appends records to the JSONL log. It is append-only by construction — O_APPEND|O_CREATE|O_WRONLY, never O_TRUNC — so no writer can ever destroy evidence a previous one recorded. An exclusive non-blocking flock makes a second writer fail immediately rather than interleave.
func NewWriter ¶
NewWriter opens path for appending and takes the exclusive lock. A second writer on the same path fails here, immediately — it never blocks and never waits, because two recorders on one log means one of them is recording a window nobody will classify.
func (*Writer) Close ¶
Close releases the file and the lock WITHOUT writing a sentinel. It exists for exactly one caller: watch's parent, which writes the header and then hands the log to the detached child that will finish it. A sentinel here would tell check the recording ended before the child had even started.
func (*Writer) Stop ¶
Stop finishes recording in a fixed order that must not be rearranged: let the in-flight write finish, append the sentinel, fsync, release — any other order can leave a sentinel that was never preceded by the polls it vouches for. The sentinel uses the writer's OWN stop time; check does the `to` comparison after this has exited. Calling Stop twice is a no-op (watch reaches it from a signal handler and a defer).
func (*Writer) WriteHeader ¶
WriteHeader writes line 1 and stamps the current schema version, so no caller can leave it at zero. It refuses a non-empty file: the log already has a header, and a second one would make ReadLog's "header is line 1" contract a lie. In watch's handoff the parent writes the header and the detached child only appends polls.