alerting

func NewSuppressedACMESource ¶ added in v0.2.0

func NewSuppressedACMESource(inner ACMEFailureSource, suppress CertSuppressor) ACMEFailureSource

NewSuppressedACMESource reports no failure for hostnames suppress names, so a stale attempt made before a proxy took over TLS never raises an alert.

func NewAutoRollbackTracker ¶ added in v0.2.0

func NewAutoRollbackTracker() *AutoRollbackTracker

NewAutoRollbackTracker builds an empty AutoRollbackTracker.

func ListCertificates ¶

func ListCertificates(ctx context.Context, source CertSource, warningWindow time.Duration, now time.Time, logger *slog.Logger) ([]CertInfo, error)

ListCertificates returns every certificate currently in source, parsed from its own stored PEM bytes, sorted by domain. A control plane that has never issued a certificate returns an empty slice, not an error. A single malformed or since-deleted entry is skipped and logged, never fails the whole list, the same "one broken resource must not block the rest" shape used throughout this codebase.

func NewSuppressedCertSource ¶ added in v0.2.0

func NewSuppressedCertSource(inner CertSource, suppress CertSuppressor) CertSource

NewSuppressedCertSource hides stored certificates whose hostname suppress reports, so cert expiry and renewal rules never fire on a certificate visitors do not receive. A nil suppress returns inner unchanged.

func Open ¶

func Open(ctx context.Context, path string) (*DB, error)

Open opens (creating if needed) the SQLite database at path, applies pragmas, and runs every pending migration, matching internal/store and internal/telemetry's own Open functions exactly.

func NewDeployDispatcher ¶

func NewDeployDispatcher(db *DB, client *http.Client, sender email.Sender, pushSender PushSender, logger *slog.Logger) *DeployDispatcher

NewDeployDispatcher builds a DeployDispatcher. client and logger default if nil; sender may be nil (an email-kind target then fails with a clear "not configured" error), and pushSender may be nil with the identical failure shape for a webpush-kind target.

func NewEngine ¶

func NewEngine(rules RuleStore, metrics MetricsSource, logs LogsSource, tracker *RestartTracker, certs CertSource, scheduledTasks ScheduledTaskSource, certExpiryWarningWindow, certRenewalStalledThreshold time.Duration, nodes NodeSource, patchStatusThreshold, nodeDiskSpaceThreshold float64, nodeServices NodeServiceSource, nodeCPUThreshold, nodeMemoryThreshold float64, domainApps AppDomainSource, domainChecker DomainCheckSource, domainHealthCheckInterval time.Duration, backups BackupSource, backupMissingGracePeriod time.Duration, newNotifier func(Rule) Notifier, logger *slog.Logger) *Engine

NewEngine builds an Engine. newNotifier defaults to a Notifier with no email capability configured if nil; a real caller passes a closure capturing an email.Sender instead. certs may be nil if no cert storage is configured; a kind=cert_expiry rule then logs a warning and is skipped each tick rather than evaluated. nodes and scheduledTasks may likewise be nil, in which case a kind=patch_status or kind=scheduled_task_failure rule is skipped the same way. certExpiryWarningWindow and certRenewalStalledThreshold fall back to DefaultCertExpiryWarningWindow/DefaultCertRenewalStalledThreshold when passed as 0; patchStatusThreshold falls back to DefaultPatchStatusThreshold, nodeDiskSpaceThreshold falls back to DefaultNodeDiskSpaceThresholdPercent, and nodeCPUThreshold/ nodeMemoryThreshold fall back to DefaultNodeCPUThresholdPercent/ DefaultNodeMemoryThresholdBytes, all the same way. nodeServices may be nil, in which case a kind=node_resource_usage rule is skipped the same way a kind=patch_status rule is when nodes is nil. domainApps and domainChecker may likewise be nil, in which case a kind=domain_health rule is skipped the same way; domainHealthCheckInterval falls back to DefaultDomainHealthCheckInterval when passed as 0. backups may likewise be nil, in which case a kind=backup_missing rule is skipped the same way; backupMissingGracePeriod falls back to DefaultBackupMissingGracePeriod when passed as 0.

func LogRulePolicyFromEnv ¶ added in v0.2.0

func LogRulePolicyFromEnv() LogRulePolicy

LogRulePolicyFromEnv applies APP_LOG_RULE_* overrides to the defaults. A malformed or non-positive value keeps the default.

func MatchLogRule ¶ added in v0.2.0

func MatchLogRule(kind Kind, cfg LogRuleConfig, entries []telemetry.LogEntry, truncated bool, now time.Time, policy LogRulePolicy) (LogRuleResult, error)

MatchLogRule applies cfg to entries (oldest first, already limited to the window) and returns the result. It is pure so rules test with seeded lines.

func FindMute ¶ added in v0.2.0

func FindMute(silences []Silence, windows []MaintenanceWindow, c AlertContext, now time.Time) (MuteHit, bool)

FindMute reports whether any active silence or enabled maintenance window in effect at now mutes the alert c.

func CheckNodeAlertStatus ¶

func CheckNodeAlertStatus(ctx context.Context, node store.Node, services NodeServiceSource, metrics MetricsSource,
	patchThreshold, diskThresholdPercent, cpuThresholdPercent, memoryThresholdBytes float64, now time.Time, logger *slog.Logger) NodeAlertStatus

CheckNodeAlertStatus live-evaluates patch_status, node_disk_space, and node_resource_usage for node, right now, rather than waiting for Engine's next tick or reading a rule's stored aggregate LastValue (which only ever holds the worst value seen across every node, not which node it came from). It calls each evaluator's own per-node check helper directly so this can never drift from what a real rule would decide. A threshold of 0 falls back to that kind's own default, the same convention EvaluatePatchStatus/EvaluateNodeDiskSpace/ EvaluateNodeResourceUsage already use.

func ClassifyNodeCert ¶ added in v0.2.0

func ClassifyNodeCert(n store.Node, warning, critical time.Duration, now time.Time) (NodeCertState, time.Duration)

ClassifyNodeCert returns n's certificate state at now and the time left before it expires (negative once expired).

func NewNoiseControl ¶ added in v0.2.0

func NewNoiseControl(cfg NoiseConfig, st NoiseStore, placement PlacementSource, logger *slog.Logger) *NoiseControl

NewNoiseControl builds a NoiseControl. placement may be nil, which disables node-based matching and inhibition.

func NewNotifier ¶

func NewNotifier(client *http.Client, sender email.Sender, pushSender PushSender, r Rule) Notifier

NewNotifier builds the right Notifier for r.NotifyKind. An unknown or empty NotifyKind falls back to NotifyGeneric rather than erroring, so a typo'd notify_kind still notifies someone, diagnosable from the payload shape. sender may be nil, in which case an email-kind rule fails with a clear "not configured" error; pushSender may be nil with the identical failure shape for a webpush-kind rule.

func CountRequests ¶ added in v0.2.0

func CountRequests(ctx context.Context, src MetricsSource, app string, from, to time.Time, cfg SLOConfig) (RequestCounts, error)

CountRequests sums an app's request counters over [from, to]. Counters are per-tick deltas, so negative or non-finite values (a counter reset artefact) count as zero and Bad never exceeds Total. A window with partial data is counted over what exists.

func NewRestartTracker ¶

func NewRestartTracker() *RestartTracker

NewRestartTracker builds an empty RestartTracker with no persistence: Observe only updates this tracker's own in-memory crashloop-window state, the behavior every test in this package already exercises. Call WithRecorder to also persist each real restart as a queryable metric sample.

func EvaluateBackupMissing ¶ added in v0.2.0

func EvaluateBackupMissing(ctx context.Context, source BackupSource, r Rule, defaultGracePeriod time.Duration, now time.Time) (Rule, string, error)

EvaluateBackupMissing runs one KindBackupMissing rule and returns its updated evaluation state plus, when firing, a human-readable notice line for Engine to attach to the outgoing Event.

"Missing" is measured against the same cron schedule internal/backup.Scheduler itself reads (store.DesiredDatabase. BackupSchedule or store.ServiceVolumeBackupConfig.BackupSchedule), not a second, independently configured cadence: the interval between two consecutive scheduled fire times from now (via cronexpr.Schedule.Next, called twice) stands in for that schedule's period, since nothing in this codebase persists an "expected next run" outside Scheduler's own in-memory map (that struct's own doc comment). The rule fires once now is more than that period plus its grace period (r.ForDuration when set, reused here per Rule's own doc comment, else defaultGracePeriod) past the last successful attempt.

A target with no schedule currently configured, or with no history at all yet (a schedule just created, nothing had a chance to run), goes quiet rather than firing: there is nothing yet to compare against, the same "nothing to alert on" quiet-state EvaluateScheduledTaskFailure's own doc comment applies to a deleted watched task. A target with attempts but no success anywhere in the lookback window anchors the overdue calculation on its oldest known attempt instead, so a backup that has only ever failed still reads as overdue rather than silently never firing.

func EvaluateBudgetThreshold ¶ added in v0.2.0

func EvaluateBudgetThreshold(ctx context.Context, src BudgetSource, r Rule, now time.Time) (Rule, []string, []string, error)

EvaluateBudgetThreshold runs one KindBudgetThreshold rule: it fires while any budget has crossed its warn level, its limit, or is forecast to overrun. It returns one notice per breach and the breach keys, which the engine uses to notify again only when a new threshold is crossed.

func EvaluateCertExpiring ¶ added in v0.2.0

func EvaluateCertExpiring(ctx context.Context, certs CertSource, r Rule, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateCertExpiring runs one KindCertExpiring rule.

func EvaluateCertExpiry ¶

func EvaluateCertExpiry(ctx context.Context, certs CertSource, snapshots CertExpiryObservationStore, r Rule, warningWindow, stalledThreshold time.Duration, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateCertExpiry runs one KindCertExpiry rule against every certificate currently in certs and returns its updated evaluation state plus, when firing, one human-readable notice line per non-healthy certificate for Engine to attach to the outgoing Event.

The rule fires (satisfied, in advanceState terms) the instant any certificate is "expiring_soon" or "expired", with no ForDuration debounce: unlike a noisy metric sample, expiry status only moves monotonically with real time, so there is nothing to debounce. LastValue is the number of days until the earliest-expiring certificate (negative once expired), across every certificate, not just the non-healthy ones, so an operator can see "how close" even while healthy.

Per non-healthy domain, snapshots records the NotAfter it had when it first entered this expiry episode (EpisodeNotAfter/EpisodeStartedAt). If a later tick finds the same domain still non-healthy with an unchanged NotAfter for at least stalledThreshold, that domain's notice is marked as a renewal that appears stalled rather than a plain warning: Caddy's own renewal should have already succeeded well before the warning window even opened (DefaultCertRenewalStalledThreshold's own doc comment), so a stuck NotAfter this deep into the window is a stronger, more actionable signal than "getting close to expiry."

func EvaluateControlPlaneBackupStale ¶ added in v0.2.0

func EvaluateControlPlaneBackupStale(source ControlPlaneBackupSource, r Rule, defaultMaxAge time.Duration, now time.Time) (Rule, string, error)

EvaluateControlPlaneBackupStale runs one KindControlPlaneBackupStale rule. It fires when the newest snapshot is older than the rule's ForDuration (else defaultMaxAge). No snapshot yet stays quiet, like the doctor check.

func EvaluateCrashloop ¶

func EvaluateCrashloop(tracker *RestartTracker, r Rule, now time.Time) Rule

EvaluateCrashloop runs one KindCrashloop rule against tracker and returns its updated evaluation state, the crashloop equivalent of EvaluateThreshold, sharing the same pending/firing debounce logic via advanceState. Passes forDuration=0: a crashloop rule fires the instant its restart count crosses the threshold, with no additional debounce layered on top, because RestartWindow itself already is the debounce (requiring N restarts within a real time window is already "sustained," unlike a single noisy metric sample).

func EvaluateDomainHealth ¶

func EvaluateDomainHealth(ctx context.Context, apps AppDomainSource, checker DomainCheckSource, r Rule, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateDomainHealth runs one KindDomainHealth rule against every domain currently configured on the app it's scoped to (ResourceID) and returns its updated evaluation state plus, when firing, one human-readable notice line per unhealthy domain, for Engine to attach to the outgoing Event.

Unlike EvaluateCertExpiry/EvaluatePatchStatus/EvaluateNodeDiskSpace (platform-wide, ResourceID only a display label), a domain_health rule's ResourceID is real: it picks out exactly which app's own domains this rule checks, the same app-scoped shape EvaluateThreshold/ EvaluateCrashloop use. An app with no domains configured, or one that has since been deleted, is treated like "no recent data": neither confirms nor denies.

r.ForDuration optionally debounces a single unhealthy check the same way EvaluateThreshold's own ForDuration does, since a real DNS lookup can blip; zero (the default) fires the instant any domain is unhealthy, matching every other non-threshold kind's own default.

func EvaluateLogArchiveStale ¶ added in v0.2.0

func EvaluateLogArchiveStale(ctx context.Context, source LogArchiveSource, r Rule, now time.Time) (Rule, string, error)

EvaluateLogArchiveStale runs one KindLogArchiveStale rule. A policy is unhealthy when its last run failed or it has not succeeded within the rule's ForDuration (else three intervals, at least two hours).

func EvaluateNodeCertExpiring ¶ added in v0.2.0

func EvaluateNodeCertExpiring(ctx context.Context, nodes NodeSource, r Rule, warning, critical time.Duration, now time.Time) (Rule, []string, error)

EvaluateNodeCertExpiring runs one KindNodeCertExpiring rule: it fires while any node's agent certificate is inside the warning window (the rule's ForDuration, else warning) or already expired. Revoked nodes are left out: that was an operator's decision, not a failure. LastValue is the fewest days left across the flagged nodes.

func EvaluateNodeDiskSpace ¶

func EvaluateNodeDiskSpace(ctx context.Context, nodes NodeSource, metrics MetricsSource, r Rule, thresholdPercent float64, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateNodeDiskSpace runs one KindNodeDiskSpace rule against every node in nodes and returns its updated evaluation state plus, when firing, one human-readable notice line per node whose latest used-disk percentage meets or exceeds thresholdPercent, for Engine to attach to the outgoing Event.

Firing follows EvaluatePatchStatus's shape, not EvaluateThreshold's: the instant any node is over threshold, with no ForDuration debounce. LastValue is the highest used-disk percentage seen across every node with a recent sample, not just the ones over threshold, matching EvaluatePatchStatus's own LastValue convention.

A node missing either a disk_used_bytes or disk_total_bytes sample inside diskSpaceLookback (collector hasn't run yet), or reporting a total of zero or less, is skipped, neither confirming nor denying the condition. A node whose metrics query fails is logged and skipped too, the same "one broken resource must not block the rest" stance EvaluatePatchStatus already takes.

func EvaluateNodeOffline ¶ added in v0.2.0

func EvaluateNodeOffline(ctx context.Context, nodes NodeSource, r Rule, now time.Time) (Rule, []string, error)

EvaluateNodeOffline runs one KindNodeOffline rule: it fires while any node's status is offline and resolves once every node is back. Like the other platform-wide node kinds it has no debounce, since the heartbeat sweep already applies its own grace before marking a node offline. LastValue is the count of offline nodes.

func EvaluateNodeResourceUsage ¶

func EvaluateNodeResourceUsage(ctx context.Context, nodes NodeSource, services NodeServiceSource, metrics MetricsSource, r Rule, cpuThresholdPercent, memoryThresholdBytes float64, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateNodeResourceUsage runs one KindNodeResourceUsage rule against every node in nodes and returns its updated evaluation state plus, when firing, one human-readable notice line per node whose summed CPU and/or memory usage meets or exceeds its threshold, for Engine to attach to the outgoing Event.

There is no real host-level CPU/memory collector in this codebase (unlike disk, telemetry has no /proc-based host reading): what exists is cpu_percent/memory_usage_bytes sampled per container (telemetry.Collector) and already summed across a node's placed services by internal/api's own node-metrics endpoint (nodeSummableMetrics). This evaluator reuses that exact same signal rather than inventing a new collector, taking each placed service's latest sample (not a time-series sum) since alerting only needs "right now," not a chart.

CPU and memory are evaluated together as one rule, not two, because they are the same "is this node under load" question from an operator's perspective, but they use genuinely different-shaped thresholds: CPU is compared as a percent (cpuThresholdPercent, the metric's own natural unit), memory as an absolute byte count (memoryThresholdBytes), since no honest node-capacity percentage exists for memory (see DefaultNodeMemoryThresholdBytes). LastValue only ever holds the highest summed CPU percent seen across nodes, matching EvaluateNodeDiskSpace's own "one representative number" convention; memory readings, having no comparable unit to take a single max across with CPU, are surfaced only in the per-node notice text.

Firing follows EvaluateNodeDiskSpace's shape: the instant either signal is over its threshold on any node, with no ForDuration debounce. A node with nothing placed on it, or whose placed services have no recent sample for either metric, is skipped, neither confirming nor denying the condition. A service whose own metrics query fails is logged and skipped, the same "one broken resource must not block the rest" stance EvaluateNodeDiskSpace already takes on a node.

func EvaluatePatchStatus ¶

func EvaluatePatchStatus(ctx context.Context, nodes NodeSource, metrics MetricsSource, r Rule, threshold float64, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluatePatchStatus runs one KindPatchStatus rule against every node in nodes and returns its updated evaluation state plus, when firing, one human-readable notice line per node whose latest security-patch count meets or exceeds threshold, for Engine to attach to the outgoing Event.

Firing follows EvaluateCertExpiry's shape, not EvaluateThreshold's: the instant any node is over threshold, with no ForDuration debounce. HostPatchCollector already only samples once an hour, so there is nothing left for a per-tick debounce to smooth out. LastValue is the highest security-patch count seen across every node with a recent sample, not just the ones over threshold, so an operator can see "how close" the fleet is even while every node is healthy, matching EvaluateCertExpiry's own LastValue convention.

A node with no sample inside patchStatusLookback (no supported package manager, or the collector hasn't run yet) is skipped, neither confirming nor denying the condition, the same "no recent data" stance EvaluateThreshold takes. A node whose own metrics query fails is logged and skipped too, rather than failing the whole rule: the same "one broken resource must not block the rest" stance ListCertificates takes on a single malformed certificate.

func EvaluateRestoreDrillFailed ¶ added in v0.2.0

func EvaluateRestoreDrillFailed(ctx context.Context, source RestoreDrillSource, r Rule, now time.Time) (Rule, string, error)

EvaluateRestoreDrillFailed runs one KindRestoreDrillFailed rule. It fires while any resource's latest restore drill is failed and resolves once a later drill passes.

func EvaluateSLOBurn ¶ added in v0.2.0

func EvaluateSLOBurn(ctx context.Context, src MetricsSource, r Rule, p SLOPolicy, now time.Time) (Rule, string, error)

EvaluateSLOBurn runs one KindSLOBurn rule. It returns the rule's next state and, while firing, the notice for the notification. A rule with no SLO config never fires.

func EvaluateScheduledTaskFailure ¶

func EvaluateScheduledTaskFailure(ctx context.Context, tasks ScheduledTaskSource, r Rule, now time.Time) (Rule, string, error)

EvaluateScheduledTaskFailure runs one KindScheduledTaskFailure rule against r.ScheduledTaskID's current ConsecutiveFailures count and returns its updated evaluation state plus, when firing, a human-readable notice line for Engine to attach to the outgoing Event.

Polls the task's own persisted counter rather than tracking anything itself, the same "read current state, no local history" shape EvaluateCertExpiry already uses: the counter is maintained by store.RecordScheduledTaskRun at the moment a run actually happens, decoupled from how often Engine.Tick itself runs.

Fires the instant ConsecutiveFailures crosses RestartCountThreshold, with no ForDuration debounce: requiring N consecutive failures is already the debounce, the same reasoning EvaluateCrashloop's own doc comment gives for RestartWindow.

func EvaluateScheduledTaskHealth ¶ added in v0.2.0

func EvaluateScheduledTaskHealth(ctx context.Context, tasks ScheduledTaskSource, r Rule, now time.Time) (Rule, string, error)

EvaluateScheduledTaskHealth evaluates a KindScheduledTaskMissed or KindScheduledTaskRunningLong rule from the job's persisted counters, so firing does not depend on how often the engine ticks.

func EvaluateThreshold ¶

func EvaluateThreshold(ctx context.Context, source MetricsSource, r Rule, now time.Time) (Rule, error)

EvaluateThreshold runs one KindThreshold rule against source and returns its updated evaluation state, to be persisted via DB.UpdateState by the caller (engine.go). Pure with respect to persistence: this function never writes to the database itself, so its state-machine logic (the part actually worth testing precisely) is testable without a real DB.

Semantics: the rule's condition must hold continuously, checked on every evaluation tick, for at least ForDuration before it graduates from "pending" to "firing" (avoiding a single noisy sample flapping a brand-new alert into existence). Once firing, it stays firing until a tick observes the condition no longer holding, at which point it resets to not-pending/not-firing immediately: there's no separate "resolving" debounce, a single good sample is enough to consider it recovered, asymmetric on purpose since false-positive resolution (briefly saying "recovered" when it isn't) is a much smaller problem than false-positive firing (paging someone for a blip).

func EvaluateVersionSkew ¶ added in v0.2.0

func EvaluateVersionSkew(ctx context.Context, settings VersionSkewSettingsSource, fetchers upgrade.Fetchers, cache *upgrade.Cache, r Rule, currentVersion string, now time.Time) (Rule, string, error)

EvaluateVersionSkew runs one KindVersionSkew rule: it fires while currentVersion is behind the configured channel's latest release. cache is checked before calling fetchers, so a rule enabled alongside Engine's own short evaluation interval never hits GitHub more often than upgrade.DefaultCacheTTL, the same cadence GET /api/v1/updates and updatecheck.Scheduler already hold themselves to.

func DefaultSLOPolicy ¶ added in v0.2.0

func DefaultSLOPolicy() SLOPolicy

DefaultSLOPolicy returns the SRE workbook defaults: 14.4x over 1h and 5m and 6x over 6h and 30m page, 3x over 1d and 2h and 1x over 3d and 6h open a ticket.

func SLOPolicyFromEnv ¶ added in v0.2.0

func SLOPolicyFromEnv() SLOPolicy

SLOPolicyFromEnv applies APP_SLO_* overrides to the defaults. A malformed or non-positive value keeps the default.

func ComputeSLOStatus ¶ added in v0.2.0

func ComputeSLOStatus(ctx context.Context, src MetricsSource, app string, cfg SLOConfig, p SLOPolicy, now time.Time) (SLOStatus, error)

ComputeSLOStatus queries every distinct window once and evaluates all tiers. Partial windows (an app newer than the window) are judged on the traffic that exists, and a tier whose long window saw fewer than MinRequests cannot fire.