alerting

func ListCertificates ¶

func ListCertificates(ctx context.Context, source CertSource, warningWindow time.Duration, now time.Time, logger *slog.Logger) ([]CertInfo, error)

ListCertificates returns every certificate currently in source, parsed from its own stored PEM bytes, sorted by domain. A control plane that has never issued a certificate returns an empty slice, not an error. A single malformed or since-deleted entry is skipped and logged, never fails the whole list, the same "one broken resource must not block the rest" shape used throughout this codebase.

func Open ¶

func Open(ctx context.Context, path string) (*DB, error)

Open opens (creating if needed) the SQLite database at path, applies pragmas, and runs every pending migration, matching internal/store and internal/telemetry's own Open functions exactly.

func NewDeployDispatcher ¶

func NewDeployDispatcher(db *DB, client *http.Client, sender email.Sender, logger *slog.Logger) *DeployDispatcher

NewDeployDispatcher builds a DeployDispatcher. client and logger default if nil; sender may be nil (an email-kind target then fails with a clear "not configured" error).

func NewEngine ¶

func NewEngine(rules RuleStore, metrics MetricsSource, logs LogsSource, tracker *RestartTracker, certs CertSource, scheduledTasks ScheduledTaskSource, certExpiryWarningWindow, certRenewalStalledThreshold time.Duration, nodes NodeSource, patchStatusThreshold, nodeDiskSpaceThreshold float64, nodeServices NodeServiceSource, nodeCPUThreshold, nodeMemoryThreshold float64, domainApps AppDomainSource, domainChecker DomainCheckSource, domainHealthCheckInterval time.Duration, newNotifier func(Rule) Notifier, logger *slog.Logger) *Engine

NewEngine builds an Engine. newNotifier defaults to a Notifier with no email capability configured if nil; a real caller passes a closure capturing an email.Sender instead. certs may be nil if no cert storage is configured; a kind=cert_expiry rule then logs a warning and is skipped each tick rather than evaluated. nodes and scheduledTasks may likewise be nil, in which case a kind=patch_status or kind=scheduled_task_failure rule is skipped the same way. certExpiryWarningWindow and certRenewalStalledThreshold fall back to DefaultCertExpiryWarningWindow/DefaultCertRenewalStalledThreshold when passed as 0; patchStatusThreshold falls back to DefaultPatchStatusThreshold, nodeDiskSpaceThreshold falls back to DefaultNodeDiskSpaceThresholdPercent, and nodeCPUThreshold/ nodeMemoryThreshold fall back to DefaultNodeCPUThresholdPercent/ DefaultNodeMemoryThresholdBytes, all the same way. nodeServices may be nil, in which case a kind=node_resource_usage rule is skipped the same way a kind=patch_status rule is when nodes is nil. domainApps and domainChecker may likewise be nil, in which case a kind=domain_health rule is skipped the same way; domainHealthCheckInterval falls back to DefaultDomainHealthCheckInterval when passed as 0.

func CheckNodeAlertStatus ¶

func CheckNodeAlertStatus(ctx context.Context, node store.Node, services NodeServiceSource, metrics MetricsSource,
	patchThreshold, diskThresholdPercent, cpuThresholdPercent, memoryThresholdBytes float64, now time.Time, logger *slog.Logger) NodeAlertStatus

CheckNodeAlertStatus live-evaluates patch_status, node_disk_space, and node_resource_usage for node, right now, rather than waiting for Engine's next tick or reading a rule's stored aggregate LastValue (which only ever holds the worst value seen across every node, not which node it came from). It calls each evaluator's own per-node check helper directly so this can never drift from what a real rule would decide. A threshold of 0 falls back to that kind's own default, the same convention EvaluatePatchStatus/EvaluateNodeDiskSpace/ EvaluateNodeResourceUsage already use.

func NewNotifier ¶

func NewNotifier(client *http.Client, sender email.Sender, r Rule) Notifier

NewNotifier builds the right Notifier for r.NotifyKind. An unknown or empty NotifyKind falls back to NotifyGeneric rather than erroring, so a typo'd notify_kind still notifies someone, diagnosable from the payload shape. sender may be nil, in which case an email-kind rule fails with a clear "not configured" error.

func NewRestartTracker ¶

func NewRestartTracker() *RestartTracker

NewRestartTracker builds an empty RestartTracker.

func EvaluateCertExpiry ¶

func EvaluateCertExpiry(ctx context.Context, certs CertSource, snapshots CertExpiryObservationStore, r Rule, warningWindow, stalledThreshold time.Duration, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateCertExpiry runs one KindCertExpiry rule against every certificate currently in certs and returns its updated evaluation state plus, when firing, one human-readable notice line per non-healthy certificate for Engine to attach to the outgoing Event.

The rule fires (satisfied, in advanceState terms) the instant any certificate is "expiring_soon" or "expired", with no ForDuration debounce: unlike a noisy metric sample, expiry status only moves monotonically with real time, so there is nothing to debounce. LastValue is the number of days until the earliest-expiring certificate (negative once expired), across every certificate, not just the non-healthy ones, so an operator can see "how close" even while healthy.

Per non-healthy domain, snapshots records the NotAfter it had when it first entered this expiry episode (EpisodeNotAfter/EpisodeStartedAt). If a later tick finds the same domain still non-healthy with an unchanged NotAfter for at least stalledThreshold, that domain's notice is marked as a renewal that appears stalled rather than a plain warning: Caddy's own renewal should have already succeeded well before the warning window even opened (DefaultCertRenewalStalledThreshold's own doc comment), so a stuck NotAfter this deep into the window is a stronger, more actionable signal than "getting close to expiry."

func EvaluateCrashloop ¶

func EvaluateCrashloop(tracker *RestartTracker, r Rule, now time.Time) Rule

EvaluateCrashloop runs one KindCrashloop rule against tracker and returns its updated evaluation state, the crashloop equivalent of EvaluateThreshold, sharing the same pending/firing debounce logic via advanceState. Passes forDuration=0: a crashloop rule fires the instant its restart count crosses the threshold, with no additional debounce layered on top, because RestartWindow itself already is the debounce (requiring N restarts within a real time window is already "sustained," unlike a single noisy metric sample).

func EvaluateDomainHealth ¶

func EvaluateDomainHealth(ctx context.Context, apps AppDomainSource, checker DomainCheckSource, r Rule, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateDomainHealth runs one KindDomainHealth rule against every domain currently configured on the app it's scoped to (ResourceID) and returns its updated evaluation state plus, when firing, one human-readable notice line per unhealthy domain, for Engine to attach to the outgoing Event.

Unlike EvaluateCertExpiry/EvaluatePatchStatus/EvaluateNodeDiskSpace (platform-wide, ResourceID only a display label), a domain_health rule's ResourceID is real: it picks out exactly which app's own domains this rule checks, the same app-scoped shape EvaluateThreshold/ EvaluateCrashloop use. An app with no domains configured, or one that has since been deleted, is treated like "no recent data": neither confirms nor denies.

r.ForDuration optionally debounces a single unhealthy check the same way EvaluateThreshold's own ForDuration does, since a real DNS lookup can blip; zero (the default) fires the instant any domain is unhealthy, matching every other non-threshold kind's own default.

func EvaluateNodeDiskSpace ¶

func EvaluateNodeDiskSpace(ctx context.Context, nodes NodeSource, metrics MetricsSource, r Rule, thresholdPercent float64, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateNodeDiskSpace runs one KindNodeDiskSpace rule against every node in nodes and returns its updated evaluation state plus, when firing, one human-readable notice line per node whose latest used-disk percentage meets or exceeds thresholdPercent, for Engine to attach to the outgoing Event.

Firing follows EvaluatePatchStatus's shape, not EvaluateThreshold's: the instant any node is over threshold, with no ForDuration debounce. LastValue is the highest used-disk percentage seen across every node with a recent sample, not just the ones over threshold, matching EvaluatePatchStatus's own LastValue convention.

A node missing either a disk_used_bytes or disk_total_bytes sample inside diskSpaceLookback (collector hasn't run yet), or reporting a total of zero or less, is skipped, neither confirming nor denying the condition. A node whose metrics query fails is logged and skipped too, the same "one broken resource must not block the rest" stance EvaluatePatchStatus already takes.

func EvaluateNodeResourceUsage ¶

func EvaluateNodeResourceUsage(ctx context.Context, nodes NodeSource, services NodeServiceSource, metrics MetricsSource, r Rule, cpuThresholdPercent, memoryThresholdBytes float64, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluateNodeResourceUsage runs one KindNodeResourceUsage rule against every node in nodes and returns its updated evaluation state plus, when firing, one human-readable notice line per node whose summed CPU and/or memory usage meets or exceeds its threshold, for Engine to attach to the outgoing Event.

There is no real host-level CPU/memory collector in this codebase (unlike disk, telemetry has no /proc-based host reading): what exists is cpu_percent/memory_usage_bytes sampled per container (telemetry.Collector) and already summed across a node's placed services by internal/api's own node-metrics endpoint (nodeSummableMetrics). This evaluator reuses that exact same signal rather than inventing a new collector, taking each placed service's latest sample (not a time-series sum) since alerting only needs "right now," not a chart.

CPU and memory are evaluated together as one rule, not two, because they are the same "is this node under load" question from an operator's perspective, but they use genuinely different-shaped thresholds: CPU is compared as a percent (cpuThresholdPercent, the metric's own natural unit), memory as an absolute byte count (memoryThresholdBytes), since no honest node-capacity percentage exists for memory (see DefaultNodeMemoryThresholdBytes). LastValue only ever holds the highest summed CPU percent seen across nodes, matching EvaluateNodeDiskSpace's own "one representative number" convention; memory readings, having no comparable unit to take a single max across with CPU, are surfaced only in the per-node notice text.

Firing follows EvaluateNodeDiskSpace's shape: the instant either signal is over its threshold on any node, with no ForDuration debounce. A node with nothing placed on it, or whose placed services have no recent sample for either metric, is skipped, neither confirming nor denying the condition. A service whose own metrics query fails is logged and skipped, the same "one broken resource must not block the rest" stance EvaluateNodeDiskSpace already takes on a node.

func EvaluatePatchStatus ¶

func EvaluatePatchStatus(ctx context.Context, nodes NodeSource, metrics MetricsSource, r Rule, threshold float64, now time.Time, logger *slog.Logger) (Rule, []string, error)

EvaluatePatchStatus runs one KindPatchStatus rule against every node in nodes and returns its updated evaluation state plus, when firing, one human-readable notice line per node whose latest security-patch count meets or exceeds threshold, for Engine to attach to the outgoing Event.

Firing follows EvaluateCertExpiry's shape, not EvaluateThreshold's: the instant any node is over threshold, with no ForDuration debounce. HostPatchCollector already only samples once an hour, so there is nothing left for a per-tick debounce to smooth out. LastValue is the highest security-patch count seen across every node with a recent sample, not just the ones over threshold, so an operator can see "how close" the fleet is even while every node is healthy, matching EvaluateCertExpiry's own LastValue convention.

A node with no sample inside patchStatusLookback (no supported package manager, or the collector hasn't run yet) is skipped, neither confirming nor denying the condition, the same "no recent data" stance EvaluateThreshold takes. A node whose own metrics query fails is logged and skipped too, rather than failing the whole rule: the same "one broken resource must not block the rest" stance ListCertificates takes on a single malformed certificate.

func EvaluateScheduledTaskFailure ¶

func EvaluateScheduledTaskFailure(ctx context.Context, tasks ScheduledTaskSource, r Rule, now time.Time) (Rule, string, error)

EvaluateScheduledTaskFailure runs one KindScheduledTaskFailure rule against r.ScheduledTaskID's current ConsecutiveFailures count and returns its updated evaluation state plus, when firing, a human-readable notice line for Engine to attach to the outgoing Event.

Polls the task's own persisted counter rather than tracking anything itself, the same "read current state, no local history" shape EvaluateCertExpiry already uses: the counter is maintained by store.RecordScheduledTaskRun at the moment a run actually happens, decoupled from how often Engine.Tick itself runs.

Fires the instant ConsecutiveFailures crosses RestartCountThreshold, with no ForDuration debounce: requiring N consecutive failures is already the debounce, the same reasoning EvaluateCrashloop's own doc comment gives for RestartWindow.

func EvaluateThreshold ¶

func EvaluateThreshold(ctx context.Context, source MetricsSource, r Rule, now time.Time) (Rule, error)

EvaluateThreshold runs one KindThreshold rule against source and returns its updated evaluation state, to be persisted via DB.UpdateState by the caller (engine.go). Pure with respect to persistence: this function never writes to the database itself, so its state-machine logic (the part actually worth testing precisely) is testable without a real DB.

Semantics: the rule's condition must hold continuously, checked on every evaluation tick, for at least ForDuration before it graduates from "pending" to "firing" (avoiding a single noisy sample flapping a brand-new alert into existence). Once firing, it stays firing until a tick observes the condition no longer holding, at which point it resets to not-pending/not-firing immediately: there's no separate "resolving" debounce, a single good sample is enough to consider it recovered, asymmetric on purpose since false-positive resolution (briefly saying "recovered" when it isn't) is a much smaller problem than false-positive firing (paging someone for a blip).