alerting

package
v0.1.45 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package alerting is a SQL alert-rule evaluator in the Prometheus shape.

A Rule carries an `expr` that is a single SELECT. Every row it returns is one firing instance: the non-value columns become the instance's labels and the `value` column its sample, e.g.

SELECT ServiceName AS service, quantile(0.95)(Duration)/1e6 AS value
FROM otel_traces GROUP BY service HAVING value > 800

The package owns what is the same for every deployment: the SQL guard, the label rules, the inactive → pending → firing → resolved state machine with flap damping, a jittered scheduler sharded over forge/pkg/lease, and an Alertmanager v2 client. It owns nothing about WHERE rules come from, WHICH database an owner's query runs against, or where state is kept; those are the four interfaces below, declared here at the consumer.

An owner is an opaque string. The package never interprets it: it is handed to the Querier and recorded beside every instance, nothing more. Identity labels (who an alert belongs to) are supplied by the RuleSource on the Rule and are stamped LAST, so no query column or user label can override them.

A self-hoster runs it with chquery (a single-owner ClickHouse Querier), the pgstore Postgres StateStore (or NewMemoryStore) and any RuleSource.

Example

A single-owner deployment: a rules file, one ClickHouse, in-memory state. Swap oneRow for chquery.New(conn), printNotifier for alerting.NewAlertmanagerNotifier(nil, "http://alertmanager:9093"), and the MemoryStore for pgstore.New(db, "") to run it for real.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/reliant-labs/forge/pkg/alerting"
)

type oneRow struct{}

func (oneRow) Query(context.Context, string, string, alerting.QueryOptions) ([]alerting.Row, error) {
	return []alerting.Row{{"service": "api", "value": 912.0}}, nil
}

type printNotifier struct{}

func (printNotifier) Notify(_ context.Context, alerts []alerting.Alert) error {
	for _, a := range alerts {
		fmt.Println(a.Labels["alertname"], a.Labels["service"])
	}
	return nil
}

// A single-owner deployment: a rules file, one ClickHouse, in-memory state.
// Swap oneRow for chquery.New(conn), printNotifier for
// alerting.NewAlertmanagerNotifier(nil, "http://alertmanager:9093"), and the
// MemoryStore for pgstore.New(db, "") to run it for real.
func main() {
	ev, _ := alerting.NewEvaluator(alerting.EvaluatorDeps{
		Querier: oneRow{}, Notifier: printNotifier{}, Store: alerting.NewMemoryStore(),
	})
	rule := alerting.RuleSpec{
		Alert: "SlowService",
		Expr:  "SELECT service, quantile(0.95)(duration_ms) AS value FROM spans GROUP BY service HAVING value > 800",
	}.Rule("self", "prod", map[string]string{"reliant_org_id": "self"})

	_, _ = ev.EvaluateRule(context.Background(), rule, nil)
	_ = time.Second
}
Output:
SlowService api

Index

Examples

Constants

View Source
const MaxExprBytes = 16 << 10

MaxExprBytes bounds a rule's expr.

View Source
const MaxNameLen = 128

MaxNameLen bounds a rule name.

View Source
const ReservedPrefix = "reliant_"

ReservedPrefix is the label-name prefix only the RuleSource may set, through Rule.Identity. A column or label a rule author writes with this prefix is dropped, in any letter case.

View Source
const ValueColumn = "value"

ValueColumn is the column whose number is an instance's sample.

Variables

View Source
var ErrBadExpr = errors.New("alerting: expr refused")

ErrBadExpr marks an expr the guard refused.

View Source
var (
	// ErrBadResult marks a result the evaluator could not turn into instances.
	ErrBadResult = errors.New("alerting: unusable result")
)

Functions

func ValidateExpr

func ValidateExpr(expr string) error

ValidateExpr accepts exactly one read-only SELECT (optionally with a WITH prefix). It is a lexical check, not a parser, and is the FIRST of two lines of defence: the Querier must still run the statement with readonly settings and the evaluator's resource limits, which is what actually contains a query.

Types

type Alert

type Alert struct {
	Labels      map[string]string
	Annotations map[string]string
	StartsAt    time.Time
	// EndsAt is when the receiver should consider the alert resolved if it is
	// not refreshed: now+3×interval while firing, now once resolved.
	EndsAt time.Time
}

Alert is one alert as posted to a Notifier.

type AlertmanagerNotifier

type AlertmanagerNotifier struct {

	// GeneratorURL, when set, is attached to every alert.
	GeneratorURL string
	// contains filtered or unexported fields
}

AlertmanagerNotifier posts to an Alertmanager v2 API. Give it every replica of an HA Alertmanager (they gossip and de-duplicate); an alert is delivered if ANY replica accepts it.

func NewAlertmanagerNotifier

func NewAlertmanagerNotifier(client *http.Client, baseURLs ...string) *AlertmanagerNotifier

NewAlertmanagerNotifier builds a notifier. baseURLs are scheme://host:port. A nil client gets a 10s-timeout default.

func (*AlertmanagerNotifier) Notify

func (n *AlertmanagerNotifier) Notify(ctx context.Context, alerts []Alert) error

Notify posts alerts to every replica.

type Changes

type Changes struct {
	Upsert []Instance
	Delete []InstanceRef
	Events []Event
}

Changes is what one evaluation step writes.

type Duration

type Duration time.Duration

Duration is a time.Duration that JSON-encodes as a Go duration string ("5m", "90s"), the way Prometheus rule files spell `for`. A bare number is read as seconds.

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (Duration) Std

func (d Duration) Std() time.Duration

Std is the time.Duration.

func (*Duration) UnmarshalJSON

func (d *Duration) UnmarshalJSON(b []byte) error

UnmarshalJSON implements json.Unmarshaler.

type Evaluator

type Evaluator struct {
	// contains filtered or unexported fields
}

Evaluator runs rules one step at a time. It holds no timers; Scheduler drives it. Safe for concurrent use as long as one rule is never evaluated by two callers at once, which the Scheduler's lease guarantees.

func NewEvaluator

func NewEvaluator(d EvaluatorDeps) (*Evaluator, error)

NewEvaluator validates deps and builds an Evaluator.

func (*Evaluator) EffectiveInterval

func (e *Evaluator) EffectiveInterval(r Rule) time.Duration

EffectiveInterval is the interval a rule is evaluated at.

func (*Evaluator) EvaluateRule

func (e *Evaluator) EvaluateRule(ctx context.Context, rule Rule, prev []Instance) ([]Instance, error)

EvaluateRule runs one rule once. prev is the stored instances of this rule.

A rule that cannot be evaluated (a refused expr, a query error, an oversized or malformed result) changes nothing: its instances keep their state and any firing alert keeps being re-sent until it lapses on its own `endsAt`. Treating an error as "nothing returned" would resolve real incidents whenever the database hiccuped.

func (*Evaluator) Retire

func (e *Evaluator) Retire(ctx context.Context, key RuleKey, prev []Instance, interval time.Duration) ([]Instance, error)

Retire resolves the live instances of a rule that was removed.

func (*Evaluator) Select

func (e *Evaluator) Select(rules []Rule) (kept []Rule, dropped int)

Select applies the per-owner rule cap and orders rules by key, so which rules the cap drops is deterministic.

type EvaluatorDeps

type EvaluatorDeps struct {
	Querier  Querier
	Notifier Notifier
	Store    StateStore
	Limits   Limits
	Logger   *slog.Logger
	// Now is the clock; time.Now when nil.
	Now func() time.Time
}

EvaluatorDeps are the evaluator's collaborators.

type Event

type Event struct {
	RuleKey
	Fingerprint string
	Labels      map[string]string
	Transition  Transition
	Value       float64
	At          time.Time
}

Event is one history row.

type EventQuery

type EventQuery struct {
	Owner string
	// Env narrows to one env; empty means all of the owner's envs.
	Env string
	// Since is the oldest event wanted.
	Since time.Time
	// Limit defaults to 200 and is capped at 1000.
	Limit int
}

EventQuery selects history rows. Owner is required; the store never returns another owner's rows.

type FileRuleSource

type FileRuleSource struct {
	// Path of the rules document.
	Path string
	// Owner and Env key every rule. Owner may be any non-empty opaque
	// string; it only has to match what the Querier expects.
	Owner, Env string
	// Identity is stamped on every rule (see Rule.Identity).
	Identity map[string]string
	// Recheck is how often to stat the file. Default 5s.
	Recheck time.Duration
	Logger  *slog.Logger
	// contains filtered or unexported fields
}

FileRuleSource is a single-owner RuleSource that reads a rules document (RulesDocument, JSON) from a path, typically a mounted ConfigMap rendered by forge from the env's declared alerts. It is how a self-hosted, in-cluster evaluator gets its rules through the normal release path.

The file is re-stat'ed at most every Recheck and re-read only when its identity, size or mtime changed (a ConfigMap update swaps a symlink, which os.SameFile sees). A file that is missing, unreadable or invalid KEEPS the last good rules and warns, rate-limited: a broken edit must not silently delete every alert. Before the first good read there are no rules.

func (*FileRuleSource) Owners

func (f *FileRuleSource) Owners(context.Context) ([]string, error)

Owners implements RuleSource.

func (*FileRuleSource) Rules

func (f *FileRuleSource) Rules(_ context.Context, owner string) ([]Rule, error)

Rules implements RuleSource. An owner other than f.Owner has none.

type Instance

type Instance struct {
	RuleKey
	Fingerprint string
	// Labels are the final labels, identity included.
	Labels      map[string]string
	Annotations map[string]string
	State       State
	// ActiveAt is when the series was first seen in its current run.
	ActiveAt time.Time
	// FiringAt is when it became firing; zero while pending.
	FiringAt time.Time
	// ClearedAt is when a firing instance first stopped returning, while flap
	// damping holds it open; zero when it is returning.
	ClearedAt  time.Time
	ResolvedAt time.Time
	// ResolveSent records that the resolved notification was delivered.
	ResolveSent bool
	LastEvalAt  time.Time
	LastValue   float64
}

Instance is one label set of one rule, with its state.

type InstanceRef

type InstanceRef struct {
	RuleKey
	Fingerprint string
}

InstanceRef addresses one stored instance.

type Limits

type Limits struct {
	// MinInterval is the floor on a rule's interval.
	MinInterval time.Duration
	// MaxExecutionTime is the per-query timeout, applied server-side and to the
	// call's context.
	MaxExecutionTime time.Duration
	// MaxInstancesPerRule: a result with more rows than this is an error, and the
	// rule's existing instances are left untouched for that evaluation.
	MaxInstancesPerRule int
	// MaxRulesPerOwner: rules past this count (in name order) are not evaluated.
	MaxRulesPerOwner int
}

Limits bound what one rule and one owner may cost.

func DefaultLimits

func DefaultLimits() Limits

DefaultLimits are conservative.

type Locker

type Locker interface {
	Run(ctx context.Context, name string, fn func(ctx context.Context) error) error
}

Locker runs fn while holding a named, cluster-wide lease. forge/pkg/lease's *Locker satisfies it.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is a StateStore that keeps everything in process: for tests and for a single-process deployment that accepts losing state on restart.

func NewMemoryStore

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an empty store.

func (*MemoryStore) Apply

func (m *MemoryStore) Apply(_ context.Context, ch Changes) error

Apply implements StateStore.

func (*MemoryStore) Events

func (m *MemoryStore) Events(_ context.Context, q EventQuery) ([]Event, error)

Events implements StateStore. History is kept in memory without bound; a long-running process should use a persistent store.

func (*MemoryStore) LoadOwner

func (m *MemoryStore) LoadOwner(_ context.Context, owner string) ([]Instance, error)

LoadOwner implements StateStore.

func (*MemoryStore) Owners

func (m *MemoryStore) Owners(context.Context) ([]string, error)

Owners implements StateStore.

type Notifier

type Notifier interface {
	Notify(ctx context.Context, alerts []Alert) error
}

Notifier delivers alerts.

type Querier

type Querier interface {
	Query(ctx context.Context, owner, query string, opts QueryOptions) ([]Row, error)
}

Querier runs one guarded SELECT for an owner.

type QueryOptions

type QueryOptions struct {
	MaxExecutionTime time.Duration
	// MaxResultRows is one more than the instance cap, so exceeding the cap
	// surfaces as an error instead of a silently truncated result.
	MaxResultRows int
}

QueryOptions bound one evaluation. The Querier must apply them server-side.

func (QueryOptions) Limits

func (o QueryOptions) Limits() map[string]any

Limits are the per-query resource settings: a timeout and a row cap, both throwing on overflow rather than truncating.

A Querier that runs as an IMPERSONATED user (ClickHouse `EXECUTE AS`) sends only these. It must not send `readonly`: a readonly session refuses the EXECUTE AS statement itself, and the target user's settings profile (which should say readonly = 2 with MAX constraints) is what contains the query.

func (QueryOptions) Settings

func (o QueryOptions) Settings() map[string]any

Settings are Limits plus readonly=2 (no writes or DDL, while the per-query limits stay settable): what a Querier on its own, directly-authenticated connection sends.

type Row

type Row map[string]any

Row is one row a Querier returned: column name to value.

type Rule

type Rule struct {
	RuleKey

	// Expr is a single ClickHouse SELECT; see ValidateExpr.
	Expr string
	// For is how long a series must keep returning before it fires. Zero fires
	// on the first evaluation that returns it.
	For time.Duration
	// Interval is the evaluation period, raised to the scheduler's MinInterval.
	Interval time.Duration
	// KeepFiringFor holds a firing alert open this long after its series stops
	// returning (Prometheus's keep_firing_for). The hold is never shorter than
	// one evaluation, which is the flap damping every rule gets.
	KeepFiringFor time.Duration

	// Source is where the rule was declared.
	Source Source

	// Labels and Annotations are the rule author's. Labels whose name starts
	// with "reliant_" are dropped, and `alertname` is always the rule's name.
	Labels      map[string]string
	Annotations map[string]string

	// Identity is set by the RuleSource, never by the rule author, and is
	// applied after every other label: it is how a consumer routes an alert to
	// the owner it belongs to.
	Identity map[string]string
}

Rule is one alert rule, as declared.

type RuleKey

type RuleKey struct {
	Owner string
	Env   string
	Name  string
}

RuleKey identifies one rule. Owner and Env are opaque to this package.

func (RuleKey) String

func (k RuleKey) String() string

String is the key as a log field.

type RuleSet

type RuleSet []RuleSpec

RuleSet is a list of specs, the "alerts" section of a rules document.

func ParseRulesDocument

func ParseRulesDocument(b []byte) (RuleSet, error)

ParseRulesDocument decodes and validates a document.

func (RuleSet) Validate

func (rs RuleSet) Validate() error

Validate checks every spec and that names are unique.

type RuleSource

type RuleSource interface {
	// Owners lists every owner that has rules.
	Owners(ctx context.Context) ([]string, error)
	// Rules lists an owner's rules across all of its envs.
	Rules(ctx context.Context, owner string) ([]Rule, error)
}

RuleSource lists what to evaluate.

type RuleSpec

type RuleSpec struct {
	// Alert is the rule's name, unique within an env: becomes `alertname`.
	Alert string `json:"alert"`
	// Expr is a single ClickHouse SELECT; every row it returns is a firing
	// instance. See ValidateExpr and the package doc.
	Expr string `json:"expr"`
	// For is how long a series must keep returning before it fires.
	For Duration `json:"for,omitempty"`
	// Interval is the evaluation period; zero means the evaluator's floor.
	Interval Duration `json:"interval,omitempty"`
	// KeepFiringFor holds a firing alert open this long after it stops
	// returning, so a flapping series does not page on every recovery.
	KeepFiringFor Duration `json:"keep_firing_for,omitempty"`
	// Labels carry routing data (`severity` is a label). Names starting with
	// "reliant_" are reserved and dropped.
	Labels      map[string]string `json:"labels,omitempty"`
	Annotations map[string]string `json:"annotations,omitempty"`
	// Source is user or default; empty means user.
	Source Source `json:"source,omitempty"`
}

RuleSpec is one declared alert rule, in the Prometheus rule shape with a SQL `expr`. It is the unit a release's declared shape records and a rules file holds: no owner, no env and no identity, because those belong to WHERE the rule is deployed, not to what it says. A RuleSource turns a spec into a Rule with Rule.Identity set.

func (RuleSpec) Rule

func (s RuleSpec) Rule(owner, env string, identity map[string]string) Rule

Rule binds the spec to an owner and env, with the identity labels the RuleSource vouches for.

func (RuleSpec) Validate

func (s RuleSpec) Validate() error

Validate checks everything that can be checked without a database. A bad spec is refused at render/lint time instead of failing every evaluation.

type RulesDocument

type RulesDocument struct {
	Alerts RuleSet `json:"alerts"`
}

RulesDocument is the JSON document FileRuleSource reads: the alerts section of a release's declared shape, serialized. Other sections (routes, say) are ignored, so the same file can carry them.

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler evaluates every owner's rules on their intervals.

Owners hash into Shards partitions and each partition is a lease, so exactly one replica evaluates an owner at a time and a replica that dies hands its partitions over within the lease's retry interval. Every replica contends for every shard, each starting after a random delay so partitions spread across replicas rather than the fastest one taking all.

Within a shard each rule is due on its own interval, first offset by a jitter derived from its key so a restart does not fire every rule in the same second.

func NewScheduler

func NewScheduler(d SchedulerDeps) (*Scheduler, error)

NewScheduler validates deps.

func (*Scheduler) Run

func (s *Scheduler) Run(ctx context.Context) error

Run blocks until ctx is done.

func (*Scheduler) ShardOf

func (s *Scheduler) ShardOf(owner string) int

ShardOf is the partition an owner belongs to.

type SchedulerDeps

type SchedulerDeps struct {
	Rules RuleSource
	Store StateStore
	Eval  *Evaluator
	// Locker shards evaluation across replicas. Nil runs every shard in this
	// process, which is right for a single replica.
	Locker Locker
	// Shards is how many lease-owned partitions owners hash into. Default 16.
	Shards int
	// LeasePrefix names the leases: "<prefix>/shard/<n>". Default "alerting".
	LeasePrefix string
	// Tick is how often a shard looks for due rules. Default 10s.
	Tick time.Duration
	// Concurrency bounds owners evaluated at once within a shard. Default 4.
	Concurrency int
	Logger      *slog.Logger
}

SchedulerDeps are the scheduler's collaborators.

type Source

type Source string

Source says where a declared rule came from.

const (
	// SourceUser: written by the project's author in its own deploy config.
	SourceUser Source = "user"
	// SourceDefault: generated by forge for every project; the author may
	// disable or override it.
	SourceDefault Source = "default"
)

type State

type State string

State is an instance's position in the state machine. An instance that is not stored is inactive.

const (
	StatePending  State = "pending"
	StateFiring   State = "firing"
	StateResolved State = "resolved"
)

type StateStore

type StateStore interface {
	// LoadOwner returns every stored instance of an owner, resolved included.
	LoadOwner(ctx context.Context, owner string) ([]Instance, error)
	// Owners lists owners that have stored instances.
	Owners(ctx context.Context) ([]string, error)
	// Apply persists one rule's changes atomically.
	Apply(ctx context.Context, ch Changes) error
	// Events reads the transition history, newest first.
	Events(ctx context.Context, q EventQuery) ([]Event, error)
}

StateStore keeps instance state and the transition history.

type StoreError

type StoreError struct {
	Rule RuleKey
	Err  error
}

StoreError is a failed write to the StateStore. The caller's in-memory copy of the rule's state may be ahead of what was stored and must be reloaded.

func (*StoreError) Error

func (e *StoreError) Error() string

func (*StoreError) Unwrap

func (e *StoreError) Unwrap() error

type Transition

type Transition string

Transition names a state change in the history.

const (
	TransitionPending  Transition = "pending"
	TransitionFiring   Transition = "firing"
	TransitionResolved Transition = "resolved"
	// TransitionInactive: a pending series stopped returning before `for` held.
	TransitionInactive Transition = "inactive"
	// TransitionRemoved: the rule was deleted while the instance was live.
	TransitionRemoved Transition = "removed"
)

Directories

Path Synopsis
Package chquery is a single-owner alerting.Querier over one ClickHouse connection: the owner key is ignored and every rule runs against the connection's database.
Package chquery is a single-owner alerting.Querier over one ClickHouse connection: the owner key is ignored and every rule runs against the connection's database.
Package pgstore is the Postgres alerting.StateStore.
Package pgstore is the Postgres alerting.StateStore.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL