Documentation
¶
Overview ¶
Package alerting is a SQL alert-rule evaluator in the Prometheus shape.
A Rule carries an `expr` that is a single SELECT. Every row it returns is one firing instance: the non-value columns become the instance's labels and the `value` column its sample, e.g.
SELECT ServiceName AS service, quantile(0.95)(Duration)/1e6 AS value FROM otel_traces GROUP BY service HAVING value > 800
The package owns what is the same for every deployment: the SQL guard, the label rules, the inactive → pending → firing → resolved state machine with flap damping, a jittered scheduler sharded over forge/pkg/lease, and an Alertmanager v2 client. It owns nothing about WHERE rules come from, WHICH database an owner's query runs against, or where state is kept; those are the four interfaces below, declared here at the consumer.
An owner is an opaque string. The package never interprets it: it is handed to the Querier and recorded beside every instance, nothing more. Identity labels (who an alert belongs to) are supplied by the RuleSource on the Rule and are stamped LAST, so no query column or user label can override them.
A self-hoster runs it with chquery (a single-owner ClickHouse Querier), the pgstore Postgres StateStore (or NewMemoryStore) and any RuleSource.
Example ¶
A single-owner deployment: a rules file, one ClickHouse, in-memory state. Swap oneRow for chquery.New(conn), printNotifier for alerting.NewAlertmanagerNotifier(nil, "http://alertmanager:9093"), and the MemoryStore for pgstore.New(db, "") to run it for real.
package main
import (
"context"
"fmt"
"time"
"github.com/reliant-labs/forge/pkg/alerting"
)
type oneRow struct{}
func (oneRow) Query(context.Context, string, string, alerting.QueryOptions) ([]alerting.Row, error) {
return []alerting.Row{{"service": "api", "value": 912.0}}, nil
}
type printNotifier struct{}
func (printNotifier) Notify(_ context.Context, alerts []alerting.Alert) error {
for _, a := range alerts {
fmt.Println(a.Labels["alertname"], a.Labels["service"])
}
return nil
}
// A single-owner deployment: a rules file, one ClickHouse, in-memory state.
// Swap oneRow for chquery.New(conn), printNotifier for
// alerting.NewAlertmanagerNotifier(nil, "http://alertmanager:9093"), and the
// MemoryStore for pgstore.New(db, "") to run it for real.
func main() {
ev, _ := alerting.NewEvaluator(alerting.EvaluatorDeps{
Querier: oneRow{}, Notifier: printNotifier{}, Store: alerting.NewMemoryStore(),
})
rule := alerting.RuleSpec{
Alert: "SlowService",
Expr: "SELECT service, quantile(0.95)(duration_ms) AS value FROM spans GROUP BY service HAVING value > 800",
}.Rule("self", "prod", map[string]string{"reliant_org_id": "self"})
_, _ = ev.EvaluateRule(context.Background(), rule, nil)
_ = time.Second
}
Output: SlowService api
Index ¶
- Constants
- Variables
- func ValidateExpr(expr string) error
- type Alert
- type AlertmanagerNotifier
- type Changes
- type Duration
- type Evaluator
- func (e *Evaluator) EffectiveInterval(r Rule) time.Duration
- func (e *Evaluator) EvaluateRule(ctx context.Context, rule Rule, prev []Instance) ([]Instance, error)
- func (e *Evaluator) Retire(ctx context.Context, key RuleKey, prev []Instance, interval time.Duration) ([]Instance, error)
- func (e *Evaluator) Select(rules []Rule) (kept []Rule, dropped int)
- type EvaluatorDeps
- type Event
- type EventQuery
- type FileRuleSource
- type Instance
- type InstanceRef
- type Limits
- type Locker
- type MemoryStore
- type Notifier
- type Querier
- type QueryOptions
- type Row
- type Rule
- type RuleKey
- type RuleSet
- type RuleSource
- type RuleSpec
- type RulesDocument
- type Scheduler
- type SchedulerDeps
- type Source
- type State
- type StateStore
- type StoreError
- type Transition
Examples ¶
Constants ¶
const MaxExprBytes = 16 << 10
MaxExprBytes bounds a rule's expr.
const MaxNameLen = 128
MaxNameLen bounds a rule name.
const ReservedPrefix = "reliant_"
ReservedPrefix is the label-name prefix only the RuleSource may set, through Rule.Identity. A column or label a rule author writes with this prefix is dropped, in any letter case.
const ValueColumn = "value"
ValueColumn is the column whose number is an instance's sample.
Variables ¶
var ErrBadExpr = errors.New("alerting: expr refused")
ErrBadExpr marks an expr the guard refused.
var ( // ErrBadResult marks a result the evaluator could not turn into instances. ErrBadResult = errors.New("alerting: unusable result") )
Functions ¶
func ValidateExpr ¶
ValidateExpr accepts exactly one read-only SELECT (optionally with a WITH prefix). It is a lexical check, not a parser, and is the FIRST of two lines of defence: the Querier must still run the statement with readonly settings and the evaluator's resource limits, which is what actually contains a query.
Types ¶
type Alert ¶
type Alert struct {
Labels map[string]string
Annotations map[string]string
StartsAt time.Time
// EndsAt is when the receiver should consider the alert resolved if it is
// not refreshed: now+3×interval while firing, now once resolved.
EndsAt time.Time
}
Alert is one alert as posted to a Notifier.
type AlertmanagerNotifier ¶
type AlertmanagerNotifier struct {
// GeneratorURL, when set, is attached to every alert.
GeneratorURL string
// contains filtered or unexported fields
}
AlertmanagerNotifier posts to an Alertmanager v2 API. Give it every replica of an HA Alertmanager (they gossip and de-duplicate); an alert is delivered if ANY replica accepts it.
func NewAlertmanagerNotifier ¶
func NewAlertmanagerNotifier(client *http.Client, baseURLs ...string) *AlertmanagerNotifier
NewAlertmanagerNotifier builds a notifier. baseURLs are scheme://host:port. A nil client gets a 10s-timeout default.
type Changes ¶
type Changes struct {
Upsert []Instance
Delete []InstanceRef
Events []Event
}
Changes is what one evaluation step writes.
type Duration ¶
Duration is a time.Duration that JSON-encodes as a Go duration string ("5m", "90s"), the way Prometheus rule files spell `for`. A bare number is read as seconds.
func (Duration) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*Duration) UnmarshalJSON ¶
UnmarshalJSON implements json.Unmarshaler.
type Evaluator ¶
type Evaluator struct {
// contains filtered or unexported fields
}
Evaluator runs rules one step at a time. It holds no timers; Scheduler drives it. Safe for concurrent use as long as one rule is never evaluated by two callers at once, which the Scheduler's lease guarantees.
func NewEvaluator ¶
func NewEvaluator(d EvaluatorDeps) (*Evaluator, error)
NewEvaluator validates deps and builds an Evaluator.
func (*Evaluator) EffectiveInterval ¶
EffectiveInterval is the interval a rule is evaluated at.
func (*Evaluator) EvaluateRule ¶
func (e *Evaluator) EvaluateRule(ctx context.Context, rule Rule, prev []Instance) ([]Instance, error)
EvaluateRule runs one rule once. prev is the stored instances of this rule.
A rule that cannot be evaluated (a refused expr, a query error, an oversized or malformed result) changes nothing: its instances keep their state and any firing alert keeps being re-sent until it lapses on its own `endsAt`. Treating an error as "nothing returned" would resolve real incidents whenever the database hiccuped.
type EvaluatorDeps ¶
type EvaluatorDeps struct {
Querier Querier
Notifier Notifier
Store StateStore
Limits Limits
Logger *slog.Logger
// Now is the clock; time.Now when nil.
Now func() time.Time
}
EvaluatorDeps are the evaluator's collaborators.
type Event ¶
type Event struct {
RuleKey
Fingerprint string
Labels map[string]string
Transition Transition
Value float64
At time.Time
}
Event is one history row.
type EventQuery ¶
type EventQuery struct {
Owner string
// Env narrows to one env; empty means all of the owner's envs.
Env string
// Since is the oldest event wanted.
Since time.Time
// Limit defaults to 200 and is capped at 1000.
Limit int
}
EventQuery selects history rows. Owner is required; the store never returns another owner's rows.
type FileRuleSource ¶
type FileRuleSource struct {
// Path of the rules document.
Path string
// Owner and Env key every rule. Owner may be any non-empty opaque
// string; it only has to match what the Querier expects.
Owner, Env string
// Identity is stamped on every rule (see Rule.Identity).
Identity map[string]string
// Recheck is how often to stat the file. Default 5s.
Recheck time.Duration
Logger *slog.Logger
// contains filtered or unexported fields
}
FileRuleSource is a single-owner RuleSource that reads a rules document (RulesDocument, JSON) from a path, typically a mounted ConfigMap rendered by forge from the env's declared alerts. It is how a self-hosted, in-cluster evaluator gets its rules through the normal release path.
The file is re-stat'ed at most every Recheck and re-read only when its identity, size or mtime changed (a ConfigMap update swaps a symlink, which os.SameFile sees). A file that is missing, unreadable or invalid KEEPS the last good rules and warns, rate-limited: a broken edit must not silently delete every alert. Before the first good read there are no rules.
type Instance ¶
type Instance struct {
RuleKey
Fingerprint string
// Labels are the final labels, identity included.
Labels map[string]string
Annotations map[string]string
State State
// ActiveAt is when the series was first seen in its current run.
ActiveAt time.Time
// FiringAt is when it became firing; zero while pending.
FiringAt time.Time
// ClearedAt is when a firing instance first stopped returning, while flap
// damping holds it open; zero when it is returning.
ClearedAt time.Time
ResolvedAt time.Time
// ResolveSent records that the resolved notification was delivered.
ResolveSent bool
LastEvalAt time.Time
LastValue float64
}
Instance is one label set of one rule, with its state.
type InstanceRef ¶
InstanceRef addresses one stored instance.
type Limits ¶
type Limits struct {
// MinInterval is the floor on a rule's interval.
MinInterval time.Duration
// MaxExecutionTime is the per-query timeout, applied server-side and to the
// call's context.
MaxExecutionTime time.Duration
// MaxInstancesPerRule: a result with more rows than this is an error, and the
// rule's existing instances are left untouched for that evaluation.
MaxInstancesPerRule int
// MaxRulesPerOwner: rules past this count (in name order) are not evaluated.
MaxRulesPerOwner int
}
Limits bound what one rule and one owner may cost.
type Locker ¶
type Locker interface {
Run(ctx context.Context, name string, fn func(ctx context.Context) error) error
}
Locker runs fn while holding a named, cluster-wide lease. forge/pkg/lease's *Locker satisfies it.
type MemoryStore ¶
type MemoryStore struct {
// contains filtered or unexported fields
}
MemoryStore is a StateStore that keeps everything in process: for tests and for a single-process deployment that accepts losing state on restart.
func (*MemoryStore) Apply ¶
func (m *MemoryStore) Apply(_ context.Context, ch Changes) error
Apply implements StateStore.
func (*MemoryStore) Events ¶
func (m *MemoryStore) Events(_ context.Context, q EventQuery) ([]Event, error)
Events implements StateStore. History is kept in memory without bound; a long-running process should use a persistent store.
type Querier ¶
type Querier interface {
Query(ctx context.Context, owner, query string, opts QueryOptions) ([]Row, error)
}
Querier runs one guarded SELECT for an owner.
type QueryOptions ¶
type QueryOptions struct {
MaxExecutionTime time.Duration
// MaxResultRows is one more than the instance cap, so exceeding the cap
// surfaces as an error instead of a silently truncated result.
MaxResultRows int
}
QueryOptions bound one evaluation. The Querier must apply them server-side.
func (QueryOptions) Limits ¶
func (o QueryOptions) Limits() map[string]any
Limits are the per-query resource settings: a timeout and a row cap, both throwing on overflow rather than truncating.
A Querier that runs as an IMPERSONATED user (ClickHouse `EXECUTE AS`) sends only these. It must not send `readonly`: a readonly session refuses the EXECUTE AS statement itself, and the target user's settings profile (which should say readonly = 2 with MAX constraints) is what contains the query.
func (QueryOptions) Settings ¶
func (o QueryOptions) Settings() map[string]any
Settings are Limits plus readonly=2 (no writes or DDL, while the per-query limits stay settable): what a Querier on its own, directly-authenticated connection sends.
type Rule ¶
type Rule struct {
RuleKey
// Expr is a single ClickHouse SELECT; see ValidateExpr.
Expr string
// For is how long a series must keep returning before it fires. Zero fires
// on the first evaluation that returns it.
For time.Duration
// Interval is the evaluation period, raised to the scheduler's MinInterval.
Interval time.Duration
// KeepFiringFor holds a firing alert open this long after its series stops
// returning (Prometheus's keep_firing_for). The hold is never shorter than
// one evaluation, which is the flap damping every rule gets.
KeepFiringFor time.Duration
// Source is where the rule was declared.
Source Source
// Labels and Annotations are the rule author's. Labels whose name starts
// with "reliant_" are dropped, and `alertname` is always the rule's name.
Labels map[string]string
Annotations map[string]string
// Identity is set by the RuleSource, never by the rule author, and is
// applied after every other label: it is how a consumer routes an alert to
// the owner it belongs to.
Identity map[string]string
}
Rule is one alert rule, as declared.
type RuleSet ¶
type RuleSet []RuleSpec
RuleSet is a list of specs, the "alerts" section of a rules document.
func ParseRulesDocument ¶
ParseRulesDocument decodes and validates a document.
type RuleSource ¶
type RuleSource interface {
// Owners lists every owner that has rules.
Owners(ctx context.Context) ([]string, error)
// Rules lists an owner's rules across all of its envs.
Rules(ctx context.Context, owner string) ([]Rule, error)
}
RuleSource lists what to evaluate.
type RuleSpec ¶
type RuleSpec struct {
// Alert is the rule's name, unique within an env: becomes `alertname`.
Alert string `json:"alert"`
// Expr is a single ClickHouse SELECT; every row it returns is a firing
// instance. See ValidateExpr and the package doc.
Expr string `json:"expr"`
// For is how long a series must keep returning before it fires.
For Duration `json:"for,omitempty"`
// Interval is the evaluation period; zero means the evaluator's floor.
Interval Duration `json:"interval,omitempty"`
// KeepFiringFor holds a firing alert open this long after it stops
// returning, so a flapping series does not page on every recovery.
KeepFiringFor Duration `json:"keep_firing_for,omitempty"`
// Labels carry routing data (`severity` is a label). Names starting with
// "reliant_" are reserved and dropped.
Labels map[string]string `json:"labels,omitempty"`
Annotations map[string]string `json:"annotations,omitempty"`
// Source is user or default; empty means user.
Source Source `json:"source,omitempty"`
}
RuleSpec is one declared alert rule, in the Prometheus rule shape with a SQL `expr`. It is the unit a release's declared shape records and a rules file holds: no owner, no env and no identity, because those belong to WHERE the rule is deployed, not to what it says. A RuleSource turns a spec into a Rule with Rule.Identity set.
type RulesDocument ¶
type RulesDocument struct {
Alerts RuleSet `json:"alerts"`
}
RulesDocument is the JSON document FileRuleSource reads: the alerts section of a release's declared shape, serialized. Other sections (routes, say) are ignored, so the same file can carry them.
type Scheduler ¶
type Scheduler struct {
// contains filtered or unexported fields
}
Scheduler evaluates every owner's rules on their intervals.
Owners hash into Shards partitions and each partition is a lease, so exactly one replica evaluates an owner at a time and a replica that dies hands its partitions over within the lease's retry interval. Every replica contends for every shard, each starting after a random delay so partitions spread across replicas rather than the fastest one taking all.
Within a shard each rule is due on its own interval, first offset by a jitter derived from its key so a restart does not fire every rule in the same second.
func NewScheduler ¶
func NewScheduler(d SchedulerDeps) (*Scheduler, error)
NewScheduler validates deps.
type SchedulerDeps ¶
type SchedulerDeps struct {
Rules RuleSource
Store StateStore
Eval *Evaluator
// Locker shards evaluation across replicas. Nil runs every shard in this
// process, which is right for a single replica.
Locker Locker
// Shards is how many lease-owned partitions owners hash into. Default 16.
Shards int
// LeasePrefix names the leases: "<prefix>/shard/<n>". Default "alerting".
LeasePrefix string
// Tick is how often a shard looks for due rules. Default 10s.
Tick time.Duration
// Concurrency bounds owners evaluated at once within a shard. Default 4.
Concurrency int
Logger *slog.Logger
}
SchedulerDeps are the scheduler's collaborators.
type State ¶
type State string
State is an instance's position in the state machine. An instance that is not stored is inactive.
type StateStore ¶
type StateStore interface {
// LoadOwner returns every stored instance of an owner, resolved included.
LoadOwner(ctx context.Context, owner string) ([]Instance, error)
// Owners lists owners that have stored instances.
Owners(ctx context.Context) ([]string, error)
// Apply persists one rule's changes atomically.
Apply(ctx context.Context, ch Changes) error
// Events reads the transition history, newest first.
Events(ctx context.Context, q EventQuery) ([]Event, error)
}
StateStore keeps instance state and the transition history.
type StoreError ¶
StoreError is a failed write to the StateStore. The caller's in-memory copy of the rule's state may be ahead of what was stored and must be reloaded.
func (*StoreError) Error ¶
func (e *StoreError) Error() string
func (*StoreError) Unwrap ¶
func (e *StoreError) Unwrap() error
type Transition ¶
type Transition string
Transition names a state change in the history.
const ( TransitionPending Transition = "pending" TransitionFiring Transition = "firing" TransitionResolved Transition = "resolved" // TransitionInactive: a pending series stopped returning before `for` held. TransitionInactive Transition = "inactive" // TransitionRemoved: the rule was deleted while the instance was live. TransitionRemoved Transition = "removed" )
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package chquery is a single-owner alerting.Querier over one ClickHouse connection: the owner key is ignored and every rule runs against the connection's database.
|
Package chquery is a single-owner alerting.Querier over one ClickHouse connection: the owner key is ignored and every rule runs against the connection's database. |
|
Package pgstore is the Postgres alerting.StateStore.
|
Package pgstore is the Postgres alerting.StateStore. |