whalert

package
v1.147.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 17, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package whalert routes the webhook receiver's conditions that end in lost or late data to notification channels, at a warning and a critical level (#2149): the spool filling or holding events too long, the drain failing, a replica holding events that stopped reporting, refusals, unauthorized requests, slow acknowledgements, compaction failing or behind, a source gone silent, and the spool volume running out of space. A sender does not wait, so these are told to someone as they happen rather than found on the page.

Index

Constants

View Source
const (
	// UnauthorizedSpan is the span the unauthorized share is judged over.
	UnauthorizedSpan = 15 * time.Minute
	// AckSpan is the span the acknowledgement p99 is judged over.
	AckSpan = 5 * time.Minute
)

The spans the per-source conditions are judged over.

View Source
const AlertFamily = "webhook"

AlertFamily is the levelalert family the webhook alerts are recorded under.

View Source
const DefaultInterval = time.Minute

DefaultInterval is how often the conditions are evaluated: a sender does not wait, so this is minutes, not the storage alerts' quarter hour.

View Source
const LockKey int64 = 4713210403

LockKey is the advisory lock one replica evaluates the alerts under. The storage alerts hold 4713210402.

View Source
const SettingsSection = "webhook_alert"

SettingsSection is the platform_settings section holding the settings. It is a stored key.

Variables

This section is empty.

Functions

func Evaluate

func Evaluate(settings Settings, r Readings) []levelalert.Finding

Evaluate returns every condition at warning or critical under settings. It reads nothing; Readings carries it all.

Types

type Deps

type Deps struct {
	DB       *sql.DB
	Settings SettingsStore
	Reader   Reader
	Sources  SourceLister
	Engine   *levelalert.Engine
	// Grace is the compactor's grace after a window ends.
	Grace    time.Duration
	BaseURL  string
	Interval time.Duration
	Now      func() time.Time
}

Deps is what the service reads and sends through.

type Input

type Input struct {
	Channels               *[]string               `json:"channels,omitempty"`
	SpoolFillPercent       *levelalert.Thresholds  `json:"spool_fill_percent,omitempty"`
	SpoolAgePercent        *levelalert.Thresholds  `json:"spool_age_percent,omitempty"`
	DrainFailures          *levelalert.Thresholds  `json:"drain_failures,omitempty"`
	DrainFailingMinutes    *float64                `json:"drain_failing_minutes,omitempty"`
	NotReportingSeconds    *float64                `json:"not_reporting_seconds,omitempty"`
	SpoolVolumeFreePercent *levelalert.Thresholds  `json:"spool_volume_free_percent,omitempty"`
	Defaults               *SourceRules            `json:"defaults,omitempty"`
	Sources                *map[string]SourceRules `json:"sources,omitempty"`
}

Input is the write shape. An absent field keeps the stored value; Sources, when present, replaces every source's rules.

func (Input) Apply

func (in Input) Apply(stored Settings) (Settings, error)

Apply returns stored with in written over it, or why in is invalid.

type Reader

type Reader interface {
	SpoolStates(ctx context.Context) ([]whstore.SpoolState, error)
	AlertReadings(ctx context.Context, countsSince, ackSince time.Time) (map[string]*whstore.SourceReadings, error)
}

Reader is what an evaluation reads. *whstore.Store satisfies it.

type Readings

type Readings struct {
	Now time.Time
	// Sources are the sources that exist, by name; Enabled is whether each
	// receives.
	Sources map[string]bool
	Spools  []whstore.SpoolState
	// PerSource is each source's request counts, latency and compaction.
	PerSource map[string]*whstore.SourceReadings
	// Grace is how long after a window ends the compactor waits for its
	// segments, so a window is late only past it.
	Grace time.Duration
	// BaseURL is the portal's address; the alerts link to the webhooks
	// page under it. Empty omits the link.
	BaseURL string
}

Readings is what one evaluation judges.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service evaluates the alerts on one replica at a time.

func New

func New(d Deps) *Service

New returns the service, or nil without a database.

func (*Service) Pass

func (s *Service) Pass(ctx context.Context) error

Pass is one evaluation: read, judge, and route what it found.

func (*Service) Start

func (s *Service) Start(ctx context.Context)

Start runs the evaluation loop. Nil-safe.

func (*Service) Stop

func (s *Service) Stop()

Stop ends the loop and waits for it. Nil-safe; a second call is a no-op.

type Settings

type Settings struct {
	Channels []string `json:"channels"`
	// SpoolFillPercent is a replica's spool bytes as a percent of its
	// max_bytes.
	SpoolFillPercent levelalert.Thresholds `json:"spool_fill_percent"`
	// SpoolAgePercent is the oldest event waiting as a percent of max_age.
	SpoolAgePercent levelalert.Thresholds `json:"spool_age_percent"`
	// DrainFailures is how many drains in a row have failed; a streak
	// longer than DrainFailingMinutes is critical however many it counts.
	DrainFailures       levelalert.Thresholds `json:"drain_failures"`
	DrainFailingMinutes float64               `json:"drain_failing_minutes"`
	// NotReportingSeconds is how long a replica whose spool holds events
	// may go without reporting before it is critical.
	NotReportingSeconds float64 `json:"not_reporting_seconds"`
	// SpoolVolumeFreePercent is the free space left on the volume a spool
	// is on.
	SpoolVolumeFreePercent levelalert.Thresholds `json:"spool_volume_free_percent"`
	// Sources holds the per-source conditions: the defaults every source is
	// judged by, and any source's own.
	Defaults SourceRules            `json:"defaults"`
	Sources  map[string]SourceRules `json:"sources"`
}

Settings is the configuration: the channels, and each condition's thresholds. A zero threshold is off.

func Defaults

func Defaults() Settings

Defaults is the configuration before an administrator writes any.

func (Settings) RulesFor

func (s Settings) RulesFor(source string) SourceRules

RulesFor is the rules source is judged by: its own over the defaults.

type SettingsStore

type SettingsStore interface {
	Get(ctx context.Context) (settingsjson.Stored[Settings], error)
	Set(ctx context.Context, v Settings, author string) error
}

SettingsStore reads and writes the section.

func NewSettingsStore

func NewSettingsStore(db *sql.DB) SettingsStore

NewSettingsStore returns the section over db.

type SourceLister

type SourceLister interface {
	List(ctx context.Context) ([]whsource.Source, error)
}

SourceLister lists the sources. *whsource.Store satisfies it.

type SourceRules

type SourceRules struct {
	// RefusalMinutes is the span refusals (spool_full, write_failed,
	// buffer_full) are judged over: any in it is a warning, one in every
	// minute of it is critical. Zero is off.
	RefusalMinutes *int `json:"refusal_minutes,omitempty"`
	// UnauthorizedPercent is the share of the source's requests over the
	// last 15 minutes that were refused as unauthorized.
	UnauthorizedPercent *levelalert.Thresholds `json:"unauthorized_percent,omitempty"`
	// AckP99MS is the 99th percentile acknowledgement over the last five
	// minutes, in milliseconds.
	AckP99MS *levelalert.Thresholds `json:"ack_p99_ms,omitempty"`
	// CompactionOverdueMinutes is how long past its end and the compactor's
	// grace the oldest window owed a compaction has waited.
	CompactionOverdueMinutes *levelalert.Thresholds `json:"compaction_overdue_minutes,omitempty"`
	// SilentAfterMinutes, when set, warns once the source has written no
	// segment for that long. Off unless a source opts in.
	SilentAfterMinutes *int `json:"silent_after_minutes,omitempty"`
}

SourceRules are the conditions judged per source.

type View

type View struct {
	Settings
	UpdatedBy string `json:"updated_by,omitempty"`
	UpdatedAt string `json:"updated_at,omitempty"`
}

View is the read shape.

func Read

func Read(ctx context.Context, store SettingsStore) (View, error)

Read returns the stored settings, or the defaults before any are written.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL