Documentation
¶
Overview ¶
Package whalert routes the webhook receiver's conditions that end in lost or late data to notification channels, at a warning and a critical level (#2149): the spool filling or holding events too long, the drain failing, a replica holding events that stopped reporting, refusals, unauthorized requests, slow acknowledgements, compaction failing or behind, a source gone silent, and the spool volume running out of space. A sender does not wait, so these are told to someone as they happen rather than found on the page.
Index ¶
Constants ¶
const ( UnauthorizedSpan = 15 * time.Minute // AckSpan is the span the acknowledgement p99 is judged over. AckSpan = 5 * time.Minute )
The spans the per-source conditions are judged over.
const AlertFamily = "webhook"
AlertFamily is the levelalert family the webhook alerts are recorded under.
const DefaultInterval = time.Minute
DefaultInterval is how often the conditions are evaluated: a sender does not wait, so this is minutes, not the storage alerts' quarter hour.
const LockKey int64 = 4713210403
LockKey is the advisory lock one replica evaluates the alerts under. The storage alerts hold 4713210402.
const SettingsSection = "webhook_alert"
SettingsSection is the platform_settings section holding the settings. It is a stored key.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Deps ¶
type Deps struct {
DB *sql.DB
Settings SettingsStore
Reader Reader
Sources SourceLister
Engine *levelalert.Engine
// Grace is the compactor's grace after a window ends.
Grace time.Duration
BaseURL string
Interval time.Duration
Now func() time.Time
}
Deps is what the service reads and sends through.
type Input ¶
type Input struct {
Channels *[]string `json:"channels,omitempty"`
SpoolFillPercent *levelalert.Thresholds `json:"spool_fill_percent,omitempty"`
SpoolAgePercent *levelalert.Thresholds `json:"spool_age_percent,omitempty"`
DrainFailures *levelalert.Thresholds `json:"drain_failures,omitempty"`
DrainFailingMinutes *float64 `json:"drain_failing_minutes,omitempty"`
NotReportingSeconds *float64 `json:"not_reporting_seconds,omitempty"`
SpoolVolumeFreePercent *levelalert.Thresholds `json:"spool_volume_free_percent,omitempty"`
Defaults *SourceRules `json:"defaults,omitempty"`
Sources *map[string]SourceRules `json:"sources,omitempty"`
}
Input is the write shape. An absent field keeps the stored value; Sources, when present, replaces every source's rules.
type Reader ¶
type Reader interface {
SpoolStates(ctx context.Context) ([]whstore.SpoolState, error)
AlertReadings(ctx context.Context, countsSince, ackSince time.Time) (map[string]*whstore.SourceReadings, error)
}
Reader is what an evaluation reads. *whstore.Store satisfies it.
type Readings ¶
type Readings struct {
Now time.Time
// Sources are the sources that exist, by name; Enabled is whether each
// receives.
Sources map[string]bool
Spools []whstore.SpoolState
// PerSource is each source's request counts, latency and compaction.
PerSource map[string]*whstore.SourceReadings
// Grace is how long after a window ends the compactor waits for its
// segments, so a window is late only past it.
Grace time.Duration
// BaseURL is the portal's address; the alerts link to the webhooks
// page under it. Empty omits the link.
BaseURL string
}
Readings is what one evaluation judges.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service evaluates the alerts on one replica at a time.
type Settings ¶
type Settings struct {
Channels []string `json:"channels"`
// SpoolFillPercent is a replica's spool bytes as a percent of its
// max_bytes.
SpoolFillPercent levelalert.Thresholds `json:"spool_fill_percent"`
// SpoolAgePercent is the oldest event waiting as a percent of max_age.
SpoolAgePercent levelalert.Thresholds `json:"spool_age_percent"`
// DrainFailures is how many drains in a row have failed; a streak
// longer than DrainFailingMinutes is critical however many it counts.
DrainFailures levelalert.Thresholds `json:"drain_failures"`
DrainFailingMinutes float64 `json:"drain_failing_minutes"`
// NotReportingSeconds is how long a replica whose spool holds events
// may go without reporting before it is critical.
NotReportingSeconds float64 `json:"not_reporting_seconds"`
// SpoolVolumeFreePercent is the free space left on the volume a spool
// is on.
SpoolVolumeFreePercent levelalert.Thresholds `json:"spool_volume_free_percent"`
// Sources holds the per-source conditions: the defaults every source is
// judged by, and any source's own.
Defaults SourceRules `json:"defaults"`
Sources map[string]SourceRules `json:"sources"`
}
Settings is the configuration: the channels, and each condition's thresholds. A zero threshold is off.
func Defaults ¶
func Defaults() Settings
Defaults is the configuration before an administrator writes any.
func (Settings) RulesFor ¶
func (s Settings) RulesFor(source string) SourceRules
RulesFor is the rules source is judged by: its own over the defaults.
type SettingsStore ¶
type SettingsStore interface {
Get(ctx context.Context) (settingsjson.Stored[Settings], error)
Set(ctx context.Context, v Settings, author string) error
}
SettingsStore reads and writes the section.
func NewSettingsStore ¶
func NewSettingsStore(db *sql.DB) SettingsStore
NewSettingsStore returns the section over db.
type SourceLister ¶
SourceLister lists the sources. *whsource.Store satisfies it.
type SourceRules ¶
type SourceRules struct {
// RefusalMinutes is the span refusals (spool_full, write_failed,
// buffer_full) are judged over: any in it is a warning, one in every
// minute of it is critical. Zero is off.
RefusalMinutes *int `json:"refusal_minutes,omitempty"`
// last 15 minutes that were refused as unauthorized.
UnauthorizedPercent *levelalert.Thresholds `json:"unauthorized_percent,omitempty"`
// AckP99MS is the 99th percentile acknowledgement over the last five
// minutes, in milliseconds.
AckP99MS *levelalert.Thresholds `json:"ack_p99_ms,omitempty"`
// CompactionOverdueMinutes is how long past its end and the compactor's
// grace the oldest window owed a compaction has waited.
CompactionOverdueMinutes *levelalert.Thresholds `json:"compaction_overdue_minutes,omitempty"`
// SilentAfterMinutes, when set, warns once the source has written no
// segment for that long. Off unless a source opts in.
SilentAfterMinutes *int `json:"silent_after_minutes,omitempty"`
}
SourceRules are the conditions judged per source.