Documentation
¶
Overview ¶
Package alerts sends mail and webhook alerts when watched visors go offline, when they come back, and when their storage starts to fail.
Index ¶
- Constants
- Variables
- func CheckStorage() (readOnly bool, ioErrors int, last string)
- func LANProbe(ctx context.Context, ip string) (bool, error)
- func RecoveryReason(uptime, outage time.Duration) string
- func RootReadOnly(r io.Reader) bool
- func StorageErrors(lines []string, since time.Duration) (n int, last string)
- func Subject(ev Event) string
- type Duration
- type Event
- type Result
- type SMTP
- type Saved
- type Sender
- type Service
- func (s *Service) Run(ctx context.Context, interval, grace time.Duration)
- func (s *Service) SetSettings(next Settings) error
- func (s *Service) Settings() Settings
- func (s *Service) Status() Status
- func (s *Service) Test(ctx context.Context) ([]Result, error)
- func (s *Service) Tick(ctx context.Context, now time.Time)
- type Settings
- type Status
- type Storage
- type Target
- type TargetState
- type Tracker
- type WebhookPayload
Constants ¶
const ( DiagLANUp = "board answers on the LAN, skywire is down" DiagLANDown = "board does not answer on the LAN: powered off, hung or unplugged" DiagNoNeighbor = "no neighbor to check from" DiagNoLocalIP = "board's LAN address is unknown, no LAN check" )
LAN check results for a board alert.
const ( MinOfflineAfter = time.Minute MaxOfflineAfter = time.Hour DefaultOfflineAfter = 5 * time.Minute )
Offline delay bounds. A visor that comes back within the delay sends nothing.
const ( KindOffline = "offline" KindOnline = "online" KindStorage = "storage" KindConflict = "ip-conflict" KindTest = "test" // KindLinkLost is a visor that is up but has not reconnected to its // hypervisor for the offline delay. KindLinkLost = "link-lost" )
Kind of an alert event.
const ( StateOnline = "online" // StateReconnecting is a visor whose link to the hypervisor dropped // but which still answers the hypervisor's call back. StateReconnecting = "reconnecting" // StateRestarting is a visor that said it is going down to restart. StateRestarting = "restarting" // StateDetached is a visor that removed this hypervisor. It is not // watched. StateDetached = "detached" StateOffline = "offline" )
States of a watched visor. A hypervisor reports all five for the visors it manages; a watch list visor is only online or offline.
const ( ScopeBoard = "board" ScopeSite = "site" )
Scope of an outage: one board, or every watched visor behind one public IP.
const ( ReasonRebooted = "rebooted during the outage (likely power loss or crash)" ReasonStayedUp = "stayed up (network or internet outage)" )
Recovery reasons, from the board's uptime against the outage length.
const CheckInterval = 30 * time.Second
CheckInterval is how often the watched set is checked.
const ConflictWindow = 10 * time.Minute
ConflictWindow is how recently a visor must have been online for its LAN IP to count in a conflict, so a dead board's stale address does not.
const DefaultDiskWarn = 90
DefaultDiskWarn is the disk use percent that raises a storage warning.
const DiagLinkRestored = "the visor reconnected to its hypervisor"
DiagLinkRestored is the diagnosis of the recovery after KindLinkLost.
const IOErrorWindow = 24 * time.Hour
IOErrorWindow is how far back the kernel log counts storage errors.
const SiteWindow = 2 * time.Minute
SiteWindow is how close together every watched visor behind one public IP must go offline to count as one site outage.
Variables ¶
var ProbePorts = []int{22, 80, 443, 8000}
ProbePorts are the TCP ports LANProbe tries: ssh, http, https and the hypervisor UI.
Functions ¶
func CheckStorage ¶
CheckStorage reports whether the root filesystem is read-only and the storage errors in the kernel log over the last IOErrorWindow. Reading the kernel log needs root or CAP_SYSLOG; without it the count is zero.
func LANProbe ¶
LANProbe reports whether ip answers on the local network. A port that accepts or refuses the connection both prove a live host, so no ICMP and no privileges are needed. Only private, link-local and loopback addresses are probed, so it cannot be used to scan the internet.
func RecoveryReason ¶
RecoveryReason says whether a board rebooted during an outage of the given length: an uptime shorter than the outage means it booted after it went down. Empty when the uptime is unknown.
func RootReadOnly ¶
RootReadOnly reports whether /proc/mounts, read from r, has the root filesystem mounted read-only. The last mount on / wins.
func StorageErrors ¶
StorageErrors counts the storage error records among /dev/kmsg lines ("prio,seq,usec,flags;message", Documentation/ABI/testing/dev-kmsg) logged at or after since, a time since boot, and returns the last one.
Types ¶
type Duration ¶
Duration is a time.Duration written as a string such as "5m" in JSON.
func (Duration) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*Duration) UnmarshalJSON ¶
UnmarshalJSON accepts "5m" or a number of nanoseconds.
type Event ¶
type Event struct {
Kind string `json:"event"`
Scope string `json:"kind,omitempty"`
// Target is the board, or for a site only its public IP.
Target Target `json:"target"`
// Site is the public IP of a site event and Members its visors.
Site string `json:"site,omitempty"`
Members []Target `json:"members,omitempty"`
Since time.Time `json:"offline_since,omitzero"`
At time.Time `json:"at"`
Diagnosis string `json:"diagnosis,omitempty"`
RecoveryReason string `json:"recovery_reason,omitempty"`
Warnings []string `json:"warnings,omitempty"`
}
Event is one alert to deliver.
type Result ¶
type Result struct {
Channel string `json:"channel"`
Target string `json:"target"`
Err string `json:"error,omitempty"`
}
Result is the outcome of one delivery channel.
type SMTP ¶
type SMTP struct {
Host string `json:"host"`
Port int `json:"port,omitempty"`
Username string `json:"username,omitempty"`
Password string `json:"password,omitempty"`
StartTLS bool `json:"starttls,omitempty"`
// PasswordSet is only set in reads, which never carry the password.
PasswordSet bool `json:"password_set,omitempty"`
}
SMTP is an optional mail relay. Without one, mail goes straight to each recipient domain's MX on port 25.
type Saved ¶
type Saved struct {
Since time.Time `json:"since,omitzero"`
Site string `json:"site,omitempty"`
Storage []string `json:"storage,omitempty"`
// Lost is when the link lost alert's outage began.
Lost time.Time `json:"lost,omitzero"`
}
Saved is one visor's alert state kept across restarts.
type Sender ¶
type Sender struct {
// Watcher is the public key of the visor doing the watching.
Watcher cipher.PubKey
// Hostname is the watcher's host name, for the default sender address.
Hostname string
// LookupMX and MXPort are overridable for tests. Defaults are
// net.DefaultResolver.LookupMX and 25.
LookupMX func(ctx context.Context, domain string) ([]*net.MX, error)
MXPort int
// TLSConfig is used for STARTTLS and implicit TLS. Nil verifies the
// server name against the system roots.
TLSConfig *tls.Config
// HTTPClient posts webhooks. Nil is a client with a 15s timeout.
HTTPClient *http.Client
}
Sender delivers events by mail and webhook.
type Service ¶
type Service struct {
Sender *Sender
// Targets returns the current liveness of every watched visor.
Targets func(ctx context.Context, s Settings) []Target
// Probe asks the visor via to check whether ip answers on its LAN.
// Nil skips the LAN check.
Probe func(ctx context.Context, via cipher.PubKey, ip string) (bool, error)
// Self is the watching visor, a LAN neighbor when it shares a site.
// Nil means it is never one.
Self func() Target
// Logf logs delivery failures. Nil discards.
Logf func(format string, args ...any)
// contains filtered or unexported fields
}
Service checks the watched visors and delivers alerts.
func NewService ¶
NewService returns a Service whose settings and alert state live in dir. base is the config's alerts block, used until settings are saved at runtime.
func (*Service) Run ¶
Run checks every interval until ctx is done. Visors that went down while nothing watched get floor-grace plus the delay before an alert.
func (*Service) SetSettings ¶
SetSettings validates, applies and saves new settings. An empty relay password keeps the current one.
type Settings ¶
type Settings struct {
EmailTo []string `json:"email_to,omitempty"`
OfflineAfter Duration `json:"offline_after,omitempty"`
SMTP *SMTP `json:"smtp,omitempty"`
Webhook string `json:"webhook,omitempty"`
Watch []cipher.PubKey `json:"watch,omitempty"`
// From is the sender address. Default: skywire-alerts@<hostname>.
From string `json:"from,omitempty"`
// DiskWarnPercent is the disk use that raises a storage warning.
// Default: DefaultDiskWarn.
DiskWarnPercent float64 `json:"disk_warn_percent,omitempty"`
}
Settings is the alerts block of the visor config.
func Merge ¶
Merge returns next, keeping the current relay password when next names the same relay and leaves the password empty.
type Status ¶
type Status struct {
Settings Settings `json:"settings"`
Enabled bool `json:"enabled"`
OfflineAfter string `json:"offline_after"`
Watched []TargetState `json:"watched"`
LastCheck time.Time `json:"last_check,omitzero"`
LastSent time.Time `json:"last_sent,omitzero"`
LastResults []Result `json:"last_results,omitempty"`
Pending int `json:"pending,omitempty"`
// LastEvents are the most recent alerts, newest last.
LastEvents []Event `json:"last_events,omitempty"`
}
Status is what a read of the alerts returns. It never carries the relay password.
type Storage ¶
type Storage struct {
ReadOnlyRoot bool `json:"root_read_only,omitempty"`
IOErrors int `json:"io_errors,omitempty"`
LastIOError string `json:"last_io_error,omitempty"`
DiskUsedPercent float64 `json:"disk_used_percent,omitempty"`
}
Storage is a board's storage health, the early signs of a failing SD card.
type Target ¶
type Target struct {
PK cipher.PubKey `json:"pk"`
Hostname string `json:"hostname,omitempty"`
PublicIP string `json:"public_ip,omitempty"`
LocalIP string `json:"local_ip,omitempty"`
// LastSeen is when the visor was last seen online, if known.
LastSeen time.Time `json:"last_seen,omitzero"`
Online bool `json:"online"`
// State is one of the State values. Empty means StateOnline or
// StateOffline, from Online.
State string `json:"state,omitempty"`
// Source is "hypervisor" or "watch".
Source string `json:"source,omitempty"`
// Uptime is how long the board has been up at the check, the host's
// uptime when the visor reports it and the visor's own otherwise.
Uptime Duration `json:"uptime,omitempty"`
// Storage is the board's last reported storage health.
Storage *Storage `json:"storage,omitempty"`
}
Target is one watched visor as seen in one check.
type TargetState ¶
type TargetState struct {
Target
OfflineSince time.Time `json:"offline_since,omitzero"`
Alerted bool `json:"alerted,omitempty"`
// Site is set while the visor is part of an alerted site outage.
Site string `json:"site,omitempty"`
StorageWarnings []string `json:"storage_warnings,omitempty"`
// LinkLostSince is when a reconnecting visor lost its hypervisor link.
LinkLostSince time.Time `json:"link_lost_since,omitzero"`
LinkAlerted bool `json:"link_alerted,omitempty"`
}
TargetState is a watched visor's alert state, for status reads.
type Tracker ¶
type Tracker struct {
// Floor is the earliest time an offline period counts from, so visors
// that went down while nothing watched get the full delay to return.
Floor time.Time
// DiskWarn is the disk use percent that raises a storage warning.
// Zero is DefaultDiskWarn.
DiskWarn float64
// contains filtered or unexported fields
}
Tracker turns liveness checks into offline and recovery events. One offline event fires once a visor has been offline for the delay, one recovery event when an alerted visor is back, and nothing for a visor that comes back within the delay. When every watched visor behind one public IP goes down together, one site event replaces the board events.
func NewTracker ¶
NewTracker returns a Tracker restored from saved, so a restart neither repeats an alert nor loses the recovery mail.
func (*Tracker) Observe ¶
Observe records one check of the whole watched set at now and returns the events it causes. Visors no longer in targets, or detached, are dropped silently. Reconnecting and restarting visors are held: their offline delay does not run.
func (*Tracker) States ¶
func (t *Tracker) States() []TargetState
States lists every tracked visor, sorted by public key.
type WebhookPayload ¶
type WebhookPayload struct {
Event string `json:"event"`
// Kind is "board" or "site".
Kind string `json:"kind,omitempty"`
PK string `json:"pk,omitempty"`
Hostname string `json:"hostname,omitempty"`
PublicIP string `json:"public_ip,omitempty"`
LocalIP string `json:"local_ip,omitempty"`
LastSeen time.Time `json:"last_seen,omitzero"`
OfflineSince time.Time `json:"offline_since,omitzero"`
At time.Time `json:"at"`
Diagnosis string `json:"diagnosis,omitempty"`
RecoveryReason string `json:"recovery_reason,omitempty"`
Warnings []string `json:"warnings,omitempty"`
// Members are the visors of a site event.
Members []Target `json:"members,omitempty"`
Watcher string `json:"watcher"`
Text string `json:"text"`
}
WebhookPayload is the JSON body of a webhook POST.