Documentation
¶
Overview ¶
Package alerts sends mail and webhook alerts when watched visors go offline, when they come back, and when their storage starts to fail.
Index ¶
- Constants
- Variables
- func CheckStorage() (readOnly bool, ioErrors int, last string)
- func DataRoot() string
- func LANProbe(ctx context.Context, ip string) (bool, error)
- func PushID(endpoint string) string
- func RecoveryReason(uptime, outage time.Duration) string
- func RootReadOnly(r io.Reader) bool
- func StorageErrors(lines []string, since time.Duration) (n int, last string)
- func Subject(ev Event) string
- func VAPIDPublicKey(key *ecdsa.PrivateKey) string
- type Duration
- type Event
- type MX
- type PushKeys
- type PushPayload
- type PushSubscription
- type Result
- type SMTP
- type Saved
- type Sender
- type Service
- func (s *Service) PushAdd(sub PushSubscription) (PushSubscription, error)
- func (s *Service) PushRemove(id string) (int, error)
- func (s *Service) Run(ctx context.Context, interval, grace time.Duration)
- func (s *Service) SetSettings(next Settings) error
- func (s *Service) Settings() Settings
- func (s *Service) Status() Status
- func (s *Service) Test(ctx context.Context) ([]Result, error)
- func (s *Service) Tick(ctx context.Context, now time.Time)
- type Settings
- type Status
- type Storage
- type Target
- type TargetState
- type Tracker
- type WebhookPayload
Constants ¶
const ( DiagLANUp = "board answers on the LAN, skywire is down" DiagLANDown = "board does not answer on the LAN: powered off, hung or unplugged" DiagNoNeighbor = "no neighbor to check from" DiagNoLocalIP = "board's LAN address is unknown, no LAN check" )
LAN check results for a board alert.
const ( MinOfflineAfter = time.Minute MaxOfflineAfter = time.Hour DefaultOfflineAfter = 5 * time.Minute )
Offline delay bounds. A visor that comes back within the delay sends nothing.
const ( KindOffline = "offline" KindOnline = "online" KindStorage = "storage" KindConflict = "ip-conflict" KindTest = "test" // KindLinkLost is a visor that is up but has not reconnected to its // hypervisor for the offline delay. KindLinkLost = "link-lost" )
Kind of an alert event.
const ( StateOnline = "online" // StateReconnecting is a visor whose link to the hypervisor dropped // but which still answers the hypervisor's call back. StateReconnecting = "reconnecting" // StateRestarting is a visor that said it is going down to restart. StateRestarting = "restarting" // StateDetached is a visor that removed this hypervisor. It is not // watched. StateDetached = "detached" StateOffline = "offline" )
States of a watched visor. A hypervisor reports all five for the visors it manages; a watch list visor is only online or offline.
const ( ScopeBoard = "board" ScopeSite = "site" )
Scope of an outage: one board, or every watched visor behind one public IP.
const ( ReasonRebooted = "rebooted during the outage (likely power loss or crash)" ReasonStayedUp = "stayed up (network or internet outage)" )
Recovery reasons, from the board's uptime against the outage length.
const CheckInterval = 30 * time.Second
CheckInterval is how often the watched set is checked.
const ConflictWindow = 10 * time.Minute
ConflictWindow is how recently a visor must have been online for its LAN IP to count in a conflict, so a dead board's stale address does not.
const DefaultDiskWarn = 90
DefaultDiskWarn is the disk use percent that raises a storage warning.
const DefaultPushSubject = "https://github.com/skycoin/skywire"
DefaultPushSubject is the VAPID contact (RFC 8292 section 2.1) when no email address is set.
const DiagLinkRestored = "the visor reconnected to its hypervisor"
DiagLinkRestored is the diagnosis of the recovery after KindLinkLost.
const IOErrorWindow = 24 * time.Hour
IOErrorWindow is how far back the kernel log counts storage errors.
const SiteWindow = 2 * time.Minute
SiteWindow is how close together every watched visor behind one public IP must go offline to count as one site outage.
Variables ¶
var ProbePorts = []int{22, 80, 443, 8000}
ProbePorts are the TCP ports LANProbe tries: ssh, http, https and the hypervisor UI.
Functions ¶
func CheckStorage ¶
CheckStorage reports whether the root filesystem is read-only and the storage errors in the kernel log over the last IOErrorWindow. Reading the kernel log needs root or CAP_SYSLOG; without it the count is zero.
func DataRoot ¶
func DataRoot() string
DataRoot is the filesystem whose use the visor reports. On Android / is the read-only system image and always nearly full, so it reports /data.
func LANProbe ¶
LANProbe reports whether ip answers on the local network. A port that accepts or refuses the connection both prove a live host, so no ICMP and no privileges are needed. Only private, link-local and loopback addresses are probed, so it cannot be used to scan the internet.
func PushID ¶
PushID is the stable id of a subscription endpoint, so the same browser subscribing again replaces its old entry.
func RecoveryReason ¶
RecoveryReason says whether a board rebooted during an outage of the given length: an uptime shorter than the outage means it booted after it went down. Empty when the uptime is unknown.
func RootReadOnly ¶
RootReadOnly reports whether /proc/mounts, read from r, has the root filesystem mounted read-only. The last mount on / wins.
func StorageErrors ¶
StorageErrors counts the storage error records among /dev/kmsg lines ("prio,seq,usec,flags;message", Documentation/ABI/testing/dev-kmsg) logged at or after since, a time since boot, and returns the last one.
func VAPIDPublicKey ¶
func VAPIDPublicKey(key *ecdsa.PrivateKey) string
VAPIDPublicKey is the uncompressed public point, base64url, as the browser's applicationServerKey wants it.
Types ¶
type Duration ¶
Duration is a time.Duration written as a string such as "5m" in JSON.
func (Duration) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*Duration) UnmarshalJSON ¶
UnmarshalJSON accepts "5m" or a number of nanoseconds.
type Event ¶
type Event struct {
Kind string `json:"event"`
Scope string `json:"kind,omitempty"`
// Target is the board, or for a site only its public IP.
Target Target `json:"target"`
// Site is the public IP of a site event and Members its visors.
Site string `json:"site,omitempty"`
Members []Target `json:"members,omitempty"`
Since time.Time `json:"offline_since,omitzero"`
At time.Time `json:"at"`
Diagnosis string `json:"diagnosis,omitempty"`
RecoveryReason string `json:"recovery_reason,omitempty"`
Warnings []string `json:"warnings,omitempty"`
}
Event is one alert to deliver.
type PushKeys ¶
type PushKeys struct {
P256DH string `json:"p256dh,omitempty"`
Auth string `json:"auth,omitempty"`
}
PushKeys are a browser subscription's keys, base64url as the browser gives them.
type PushPayload ¶
type PushPayload struct {
Title string `json:"title"`
Body string `json:"body"`
Event string `json:"event"`
PK string `json:"pk,omitempty"`
Watcher string `json:"watcher"`
At time.Time `json:"at"`
// Tag groups the notifications of one visor or site.
Tag string `json:"tag"`
}
PushPayload is the JSON a subscribed browser receives.
type PushSubscription ¶
type PushSubscription struct {
ID string `json:"id,omitempty"`
Endpoint string `json:"endpoint"`
Keys PushKeys `json:"keys"`
Label string `json:"label,omitempty"`
Created time.Time `json:"created,omitzero"`
// ExpirationTime is the browser's, in ms since the epoch, if any.
ExpirationTime *int64 `json:"expirationTime,omitempty"`
}
PushSubscription is one browser that gets Web Push alerts. The fields endpoint and keys match PushSubscription.toJSON() in the browser.
func (*PushSubscription) Host ¶
func (p *PushSubscription) Host() string
Host is the push service host, for display.
func (*PushSubscription) Validate ¶
func (p *PushSubscription) Validate() error
Validate checks the endpoint and the key lengths.
type Result ¶
type Result struct {
Channel string `json:"channel"`
Target string `json:"target"`
Err string `json:"error,omitempty"`
// contains filtered or unexported fields
}
Result is the outcome of one delivery channel.
type SMTP ¶
type SMTP struct {
Host string `json:"host"`
Port int `json:"port,omitempty"`
Username string `json:"username,omitempty"`
Password string `json:"password,omitempty"`
StartTLS bool `json:"starttls,omitempty"`
// PasswordSet is only set in reads, which never carry the password.
PasswordSet bool `json:"password_set,omitempty"`
}
SMTP is an optional mail relay. Without one, mail goes straight to each recipient domain's MX on port 25.
type Saved ¶
type Saved struct {
Since time.Time `json:"since,omitzero"`
Site string `json:"site,omitempty"`
Storage []string `json:"storage,omitempty"`
// Lost is when the link lost alert's outage began.
Lost time.Time `json:"lost,omitzero"`
}
Saved is one visor's alert state kept across restarts.
type Sender ¶
type Sender struct {
// Watcher is the public key of the visor doing the watching.
Watcher cipher.PubKey
// Hostname is the watcher's host name, for the default sender address.
Hostname string
// LookupMX and MXPort are overridable for tests. Defaults are the
// system resolver and 25.
LookupMX func(ctx context.Context, domain string) ([]*MX, error)
MXPort int
// TLSConfig is used for STARTTLS and implicit TLS. Nil verifies the
// server name against the system roots.
TLSConfig *tls.Config
// HTTPClient posts webhooks and pushes. Nil is a client with a 15s
// timeout.
HTTPClient *http.Client
// VAPID signs Web Push requests (RFC 8292). Nil sends no pushes.
VAPID *ecdsa.PrivateKey
}
Sender delivers events by mail, webhook and Web Push.
type Service ¶
type Service struct {
Sender *Sender
// Targets returns the current liveness of every watched visor.
Targets func(ctx context.Context, s Settings) []Target
// Probe asks the visor via to check whether ip answers on its LAN.
// Nil skips the LAN check.
Probe func(ctx context.Context, via cipher.PubKey, ip string) (bool, error)
// Self is the watching visor, a LAN neighbor when it shares a site.
// Nil means it is never one.
Self func() Target
// Logf logs delivery failures. Nil discards.
Logf func(format string, args ...any)
// contains filtered or unexported fields
}
Service checks the watched visors and delivers alerts.
func NewService ¶
NewService returns a Service whose settings and alert state live in dir. base is the config's alerts block, used until settings are saved at runtime.
func (*Service) PushAdd ¶
func (s *Service) PushAdd(sub PushSubscription) (PushSubscription, error)
PushAdd adds or replaces the subscription of one browser and returns it with its id.
func (*Service) PushRemove ¶
PushRemove removes the subscription id, or every one for "all", and returns how many it removed.
func (*Service) Run ¶
Run checks every interval until ctx is done. Visors that went down while nothing watched get floor-grace plus the delay before an alert.
func (*Service) SetSettings ¶
SetSettings validates, applies and saves new settings. An empty relay password keeps the current one.
type Settings ¶
type Settings struct {
EmailTo []string `json:"email_to,omitempty"`
OfflineAfter Duration `json:"offline_after,omitempty"`
SMTP *SMTP `json:"smtp,omitempty"`
Webhook string `json:"webhook,omitempty"`
Watch []cipher.PubKey `json:"watch,omitempty"`
// From is the sender address. Default: skywire-alerts@<hostname>.
From string `json:"from,omitempty"`
// DiskWarnPercent is the disk use that raises a storage warning.
// Default: DefaultDiskWarn.
DiskWarnPercent float64 `json:"disk_warn_percent,omitempty"`
// Push are the browsers that get Web Push alerts. They change only
// through Service.PushAdd and Service.PushRemove.
Push []PushSubscription `json:"push,omitempty"`
}
Settings is the alerts block of the visor config.
func Merge ¶
Merge returns next, keeping the current relay password when next names the same relay and leaves the password empty. The push subscriptions are always the current ones.
type Status ¶
type Status struct {
Settings Settings `json:"settings"`
Enabled bool `json:"enabled"`
OfflineAfter string `json:"offline_after"`
Watched []TargetState `json:"watched"`
LastCheck time.Time `json:"last_check,omitzero"`
LastSent time.Time `json:"last_sent,omitzero"`
LastResults []Result `json:"last_results,omitempty"`
Pending int `json:"pending,omitempty"`
// LastEvents are the most recent alerts, newest last.
LastEvents []Event `json:"last_events,omitempty"`
// PushPublicKey is the VAPID public key a browser subscribes with.
PushPublicKey string `json:"push_public_key,omitempty"`
}
Status is what a read of the alerts returns. It never carries the relay password.
type Storage ¶
type Storage struct {
ReadOnlyRoot bool `json:"root_read_only,omitempty"`
IOErrors int `json:"io_errors,omitempty"`
LastIOError string `json:"last_io_error,omitempty"`
DiskUsedPercent float64 `json:"disk_used_percent,omitempty"`
}
Storage is a board's storage health, the early signs of a failing SD card.
type Target ¶
type Target struct {
PK cipher.PubKey `json:"pk"`
Hostname string `json:"hostname,omitempty"`
PublicIP string `json:"public_ip,omitempty"`
LocalIP string `json:"local_ip,omitempty"`
// LastSeen is when the visor was last seen online, if known.
LastSeen time.Time `json:"last_seen,omitzero"`
Online bool `json:"online"`
// State is one of the State values. Empty means StateOnline or
// StateOffline, from Online.
State string `json:"state,omitempty"`
// Source is "hypervisor" or "watch".
Source string `json:"source,omitempty"`
// Uptime is how long the board has been up at the check, the host's
// uptime when the visor reports it and the visor's own otherwise.
Uptime Duration `json:"uptime,omitempty"`
// Storage is the board's last reported storage health.
Storage *Storage `json:"storage,omitempty"`
}
Target is one watched visor as seen in one check.
type TargetState ¶
type TargetState struct {
Target
OfflineSince time.Time `json:"offline_since,omitzero"`
Alerted bool `json:"alerted,omitempty"`
// Site is set while the visor is part of an alerted site outage.
Site string `json:"site,omitempty"`
StorageWarnings []string `json:"storage_warnings,omitempty"`
// LinkLostSince is when a reconnecting visor lost its hypervisor link.
LinkLostSince time.Time `json:"link_lost_since,omitzero"`
LinkAlerted bool `json:"link_alerted,omitempty"`
}
TargetState is a watched visor's alert state, for status reads.
type Tracker ¶
type Tracker struct {
// Floor is the earliest time an offline period counts from, so visors
// that went down while nothing watched get the full delay to return.
Floor time.Time
// DiskWarn is the disk use percent that raises a storage warning.
// Zero is DefaultDiskWarn.
DiskWarn float64
// contains filtered or unexported fields
}
Tracker turns liveness checks into offline and recovery events. One offline event fires once a visor has been offline for the delay, one recovery event when an alerted visor is back, and nothing for a visor that comes back within the delay. When every watched visor behind one public IP goes down together, one site event replaces the board events.
func NewTracker ¶
NewTracker returns a Tracker restored from saved, so a restart neither repeats an alert nor loses the recovery mail.
func (*Tracker) Observe ¶
Observe records one check of the whole watched set at now and returns the events it causes. Visors no longer in targets, or detached, are dropped silently. Reconnecting and restarting visors are held: their offline delay does not run.
func (*Tracker) States ¶
func (t *Tracker) States() []TargetState
States lists every tracked visor, sorted by public key.
type WebhookPayload ¶
type WebhookPayload struct {
Event string `json:"event"`
// Kind is "board" or "site".
Kind string `json:"kind,omitempty"`
PK string `json:"pk,omitempty"`
Hostname string `json:"hostname,omitempty"`
PublicIP string `json:"public_ip,omitempty"`
LocalIP string `json:"local_ip,omitempty"`
LastSeen time.Time `json:"last_seen,omitzero"`
OfflineSince time.Time `json:"offline_since,omitzero"`
At time.Time `json:"at"`
Diagnosis string `json:"diagnosis,omitempty"`
RecoveryReason string `json:"recovery_reason,omitempty"`
Warnings []string `json:"warnings,omitempty"`
// Members are the visors of a site event.
Members []Target `json:"members,omitempty"`
Watcher string `json:"watcher"`
Text string `json:"text"`
}
WebhookPayload is the JSON body of a webhook POST.