alerts

package
v1.3.109 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: AGPL-3.0 Imports: 26 Imported by: 0

Documentation

Overview

Package alerts sends mail and webhook alerts when watched visors go offline, when they come back, and when their storage starts to fail.

Index

Constants

View Source
const (
	DiagLANUp      = "board answers on the LAN, skywire is down"
	DiagLANDown    = "board does not answer on the LAN: powered off, hung or unplugged"
	DiagNoNeighbor = "no neighbor to check from"
	DiagNoLocalIP  = "board's LAN address is unknown, no LAN check"
)

LAN check results for a board alert.

View Source
const (
	MinOfflineAfter     = time.Minute
	MaxOfflineAfter     = time.Hour
	DefaultOfflineAfter = 5 * time.Minute
)

Offline delay bounds. A visor that comes back within the delay sends nothing.

View Source
const (
	KindOffline  = "offline"
	KindOnline   = "online"
	KindStorage  = "storage"
	KindConflict = "ip-conflict"
	KindTest     = "test"
	// KindLinkLost is a visor that is up but has not reconnected to its
	// hypervisor for the offline delay.
	KindLinkLost = "link-lost"
)

Kind of an alert event.

View Source
const (
	StateOnline = "online"
	// StateReconnecting is a visor whose link to the hypervisor dropped
	// but which still answers the hypervisor's call back.
	StateReconnecting = "reconnecting"
	// StateRestarting is a visor that said it is going down to restart.
	StateRestarting = "restarting"
	// StateDetached is a visor that removed this hypervisor. It is not
	// watched.
	StateDetached = "detached"
	StateOffline  = "offline"
)

States of a watched visor. A hypervisor reports all five for the visors it manages; a watch list visor is only online or offline.

View Source
const (
	ScopeBoard = "board"
	ScopeSite  = "site"
)

Scope of an outage: one board, or every watched visor behind one public IP.

View Source
const (
	ReasonRebooted = "rebooted during the outage (likely power loss or crash)"
	ReasonStayedUp = "stayed up (network or internet outage)"
)

Recovery reasons, from the board's uptime against the outage length.

View Source
const CheckInterval = 30 * time.Second

CheckInterval is how often the watched set is checked.

View Source
const ConflictWindow = 10 * time.Minute

ConflictWindow is how recently a visor must have been online for its LAN IP to count in a conflict, so a dead board's stale address does not.

View Source
const DefaultDiskWarn = 90

DefaultDiskWarn is the disk use percent that raises a storage warning.

View Source
const DiagLinkRestored = "the visor reconnected to its hypervisor"

DiagLinkRestored is the diagnosis of the recovery after KindLinkLost.

View Source
const IOErrorWindow = 24 * time.Hour

IOErrorWindow is how far back the kernel log counts storage errors.

View Source
const SiteWindow = 2 * time.Minute

SiteWindow is how close together every watched visor behind one public IP must go offline to count as one site outage.

Variables

View Source
var ProbePorts = []int{22, 80, 443, 8000}

ProbePorts are the TCP ports LANProbe tries: ssh, http, https and the hypervisor UI.

Functions

func CheckStorage

func CheckStorage() (readOnly bool, ioErrors int, last string)

CheckStorage reports whether the root filesystem is read-only and the storage errors in the kernel log over the last IOErrorWindow. Reading the kernel log needs root or CAP_SYSLOG; without it the count is zero.

func LANProbe

func LANProbe(ctx context.Context, ip string) (bool, error)

LANProbe reports whether ip answers on the local network. A port that accepts or refuses the connection both prove a live host, so no ICMP and no privileges are needed. Only private, link-local and loopback addresses are probed, so it cannot be used to scan the internet.

func RecoveryReason

func RecoveryReason(uptime, outage time.Duration) string

RecoveryReason says whether a board rebooted during an outage of the given length: an uptime shorter than the outage means it booted after it went down. Empty when the uptime is unknown.

func RootReadOnly

func RootReadOnly(r io.Reader) bool

RootReadOnly reports whether /proc/mounts, read from r, has the root filesystem mounted read-only. The last mount on / wins.

func StorageErrors

func StorageErrors(lines []string, since time.Duration) (n int, last string)

StorageErrors counts the storage error records among /dev/kmsg lines ("prio,seq,usec,flags;message", Documentation/ABI/testing/dev-kmsg) logged at or after since, a time since boot, and returns the last one.

func Subject

func Subject(ev Event) string

Subject is the mail subject for ev.

Types

type Duration

type Duration time.Duration

Duration is a time.Duration written as a string such as "5m" in JSON.

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*Duration) UnmarshalJSON

func (d *Duration) UnmarshalJSON(b []byte) error

UnmarshalJSON accepts "5m" or a number of nanoseconds.

type Event

type Event struct {
	Kind  string `json:"event"`
	Scope string `json:"kind,omitempty"`
	// Target is the board, or for a site only its public IP.
	Target Target `json:"target"`
	// Site is the public IP of a site event and Members its visors.
	Site           string    `json:"site,omitempty"`
	Members        []Target  `json:"members,omitempty"`
	Since          time.Time `json:"offline_since,omitzero"`
	At             time.Time `json:"at"`
	Diagnosis      string    `json:"diagnosis,omitempty"`
	RecoveryReason string    `json:"recovery_reason,omitempty"`
	Warnings       []string  `json:"warnings,omitempty"`
}

Event is one alert to deliver.

type Result

type Result struct {
	Channel string `json:"channel"`
	Target  string `json:"target"`
	Err     string `json:"error,omitempty"`
}

Result is the outcome of one delivery channel.

type SMTP

type SMTP struct {
	Host     string `json:"host"`
	Port     int    `json:"port,omitempty"`
	Username string `json:"username,omitempty"`
	Password string `json:"password,omitempty"`
	StartTLS bool   `json:"starttls,omitempty"`
	// PasswordSet is only set in reads, which never carry the password.
	PasswordSet bool `json:"password_set,omitempty"`
}

SMTP is an optional mail relay. Without one, mail goes straight to each recipient domain's MX on port 25.

type Saved

type Saved struct {
	Since   time.Time `json:"since,omitzero"`
	Site    string    `json:"site,omitempty"`
	Storage []string  `json:"storage,omitempty"`
	// Lost is when the link lost alert's outage began.
	Lost time.Time `json:"lost,omitzero"`
}

Saved is one visor's alert state kept across restarts.

type Sender

type Sender struct {
	// Watcher is the public key of the visor doing the watching.
	Watcher cipher.PubKey
	// Hostname is the watcher's host name, for the default sender address.
	Hostname string
	// LookupMX and MXPort are overridable for tests. Defaults are
	// net.DefaultResolver.LookupMX and 25.
	LookupMX func(ctx context.Context, domain string) ([]*net.MX, error)
	MXPort   int
	// TLSConfig is used for STARTTLS and implicit TLS. Nil verifies the
	// server name against the system roots.
	TLSConfig *tls.Config
	// HTTPClient posts webhooks. Nil is a client with a 15s timeout.
	HTTPClient *http.Client
}

Sender delivers events by mail and webhook.

func (*Sender) Body

func (s *Sender) Body(ev Event) string

Body is the plain text mail body for ev.

func (*Sender) Format

func (s *Sender) Format(set Settings, ev Event) []byte

Format builds the full RFC 5322 message for ev.

func (*Sender) Send

func (s *Sender) Send(ctx context.Context, set Settings, ev Event) []Result

Send delivers ev to every configured channel and reports each one.

type Service

type Service struct {
	Sender *Sender
	// Targets returns the current liveness of every watched visor.
	Targets func(ctx context.Context, s Settings) []Target
	// Probe asks the visor via to check whether ip answers on its LAN.
	// Nil skips the LAN check.
	Probe func(ctx context.Context, via cipher.PubKey, ip string) (bool, error)
	// Self is the watching visor, a LAN neighbor when it shares a site.
	// Nil means it is never one.
	Self func() Target
	// Logf logs delivery failures. Nil discards.
	Logf func(format string, args ...any)
	// contains filtered or unexported fields
}

Service checks the watched visors and delivers alerts.

func NewService

func NewService(dir string, base *Settings, sender *Sender) (*Service, error)

NewService returns a Service whose settings and alert state live in dir. base is the config's alerts block, used until settings are saved at runtime.

func (*Service) Run

func (s *Service) Run(ctx context.Context, interval, grace time.Duration)

Run checks every interval until ctx is done. Visors that went down while nothing watched get floor-grace plus the delay before an alert.

func (*Service) SetSettings

func (s *Service) SetSettings(next Settings) error

SetSettings validates, applies and saves new settings. An empty relay password keeps the current one.

func (*Service) Settings

func (s *Service) Settings() Settings

Settings returns the current settings, password included.

func (*Service) Status

func (s *Service) Status() Status

Status returns the settings without the password and the watch state.

func (*Service) Test

func (s *Service) Test(ctx context.Context) ([]Result, error)

Test sends a test alert to every configured channel now.

func (*Service) Tick

func (s *Service) Tick(ctx context.Context, now time.Time)

Tick runs one check at now and delivers what it causes.

type Settings

type Settings struct {
	EmailTo      []string        `json:"email_to,omitempty"`
	OfflineAfter Duration        `json:"offline_after,omitempty"`
	SMTP         *SMTP           `json:"smtp,omitempty"`
	Webhook      string          `json:"webhook,omitempty"`
	Watch        []cipher.PubKey `json:"watch,omitempty"`
	// From is the sender address. Default: skywire-alerts@<hostname>.
	From string `json:"from,omitempty"`
	// DiskWarnPercent is the disk use that raises a storage warning.
	// Default: DefaultDiskWarn.
	DiskWarnPercent float64 `json:"disk_warn_percent,omitempty"`
}

Settings is the alerts block of the visor config.

func Merge

func Merge(cur *Settings, next Settings) Settings

Merge returns next, keeping the current relay password when next names the same relay and leaves the password empty.

func (*Settings) Delay

func (s *Settings) Delay() time.Duration

Delay is the offline delay, the default when unset.

func (*Settings) DiskWarn

func (s *Settings) DiskWarn() float64

DiskWarn is the disk warning threshold, the default when unset.

func (*Settings) Enabled

func (s *Settings) Enabled() bool

Enabled reports whether there is anywhere to send an alert.

func (Settings) Redacted

func (s Settings) Redacted() Settings

Redacted is a copy without the relay password.

func (*Settings) Validate

func (s *Settings) Validate() error

Validate checks addresses, the delay range and the webhook URL.

type Status

type Status struct {
	Settings     Settings      `json:"settings"`
	Enabled      bool          `json:"enabled"`
	OfflineAfter string        `json:"offline_after"`
	Watched      []TargetState `json:"watched"`
	LastCheck    time.Time     `json:"last_check,omitzero"`
	LastSent     time.Time     `json:"last_sent,omitzero"`
	LastResults  []Result      `json:"last_results,omitempty"`
	Pending      int           `json:"pending,omitempty"`
	// LastEvents are the most recent alerts, newest last.
	LastEvents []Event `json:"last_events,omitempty"`
}

Status is what a read of the alerts returns. It never carries the relay password.

type Storage

type Storage struct {
	ReadOnlyRoot    bool    `json:"root_read_only,omitempty"`
	IOErrors        int     `json:"io_errors,omitempty"`
	LastIOError     string  `json:"last_io_error,omitempty"`
	DiskUsedPercent float64 `json:"disk_used_percent,omitempty"`
}

Storage is a board's storage health, the early signs of a failing SD card.

type Target

type Target struct {
	PK       cipher.PubKey `json:"pk"`
	Hostname string        `json:"hostname,omitempty"`
	PublicIP string        `json:"public_ip,omitempty"`
	LocalIP  string        `json:"local_ip,omitempty"`
	// LastSeen is when the visor was last seen online, if known.
	LastSeen time.Time `json:"last_seen,omitzero"`
	Online   bool      `json:"online"`
	// State is one of the State values. Empty means StateOnline or
	// StateOffline, from Online.
	State string `json:"state,omitempty"`
	// Source is "hypervisor" or "watch".
	Source string `json:"source,omitempty"`
	// Uptime is how long the board has been up at the check, the host's
	// uptime when the visor reports it and the visor's own otherwise.
	Uptime Duration `json:"uptime,omitempty"`
	// Storage is the board's last reported storage health.
	Storage *Storage `json:"storage,omitempty"`
}

Target is one watched visor as seen in one check.

func Neighbors

func Neighbors(board Target, targets []Target) []Target

Neighbors lists the online visors in targets at board's site: the same public IP and the same LAN subnet, a /24 for IPv4 and a /64 for IPv6.

type TargetState

type TargetState struct {
	Target
	OfflineSince time.Time `json:"offline_since,omitzero"`
	Alerted      bool      `json:"alerted,omitempty"`
	// Site is set while the visor is part of an alerted site outage.
	Site            string   `json:"site,omitempty"`
	StorageWarnings []string `json:"storage_warnings,omitempty"`
	// LinkLostSince is when a reconnecting visor lost its hypervisor link.
	LinkLostSince time.Time `json:"link_lost_since,omitzero"`
	LinkAlerted   bool      `json:"link_alerted,omitempty"`
}

TargetState is a watched visor's alert state, for status reads.

type Tracker

type Tracker struct {

	// Floor is the earliest time an offline period counts from, so visors
	// that went down while nothing watched get the full delay to return.
	Floor time.Time
	// DiskWarn is the disk use percent that raises a storage warning.
	// Zero is DefaultDiskWarn.
	DiskWarn float64
	// contains filtered or unexported fields
}

Tracker turns liveness checks into offline and recovery events. One offline event fires once a visor has been offline for the delay, one recovery event when an alerted visor is back, and nothing for a visor that comes back within the delay. When every watched visor behind one public IP goes down together, one site event replaces the board events.

func NewTracker

func NewTracker(saved map[cipher.PubKey]Saved) *Tracker

NewTracker returns a Tracker restored from saved, so a restart neither repeats an alert nor loses the recovery mail.

func (*Tracker) Observe

func (t *Tracker) Observe(now time.Time, delay time.Duration, targets []Target) []Event

Observe records one check of the whole watched set at now and returns the events it causes. Visors no longer in targets, or detached, are dropped silently. Reconnecting and restarting visors are held: their offline delay does not run.

func (*Tracker) Saved

func (t *Tracker) Saved() map[cipher.PubKey]Saved

Saved lists the state worth keeping across a restart.

func (*Tracker) States

func (t *Tracker) States() []TargetState

States lists every tracked visor, sorted by public key.

type WebhookPayload

type WebhookPayload struct {
	Event string `json:"event"`
	// Kind is "board" or "site".
	Kind           string    `json:"kind,omitempty"`
	PK             string    `json:"pk,omitempty"`
	Hostname       string    `json:"hostname,omitempty"`
	PublicIP       string    `json:"public_ip,omitempty"`
	LocalIP        string    `json:"local_ip,omitempty"`
	LastSeen       time.Time `json:"last_seen,omitzero"`
	OfflineSince   time.Time `json:"offline_since,omitzero"`
	At             time.Time `json:"at"`
	Diagnosis      string    `json:"diagnosis,omitempty"`
	RecoveryReason string    `json:"recovery_reason,omitempty"`
	Warnings       []string  `json:"warnings,omitempty"`
	// Members are the visors of a site event.
	Members []Target `json:"members,omitempty"`
	Watcher string   `json:"watcher"`
	Text    string   `json:"text"`
}

WebhookPayload is the JSON body of a webhook POST.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL