monitor

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package monitor runs operator-defined uptime checks and records their results.

Index

Constants

View Source
const (
	ReasonTimeout       = "timeout"
	ReasonBlocked       = "blocked_address"
	ReasonConnect       = "connect_failed"
	ReasonStatus        = "status_mismatch"
	ReasonKeyword       = "keyword_missing"
	ReasonRedirects     = "redirect_limit"
	ReasonBadRequest    = "bad_request"
	ReasonSecretMissing = "secret_unavailable"
	ReasonCertInvalid   = "cert_invalid"
	ReasonCertExpired   = "cert_expired"
	ReasonCertWarn      = "cert_expiring"
	ReasonCertCritical  = "cert_expiring_critical"
)

Failure classes stored in Result.Reason.

View Source
const (
	DefaultWorkers              = 4
	DefaultConfirmFailures      = 2
	DefaultRecoveryOKs          = 2
	DefaultMinIntervalHTTP      = 30 * time.Second
	DefaultMinIntervalTCP       = 30 * time.Second
	DefaultMaxTimeout           = 30 * time.Second
	DefaultMaxCount             = 100
	DefaultBodyMaxBytes         = 64 * 1024
	DefaultMaxHeaders           = 16
	DefaultHeaderMaxBytes       = 4096
	DefaultMaxRedirects         = 5
	DefaultFlushInterval        = 5 * time.Second
	DefaultBufferMax            = 10000
	DefaultRawRetention         = 48 * time.Hour
	DefaultHourRetention        = 400 * 24 * time.Hour
	DefaultPruneInterval        = 10 * time.Minute
	DefaultPruneBatch           = 1000
	DefaultJitterPercent        = 10
	DefaultTickInterval         = time.Second
	DefaultResultsMaxLimit      = 1000
	DefaultMinIntervalHeartbeat = 30 * time.Second
	DefaultMinIntervalTLS       = time.Hour
	DefaultHeartbeatGrace       = 5 * time.Minute
	DefaultHeartbeatMaxGrace    = 7 * 24 * time.Hour
	DefaultHeartbeatCheck       = 15 * time.Second
	DefaultHeartbeatPerMinute   = 60
	DefaultHeartbeatBodyMax     = 1024
	DefaultHeartbeatMinResponse = 25 * time.Millisecond
	DefaultTLSWarnDays          = 21
	DefaultTLSCritDays          = 7
	DefaultTLSTimeout           = 10 * time.Second
)

Defaults apply only when the matching APP_MONITOR_* variable is unset.

View Source
const (
	PingOK    = ""
	PingStart = "start"
	PingFail  = "fail"
)

Heartbeat signals carried by the ping URL suffix.

View Source
const (
	ReasonHeartbeatMissed = "heartbeat_missed"
	ReasonJobFailed       = "job_failed"
)

Heartbeat failure classes stored in Result.Reason.

View Source
const (
	KindHTTP = "http"
	KindTCP  = "tcp"
	// KindHeartbeat expects an inbound ping from a cron job within interval plus grace.
	KindHeartbeat = "heartbeat"
	// KindTLS watches the certificate expiry of a host and port.
	KindTLS = "tls"
)

Monitor kinds.

View Source
const (
	StateUp         = "up"
	StateSuspect    = "suspect"
	StateDown       = "down"
	StateRecovering = "recovering"
)

Confirmed monitor states (ADR 054). Suspect and recovering are the unconfirmed steps either side of down.

View Source
const Location = "cp"

Location labels results from this control plane, the only vantage point so far.

View Source
const StateDegraded = "degraded"

StateDegraded is reported for a passing monitor that is near failing, such as a certificate inside its warning window. Confirmed tracking itself stays up while degraded.

Variables

View Source
var (
	ErrInvalid  = errors.New("monitor: invalid")
	ErrNotFound = errors.New("monitor: not found")
	ErrLimit    = errors.New("monitor: limit reached")
)

Sentinel errors the API maps to status codes.

Functions

func HashPingToken

func HashPingToken(token string) string

HashPingToken is the stored form of a ping token.

func HourBucket

func HourBucket(at int64) int64

HourBucket is the start of the hour containing unix second at.

func PingPath

func PingPath(token string) string

PingPath is the public path a job calls; it embeds the secret token.

func Validate

func Validate(m Monitor, cfg Config) error

Validate checks a normalized monitor against the configured limits. Errors wrap ErrInvalid.

func ValidateTarget

func ValidateTarget(ctx context.Context, m Monitor) error

ValidateTarget refuses, at configuration time, a target every probe would refuse.

Types

type Batcher

type Batcher struct {
	// contains filtered or unexported fields
}

Batcher buffers results so the single SQLite writer sees one transaction per flush.

func NewBatcher

func NewBatcher(sink ResultSink, capacity int) *Batcher

NewBatcher builds a Batcher that holds at most capacity results.

func (*Batcher) Add

func (b *Batcher) Add(r Result)

Add queues a result, dropping the oldest when the buffer is full.

func (*Batcher) Dropped

func (b *Batcher) Dropped() uint64

Dropped counts results discarded because the buffer was full.

func (*Batcher) Flush

func (b *Batcher) Flush(ctx context.Context) error

Flush writes the queued results. On failure they are put back, bounded by the buffer size.

func (*Batcher) Pending

func (b *Batcher) Pending() int

Pending is the number of queued results.

type Checker

type Checker struct {
	// contains filtered or unexported fields
}

Checker runs one probe. Every dial goes through the netguard address check.

func NewChecker

func NewChecker(cfg Config) *Checker

NewChecker builds a Checker whose connections refuse internal addresses unless the netguard private opt-in env is set.

func (*Checker) Check

func (c *Checker) Check(ctx context.Context, m Monitor, headers map[string]string) Result

Check probes m once. headers holds resolved header values by canonical name.

type Config

type Config struct {
	Enabled         bool
	Workers         int
	MinIntervalHTTP time.Duration
	MinIntervalTCP  time.Duration
	MaxTimeout      time.Duration
	MaxCount        int
	BodyMaxBytes    int
	MaxHeaders      int
	HeaderMaxBytes  int
	MaxRedirects    int
	FlushInterval   time.Duration
	BufferMax       int
	RawRetention    time.Duration
	HourRetention   time.Duration
	PruneInterval   time.Duration
	PruneBatch      int
	JitterPercent   int
	TickInterval    time.Duration
	ResultsMaxLimit int
	ConfirmFailures int
	RecoveryOKs     int
	UserAgent       string

	MinIntervalHeartbeat time.Duration
	MinIntervalTLS       time.Duration
	HeartbeatGrace       time.Duration
	HeartbeatMaxGrace    time.Duration
	HeartbeatCheck       time.Duration
	HeartbeatPerMinute   int
	HeartbeatBodyMax     int
	HeartbeatMinResponse time.Duration
	TLSWarnDays          int
	TLSCritDays          int
}

Config holds every limit of the monitor engine, all read from APP_MONITOR* variables.

func ConfigFromEnv

func ConfigFromEnv(lookup func(string) (string, bool)) (Config, error)

ConfigFromEnv reads the engine settings. Unset variables take the defaults above.

func (Config) MinInterval

func (c Config) MinInterval(kind string) time.Duration

MinInterval is the per-kind floor, or zero for an unknown kind.

type CreateInput

type CreateInput struct {
	Monitor Monitor
	Headers map[string]string
}

CreateInput is a monitor definition plus its header values, which are stored encrypted and never returned.

type HeartbeatState

type HeartbeatState struct {
	MonitorID string
	TokenHash string
	ArmedAt   int64
	LastOK    int64
	LastStart int64
	LastFail  int64
}

HeartbeatState is the persisted ping history of one heartbeat monitor, unix seconds.

type HeartbeatStore

type HeartbeatStore interface {
	SaveHeartbeatToken(ctx context.Context, monitorID, tokenHash string, armedAt int64) error
	FindHeartbeat(ctx context.Context, tokenHash string) (HeartbeatState, error)
	GetHeartbeat(ctx context.Context, monitorID string) (HeartbeatState, error)
	RecordHeartbeat(ctx context.Context, monitorID, signal string, at int64) error
}

HeartbeatStore persists token hashes and ping times. FindHeartbeat and GetHeartbeat return ErrNotFound.

type History

type History struct {
	Raw    []Result
	Rollup []Rollup
}

History is the recent raw results plus hourly rollups for a window.

type Monitor

type Monitor struct {
	ID        string
	Name      string
	Kind      string
	Interval  time.Duration
	Timeout   time.Duration
	Enabled   bool
	Spec      Spec
	CreatedAt time.Time
	// PingToken is the plaintext heartbeat token, set only on the Monitor returned by Create
	// and RotateToken. It is never stored: only its hash is.
	PingToken string
}

Monitor is one check definition.

func Normalize

func Normalize(m Monitor) Monitor

Normalize fills defaults so stored and scheduled monitors are complete.

type Notifier

type Notifier func(Transition)

Notifier receives confirmed transitions. It must not block the probe worker.

type Options

type Options struct {
	Config  Config
	Store   Store
	Secrets Secrets
	Logger  *slog.Logger
	// Notify receives confirmed down and recovered transitions; nil drops them.
	Notify Notifier
	// OnDelete runs after a monitor is removed so alert state tied to it can clear.
	OnDelete func(id string)
	// Heartbeats stores heartbeat tokens and pings; nil rejects heartbeat monitors.
	Heartbeats HeartbeatStore
}

Options configures a Service. Secrets may be nil, which rejects monitors that carry headers.

type Result

type Result struct {
	MonitorID string
	At        int64
	OK        bool
	LatencyMS int
	Code      int
	Reason    string
	// Degraded marks a passing check that is close to failing, such as a certificate
	// inside its warning window. It is not stored, the Reason is.
	Degraded bool
}

Result is one probe outcome. Reason is a short class, never a body or header value.

type ResultSink

type ResultSink interface {
	InsertResults(ctx context.Context, results []Result) error
}

ResultSink persists a batch of results and updates the hourly rollups in the same transaction.

type Rollup

type Rollup struct {
	MonitorID string
	Bucket    int64
	Total     int
	Failed    int
	LatSum    int64
	LatMax    int
}

Rollup is one hour bucket of results for a monitor.

func Aggregate

func Aggregate(results []Result) []Rollup

Aggregate folds results into hour rollups. Latency counts only successful probes.

type Scheduler

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler keeps monitors in a min-heap by due time. It takes the time as an argument so tests drive it with a fake clock.

func NewScheduler

func NewScheduler(cfg Config, jitter func(time.Duration) time.Duration) *Scheduler

NewScheduler builds a Scheduler. A nil jitter uses a random share of the interval.

func (*Scheduler) Add

func (s *Scheduler) Add(m Monitor, now time.Time)

Add schedules m, replacing an earlier entry with the same id. A disabled monitor is only removed.

func (*Scheduler) Done

func (s *Scheduler) Done(id string)

Done marks a probe finished so the monitor may be dispatched again.

func (*Scheduler) Due

func (s *Scheduler) Due(now time.Time) []Monitor

Due returns the monitors to probe at now and schedules their next run. A monitor with a probe still in flight is pushed back, never run twice at once.

func (*Scheduler) Len

func (s *Scheduler) Len() int

Len is the number of scheduled monitors.

func (*Scheduler) Remove

func (s *Scheduler) Remove(id string)

Remove drops a monitor from the schedule.

type Secrets

type Secrets interface {
	SetValue(ctx context.Context, serviceName, envKey, plaintext string) error
	Resolve(ctx context.Context, serviceName, envKey string) (string, error)
	DeleteAll(ctx context.Context, serviceName string) error
}

Secrets keeps header values envelope-encrypted.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service owns the scheduler, workers, batcher and pruner.

func New

func New(o Options) *Service

New builds a Service.

func (*Service) Config

func (s *Service) Config() Config

Config returns the engine settings.

func (*Service) Create

func (s *Service) Create(ctx context.Context, in CreateInput) (Monitor, error)

Create validates, stores and schedules a monitor.

func (*Service) Delete

func (s *Service) Delete(ctx context.Context, id string) error

Delete removes a monitor, its history and its stored header values.

func (*Service) Get

func (s *Service) Get(ctx context.Context, id string) (Monitor, error)

Get returns one monitor or ErrNotFound.

func (*Service) List

func (s *Service) List(ctx context.Context) ([]Monitor, error)

List returns every monitor.

func (*Service) Ping

func (s *Service) Ping(ctx context.Context, token, signal string) error

Ping records a signal for the monitor that owns token. Every miss (unknown token, wrong kind, disabled monitor, no heartbeat store) is the same ErrNotFound after the same hash and lookup work.

func (*Service) Prune

func (s *Service) Prune(ctx context.Context) error

Prune deletes expired raw results and hourly rollups in bounded batches.

func (*Service) Results

func (s *Service) Results(ctx context.Context, id string, window time.Duration, limit int) (History, error)

Results returns history for the last window, clamped to the retention periods and the limit cap.

func (*Service) RotateToken

func (s *Service) RotateToken(ctx context.Context, id string) (Monitor, error)

RotateToken replaces a heartbeat monitor's ping token; the old URL stops working at once.

func (*Service) Run

func (s *Service) Run(ctx context.Context)

Run schedules every stored monitor and blocks until ctx ends.

func (*Service) SetRawRetentionSource

func (s *Service) SetRawRetentionSource(fn func() time.Duration)

SetRawRetentionSource makes raw result retention follow fn on each prune and history read. Call before Run.

func (s *Service) SetStatusLink(ctx context.Context, id string, l StatusLink) (Monitor, error)

SetStatusLink changes only the status page link settings of a stored monitor.

func (*Service) State

func (s *Service) State(id string) string

State returns the confirmed state of a monitor, or "" before its first probe.

func (*Service) Status

func (s *Service) Status(id string) Status

Status returns the confirmed state with its consecutive failure and success counts.

type Spec

type Spec struct {
	URL             string        `json:"url,omitempty"`
	Method          string        `json:"method,omitempty"`
	HeaderNames     []string      `json:"header_names,omitempty"`
	Expect          []StatusRange `json:"expect,omitempty"`
	Contains        string        `json:"contains,omitempty"`
	FollowRedirects bool          `json:"follow_redirects,omitempty"`
	Host            string        `json:"host,omitempty"`
	Port            int           `json:"port,omitempty"`
	// StatusLink opts the monitor in to opening and resolving a status page incident. Zero
	// confirm and recovery values fall back to the APP_MONITOR_LINK_* settings.
	StatusLink          bool `json:"status_link,omitempty"`
	LinkConfirmFailures int  `json:"link_confirm_failures,omitempty"`
	LinkRecoverySeconds int  `json:"link_recovery_seconds,omitempty"`
	// GraceSeconds is the heartbeat grace period; zero uses APP_MONITOR_HEARTBEAT_GRACE.
	GraceSeconds int `json:"grace_seconds,omitempty"`
	// WarnDays and CritDays override the APP_MONITOR_TLS_* thresholds for one TLS monitor.
	WarnDays int `json:"warn_days,omitempty"`
	CritDays int `json:"crit_days,omitempty"`
}

Spec is the kind specific part of a monitor. Header values never live here, only their names.

type Status

type Status struct {
	State         string
	FailStreak    int
	SuccessStreak int
}

Status is a monitor's confirmed state plus the consecutive counts behind it.

type StatusLink struct {
	Enabled         bool
	ConfirmFailures int
	RecoverySeconds int
}

StatusLink is the per-monitor status page link setting.

type StatusRange

type StatusRange struct {
	Min int `json:"min"`
	Max int `json:"max"`
}

StatusRange is an inclusive HTTP status interval.

type Store

type Store interface {
	ResultSink
	ListMonitors(ctx context.Context) ([]Monitor, error)
	GetMonitor(ctx context.Context, id string) (Monitor, error)
	CountMonitors(ctx context.Context) (int, error)
	SaveMonitor(ctx context.Context, m Monitor) error
	DeleteMonitor(ctx context.Context, id string) error
	ListResults(ctx context.Context, id string, since int64, limit int) ([]Result, error)
	ListRollups(ctx context.Context, id string, sinceBucket int64, limit int) ([]Rollup, error)
	PruneResults(ctx context.Context, id string, before int64, limit int) (int, error)
	PruneRollups(ctx context.Context, id string, before int64, limit int) (int, error)
}

Store is the persistence the engine needs.

type Tracker

type Tracker struct {
	// contains filtered or unexported fields
}

Tracker turns raw probe results into confirmed states so one bad sample never pages. Flap control stays in alerting, this only confirms.

func NewTracker

func NewTracker(confirmFailures, recoverySuccesses int, n Notifier) *Tracker

NewTracker builds a Tracker; counts below 1 are treated as 1.

func (*Tracker) Degraded

func (t *Tracker) Degraded(id string) bool

Degraded reports whether the last passing result was flagged degraded.

func (*Tracker) Force

func (t *Tracker) Force(m Monitor, down bool, reason string, degraded bool) string

Force sets a monitor up or down at once, for kinds whose own timing already confirms the change. It notifies on a real transition and returns the state afterwards.

func (*Tracker) Forget

func (t *Tracker) Forget(id string)

Forget drops a deleted monitor.

func (*Tracker) Observe

func (t *Tracker) Observe(m Monitor, r Result) string

Observe feeds one result and returns the monitor's state afterwards.

func (*Tracker) State

func (t *Tracker) State(id string) string

State returns the current state, or "" for a monitor not probed yet.

func (*Tracker) Status

func (t *Tracker) Status(id string) Status

Status returns the state and streaks, or a zero Status for a monitor not probed yet.

type Transition

type Transition struct {
	Monitor Monitor
	State   string
	Reason  string
}

Transition is a confirmed change worth telling someone about: down or back up.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL