Documentation
¶
Overview ¶
Package monitor runs operator-defined uptime checks and records their results.
Index ¶
- Constants
- Variables
- func HashPingToken(token string) string
- func HourBucket(at int64) int64
- func PingPath(token string) string
- func Validate(m Monitor, cfg Config) error
- func ValidateTarget(ctx context.Context, m Monitor) error
- type Batcher
- type Checker
- type Config
- type CreateInput
- type HeartbeatState
- type HeartbeatStore
- type History
- type Monitor
- type Notifier
- type Options
- type Result
- type ResultSink
- type Rollup
- type Scheduler
- type Secrets
- type Service
- func (s *Service) Config() Config
- func (s *Service) Create(ctx context.Context, in CreateInput) (Monitor, error)
- func (s *Service) Delete(ctx context.Context, id string) error
- func (s *Service) Get(ctx context.Context, id string) (Monitor, error)
- func (s *Service) List(ctx context.Context) ([]Monitor, error)
- func (s *Service) Ping(ctx context.Context, token, signal string) error
- func (s *Service) Prune(ctx context.Context) error
- func (s *Service) Results(ctx context.Context, id string, window time.Duration, limit int) (History, error)
- func (s *Service) RotateToken(ctx context.Context, id string) (Monitor, error)
- func (s *Service) Run(ctx context.Context)
- func (s *Service) SetRawRetentionSource(fn func() time.Duration)
- func (s *Service) SetStatusLink(ctx context.Context, id string, l StatusLink) (Monitor, error)
- func (s *Service) State(id string) string
- func (s *Service) Status(id string) Status
- type Spec
- type Status
- type StatusLink
- type StatusRange
- type Store
- type Tracker
- func (t *Tracker) Degraded(id string) bool
- func (t *Tracker) Force(m Monitor, down bool, reason string, degraded bool) string
- func (t *Tracker) Forget(id string)
- func (t *Tracker) Observe(m Monitor, r Result) string
- func (t *Tracker) State(id string) string
- func (t *Tracker) Status(id string) Status
- type Transition
Constants ¶
const ( ReasonTimeout = "timeout" ReasonBlocked = "blocked_address" ReasonConnect = "connect_failed" ReasonStatus = "status_mismatch" ReasonKeyword = "keyword_missing" ReasonRedirects = "redirect_limit" ReasonBadRequest = "bad_request" ReasonSecretMissing = "secret_unavailable" ReasonCertInvalid = "cert_invalid" ReasonCertExpired = "cert_expired" ReasonCertWarn = "cert_expiring" ReasonCertCritical = "cert_expiring_critical" )
Failure classes stored in Result.Reason.
const ( DefaultWorkers = 4 DefaultConfirmFailures = 2 DefaultRecoveryOKs = 2 DefaultMinIntervalHTTP = 30 * time.Second DefaultMinIntervalTCP = 30 * time.Second DefaultMaxTimeout = 30 * time.Second DefaultMaxCount = 100 DefaultBodyMaxBytes = 64 * 1024 DefaultMaxHeaders = 16 DefaultHeaderMaxBytes = 4096 DefaultMaxRedirects = 5 DefaultFlushInterval = 5 * time.Second DefaultBufferMax = 10000 DefaultRawRetention = 48 * time.Hour DefaultHourRetention = 400 * 24 * time.Hour DefaultPruneInterval = 10 * time.Minute DefaultPruneBatch = 1000 DefaultJitterPercent = 10 DefaultTickInterval = time.Second DefaultResultsMaxLimit = 1000 DefaultMinIntervalHeartbeat = 30 * time.Second DefaultMinIntervalTLS = time.Hour DefaultHeartbeatGrace = 5 * time.Minute DefaultHeartbeatMaxGrace = 7 * 24 * time.Hour DefaultHeartbeatCheck = 15 * time.Second DefaultHeartbeatPerMinute = 60 DefaultHeartbeatBodyMax = 1024 DefaultHeartbeatMinResponse = 25 * time.Millisecond DefaultTLSWarnDays = 21 DefaultTLSCritDays = 7 DefaultTLSTimeout = 10 * time.Second )
Defaults apply only when the matching APP_MONITOR_* variable is unset.
const ( PingOK = "" PingStart = "start" PingFail = "fail" )
Heartbeat signals carried by the ping URL suffix.
const ( ReasonHeartbeatMissed = "heartbeat_missed" ReasonJobFailed = "job_failed" )
Heartbeat failure classes stored in Result.Reason.
const ( KindHTTP = "http" KindTCP = "tcp" // KindHeartbeat expects an inbound ping from a cron job within interval plus grace. KindHeartbeat = "heartbeat" // KindTLS watches the certificate expiry of a host and port. KindTLS = "tls" )
Monitor kinds.
const ( StateUp = "up" StateSuspect = "suspect" StateDown = "down" StateRecovering = "recovering" )
Confirmed monitor states (ADR 054). Suspect and recovering are the unconfirmed steps either side of down.
const Location = "cp"
Location labels results from this control plane, the only vantage point so far.
const StateDegraded = "degraded"
StateDegraded is reported for a passing monitor that is near failing, such as a certificate inside its warning window. Confirmed tracking itself stays up while degraded.
Variables ¶
var ( ErrInvalid = errors.New("monitor: invalid") ErrNotFound = errors.New("monitor: not found") ErrLimit = errors.New("monitor: limit reached") )
Sentinel errors the API maps to status codes.
Functions ¶
func HashPingToken ¶
HashPingToken is the stored form of a ping token.
func HourBucket ¶
HourBucket is the start of the hour containing unix second at.
Types ¶
type Batcher ¶
type Batcher struct {
// contains filtered or unexported fields
}
Batcher buffers results so the single SQLite writer sees one transaction per flush.
func NewBatcher ¶
func NewBatcher(sink ResultSink, capacity int) *Batcher
NewBatcher builds a Batcher that holds at most capacity results.
type Checker ¶
type Checker struct {
// contains filtered or unexported fields
}
Checker runs one probe. Every dial goes through the netguard address check.
func NewChecker ¶
NewChecker builds a Checker whose connections refuse internal addresses unless the netguard private opt-in env is set.
type Config ¶
type Config struct {
Enabled bool
Workers int
MinIntervalHTTP time.Duration
MinIntervalTCP time.Duration
MaxTimeout time.Duration
MaxCount int
BodyMaxBytes int
MaxHeaders int
HeaderMaxBytes int
MaxRedirects int
FlushInterval time.Duration
BufferMax int
RawRetention time.Duration
HourRetention time.Duration
PruneInterval time.Duration
PruneBatch int
JitterPercent int
TickInterval time.Duration
ResultsMaxLimit int
ConfirmFailures int
RecoveryOKs int
UserAgent string
MinIntervalHeartbeat time.Duration
MinIntervalTLS time.Duration
HeartbeatGrace time.Duration
HeartbeatMaxGrace time.Duration
HeartbeatCheck time.Duration
HeartbeatPerMinute int
HeartbeatBodyMax int
HeartbeatMinResponse time.Duration
TLSWarnDays int
TLSCritDays int
}
Config holds every limit of the monitor engine, all read from APP_MONITOR* variables.
func ConfigFromEnv ¶
ConfigFromEnv reads the engine settings. Unset variables take the defaults above.
type CreateInput ¶
CreateInput is a monitor definition plus its header values, which are stored encrypted and never returned.
type HeartbeatState ¶
type HeartbeatState struct {
MonitorID string
TokenHash string
ArmedAt int64
LastOK int64
LastStart int64
LastFail int64
}
HeartbeatState is the persisted ping history of one heartbeat monitor, unix seconds.
type HeartbeatStore ¶
type HeartbeatStore interface {
SaveHeartbeatToken(ctx context.Context, monitorID, tokenHash string, armedAt int64) error
FindHeartbeat(ctx context.Context, tokenHash string) (HeartbeatState, error)
GetHeartbeat(ctx context.Context, monitorID string) (HeartbeatState, error)
RecordHeartbeat(ctx context.Context, monitorID, signal string, at int64) error
}
HeartbeatStore persists token hashes and ping times. FindHeartbeat and GetHeartbeat return ErrNotFound.
type Monitor ¶
type Monitor struct {
ID string
Name string
Kind string
Interval time.Duration
Timeout time.Duration
Enabled bool
Spec Spec
CreatedAt time.Time
// PingToken is the plaintext heartbeat token, set only on the Monitor returned by Create
// and RotateToken. It is never stored: only its hash is.
PingToken string
}
Monitor is one check definition.
type Notifier ¶
type Notifier func(Transition)
Notifier receives confirmed transitions. It must not block the probe worker.
type Options ¶
type Options struct {
Config Config
Store Store
Secrets Secrets
Logger *slog.Logger
// Notify receives confirmed down and recovered transitions; nil drops them.
Notify Notifier
// OnDelete runs after a monitor is removed so alert state tied to it can clear.
OnDelete func(id string)
// Heartbeats stores heartbeat tokens and pings; nil rejects heartbeat monitors.
Heartbeats HeartbeatStore
}
Options configures a Service. Secrets may be nil, which rejects monitors that carry headers.
type Result ¶
type Result struct {
MonitorID string
At int64
OK bool
LatencyMS int
Code int
Reason string
// Degraded marks a passing check that is close to failing, such as a certificate
// inside its warning window. It is not stored, the Reason is.
Degraded bool
}
Result is one probe outcome. Reason is a short class, never a body or header value.
type ResultSink ¶
ResultSink persists a batch of results and updates the hourly rollups in the same transaction.
type Scheduler ¶
type Scheduler struct {
// contains filtered or unexported fields
}
Scheduler keeps monitors in a min-heap by due time. It takes the time as an argument so tests drive it with a fake clock.
func NewScheduler ¶
NewScheduler builds a Scheduler. A nil jitter uses a random share of the interval.
func (*Scheduler) Add ¶
Add schedules m, replacing an earlier entry with the same id. A disabled monitor is only removed.
type Secrets ¶
type Secrets interface {
SetValue(ctx context.Context, serviceName, envKey, plaintext string) error
Resolve(ctx context.Context, serviceName, envKey string) (string, error)
DeleteAll(ctx context.Context, serviceName string) error
}
Secrets keeps header values envelope-encrypted.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service owns the scheduler, workers, batcher and pruner.
func (*Service) Ping ¶
Ping records a signal for the monitor that owns token. Every miss (unknown token, wrong kind, disabled monitor, no heartbeat store) is the same ErrNotFound after the same hash and lookup work.
func (*Service) Results ¶
func (s *Service) Results(ctx context.Context, id string, window time.Duration, limit int) (History, error)
Results returns history for the last window, clamped to the retention periods and the limit cap.
func (*Service) RotateToken ¶
RotateToken replaces a heartbeat monitor's ping token; the old URL stops working at once.
func (*Service) SetRawRetentionSource ¶
SetRawRetentionSource makes raw result retention follow fn on each prune and history read. Call before Run.
func (*Service) SetStatusLink ¶
SetStatusLink changes only the status page link settings of a stored monitor.
type Spec ¶
type Spec struct {
URL string `json:"url,omitempty"`
Method string `json:"method,omitempty"`
HeaderNames []string `json:"header_names,omitempty"`
Expect []StatusRange `json:"expect,omitempty"`
Contains string `json:"contains,omitempty"`
FollowRedirects bool `json:"follow_redirects,omitempty"`
Host string `json:"host,omitempty"`
Port int `json:"port,omitempty"`
// StatusLink opts the monitor in to opening and resolving a status page incident. Zero
// confirm and recovery values fall back to the APP_MONITOR_LINK_* settings.
StatusLink bool `json:"status_link,omitempty"`
LinkConfirmFailures int `json:"link_confirm_failures,omitempty"`
LinkRecoverySeconds int `json:"link_recovery_seconds,omitempty"`
// GraceSeconds is the heartbeat grace period; zero uses APP_MONITOR_HEARTBEAT_GRACE.
GraceSeconds int `json:"grace_seconds,omitempty"`
// WarnDays and CritDays override the APP_MONITOR_TLS_* thresholds for one TLS monitor.
WarnDays int `json:"warn_days,omitempty"`
CritDays int `json:"crit_days,omitempty"`
}
Spec is the kind specific part of a monitor. Header values never live here, only their names.
type StatusLink ¶
StatusLink is the per-monitor status page link setting.
type StatusRange ¶
StatusRange is an inclusive HTTP status interval.
type Store ¶
type Store interface {
ResultSink
ListMonitors(ctx context.Context) ([]Monitor, error)
GetMonitor(ctx context.Context, id string) (Monitor, error)
CountMonitors(ctx context.Context) (int, error)
SaveMonitor(ctx context.Context, m Monitor) error
DeleteMonitor(ctx context.Context, id string) error
ListResults(ctx context.Context, id string, since int64, limit int) ([]Result, error)
ListRollups(ctx context.Context, id string, sinceBucket int64, limit int) ([]Rollup, error)
PruneResults(ctx context.Context, id string, before int64, limit int) (int, error)
PruneRollups(ctx context.Context, id string, before int64, limit int) (int, error)
}
Store is the persistence the engine needs.
type Tracker ¶
type Tracker struct {
// contains filtered or unexported fields
}
Tracker turns raw probe results into confirmed states so one bad sample never pages. Flap control stays in alerting, this only confirms.
func NewTracker ¶
NewTracker builds a Tracker; counts below 1 are treated as 1.
func (*Tracker) Force ¶
Force sets a monitor up or down at once, for kinds whose own timing already confirms the change. It notifies on a real transition and returns the state afterwards.
type Transition ¶
Transition is a confirmed change worth telling someone about: down or back up.