Documentation
¶
Overview ¶
Package incident opens internal incidents from confirmed monitor transitions (ADR 061).
Index ¶
Constants ¶
const ( DefaultFoldWindow = 10 * time.Minute DefaultReopenWindow = 15 * time.Minute DefaultSweepInterval = time.Minute DefaultOpenMax = 200 DefaultQueueMax = 256 DefaultUpdateBodyMax = 4096 )
Defaults apply only when the matching APP_INCIDENT* variable is unset.
const ( // AuditLinkOpened is written when a monitor outage opens a status page incident. AuditLinkOpened = "monitor_incident.opened" // AuditLinkResolved is written when that incident resolves. AuditLinkResolved = "monitor_incident.resolved" )
Audit actions for system written incident rows.
const ( DefaultLinkConfirmFailures = 1 DefaultLinkRecoveryPeriod = 2 * time.Minute DefaultLinkSweepInterval = 30 * time.Second DefaultLinkImpact = "major" )
Defaults apply only when the matching APP_MONITOR_LINK_* variable is unset.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config struct {
// Enabled needs both APP_MONITORS_ENABLED and APP_INCIDENTS_ENABLED.
Enabled bool
FoldWindow time.Duration
ReopenWindow time.Duration
SweepInterval time.Duration
OpenMax int
QueueMax int
// UpdateBodyMax caps the bytes of an internal note.
UpdateBodyMax int
}
Config holds every knob of the incident opener, all from the environment.
type Guard ¶
type Guard interface {
RuleEnabled(ctx context.Context) bool
Muted(ctx context.Context, monitorID, name string, now time.Time) (detail string, muted bool)
}
Guard answers the alerting questions: is monitor_down enabled, and is this outage muted.
type LinkConfig ¶
type LinkConfig struct {
// ConfirmFailures is the consecutive failed probes a confirmed-down monitor needs before an incident opens.
ConfirmFailures int
// RecoveryPeriod is how long a monitor must stay up before its incident resolves.
RecoveryPeriod time.Duration
SweepInterval time.Duration
// Impact is the component impact the incident carries (minor or major).
Impact string
QueueMax int
}
LinkConfig tunes the monitor to status page link. Per-monitor settings override the first two.
func LinkConfigFromEnv ¶
func LinkConfigFromEnv(lookup func(string) (string, bool)) (LinkConfig, error)
LinkConfigFromEnv reads the settings; invalid values fall back to the default and are reported in the error.
type LinkStore ¶
type LinkStore interface {
GetMonitor(ctx context.Context, id string) (monitor.Monitor, error)
ListMonitors(ctx context.Context) ([]monitor.Monitor, error)
ListStatusComponents(ctx context.Context) ([]store.StatusComponent, error)
ListStatusIncidents(ctx context.Context, limit int) ([]store.StatusIncident, error)
ListOpenLinkedIncidents(ctx context.Context) ([]store.InternalStatusIncident, error)
CreateInternalStatusIncident(ctx context.Context, in store.InternalStatusIncident) error
SaveStatusIncident(ctx context.Context, i store.StatusIncident) error
SetStatusIncidentInternalState(ctx context.Context, id, state, by string, at time.Time) error
AddInternalIncidentUpdate(ctx context.Context, u store.StatusIncidentUpdate, visibility, author string) error
}
LinkStore is the persistence the Linker needs.
type Linker ¶
type Linker struct {
// contains filtered or unexported fields
}
Linker opens a public status page incident for an opted-in monitor that is confirmed down and resolves it after the recovery period. Paging stays with the monitor_down alert rule.
func NewLinker ¶
func NewLinker(cfg LinkConfig, st LinkStore, g Guard, mons MonitorState, audit Auditor, logger *slog.Logger) *Linker
NewLinker builds a Linker. guard and audit may be nil; a nil guard means nothing is muted.
func (*Linker) Notify ¶
func (l *Linker) Notify(t monitor.Transition)
Notify is a monitor.Notifier hook that never blocks the probe worker; the sweep catches drops up.
func (*Linker) SetMonitors ¶
func (l *Linker) SetMonitors(m MonitorState)
SetMonitors wires the monitor service once it exists (it needs Notify first).
func (*Linker) SetPublisher ¶
SetPublisher wires the status webhook emitter from the composition root, so this package does not import the API or alerting layers.
type MonitorState ¶
type MonitorState interface {
List(ctx context.Context) ([]monitor.Monitor, error)
Status(id string) monitor.Status
}
MonitorState is the live monitor view the sweep reconciles against.
type Opener ¶
type Opener struct {
// contains filtered or unexported fields
}
Opener turns confirmed monitor transitions into internal, unpublished incidents.
func (*Opener) Handle ¶
func (o *Opener) Handle(ctx context.Context, t monitor.Transition)
Handle applies one confirmed transition.
func (*Opener) Notify ¶
func (o *Opener) Notify(t monitor.Transition)
Notify is the monitor.Notifier hook. On overflow it drops and logs; the sweep catches up.
func (*Opener) SetMonitors ¶
func (o *Opener) SetMonitors(m MonitorState)
SetMonitors wires the monitor service once it exists (it needs Notify first).
type Publisher ¶
Publisher queues a status webhook for a public incident; opened is false on resolve.
type Store ¶
type Store interface {
ListStatusComponents(ctx context.Context) ([]store.StatusComponent, error)
ListMonitorIncidents(ctx context.Context, since time.Time) ([]store.InternalStatusIncident, error)
ListStatusIncidentLinks(ctx context.Context, incidentID string) ([]store.StatusIncidentLink, error)
CreateInternalStatusIncident(ctx context.Context, in store.InternalStatusIncident) error
SetStatusIncidentInternalState(ctx context.Context, id, state, by string, at time.Time) error
AddInternalIncidentUpdate(ctx context.Context, u store.StatusIncidentUpdate, visibility, author string) error
AddStatusIncidentLink(ctx context.Context, incidentID, monitorID string, downAt time.Time) error
AddStatusIncidentComponents(ctx context.Context, incidentID string, ids []string) error
MarkStatusIncidentLinkUp(ctx context.Context, incidentID, monitorID string, upAt time.Time) error
}
Store is the persistence the opener needs.
Source Files