alert

package
v1.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 18, 2026 License: AGPL-3.0 Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source
const (
	SourceContainer   = "container"
	SourceEndpoint    = "endpoint"
	SourceHeartbeat   = "heartbeat"
	SourceCertificate = "certificate"
	SourceResource    = "resource"
	SourceSecurity    = "security"
	SourceAgent       = "agent"
	SourceHost        = "host"
)

Alert sources.

View Source
const (
	AlertTypeDangerousConfig  = "dangerous_configuration"
	AlertTypePostureThreshold = "posture_threshold"
)

Security alert types.

View Source
const (
	StatusActive   = "active"
	StatusResolved = "resolved"
	StatusSilenced = "silenced"
)

Alert statuses.

View Source
const (
	SeverityCritical = "critical"
	SeverityWarning  = "warning"
	SeverityInfo     = "info"
)

Severity levels.

View Source
const (
	DeliveryPending   = "pending"
	DeliveryDelivered = "delivered"
	DeliveryFailed    = "failed"
)

Delivery statuses.

View Source
const AlertTypeContainerDown = "container_down"

AlertTypeContainerDown is raised for a container that has been stopped for longer than the configured threshold.

View Source
const (

	// CriticalRestartMultiplier is applied to the per-container restart
	// threshold to determine when the alert escalates from warning to
	// critical. E.g. threshold=3 → critical at 9 restarts in the window.
	CriticalRestartMultiplier = 3
)
View Source
const TelegramAPIBase = "https://api.telegram.org"

TelegramAPIBase is the destination the product calls. The operator never supplies it: that is what makes the channel immune to the SSRF question a user-supplied URL raises.

Variables

This section is empty.

Functions

func BuildTelegramMessage added in v1.5.0

func BuildTelegramMessage(eventType string, a *Alert) string

BuildTelegramMessage renders the fired or resolved message. The severity and the entity sit on the first line so a phone notification, which shows little, still says what happened and to what (SC-006).

func SendTelegram added in v1.5.0

func SendTelegram(ctx context.Context, client *http.Client, apiBase string, ch *NotificationChannel, text string) error

SendTelegram posts one message.

The client and the API base are parameters, not fields read from the caller: in production the notifier hands over its own client, SSRF guard included, and the real host; tests hand over a bare client and an httptest server, which the guard would otherwise refuse for listening on 127.0.0.1.

The URL is never logged and never wrapped into an error: it carries the bot token (FR-005).

func ValidateBotToken added in v1.5.0

func ValidateBotToken(token string) error

ValidateBotToken rejects a token whose shape cannot be a Telegram token. It says nothing about whether the token still works: that is what the test button is for. The point is to spare a network call, and a puzzling refusal from Telegram, on an obvious typo.

func ValidateChatID added in v1.5.0

func ValidateChatID(chatID string) error

ValidateChatID accepts a numeric chat id, negative or not, or a public @username. The -100 prefix supergroups carry is part of the number and is passed through untouched.

func ValidateThreadID added in v1.5.0

func ValidateThreadID(threadID string) error

ValidateThreadID accepts an empty value: a channel without a topic posts to the group's general thread.

Types

type Acknowledgment added in v1.2.12

type Acknowledgment struct {
	By string
	At time.Time
}

Acknowledgment carries info about an alert ack event.

type Alert

type Alert struct {
	ID             string     `json:"id"`
	Source         string     `json:"source"`
	AlertType      string     `json:"alert_type"`
	Severity       string     `json:"severity"`
	Status         string     `json:"status"`
	Message        string     `json:"message"`
	EntityType     string     `json:"entity_type"`
	EntityID       string     `json:"entity_id"`
	EntityName     string     `json:"entity_name"`
	Details        string     `json:"details"`
	ResolvedByID   *string    `json:"resolved_by_id"`
	FiredAt        time.Time  `json:"fired_at"`
	ResolvedAt     *time.Time `json:"resolved_at"`
	AcknowledgedAt *time.Time `json:"acknowledged_at,omitempty"`
	AcknowledgedBy string     `json:"acknowledged_by,omitempty"`
	EscalatedAt    *time.Time `json:"escalated_at,omitempty"`
	CreatedAt      time.Time  `json:"created_at"`
}

Alert represents a persisted alert record.

type AlertStore

type AlertStore interface {
	InsertAlert(ctx context.Context, a *Alert) (string, error)
	GetAlert(ctx context.Context, id string) (*Alert, error)
	ListAlerts(ctx context.Context, opts ListAlertsOpts) ([]*Alert, error)
	UpdateAlertStatus(ctx context.Context, id string, status string, resolvedAt *time.Time, resolvedByID *string) error
	// UpdateAlertOnEscalation refreshes a live alert to the latest event:
	// severity, message AND entity name/details, so an escalation never leaves a
	// record mixing two events' fields.
	UpdateAlertOnEscalation(ctx context.Context, id, severity, message, entityName, details string) error
	GetActiveAlert(ctx context.Context, source, alertType, entityType string, entityID string) (*Alert, error)
	ListActiveAlerts(ctx context.Context) ([]*Alert, error)
	DeleteAlertsOlderThan(ctx context.Context, before time.Time) (int64, error)
	AcknowledgeAlert(ctx context.Context, id string, by string, at time.Time) error
	SetEscalatedAt(ctx context.Context, id string, at time.Time) error
	ListUnacknowledgedActiveAlerts(ctx context.Context) ([]*Alert, error)
}

AlertStore defines the persistence interface for alerts.

type AlertTrigger added in v1.2.12

type AlertTrigger struct {
	ID               string    `json:"id"`
	Name             string    `json:"name"`
	FilterSeverities string    `json:"filter_severities"`
	FilterSources    string    `json:"filter_sources"`
	FilterScopes     string    `json:"filter_scopes"`
	FilterTags       string    `json:"filter_tags"`
	Enabled          bool      `json:"enabled"`
	NotifyOnResolve  bool      `json:"notify_on_resolve"`
	ChannelIDs       []string  `json:"channel_ids"`
	CreatedAt        time.Time `json:"created_at"`
	UpdatedAt        time.Time `json:"updated_at"`
}

AlertTrigger is a routing rule that maps an alert filter to one or more channels. Filters are stored as CSV strings; an empty filter matches anything. Filters are combined in AND between fields, OR within a field. FilterScopes and FilterTags are Pro-only (gated at the handler level).

type ChannelStore

type ChannelStore interface {
	InsertChannel(ctx context.Context, ch *NotificationChannel) (string, error)
	GetChannel(ctx context.Context, id string) (*NotificationChannel, error)
	ListChannels(ctx context.Context) ([]*NotificationChannel, error)
	UpdateChannel(ctx context.Context, ch *NotificationChannel) error
	DeleteChannel(ctx context.Context, id string) error
	GetChannelHealth(ctx context.Context, channelID string) (string, error)

	InsertDelivery(ctx context.Context, d *NotificationDelivery) (string, error)
	UpdateDelivery(ctx context.Context, d *NotificationDelivery) error
	ListDeliveriesByAlert(ctx context.Context, alertID string) ([]*NotificationDelivery, error)
}

ChannelStore defines the persistence interface for notification channels.

type DownDetector added in v1.5.0

type DownDetector struct {
	// contains filtered or unexported fields
}

DownDetector raises an alert for every container stopped for longer than the threshold, and resolves it once the container runs again. It reads the state from the store on each pass instead of tracking it in memory, so a container that went down while the server was off is still caught, and it asks the alert store what is already firing instead of re-emitting every pass.

func NewDownDetector added in v1.5.0

func NewDownDetector(containers container.ContainerStore, alerts AlertStore, after time.Duration, logger *slog.Logger) *DownDetector

NewDownDetector returns a detector for containers stopped longer than after.

func (*DownDetector) Check added in v1.5.0

func (d *DownDetector) Check(ctx context.Context) error

Check runs one pass: fire for what crossed the threshold, resolve for what came back.

func (*DownDetector) SetEmitter added in v1.5.0

func (d *DownDetector) SetEmitter(emit func(Event))

SetEmitter sets the sink receiving the fire and recovery events.

type EndpointAlert

type EndpointAlert struct {
	EndpointID    string
	ContainerName string
	Target        string
	Type          string // "alert" or "recovery"
	NewAlertState endpoint.AlertState
	Failures      int
	Successes     int
	Threshold     int
	LastError     string
	Timestamp     time.Time
}

EndpointAlert represents an alert or recovery event for an endpoint.

type EndpointAlertDetector

type EndpointAlertDetector struct{}

EndpointAlertDetector evaluates check results against thresholds.

func NewEndpointAlertDetector

func NewEndpointAlertDetector() *EndpointAlertDetector

NewEndpointAlertDetector creates a new detector.

func (*EndpointAlertDetector) EvaluateCheckResult

func (d *EndpointAlertDetector) EvaluateCheckResult(ep *endpoint.Endpoint, result endpoint.CheckResult) *EndpointAlert

EvaluateCheckResult checks if a threshold has been crossed and returns an alert if so. Returns nil if no alert/recovery needs to be emitted.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine consumes alert events from monitoring services, persists them, evaluates silence rules, dispatches notifications, and broadcasts via SSE.

func NewEngine

func NewEngine(d EngineDeps) *Engine

NewEngine creates a new alert engine.

func (*Engine) AlertCount

func (e *Engine) AlertCount() int

AlertCount returns the number of active alerts (for monitoring).

func (*Engine) Escalator added in v1.2.12

func (e *Engine) Escalator() Escalator

Escalator returns the current escalator implementation.

func (*Engine) EventChannel

func (e *Engine) EventChannel() chan<- Event

EventChannel returns the channel for sending alert events to the engine.

func (*Engine) ResolveByEntity

func (e *Engine) ResolveByEntity(ctx context.Context, entityType string, entityID string)

ResolveByEntity resolves all active alerts for a given entity (e.g. when a container is destroyed). This prevents stale alerts from accumulating when containers are recreated with new internal IDs.

func (*Engine) SetBroadcaster

func (e *Engine) SetBroadcaster(b SSEBroadcaster)

SetBroadcaster sets the SSE broadcaster.

func (*Engine) SetEntityRouter added in v1.0.1

func (e *Engine) SetEntityRouter(r EntityRouter)

SetEntityRouter sets the entity routing extension.

func (*Engine) SetEscalator added in v1.0.1

func (e *Engine) SetEscalator(esc Escalator)

SetEscalator sets the escalation extension.

func (*Engine) SetMaintenanceSuppressor added in v1.0.1

func (e *Engine) SetMaintenanceSuppressor(s MaintenanceSuppressor)

SetMaintenanceSuppressor sets the maintenance suppression extension.

func (*Engine) SetNotifier

func (e *Engine) SetNotifier(n *Notifier)

SetNotifier sets the webhook notifier for dispatching notifications.

func (*Engine) Start

func (e *Engine) Start(ctx context.Context)

Start begins the engine's event processing loop. Call this in a goroutine.

type EngineDeps added in v1.1.0

type EngineDeps struct {
	AlertStore   AlertStore     // required
	ChannelStore ChannelStore   // required
	TriggerStore TriggerStore   // required
	SilenceStore SilenceStore   // required
	Logger       *slog.Logger   // required
	Notifier     *Notifier      // optional — nil-safe
	Broadcaster  SSEBroadcaster // optional — nil-safe
}

EngineDeps holds all dependencies for the alert Engine.

type EntityRouter added in v1.0.1

type EntityRouter interface {
	Route(ctx context.Context, entityType string, entityID string, severity string) ([]string, error)
}

EntityRouter provides per-entity alert routing.

type Escalator added in v1.0.1

type Escalator interface {
	EvaluateCycle(ctx context.Context) error
	OnAlertCreated(ctx context.Context, a *Alert) error
	OnAlertAcknowledged(ctx context.Context, alertID string, ack Acknowledgment) error
	OnAlertResolved(ctx context.Context, alertID string, resolvedAt time.Time) error
	OnEditionDowngraded(ctx context.Context) error
}

Escalator manages Pro escalation policy execution.

type Event

type Event struct {
	Source     string         // "container", "endpoint", "heartbeat", "certificate", "resource"
	AlertType  string         // source-specific type
	Severity   string         // "critical", "warning", "info"
	IsRecover  bool           // true if this is a recovery event
	Message    string         // human-readable description
	EntityType string         // "container", "endpoint", "heartbeat", "certificate"
	EntityID   string         // UUID of the referenced entity in its source table
	EntityName string         // display name
	Details    map[string]any // source-specific metadata
	Timestamp  time.Time      // when condition was detected
}

Event represents a unified alert event sent via Go channel from any monitoring service.

type HealthAlert

type HealthAlert struct {
	ContainerID    string
	ContainerName  string
	PreviousHealth *container.HealthStatus
	NewHealth      container.HealthStatus
	Severity       container.AlertSeverity
	Channels       string
	Timestamp      time.Time
}

HealthAlert represents a health status change alert.

func CheckHealthTransition

func CheckHealthTransition(c *container.Container, previousHealth *container.HealthStatus, newHealth container.HealthStatus) *HealthAlert

CheckHealthTransition returns an alert if a container transitions from healthy to unhealthy.

type ListAlertsOpts

type ListAlertsOpts struct {
	Source   string
	Severity string
	Status   string
	Before   *time.Time
	Limit    int
}

ListAlertsOpts contains filter parameters for listing alerts.

type MaintenanceSuppressor added in v1.0.1

type MaintenanceSuppressor interface {
	IsSuppressed(ctx context.Context, source string, entityType string, entityID string) (bool, error)
}

MaintenanceSuppressor checks if an alert should be suppressed during a maintenance window.

type NotificationChannel

type NotificationChannel struct {
	ID      string `json:"id"`
	Name    string `json:"name"`
	Type    string `json:"type"`
	URL     string `json:"url"`
	Headers string `json:"headers,omitempty"`
	// Secret holds the channel credential (a Telegram bot token, today). It is
	// tagged "-" and not omitempty: omitempty would still serialize a non-empty
	// token, which is exactly the value that must never leave the process.
	Secret string `json:"-"`
	// Config holds non-secret per-type settings as JSON, e.g. {"thread_id":"42"}.
	Config string `json:"config,omitempty"`
	// HasSecret is derived at scan time, never stored. It lets the interface show
	// "token on file" without receiving the token.
	HasSecret bool      `json:"has_secret"`
	Enabled   bool      `json:"enabled"`
	Health    string    `json:"health,omitempty"`
	CreatedAt time.Time `json:"created_at"`
	UpdatedAt time.Time `json:"updated_at"`
}

NotificationChannel represents a configured delivery target. Channels are silent by default — they only receive alerts when referenced by an active AlertTrigger or by an EscalationLevel.

type NotificationDelivery

type NotificationDelivery struct {
	ID        string    `json:"id"`
	AlertID   string    `json:"alert_id"`
	ChannelID string    `json:"channel_id"`
	Status    string    `json:"status"`
	Attempts  int       `json:"attempts"`
	LastError string    `json:"last_error,omitempty"`
	CreatedAt time.Time `json:"created_at"`
	UpdatedAt time.Time `json:"updated_at"`
}

NotificationDelivery represents a delivery attempt record.

type NotificationJob

type NotificationJob struct {
	Delivery *NotificationDelivery
	Channel  *NotificationChannel
	Alert    *Alert
	// Body, when non-nil, is the pre-rendered request body sent verbatim
	// (used by webhook subscriptions, which dispatch the raw event payload
	// they already marshalled and signed). When nil, the body is formatted
	// from Alert.
	Body []byte
}

NotificationJob represents a webhook delivery job.

type Notifier

type Notifier struct {
	// contains filtered or unexported fields
}

Notifier dispatches webhook notifications with a bounded worker pool.

func NewNotifier

func NewNotifier(channelStore ChannelStore, logger *slog.Logger, allowPrivate bool) *Notifier

NewNotifier creates a new webhook notifier. Its HTTP client blocks delivery to private/internal IPs (SSRF guard) at dial time unless allowPrivate is set (dev only, via MAINTENANT_ALLOW_PRIVATE_WEBHOOKS).

func (*Notifier) Enqueue

func (n *Notifier) Enqueue(job NotificationJob)

Enqueue adds a notification job to the work queue.

func (*Notifier) SMTPConfigured

func (n *Notifier) SMTPConfigured() bool

SMTPConfigured reports whether SMTP delivery is available.

func (*Notifier) SendNow added in v1.2.12

func (n *Notifier) SendNow(ctx context.Context, a *Alert, ch *NotificationChannel) error

SendNow performs a synchronous send to the given channel with internal retries. Unlike Enqueue, it does not write to the notification_deliveries table — the caller (e.g. the escalation Runner) manages its own delivery row state. Returns nil on success, or the last error after maxRetries attempts.

func (*Notifier) SendTestWebhook

func (n *Notifier) SendTestWebhook(ctx context.Context, ch *NotificationChannel) (int, error)

SendTestWebhook sends a test notification to verify a channel is reachable.

func (*Notifier) SetSMTPSender

func (n *Notifier) SetSMTPSender(sender *SMTPSender)

SetSMTPSender configures SMTP delivery for email channels.

func (*Notifier) SetTelegramTransport added in v1.5.0

func (n *Notifier) SetTelegramTransport(apiBase string, client *http.Client)

SetTelegramTransport points Telegram delivery at another host with another client. Tests use it: the guarded client refuses 127.0.0.1, where an httptest server listens.

func (*Notifier) Start

func (n *Notifier) Start(ctx context.Context)

Start begins the worker pool. Call in a goroutine.

type RestartAlert

type RestartAlert struct {
	ContainerID   string
	ContainerName string
	RestartCount  int
	Threshold     int
	Severity      container.AlertSeverity
	Channels      string
	Timestamp     time.Time
	AgentID       string
}

RestartAlert represents a restart threshold alert.

type RestartDetector

type RestartDetector struct {
	// contains filtered or unexported fields
}

RestartDetector checks for crash-loop restart patterns.

func NewRestartDetector

func NewRestartDetector(store container.ContainerStore, logger *slog.Logger) *RestartDetector

NewRestartDetector creates a new restart detector.

func (*RestartDetector) Check

func (d *RestartDetector) Check(ctx context.Context, c *container.Container) (interface{}, error)

Check evaluates whether the container has exceeded its restart threshold. Returns an alert if threshold is exceeded, nil otherwise.

type SMTPConfig

type SMTPConfig struct {
	Host     string
	Port     string
	Username string
	Password string
	From     string
}

SMTPConfig holds SMTP connection parameters.

type SMTPSender

type SMTPSender struct {
	// contains filtered or unexported fields
}

SMTPSender sends email notifications via SMTP with STARTTLS.

func NewSMTPSender

func NewSMTPSender(cfg SMTPConfig) *SMTPSender

NewSMTPSender creates a new SMTPSender with the given configuration.

func (*SMTPSender) Send

func (s *SMTPSender) Send(_ context.Context, to, subject, textBody string) error

Send delivers an email via SMTP. The to parameter is the recipient address, subject is the email subject line, and textBody is the plain-text content.

type SSEBroadcaster

type SSEBroadcaster interface {
	Broadcast(eventType string, data interface{})
}

SSEBroadcaster is the interface for broadcasting SSE events.

func NewSSEBroadcasterFunc

func NewSSEBroadcasterFunc(fn func(eventType string, data interface{})) SSEBroadcaster

NewSSEBroadcasterFunc creates an SSEBroadcaster from a function.

type SilenceRule

type SilenceRule struct {
	ID              string     `json:"id"`
	EntityType      string     `json:"entity_type,omitempty"`
	EntityID        *string    `json:"entity_id,omitempty"`
	Source          string     `json:"source,omitempty"`
	Reason          string     `json:"reason,omitempty"`
	StartsAt        time.Time  `json:"starts_at"`
	DurationSeconds int        `json:"duration_seconds"`
	ExpiresAt       time.Time  `json:"expires_at"`
	IsActive        bool       `json:"is_active"`
	CancelledAt     *time.Time `json:"cancelled_at,omitempty"`
	CreatedAt       time.Time  `json:"created_at"`
}

SilenceRule represents a time-bounded notification suppression.

type SilenceStore

type SilenceStore interface {
	InsertSilenceRule(ctx context.Context, rule *SilenceRule) (string, error)
	ListSilenceRules(ctx context.Context, activeOnly bool) ([]*SilenceRule, error)
	CancelSilenceRule(ctx context.Context, id string) error
	GetActiveSilenceRules(ctx context.Context) ([]*SilenceRule, error)
}

SilenceStore defines the persistence interface for silence rules.

type TelegramConfig added in v1.5.0

type TelegramConfig struct {
	ThreadID string `json:"thread_id,omitempty"`
}

TelegramConfig is the non-secret part of a Telegram channel, serialized into NotificationChannel.Config.

func ParseTelegramConfig added in v1.5.0

func ParseTelegramConfig(raw string) (TelegramConfig, error)

ParseTelegramConfig reads the channel's config column. An empty column is a channel with no topic, not an error.

type TelegramRateLimitError added in v1.5.0

type TelegramRateLimitError struct {
	RetryAfter time.Duration
	Reason     string
}

TelegramRateLimitError carries the delay Telegram asked for. It is a typed error rather than a string the retry loop would have to read, so honouring the delay cannot break on a wording change (FR-014).

func (*TelegramRateLimitError) Error added in v1.5.0

func (e *TelegramRateLimitError) Error() string

type TriggerStore added in v1.2.12

type TriggerStore interface {
	InsertTrigger(ctx context.Context, t *AlertTrigger) (string, error)
	GetTrigger(ctx context.Context, id string) (*AlertTrigger, error)
	ListTriggers(ctx context.Context) ([]*AlertTrigger, error)
	ListEnabledTriggers(ctx context.Context) ([]*AlertTrigger, error)
	UpdateTrigger(ctx context.Context, t *AlertTrigger) error
	DeleteTrigger(ctx context.Context, id string) error
	SetChannels(ctx context.Context, triggerID string, channelIDs []string) error
	ListChannelsForTrigger(ctx context.Context, triggerID string) ([]string, error)
	ListTriggersForChannel(ctx context.Context, channelID string) ([]*AlertTrigger, error)
}

TriggerStore defines the persistence interface for AlertTrigger objects and their M:N relationship with channels.

type WebhookPayload

type WebhookPayload struct {
	Event     string                 `json:"event"`
	Alert     map[string]interface{} `json:"alert"`
	Timestamp string                 `json:"timestamp"`
}

WebhookPayload is the JSON body sent to generic webhook URLs.

Directories

Path Synopsis
Package escalation manages user-defined escalation policies (Pro).
Package escalation manages user-defined escalation policies (Pro).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL