Documentation
¶
Index ¶
- Constants
- func BuildTelegramMessage(eventType string, a *Alert) string
- func SendTelegram(ctx context.Context, client *http.Client, apiBase string, ...) error
- func ValidateBotToken(token string) error
- func ValidateChatID(chatID string) error
- func ValidateThreadID(threadID string) error
- type Acknowledgment
- type Alert
- type AlertStore
- type AlertTrigger
- type ChannelStore
- type DownDetector
- type EndpointAlert
- type EndpointAlertDetector
- type Engine
- func (e *Engine) AlertCount() int
- func (e *Engine) Escalator() Escalator
- func (e *Engine) EventChannel() chan<- Event
- func (e *Engine) ResolveByEntity(ctx context.Context, entityType string, entityID string)
- func (e *Engine) SetBroadcaster(b SSEBroadcaster)
- func (e *Engine) SetEntityRouter(r EntityRouter)
- func (e *Engine) SetEscalator(esc Escalator)
- func (e *Engine) SetMaintenanceSuppressor(s MaintenanceSuppressor)
- func (e *Engine) SetNotifier(n *Notifier)
- func (e *Engine) Start(ctx context.Context)
- type EngineDeps
- type EntityRouter
- type Escalator
- type Event
- type HealthAlert
- type ListAlertsOpts
- type MaintenanceSuppressor
- type NotificationChannel
- type NotificationDelivery
- type NotificationJob
- type Notifier
- func (n *Notifier) Enqueue(job NotificationJob)
- func (n *Notifier) SMTPConfigured() bool
- func (n *Notifier) SendNow(ctx context.Context, a *Alert, ch *NotificationChannel) error
- func (n *Notifier) SendTestWebhook(ctx context.Context, ch *NotificationChannel) (int, error)
- func (n *Notifier) SetSMTPSender(sender *SMTPSender)
- func (n *Notifier) SetTelegramTransport(apiBase string, client *http.Client)
- func (n *Notifier) Start(ctx context.Context)
- type RestartAlert
- type RestartDetector
- type SMTPConfig
- type SMTPSender
- type SSEBroadcaster
- type SilenceRule
- type SilenceStore
- type TelegramConfig
- type TelegramRateLimitError
- type TriggerStore
- type WebhookPayload
Constants ¶
const ( SourceContainer = "container" SourceEndpoint = "endpoint" SourceHeartbeat = "heartbeat" SourceCertificate = "certificate" SourceResource = "resource" SourceSecurity = "security" SourceAgent = "agent" SourceHost = "host" )
Alert sources.
const ( AlertTypeDangerousConfig = "dangerous_configuration" AlertTypePostureThreshold = "posture_threshold" )
Security alert types.
const ( StatusActive = "active" StatusResolved = "resolved" StatusSilenced = "silenced" )
Alert statuses.
const ( SeverityCritical = "critical" SeverityWarning = "warning" SeverityInfo = "info" )
Severity levels.
const ( DeliveryPending = "pending" DeliveryDelivered = "delivered" DeliveryFailed = "failed" )
Delivery statuses.
const AlertTypeContainerDown = "container_down"
AlertTypeContainerDown is raised for a container that has been stopped for longer than the configured threshold.
const ( // CriticalRestartMultiplier is applied to the per-container restart // threshold to determine when the alert escalates from warning to // critical. E.g. threshold=3 → critical at 9 restarts in the window. CriticalRestartMultiplier = 3 )
const TelegramAPIBase = "https://api.telegram.org"
TelegramAPIBase is the destination the product calls. The operator never supplies it: that is what makes the channel immune to the SSRF question a user-supplied URL raises.
Variables ¶
This section is empty.
Functions ¶
func BuildTelegramMessage ¶ added in v1.5.0
BuildTelegramMessage renders the fired or resolved message. The severity and the entity sit on the first line so a phone notification, which shows little, still says what happened and to what (SC-006).
func SendTelegram ¶ added in v1.5.0
func SendTelegram(ctx context.Context, client *http.Client, apiBase string, ch *NotificationChannel, text string) error
SendTelegram posts one message.
The client and the API base are parameters, not fields read from the caller: in production the notifier hands over its own client, SSRF guard included, and the real host; tests hand over a bare client and an httptest server, which the guard would otherwise refuse for listening on 127.0.0.1.
The URL is never logged and never wrapped into an error: it carries the bot token (FR-005).
func ValidateBotToken ¶ added in v1.5.0
ValidateBotToken rejects a token whose shape cannot be a Telegram token. It says nothing about whether the token still works: that is what the test button is for. The point is to spare a network call, and a puzzling refusal from Telegram, on an obvious typo.
func ValidateChatID ¶ added in v1.5.0
ValidateChatID accepts a numeric chat id, negative or not, or a public @username. The -100 prefix supergroups carry is part of the number and is passed through untouched.
func ValidateThreadID ¶ added in v1.5.0
ValidateThreadID accepts an empty value: a channel without a topic posts to the group's general thread.
Types ¶
type Acknowledgment ¶ added in v1.2.12
Acknowledgment carries info about an alert ack event.
type Alert ¶
type Alert struct {
ID string `json:"id"`
Source string `json:"source"`
AlertType string `json:"alert_type"`
Severity string `json:"severity"`
Status string `json:"status"`
Message string `json:"message"`
EntityType string `json:"entity_type"`
EntityID string `json:"entity_id"`
EntityName string `json:"entity_name"`
Details string `json:"details"`
ResolvedByID *string `json:"resolved_by_id"`
FiredAt time.Time `json:"fired_at"`
ResolvedAt *time.Time `json:"resolved_at"`
AcknowledgedAt *time.Time `json:"acknowledged_at,omitempty"`
AcknowledgedBy string `json:"acknowledged_by,omitempty"`
EscalatedAt *time.Time `json:"escalated_at,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
Alert represents a persisted alert record.
type AlertStore ¶
type AlertStore interface {
InsertAlert(ctx context.Context, a *Alert) (string, error)
GetAlert(ctx context.Context, id string) (*Alert, error)
ListAlerts(ctx context.Context, opts ListAlertsOpts) ([]*Alert, error)
UpdateAlertStatus(ctx context.Context, id string, status string, resolvedAt *time.Time, resolvedByID *string) error
// UpdateAlertOnEscalation refreshes a live alert to the latest event:
// severity, message AND entity name/details, so an escalation never leaves a
// record mixing two events' fields.
UpdateAlertOnEscalation(ctx context.Context, id, severity, message, entityName, details string) error
GetActiveAlert(ctx context.Context, source, alertType, entityType string, entityID string) (*Alert, error)
ListActiveAlerts(ctx context.Context) ([]*Alert, error)
DeleteAlertsOlderThan(ctx context.Context, before time.Time) (int64, error)
AcknowledgeAlert(ctx context.Context, id string, by string, at time.Time) error
SetEscalatedAt(ctx context.Context, id string, at time.Time) error
ListUnacknowledgedActiveAlerts(ctx context.Context) ([]*Alert, error)
}
AlertStore defines the persistence interface for alerts.
type AlertTrigger ¶ added in v1.2.12
type AlertTrigger struct {
ID string `json:"id"`
Name string `json:"name"`
FilterSeverities string `json:"filter_severities"`
FilterSources string `json:"filter_sources"`
FilterScopes string `json:"filter_scopes"`
FilterTags string `json:"filter_tags"`
Enabled bool `json:"enabled"`
NotifyOnResolve bool `json:"notify_on_resolve"`
ChannelIDs []string `json:"channel_ids"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
AlertTrigger is a routing rule that maps an alert filter to one or more channels. Filters are stored as CSV strings; an empty filter matches anything. Filters are combined in AND between fields, OR within a field. FilterScopes and FilterTags are Pro-only (gated at the handler level).
type ChannelStore ¶
type ChannelStore interface {
InsertChannel(ctx context.Context, ch *NotificationChannel) (string, error)
GetChannel(ctx context.Context, id string) (*NotificationChannel, error)
ListChannels(ctx context.Context) ([]*NotificationChannel, error)
UpdateChannel(ctx context.Context, ch *NotificationChannel) error
DeleteChannel(ctx context.Context, id string) error
GetChannelHealth(ctx context.Context, channelID string) (string, error)
InsertDelivery(ctx context.Context, d *NotificationDelivery) (string, error)
UpdateDelivery(ctx context.Context, d *NotificationDelivery) error
ListDeliveriesByAlert(ctx context.Context, alertID string) ([]*NotificationDelivery, error)
}
ChannelStore defines the persistence interface for notification channels.
type DownDetector ¶ added in v1.5.0
type DownDetector struct {
// contains filtered or unexported fields
}
DownDetector raises an alert for every container stopped for longer than the threshold, and resolves it once the container runs again. It reads the state from the store on each pass instead of tracking it in memory, so a container that went down while the server was off is still caught, and it asks the alert store what is already firing instead of re-emitting every pass.
func NewDownDetector ¶ added in v1.5.0
func NewDownDetector(containers container.ContainerStore, alerts AlertStore, after time.Duration, logger *slog.Logger) *DownDetector
NewDownDetector returns a detector for containers stopped longer than after.
func (*DownDetector) Check ¶ added in v1.5.0
func (d *DownDetector) Check(ctx context.Context) error
Check runs one pass: fire for what crossed the threshold, resolve for what came back.
func (*DownDetector) SetEmitter ¶ added in v1.5.0
func (d *DownDetector) SetEmitter(emit func(Event))
SetEmitter sets the sink receiving the fire and recovery events.
type EndpointAlert ¶
type EndpointAlert struct {
EndpointID string
ContainerName string
Target string
Type string // "alert" or "recovery"
NewAlertState endpoint.AlertState
Failures int
Successes int
Threshold int
LastError string
Timestamp time.Time
}
EndpointAlert represents an alert or recovery event for an endpoint.
type EndpointAlertDetector ¶
type EndpointAlertDetector struct{}
EndpointAlertDetector evaluates check results against thresholds.
func NewEndpointAlertDetector ¶
func NewEndpointAlertDetector() *EndpointAlertDetector
NewEndpointAlertDetector creates a new detector.
func (*EndpointAlertDetector) EvaluateCheckResult ¶
func (d *EndpointAlertDetector) EvaluateCheckResult(ep *endpoint.Endpoint, result endpoint.CheckResult) *EndpointAlert
EvaluateCheckResult checks if a threshold has been crossed and returns an alert if so. Returns nil if no alert/recovery needs to be emitted.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine consumes alert events from monitoring services, persists them, evaluates silence rules, dispatches notifications, and broadcasts via SSE.
func (*Engine) AlertCount ¶
AlertCount returns the number of active alerts (for monitoring).
func (*Engine) EventChannel ¶
EventChannel returns the channel for sending alert events to the engine.
func (*Engine) ResolveByEntity ¶
ResolveByEntity resolves all active alerts for a given entity (e.g. when a container is destroyed). This prevents stale alerts from accumulating when containers are recreated with new internal IDs.
func (*Engine) SetBroadcaster ¶
func (e *Engine) SetBroadcaster(b SSEBroadcaster)
SetBroadcaster sets the SSE broadcaster.
func (*Engine) SetEntityRouter ¶ added in v1.0.1
func (e *Engine) SetEntityRouter(r EntityRouter)
SetEntityRouter sets the entity routing extension.
func (*Engine) SetEscalator ¶ added in v1.0.1
SetEscalator sets the escalation extension.
func (*Engine) SetMaintenanceSuppressor ¶ added in v1.0.1
func (e *Engine) SetMaintenanceSuppressor(s MaintenanceSuppressor)
SetMaintenanceSuppressor sets the maintenance suppression extension.
func (*Engine) SetNotifier ¶
SetNotifier sets the webhook notifier for dispatching notifications.
type EngineDeps ¶ added in v1.1.0
type EngineDeps struct {
AlertStore AlertStore // required
ChannelStore ChannelStore // required
TriggerStore TriggerStore // required
SilenceStore SilenceStore // required
Logger *slog.Logger // required
Notifier *Notifier // optional — nil-safe
Broadcaster SSEBroadcaster // optional — nil-safe
}
EngineDeps holds all dependencies for the alert Engine.
type EntityRouter ¶ added in v1.0.1
type EntityRouter interface {
Route(ctx context.Context, entityType string, entityID string, severity string) ([]string, error)
}
EntityRouter provides per-entity alert routing.
type Escalator ¶ added in v1.0.1
type Escalator interface {
EvaluateCycle(ctx context.Context) error
OnAlertCreated(ctx context.Context, a *Alert) error
OnAlertAcknowledged(ctx context.Context, alertID string, ack Acknowledgment) error
OnAlertResolved(ctx context.Context, alertID string, resolvedAt time.Time) error
OnEditionDowngraded(ctx context.Context) error
}
Escalator manages Pro escalation policy execution.
type Event ¶
type Event struct {
Source string // "container", "endpoint", "heartbeat", "certificate", "resource"
AlertType string // source-specific type
Severity string // "critical", "warning", "info"
IsRecover bool // true if this is a recovery event
Message string // human-readable description
EntityType string // "container", "endpoint", "heartbeat", "certificate"
EntityID string // UUID of the referenced entity in its source table
EntityName string // display name
Details map[string]any // source-specific metadata
Timestamp time.Time // when condition was detected
}
Event represents a unified alert event sent via Go channel from any monitoring service.
type HealthAlert ¶
type HealthAlert struct {
ContainerID string
ContainerName string
PreviousHealth *container.HealthStatus
NewHealth container.HealthStatus
Severity container.AlertSeverity
Channels string
Timestamp time.Time
}
HealthAlert represents a health status change alert.
func CheckHealthTransition ¶
func CheckHealthTransition(c *container.Container, previousHealth *container.HealthStatus, newHealth container.HealthStatus) *HealthAlert
CheckHealthTransition returns an alert if a container transitions from healthy to unhealthy.
type ListAlertsOpts ¶
type ListAlertsOpts struct {
Source string
Severity string
Status string
Before *time.Time
Limit int
}
ListAlertsOpts contains filter parameters for listing alerts.
type MaintenanceSuppressor ¶ added in v1.0.1
type MaintenanceSuppressor interface {
IsSuppressed(ctx context.Context, source string, entityType string, entityID string) (bool, error)
}
MaintenanceSuppressor checks if an alert should be suppressed during a maintenance window.
type NotificationChannel ¶
type NotificationChannel struct {
ID string `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
URL string `json:"url"`
Headers string `json:"headers,omitempty"`
// Secret holds the channel credential (a Telegram bot token, today). It is
// tagged "-" and not omitempty: omitempty would still serialize a non-empty
// token, which is exactly the value that must never leave the process.
Secret string `json:"-"`
// Config holds non-secret per-type settings as JSON, e.g. {"thread_id":"42"}.
Config string `json:"config,omitempty"`
// HasSecret is derived at scan time, never stored. It lets the interface show
// "token on file" without receiving the token.
HasSecret bool `json:"has_secret"`
Enabled bool `json:"enabled"`
Health string `json:"health,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
NotificationChannel represents a configured delivery target. Channels are silent by default — they only receive alerts when referenced by an active AlertTrigger or by an EscalationLevel.
type NotificationDelivery ¶
type NotificationDelivery struct {
ID string `json:"id"`
AlertID string `json:"alert_id"`
ChannelID string `json:"channel_id"`
Status string `json:"status"`
Attempts int `json:"attempts"`
LastError string `json:"last_error,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
NotificationDelivery represents a delivery attempt record.
type NotificationJob ¶
type NotificationJob struct {
Delivery *NotificationDelivery
Channel *NotificationChannel
Alert *Alert
// Body, when non-nil, is the pre-rendered request body sent verbatim
// (used by webhook subscriptions, which dispatch the raw event payload
// they already marshalled and signed). When nil, the body is formatted
// from Alert.
Body []byte
}
NotificationJob represents a webhook delivery job.
type Notifier ¶
type Notifier struct {
// contains filtered or unexported fields
}
Notifier dispatches webhook notifications with a bounded worker pool.
func NewNotifier ¶
func NewNotifier(channelStore ChannelStore, logger *slog.Logger, allowPrivate bool) *Notifier
NewNotifier creates a new webhook notifier. Its HTTP client blocks delivery to private/internal IPs (SSRF guard) at dial time unless allowPrivate is set (dev only, via MAINTENANT_ALLOW_PRIVATE_WEBHOOKS).
func (*Notifier) Enqueue ¶
func (n *Notifier) Enqueue(job NotificationJob)
Enqueue adds a notification job to the work queue.
func (*Notifier) SMTPConfigured ¶
SMTPConfigured reports whether SMTP delivery is available.
func (*Notifier) SendNow ¶ added in v1.2.12
SendNow performs a synchronous send to the given channel with internal retries. Unlike Enqueue, it does not write to the notification_deliveries table — the caller (e.g. the escalation Runner) manages its own delivery row state. Returns nil on success, or the last error after maxRetries attempts.
func (*Notifier) SendTestWebhook ¶
SendTestWebhook sends a test notification to verify a channel is reachable.
func (*Notifier) SetSMTPSender ¶
func (n *Notifier) SetSMTPSender(sender *SMTPSender)
SetSMTPSender configures SMTP delivery for email channels.
func (*Notifier) SetTelegramTransport ¶ added in v1.5.0
SetTelegramTransport points Telegram delivery at another host with another client. Tests use it: the guarded client refuses 127.0.0.1, where an httptest server listens.
type RestartAlert ¶
type RestartAlert struct {
ContainerID string
ContainerName string
RestartCount int
Threshold int
Severity container.AlertSeverity
Channels string
Timestamp time.Time
AgentID string
}
RestartAlert represents a restart threshold alert.
type RestartDetector ¶
type RestartDetector struct {
// contains filtered or unexported fields
}
RestartDetector checks for crash-loop restart patterns.
func NewRestartDetector ¶
func NewRestartDetector(store container.ContainerStore, logger *slog.Logger) *RestartDetector
NewRestartDetector creates a new restart detector.
type SMTPConfig ¶
SMTPConfig holds SMTP connection parameters.
type SMTPSender ¶
type SMTPSender struct {
// contains filtered or unexported fields
}
SMTPSender sends email notifications via SMTP with STARTTLS.
func NewSMTPSender ¶
func NewSMTPSender(cfg SMTPConfig) *SMTPSender
NewSMTPSender creates a new SMTPSender with the given configuration.
type SSEBroadcaster ¶
type SSEBroadcaster interface {
Broadcast(eventType string, data interface{})
}
SSEBroadcaster is the interface for broadcasting SSE events.
func NewSSEBroadcasterFunc ¶
func NewSSEBroadcasterFunc(fn func(eventType string, data interface{})) SSEBroadcaster
NewSSEBroadcasterFunc creates an SSEBroadcaster from a function.
type SilenceRule ¶
type SilenceRule struct {
ID string `json:"id"`
EntityType string `json:"entity_type,omitempty"`
EntityID *string `json:"entity_id,omitempty"`
Source string `json:"source,omitempty"`
Reason string `json:"reason,omitempty"`
StartsAt time.Time `json:"starts_at"`
DurationSeconds int `json:"duration_seconds"`
ExpiresAt time.Time `json:"expires_at"`
IsActive bool `json:"is_active"`
CancelledAt *time.Time `json:"cancelled_at,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
SilenceRule represents a time-bounded notification suppression.
type SilenceStore ¶
type SilenceStore interface {
InsertSilenceRule(ctx context.Context, rule *SilenceRule) (string, error)
ListSilenceRules(ctx context.Context, activeOnly bool) ([]*SilenceRule, error)
CancelSilenceRule(ctx context.Context, id string) error
GetActiveSilenceRules(ctx context.Context) ([]*SilenceRule, error)
}
SilenceStore defines the persistence interface for silence rules.
type TelegramConfig ¶ added in v1.5.0
type TelegramConfig struct {
ThreadID string `json:"thread_id,omitempty"`
}
TelegramConfig is the non-secret part of a Telegram channel, serialized into NotificationChannel.Config.
func ParseTelegramConfig ¶ added in v1.5.0
func ParseTelegramConfig(raw string) (TelegramConfig, error)
ParseTelegramConfig reads the channel's config column. An empty column is a channel with no topic, not an error.
type TelegramRateLimitError ¶ added in v1.5.0
TelegramRateLimitError carries the delay Telegram asked for. It is a typed error rather than a string the retry loop would have to read, so honouring the delay cannot break on a wording change (FR-014).
func (*TelegramRateLimitError) Error ¶ added in v1.5.0
func (e *TelegramRateLimitError) Error() string
type TriggerStore ¶ added in v1.2.12
type TriggerStore interface {
InsertTrigger(ctx context.Context, t *AlertTrigger) (string, error)
GetTrigger(ctx context.Context, id string) (*AlertTrigger, error)
ListTriggers(ctx context.Context) ([]*AlertTrigger, error)
ListEnabledTriggers(ctx context.Context) ([]*AlertTrigger, error)
UpdateTrigger(ctx context.Context, t *AlertTrigger) error
DeleteTrigger(ctx context.Context, id string) error
SetChannels(ctx context.Context, triggerID string, channelIDs []string) error
ListChannelsForTrigger(ctx context.Context, triggerID string) ([]string, error)
ListTriggersForChannel(ctx context.Context, channelID string) ([]*AlertTrigger, error)
}
TriggerStore defines the persistence interface for AlertTrigger objects and their M:N relationship with channels.
type WebhookPayload ¶
type WebhookPayload struct {
Event string `json:"event"`
Alert map[string]interface{} `json:"alert"`
Timestamp string `json:"timestamp"`
}
WebhookPayload is the JSON body sent to generic webhook URLs.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package escalation manages user-defined escalation policies (Pro).
|
Package escalation manages user-defined escalation policies (Pro). |