Documentation
¶
Overview ¶
Package eventrecorder provides a structured event recorder for significant Alertmanager events. Events are serialized as JSON and fanned out to one or more configured destinations.
RecordEvent never blocks the caller: events are serialized and placed on a bounded in-memory queue. A background goroutine drains the queue and sends to destinations. If the queue is full, events are dropped and a metric is incremented.
Index ¶
- func EventRecordingEnabled(ctx context.Context) bool
- func WithEventRecording(ctx context.Context) context.Context
- type AlertGroup
- type Config
- type Destination
- type Event
- type EventData
- func NewAlertCreatedEvent(a *alert.Alert) EventData
- func NewAlertGroupedEvent(group AlertGroup, groupedAlert GroupedAlert) EventData
- func NewAlertResolvedEvent(group AlertGroup, groupedAlert GroupedAlert) EventData
- func NewAlertmanagerShutdownEvent() EventData
- func NewAlertmanagerStartupEvent(version, buildContext string) EventData
- func NewInhibitionMutedAlertEvent(rules []InhibitRule, fp model.Fingerprint, labelSet model.LabelSet, ...) EventData
- func NewNotificationEvent(notification Notification) EventData
- func NewSilenceCreatedEvent(silence *silencepb.Silence) EventData
- func NewSilenceMutedAlertEvent(silence *silencepb.Silence, fp model.Fingerprint, labelSet model.LabelSet) EventData
- func NewSilenceUpdatedEvent(silence *silencepb.Silence) EventData
- type FileOutput
- type FileOutputConfig
- type GroupedAlert
- type InhibitRule
- type KafkaOutput
- type KafkaOutputConfig
- type Notification
- type NotificationReason
- type Recorder
- type StdoutOutput
- type StdoutOutputConfig
- type WebhookOutput
- type WebhookOutputConfig
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EventRecordingEnabled ¶
EventRecordingEnabled reports whether event recording has been enabled in the given context via WithEventRecording.
Types ¶
type AlertGroup ¶ added in v0.35.0
type AlertGroup struct {
// contains filtered or unexported fields
}
AlertGroup is an immutable aggregation-group snapshot.
func NewAlertGroup ¶ added in v0.35.0
func NewAlertGroup(groupKey string, groupLabels model.LabelSet, groupID, receiverName string, matchers labels.Matchers, groupUUID string) AlertGroup
NewAlertGroup snapshots aggregation-group metadata.
type Config ¶
type Config struct {
FileOutputs []FileOutputConfig `yaml:"file_outputs,omitempty" json:"file_outputs,omitempty"`
WebhookOutputs []WebhookOutputConfig `yaml:"webhook_outputs,omitempty" json:"webhook_outputs,omitempty"`
KafkaOutputs []KafkaOutputConfig `yaml:"kafka_outputs,omitempty" json:"kafka_outputs,omitempty"`
StdoutOutputs []StdoutOutputConfig `yaml:"stdout_outputs,omitempty" json:"stdout_outputs,omitempty"`
}
Config configures the event recorder feature.
Outputs are grouped by type, one list per destination kind, mirroring the way receivers group their integrations (e.g. webhook_configs, email_configs). Every recorded event is fanned out to every output across all lists.
type Destination ¶
type Destination interface {
// Name returns the type and configured name for this destination,
// suitable for use as a Prometheus label value (e.g. "file:archive"
// or "webhook:primary").
Name() string
// SendEvent encodes and delivers the event. It returns the number
// of payload bytes written (for the bytes-written metric) and any
// delivery error. A serialization failure should be returned
// wrapped in *serializeError so the recorder can attribute it to
// the serialize-errors metric.
SendEvent(event Event) (size int, err error)
io.Closer
}
Destination is a single event destination. Each implementation owns its own serialization: it receives the structured event and is responsible for encoding it with Event.MarshalJSON or Event.MarshalProtobuf and delivering it.
Owning serialization per destination — rather than handing every destination a pre-encoded JSON blob — avoids the footgun of, say, a protobuf-configured Kafka output silently shipping a JSON payload.
type Event ¶ added in v0.35.0
type Event struct {
// contains filtered or unexported fields
}
Event is an immutable event recorder output with envelope metadata. Its protobuf representation is private so destinations cannot be passed arbitrary protobuf messages.
func (Event) MarshalJSON ¶ added in v0.35.0
MarshalJSON serializes the event using its protobuf schema.
func (Event) MarshalProtobuf ¶ added in v0.35.0
MarshalProtobuf serializes the event using its protobuf schema.
type EventData ¶ added in v0.35.0
type EventData struct {
// contains filtered or unexported fields
}
EventData is an immutable event recorder payload without envelope metadata. Its protobuf representation is private so producers cannot construct invalid output events.
func NewAlertCreatedEvent ¶
NewAlertCreatedEvent constructs alert-created event data.
func NewAlertGroupedEvent ¶ added in v0.35.0
func NewAlertGroupedEvent(group AlertGroup, groupedAlert GroupedAlert) EventData
NewAlertGroupedEvent constructs alert-grouped event data.
func NewAlertResolvedEvent ¶ added in v0.35.0
func NewAlertResolvedEvent(group AlertGroup, groupedAlert GroupedAlert) EventData
NewAlertResolvedEvent constructs alert-resolved event data.
func NewAlertmanagerShutdownEvent ¶ added in v0.35.0
func NewAlertmanagerShutdownEvent() EventData
NewAlertmanagerShutdownEvent constructs shutdown event data.
func NewAlertmanagerStartupEvent ¶ added in v0.35.0
NewAlertmanagerStartupEvent constructs startup event data.
func NewInhibitionMutedAlertEvent ¶
func NewInhibitionMutedAlertEvent(rules []InhibitRule, fp model.Fingerprint, labelSet model.LabelSet, inhibitingFPs []model.Fingerprint) EventData
NewInhibitionMutedAlertEvent constructs inhibition-muted-alert event data.
func NewNotificationEvent ¶ added in v0.35.0
func NewNotificationEvent(notification Notification) EventData
NewNotificationEvent constructs notification event data.
func NewSilenceCreatedEvent ¶
NewSilenceCreatedEvent constructs silence-created event data.
func NewSilenceMutedAlertEvent ¶
func NewSilenceMutedAlertEvent(silence *silencepb.Silence, fp model.Fingerprint, labelSet model.LabelSet) EventData
NewSilenceMutedAlertEvent constructs silence-muted-alert event data.
func NewSilenceUpdatedEvent ¶
NewSilenceUpdatedEvent constructs silence-updated event data.
type FileOutput ¶
type FileOutput struct {
// contains filtered or unexported fields
}
FileOutput writes pre-serialized JSON event bytes to a JSONL file. The file is reopened when fsnotify detects a rename or remove (e.g. logrotate).
func NewFileOutput ¶
func NewFileOutput(cfg FileOutputConfig, logger *slog.Logger) (*FileOutput, error)
NewFileOutput creates a new file-based event recorder output. The file is watched with fsnotify so that external log rotation tools (e.g., logrotate) trigger an immediate reopen.
func (*FileOutput) Close ¶
func (fo *FileOutput) Close() error
Close stops the watcher goroutine, waits for it to exit, and closes the file.
func (*FileOutput) Name ¶
func (fo *FileOutput) Name() string
Name returns a stable identifier for this output.
type FileOutputConfig ¶
type FileOutputConfig struct {
// Name identifies this output in metrics and logs.
Name string `yaml:"name" json:"name"`
// Path is the JSONL file to append events to. Created if absent.
Path string `yaml:"path" json:"path"`
}
FileOutputConfig configures a JSONL file event recorder output.
func (*FileOutputConfig) UnmarshalYAML ¶
func (c *FileOutputConfig) UnmarshalYAML(unmarshal func(any) error) error
UnmarshalYAML implements the yaml.Unmarshaler interface, validating the file output configuration.
type GroupedAlert ¶ added in v0.35.0
type GroupedAlert struct {
// contains filtered or unexported fields
}
GroupedAlert is an immutable grouped-alert snapshot.
func NewGroupedAlert ¶ added in v0.35.0
func NewGroupedAlert(a *alert.Alert) GroupedAlert
NewGroupedAlert snapshots an alert and its fingerprint.
func NewGroupedAlertReference ¶ added in v0.35.0
func NewGroupedAlertReference(fingerprint model.Fingerprint) GroupedAlert
NewGroupedAlertReference snapshots a fingerprint-only grouped-alert reference.
type InhibitRule ¶ added in v0.35.0
type InhibitRule struct {
// contains filtered or unexported fields
}
InhibitRule is an immutable inhibition-rule snapshot.
func NewInhibitRule ¶ added in v0.35.0
func NewInhibitRule(name string, sourceMatchers, targetMatchers labels.Matchers, equal map[model.LabelName]struct{}) InhibitRule
NewInhibitRule snapshots an inhibition rule.
type KafkaOutput ¶
type KafkaOutput struct {
// contains filtered or unexported fields
}
KafkaOutput delivers serialized events to a Kafka topic via franz-go. Events are buffered in a bounded local channel and produced by a single dispatcher goroutine; franz-go handles batching, compression, and retries internally.
When the local buffer is full, events are dropped (with a log message and a metric increment) so that a slow or unreachable broker cannot block the upstream event recorder pipeline.
func NewKafkaOutput ¶
func NewKafkaOutput( cfg KafkaOutputConfig, instance string, dropsCounter *prometheus.CounterVec, produceErrors *prometheus.CounterVec, logger *slog.Logger, ) (*KafkaOutput, error)
NewKafkaOutput constructs a KafkaOutput from the supplied configuration. A failure to reach the brokers at startup is logged at warn level but does not fail construction; franz-go retries connections in the background and records are buffered until delivery becomes possible.
func (*KafkaOutput) Close ¶
func (ko *KafkaOutput) Close() error
Close stops the dispatcher, drains any remaining buffered records into franz-go's producer, flushes the producer (up to flushBudget), and then closes the underlying client. Pending records that do not flush before the budget expires are dropped on client close.
Close is safe to call multiple times; subsequent calls are no-ops.
Note: franz-go's Client.Close has documented blocking behaviour around leaving consumer groups, but this client is configured as a producer only (no ConsumeTopics / ConsumePartitions / InstanceID), so the leave-group path is a no-op and Close will not block on it. The only bounded wait here is the explicit Flush above.
func (*KafkaOutput) Name ¶
func (ko *KafkaOutput) Name() string
Name returns the stable identifier for this output.
type KafkaOutputConfig ¶
type KafkaOutputConfig struct {
// Name identifies this output in metrics and logs.
Name string `yaml:"name" json:"name"`
// Brokers is the list of Kafka seed brokers in host:port form.
Brokers []string `yaml:"brokers" json:"brokers"`
// Topic is the Kafka topic to produce events to.
Topic string `yaml:"topic" json:"topic"`
// ClientID is reported to the Kafka brokers (default "alertmanager").
ClientID string `yaml:"client_id,omitempty" json:"client_id,omitempty"`
// Format selects the on-the-wire encoding of each event value:
// "json" (default, JSON via protojson) or "protobuf" (binary proto).
Format kafka.Format `yaml:"format,omitempty" json:"format,omitempty"`
// Acks controls the producer acknowledgement level:
// "none", "leader" (default), or "all".
Acks kafka.Acks `yaml:"acks,omitempty" json:"acks,omitempty"`
// Compression selects the producer compression codec:
// "" (default, no compression), "none", "gzip", "snappy", "lz4", or "zstd".
Compression kafka.Compression `yaml:"compression,omitempty" json:"compression,omitempty"`
// BufferSize is the capacity of the local channel between the event
// recorder dispatcher and the franz-go producer (default 1024).
BufferSize int `yaml:"buffer_size,omitempty" json:"buffer_size,omitempty"`
// TLSConfig configures TLS for the Kafka broker connection. If unset,
// PLAINTEXT is used.
TLSConfig *commoncfg.TLSConfig `yaml:"tls_config,omitempty" json:"tls_config,omitempty"`
}
KafkaOutputConfig configures a Kafka event recorder output.
func (*KafkaOutputConfig) UnmarshalYAML ¶
func (c *KafkaOutputConfig) UnmarshalYAML(unmarshal func(any) error) error
UnmarshalYAML implements the yaml.Unmarshaler interface, validating and normalising the Kafka output configuration. Transport-layer validation (brokers, acks, compression, TLS) is delegated to the shared kafka package so this code and a future Kafka receiver share a single source of truth.
type Notification ¶ added in v0.35.0
type Notification struct {
Alerts []GroupedAlert
FiringAlerts []GroupedAlert
ResolvedAlerts []GroupedAlert
MutedAlerts []GroupedAlert
Group AlertGroup
RepeatInterval time.Duration
Reason NotificationReason
FlushID uint64
Integration string
IntegrationIdx int64
}
Notification contains the snapshots used to construct a notification event.
type NotificationReason ¶ added in v0.35.0
type NotificationReason int
NotificationReason describes why a notification was sent.
const ( NotificationReasonUnspecified NotificationReason = iota NotificationReasonFirstNotification NotificationReasonNewAlertsInGroup NotificationReasonNewResolvedAlerts NotificationReasonAllAlertsResolved NotificationReasonRepeatIntervalElapsed )
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder is a concrete, non-nil-able handle to an event recorder. Because it is a struct (not an interface), passing nil where a Recorder is expected is a compile-time error.
The zero value (Recorder{}) is safe to use and silently discards all events, but prefer NopRecorder() for clarity.
func NewRecorderFromConfig ¶
func NewRecorderFromConfig(cfg Config, instance string, logger *slog.Logger, r prometheus.Registerer) Recorder
NewRecorderFromConfig builds a Recorder from the given configuration. A background goroutine is started to drain the event queue; call Close to stop it.
func NopRecorder ¶
func NopRecorder() Recorder
NopRecorder returns a Recorder that silently discards all events. Use this in tests or when the event recorder is not configured.
func (Recorder) ApplyConfig ¶
ApplyConfig hot-reloads the event recorder configuration. The update is sent to the writeLoop goroutine, which owns the outputs; this method blocks until the writeLoop has acknowledged the update.
func (Recorder) Close ¶
Close signals the background goroutine to drain remaining events and stop. The writeLoop closes all outputs before returning.
func (Recorder) RecordEvent ¶
RecordEvent wraps the event data with metadata and places it on a bounded queue for background serialization and delivery. If the queue is full the event is dropped (never blocks the caller). Recording only occurs when the context has been decorated with WithEventRecording.
The event data is supplied as a builder function rather than a value so that callers on hot read paths do not pay to snapshot alerts and fingerprints for an event that would only be discarded when recording is disabled. The builder is invoked only after the recording gates pass, and exactly once.
The expensive protojson.Marshal call is deferred to the write-loop goroutine so that the caller's hot path only pays for snapshot construction and a channel send.
func (Recorder) SetClusterPeer ¶
SetClusterPeer sets the cluster peer for HA position tracking.
type StdoutOutput ¶ added in v0.34.0
type StdoutOutput struct {
// contains filtered or unexported fields
}
StdoutOutput writes events as newline-delimited JSON to os.Stdout. This is the recommended output for container deployments where stdout is captured by the runtime log driver (Docker, Kubernetes, etc.).
Each event is serialized with protojson and followed by a newline so log collectors receive one self-contained JSON object per line.
func NewStdoutOutput ¶ added in v0.35.0
func NewStdoutOutput(cfg StdoutOutputConfig) (*StdoutOutput, error)
NewStdoutOutput creates a stdout event recorder output.
func (*StdoutOutput) Close ¶ added in v0.34.0
func (s *StdoutOutput) Close() error
Close is a no-op; os.Stdout is owned by the process, not by this output.
func (*StdoutOutput) Name ¶ added in v0.34.0
func (s *StdoutOutput) Name() string
Name returns the stable identifier used in Prometheus metric labels.
func (*StdoutOutput) SendEvent ¶ added in v0.34.0
func (s *StdoutOutput) SendEvent(event Event) (int, error)
SendEvent serializes the event as a JSON line and writes it to stdout. It returns the byte count written (including the trailing newline) and any write error encountered. A serialization failure is wrapped in serializeError so the recorder attributes it to the correct metric.
type StdoutOutputConfig ¶ added in v0.34.0
type StdoutOutputConfig struct {
// Name identifies this output in metrics and logs.
Name string `yaml:"name" json:"name"`
}
StdoutOutputConfig configures a stdout event recorder output.
func (*StdoutOutputConfig) UnmarshalYAML ¶ added in v0.35.0
func (c *StdoutOutputConfig) UnmarshalYAML(unmarshal func(any) error) error
UnmarshalYAML implements the yaml.Unmarshaler interface, validating the stdout output configuration.
type WebhookOutput ¶
type WebhookOutput struct {
// contains filtered or unexported fields
}
WebhookOutput POSTs each event as a JSON body to a configured URL. Events are processed by a bounded worker pool so that a slow or temporarily unavailable webhook does not block the event recorder queue. Events are dropped (with a log message) when the internal queue is full.
func NewWebhookOutput ¶
func NewWebhookOutput(cfg WebhookOutputConfig, dropsCounter *prometheus.CounterVec, logger *slog.Logger) (*WebhookOutput, error)
NewWebhookOutput creates a new webhook-based event recorder output.
func (*WebhookOutput) Close ¶
func (wo *WebhookOutput) Close() error
Close signals all workers to stop, drains remaining events, and waits. If the drain takes longer than 30 seconds, remaining retries are canceled.
func (*WebhookOutput) Name ¶
func (wo *WebhookOutput) Name() string
Name returns a stable identifier for this output.
func (*WebhookOutput) SendEvent ¶
func (wo *WebhookOutput) SendEvent(event Event) (int, error)
SendEvent serializes the event as JSON and queues it for delivery by a worker. It returns the serialized size (for the bytes-written metric). If the internal queue is full the event is dropped and counted via the output-drops metric.
type WebhookOutputConfig ¶
type WebhookOutputConfig struct {
// Name identifies this output in metrics and logs.
Name string `yaml:"name" json:"name"`
// URL is the endpoint to POST each event to.
URL *amcommoncfg.SecretURL `yaml:"url" json:"url"`
// HTTPConfig configures the HTTP client used for webhook delivery.
HTTPConfig *commoncfg.HTTPClientConfig `yaml:"http_config,omitempty" json:"http_config,omitempty"`
// Timeout for webhook HTTP requests (default 10s).
Timeout model.Duration `yaml:"timeout,omitempty" json:"timeout,omitempty"`
// Workers is the number of concurrent delivery goroutines (default 4).
Workers int `yaml:"workers,omitempty" json:"workers,omitempty"`
// MaxRetries is the maximum number of delivery attempts per event
// (default 3).
MaxRetries int `yaml:"max_retries,omitempty" json:"max_retries,omitempty"`
// RetryBackoff is the base backoff between retry attempts (default
// 500ms). Successive attempts use exponential backoff (base *
// 2^attempt).
RetryBackoff model.Duration `yaml:"retry_backoff,omitempty" json:"retry_backoff,omitempty"`
// Batch enables sending events as JSON arrays instead of individual objects.
Batch bool `yaml:"batch,omitempty" json:"batch,omitempty"`
// BatchMaxEvents is the maximum number of events in one request (default 100).
BatchMaxEvents int `yaml:"batch_max_events,omitempty" json:"batch_max_events,omitempty"`
// BatchMaxBytes is the soft maximum encoded request size (default 1 MiB).
BatchMaxBytes int `yaml:"batch_max_bytes,omitempty" json:"batch_max_bytes,omitempty"`
// BatchFlushInterval is the maximum time an incomplete batch waits (default 100ms).
BatchFlushInterval model.Duration `yaml:"batch_flush_interval,omitempty" json:"batch_flush_interval,omitempty"`
}
WebhookOutputConfig configures an HTTP webhook event recorder output.
func (*WebhookOutputConfig) UnmarshalYAML ¶
func (c *WebhookOutputConfig) UnmarshalYAML(unmarshal func(any) error) error
UnmarshalYAML implements the yaml.Unmarshaler interface, validating the webhook output configuration.
Note: SecretURL.UnmarshalYAML delegates to ParseURL, which already enforces a non-empty host and an http(s) scheme. The only way an otherwise-valid config reaches this function with a degenerate URL is via the "<secret>" placeholder shortcut in SecretURL.UnmarshalYAML, which sets URL to an empty url.URL{}. We catch that case here.