eventrecorder

package
v0.35.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package eventrecorder provides a structured event recorder for significant Alertmanager events. Events are serialized as JSON and fanned out to one or more configured destinations.

RecordEvent never blocks the caller: events are serialized and placed on a bounded in-memory queue. A background goroutine drains the queue and sends to destinations. If the queue is full, events are dropped and a metric is incremented.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EventRecordingEnabled

func EventRecordingEnabled(ctx context.Context) bool

EventRecordingEnabled reports whether event recording has been enabled in the given context via WithEventRecording.

func WithEventRecording

func WithEventRecording(ctx context.Context) context.Context

WithEventRecording returns a context that enables event recording. By default, event recording is disabled; callers must opt in by decorating their context with this function.

Types

type AlertGroup added in v0.35.0

type AlertGroup struct {
	// contains filtered or unexported fields
}

AlertGroup is an immutable aggregation-group snapshot.

func NewAlertGroup added in v0.35.0

func NewAlertGroup(groupKey string, groupLabels model.LabelSet, groupID, receiverName string, matchers labels.Matchers, groupUUID string) AlertGroup

NewAlertGroup snapshots aggregation-group metadata.

type Config

type Config struct {
	FileOutputs    []FileOutputConfig    `yaml:"file_outputs,omitempty" json:"file_outputs,omitempty"`
	WebhookOutputs []WebhookOutputConfig `yaml:"webhook_outputs,omitempty" json:"webhook_outputs,omitempty"`
	KafkaOutputs   []KafkaOutputConfig   `yaml:"kafka_outputs,omitempty" json:"kafka_outputs,omitempty"`
	StdoutOutputs  []StdoutOutputConfig  `yaml:"stdout_outputs,omitempty" json:"stdout_outputs,omitempty"`
}

Config configures the event recorder feature.

Outputs are grouped by type, one list per destination kind, mirroring the way receivers group their integrations (e.g. webhook_configs, email_configs). Every recorded event is fanned out to every output across all lists.

func (*Config) UnmarshalYAML added in v0.35.0

func (c *Config) UnmarshalYAML(unmarshal func(any) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface, validating that each output identifier is unique.

type Destination

type Destination interface {
	// Name returns the type and configured name for this destination,
	// suitable for use as a Prometheus label value (e.g. "file:archive"
	// or "webhook:primary").
	Name() string
	// SendEvent encodes and delivers the event.  It returns the number
	// of payload bytes written (for the bytes-written metric) and any
	// delivery error.  A serialization failure should be returned
	// wrapped in *serializeError so the recorder can attribute it to
	// the serialize-errors metric.
	SendEvent(event Event) (size int, err error)
	io.Closer
}

Destination is a single event destination. Each implementation owns its own serialization: it receives the structured event and is responsible for encoding it with Event.MarshalJSON or Event.MarshalProtobuf and delivering it.

Owning serialization per destination — rather than handing every destination a pre-encoded JSON blob — avoids the footgun of, say, a protobuf-configured Kafka output silently shipping a JSON payload.

type Event added in v0.35.0

type Event struct {
	// contains filtered or unexported fields
}

Event is an immutable event recorder output with envelope metadata. Its protobuf representation is private so destinations cannot be passed arbitrary protobuf messages.

func (Event) MarshalJSON added in v0.35.0

func (e Event) MarshalJSON() ([]byte, error)

MarshalJSON serializes the event using its protobuf schema.

func (Event) MarshalProtobuf added in v0.35.0

func (e Event) MarshalProtobuf() ([]byte, error)

MarshalProtobuf serializes the event using its protobuf schema.

type EventData added in v0.35.0

type EventData struct {
	// contains filtered or unexported fields
}

EventData is an immutable event recorder payload without envelope metadata. Its protobuf representation is private so producers cannot construct invalid output events.

func NewAlertCreatedEvent

func NewAlertCreatedEvent(a *alert.Alert) EventData

NewAlertCreatedEvent constructs alert-created event data.

func NewAlertGroupedEvent added in v0.35.0

func NewAlertGroupedEvent(group AlertGroup, groupedAlert GroupedAlert) EventData

NewAlertGroupedEvent constructs alert-grouped event data.

func NewAlertResolvedEvent added in v0.35.0

func NewAlertResolvedEvent(group AlertGroup, groupedAlert GroupedAlert) EventData

NewAlertResolvedEvent constructs alert-resolved event data.

func NewAlertmanagerShutdownEvent added in v0.35.0

func NewAlertmanagerShutdownEvent() EventData

NewAlertmanagerShutdownEvent constructs shutdown event data.

func NewAlertmanagerStartupEvent added in v0.35.0

func NewAlertmanagerStartupEvent(version, buildContext string) EventData

NewAlertmanagerStartupEvent constructs startup event data.

func NewInhibitionMutedAlertEvent

func NewInhibitionMutedAlertEvent(rules []InhibitRule, fp model.Fingerprint, labelSet model.LabelSet, inhibitingFPs []model.Fingerprint) EventData

NewInhibitionMutedAlertEvent constructs inhibition-muted-alert event data.

func NewNotificationEvent added in v0.35.0

func NewNotificationEvent(notification Notification) EventData

NewNotificationEvent constructs notification event data.

func NewSilenceCreatedEvent

func NewSilenceCreatedEvent(silence *silencepb.Silence) EventData

NewSilenceCreatedEvent constructs silence-created event data.

func NewSilenceMutedAlertEvent

func NewSilenceMutedAlertEvent(silence *silencepb.Silence, fp model.Fingerprint, labelSet model.LabelSet) EventData

NewSilenceMutedAlertEvent constructs silence-muted-alert event data.

func NewSilenceUpdatedEvent

func NewSilenceUpdatedEvent(silence *silencepb.Silence) EventData

NewSilenceUpdatedEvent constructs silence-updated event data.

type FileOutput

type FileOutput struct {
	// contains filtered or unexported fields
}

FileOutput writes pre-serialized JSON event bytes to a JSONL file. The file is reopened when fsnotify detects a rename or remove (e.g. logrotate).

func NewFileOutput

func NewFileOutput(cfg FileOutputConfig, logger *slog.Logger) (*FileOutput, error)

NewFileOutput creates a new file-based event recorder output. The file is watched with fsnotify so that external log rotation tools (e.g., logrotate) trigger an immediate reopen.

func (*FileOutput) Close

func (fo *FileOutput) Close() error

Close stops the watcher goroutine, waits for it to exit, and closes the file.

func (*FileOutput) Name

func (fo *FileOutput) Name() string

Name returns a stable identifier for this output.

func (*FileOutput) SendEvent

func (fo *FileOutput) SendEvent(event Event) (int, error)

SendEvent serializes the event as a JSON line and appends it to the file. It returns the number of bytes written (including the trailing newline) for the bytes-written metric.

type FileOutputConfig

type FileOutputConfig struct {
	// Name identifies this output in metrics and logs.
	Name string `yaml:"name" json:"name"`
	// Path is the JSONL file to append events to.  Created if absent.
	Path string `yaml:"path" json:"path"`
}

FileOutputConfig configures a JSONL file event recorder output.

func (*FileOutputConfig) UnmarshalYAML

func (c *FileOutputConfig) UnmarshalYAML(unmarshal func(any) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface, validating the file output configuration.

type GroupedAlert added in v0.35.0

type GroupedAlert struct {
	// contains filtered or unexported fields
}

GroupedAlert is an immutable grouped-alert snapshot.

func NewGroupedAlert added in v0.35.0

func NewGroupedAlert(a *alert.Alert) GroupedAlert

NewGroupedAlert snapshots an alert and its fingerprint.

func NewGroupedAlertReference added in v0.35.0

func NewGroupedAlertReference(fingerprint model.Fingerprint) GroupedAlert

NewGroupedAlertReference snapshots a fingerprint-only grouped-alert reference.

type InhibitRule added in v0.35.0

type InhibitRule struct {
	// contains filtered or unexported fields
}

InhibitRule is an immutable inhibition-rule snapshot.

func NewInhibitRule added in v0.35.0

func NewInhibitRule(name string, sourceMatchers, targetMatchers labels.Matchers, equal map[model.LabelName]struct{}) InhibitRule

NewInhibitRule snapshots an inhibition rule.

type KafkaOutput

type KafkaOutput struct {
	// contains filtered or unexported fields
}

KafkaOutput delivers serialized events to a Kafka topic via franz-go. Events are buffered in a bounded local channel and produced by a single dispatcher goroutine; franz-go handles batching, compression, and retries internally.

When the local buffer is full, events are dropped (with a log message and a metric increment) so that a slow or unreachable broker cannot block the upstream event recorder pipeline.

func NewKafkaOutput

func NewKafkaOutput(
	cfg KafkaOutputConfig,
	instance string,
	dropsCounter *prometheus.CounterVec,
	produceErrors *prometheus.CounterVec,
	logger *slog.Logger,
) (*KafkaOutput, error)

NewKafkaOutput constructs a KafkaOutput from the supplied configuration. A failure to reach the brokers at startup is logged at warn level but does not fail construction; franz-go retries connections in the background and records are buffered until delivery becomes possible.

func (*KafkaOutput) Close

func (ko *KafkaOutput) Close() error

Close stops the dispatcher, drains any remaining buffered records into franz-go's producer, flushes the producer (up to flushBudget), and then closes the underlying client. Pending records that do not flush before the budget expires are dropped on client close.

Close is safe to call multiple times; subsequent calls are no-ops.

Note: franz-go's Client.Close has documented blocking behaviour around leaving consumer groups, but this client is configured as a producer only (no ConsumeTopics / ConsumePartitions / InstanceID), so the leave-group path is a no-op and Close will not block on it. The only bounded wait here is the explicit Flush above.

func (*KafkaOutput) Name

func (ko *KafkaOutput) Name() string

Name returns the stable identifier for this output.

func (*KafkaOutput) SendEvent

func (ko *KafkaOutput) SendEvent(event Event) (int, error)

SendEvent serializes the event in the configured format (JSON or protobuf) and queues it for asynchronous delivery. It returns the serialized size (for the bytes-written metric).

type KafkaOutputConfig

type KafkaOutputConfig struct {
	// Name identifies this output in metrics and logs.
	Name string `yaml:"name" json:"name"`
	// Brokers is the list of Kafka seed brokers in host:port form.
	Brokers []string `yaml:"brokers" json:"brokers"`
	// Topic is the Kafka topic to produce events to.
	Topic string `yaml:"topic" json:"topic"`
	// ClientID is reported to the Kafka brokers (default "alertmanager").
	ClientID string `yaml:"client_id,omitempty" json:"client_id,omitempty"`
	// Format selects the on-the-wire encoding of each event value:
	// "json" (default, JSON via protojson) or "protobuf" (binary proto).
	Format kafka.Format `yaml:"format,omitempty" json:"format,omitempty"`
	// Acks controls the producer acknowledgement level:
	// "none", "leader" (default), or "all".
	Acks kafka.Acks `yaml:"acks,omitempty" json:"acks,omitempty"`
	// Compression selects the producer compression codec:
	// "" (default, no compression), "none", "gzip", "snappy", "lz4", or "zstd".
	Compression kafka.Compression `yaml:"compression,omitempty" json:"compression,omitempty"`
	// BufferSize is the capacity of the local channel between the event
	// recorder dispatcher and the franz-go producer (default 1024).
	BufferSize int `yaml:"buffer_size,omitempty" json:"buffer_size,omitempty"`
	// TLSConfig configures TLS for the Kafka broker connection.  If unset,
	// PLAINTEXT is used.
	TLSConfig *commoncfg.TLSConfig `yaml:"tls_config,omitempty" json:"tls_config,omitempty"`
}

KafkaOutputConfig configures a Kafka event recorder output.

func (*KafkaOutputConfig) UnmarshalYAML

func (c *KafkaOutputConfig) UnmarshalYAML(unmarshal func(any) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface, validating and normalising the Kafka output configuration. Transport-layer validation (brokers, acks, compression, TLS) is delegated to the shared kafka package so this code and a future Kafka receiver share a single source of truth.

type Notification added in v0.35.0

type Notification struct {
	Alerts         []GroupedAlert
	FiringAlerts   []GroupedAlert
	ResolvedAlerts []GroupedAlert
	MutedAlerts    []GroupedAlert
	Group          AlertGroup
	RepeatInterval time.Duration
	Reason         NotificationReason
	FlushID        uint64
	Integration    string
	IntegrationIdx int64
}

Notification contains the snapshots used to construct a notification event.

type NotificationReason added in v0.35.0

type NotificationReason int

NotificationReason describes why a notification was sent.

const (
	NotificationReasonUnspecified NotificationReason = iota
	NotificationReasonFirstNotification
	NotificationReasonNewAlertsInGroup
	NotificationReasonNewResolvedAlerts
	NotificationReasonAllAlertsResolved
	NotificationReasonRepeatIntervalElapsed
)

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder is a concrete, non-nil-able handle to an event recorder. Because it is a struct (not an interface), passing nil where a Recorder is expected is a compile-time error.

The zero value (Recorder{}) is safe to use and silently discards all events, but prefer NopRecorder() for clarity.

func NewRecorderFromConfig

func NewRecorderFromConfig(cfg Config, instance string, logger *slog.Logger, r prometheus.Registerer) Recorder

NewRecorderFromConfig builds a Recorder from the given configuration. A background goroutine is started to drain the event queue; call Close to stop it.

func NopRecorder

func NopRecorder() Recorder

NopRecorder returns a Recorder that silently discards all events. Use this in tests or when the event recorder is not configured.

func (Recorder) ApplyConfig

func (r Recorder) ApplyConfig(cfg Config)

ApplyConfig hot-reloads the event recorder configuration. The update is sent to the writeLoop goroutine, which owns the outputs; this method blocks until the writeLoop has acknowledged the update.

func (Recorder) Close

func (r Recorder) Close() error

Close signals the background goroutine to drain remaining events and stop. The writeLoop closes all outputs before returning.

func (Recorder) RecordEvent

func (r Recorder) RecordEvent(ctx context.Context, build func() EventData)

RecordEvent wraps the event data with metadata and places it on a bounded queue for background serialization and delivery. If the queue is full the event is dropped (never blocks the caller). Recording only occurs when the context has been decorated with WithEventRecording.

The event data is supplied as a builder function rather than a value so that callers on hot read paths do not pay to snapshot alerts and fingerprints for an event that would only be discarded when recording is disabled. The builder is invoked only after the recording gates pass, and exactly once.

The expensive protojson.Marshal call is deferred to the write-loop goroutine so that the caller's hot path only pays for snapshot construction and a channel send.

func (Recorder) SetClusterPeer

func (r Recorder) SetClusterPeer(peer *cluster.Peer)

SetClusterPeer sets the cluster peer for HA position tracking.

type StdoutOutput added in v0.34.0

type StdoutOutput struct {
	// contains filtered or unexported fields
}

StdoutOutput writes events as newline-delimited JSON to os.Stdout. This is the recommended output for container deployments where stdout is captured by the runtime log driver (Docker, Kubernetes, etc.).

Each event is serialized with protojson and followed by a newline so log collectors receive one self-contained JSON object per line.

func NewStdoutOutput added in v0.35.0

func NewStdoutOutput(cfg StdoutOutputConfig) (*StdoutOutput, error)

NewStdoutOutput creates a stdout event recorder output.

func (*StdoutOutput) Close added in v0.34.0

func (s *StdoutOutput) Close() error

Close is a no-op; os.Stdout is owned by the process, not by this output.

func (*StdoutOutput) Name added in v0.34.0

func (s *StdoutOutput) Name() string

Name returns the stable identifier used in Prometheus metric labels.

func (*StdoutOutput) SendEvent added in v0.34.0

func (s *StdoutOutput) SendEvent(event Event) (int, error)

SendEvent serializes the event as a JSON line and writes it to stdout. It returns the byte count written (including the trailing newline) and any write error encountered. A serialization failure is wrapped in serializeError so the recorder attributes it to the correct metric.

type StdoutOutputConfig added in v0.34.0

type StdoutOutputConfig struct {
	// Name identifies this output in metrics and logs.
	Name string `yaml:"name" json:"name"`
}

StdoutOutputConfig configures a stdout event recorder output.

func (*StdoutOutputConfig) UnmarshalYAML added in v0.35.0

func (c *StdoutOutputConfig) UnmarshalYAML(unmarshal func(any) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface, validating the stdout output configuration.

type WebhookOutput

type WebhookOutput struct {
	// contains filtered or unexported fields
}

WebhookOutput POSTs each event as a JSON body to a configured URL. Events are processed by a bounded worker pool so that a slow or temporarily unavailable webhook does not block the event recorder queue. Events are dropped (with a log message) when the internal queue is full.

func NewWebhookOutput

func NewWebhookOutput(cfg WebhookOutputConfig, dropsCounter *prometheus.CounterVec, logger *slog.Logger) (*WebhookOutput, error)

NewWebhookOutput creates a new webhook-based event recorder output.

func (*WebhookOutput) Close

func (wo *WebhookOutput) Close() error

Close signals all workers to stop, drains remaining events, and waits. If the drain takes longer than 30 seconds, remaining retries are canceled.

func (*WebhookOutput) Name

func (wo *WebhookOutput) Name() string

Name returns a stable identifier for this output.

func (*WebhookOutput) SendEvent

func (wo *WebhookOutput) SendEvent(event Event) (int, error)

SendEvent serializes the event as JSON and queues it for delivery by a worker. It returns the serialized size (for the bytes-written metric). If the internal queue is full the event is dropped and counted via the output-drops metric.

type WebhookOutputConfig

type WebhookOutputConfig struct {
	// Name identifies this output in metrics and logs.
	Name string `yaml:"name" json:"name"`
	// URL is the endpoint to POST each event to.
	URL *amcommoncfg.SecretURL `yaml:"url" json:"url"`
	// HTTPConfig configures the HTTP client used for webhook delivery.
	HTTPConfig *commoncfg.HTTPClientConfig `yaml:"http_config,omitempty" json:"http_config,omitempty"`
	// Timeout for webhook HTTP requests (default 10s).
	Timeout model.Duration `yaml:"timeout,omitempty" json:"timeout,omitempty"`
	// Workers is the number of concurrent delivery goroutines (default 4).
	Workers int `yaml:"workers,omitempty" json:"workers,omitempty"`
	// MaxRetries is the maximum number of delivery attempts per event
	// (default 3).
	MaxRetries int `yaml:"max_retries,omitempty" json:"max_retries,omitempty"`
	// RetryBackoff is the base backoff between retry attempts (default
	// 500ms).  Successive attempts use exponential backoff (base *
	// 2^attempt).
	RetryBackoff model.Duration `yaml:"retry_backoff,omitempty" json:"retry_backoff,omitempty"`
	// Batch enables sending events as JSON arrays instead of individual objects.
	Batch bool `yaml:"batch,omitempty" json:"batch,omitempty"`
	// BatchMaxEvents is the maximum number of events in one request (default 100).
	BatchMaxEvents int `yaml:"batch_max_events,omitempty" json:"batch_max_events,omitempty"`
	// BatchMaxBytes is the soft maximum encoded request size (default 1 MiB).
	BatchMaxBytes int `yaml:"batch_max_bytes,omitempty" json:"batch_max_bytes,omitempty"`
	// BatchFlushInterval is the maximum time an incomplete batch waits (default 100ms).
	BatchFlushInterval model.Duration `yaml:"batch_flush_interval,omitempty" json:"batch_flush_interval,omitempty"`
}

WebhookOutputConfig configures an HTTP webhook event recorder output.

func (*WebhookOutputConfig) UnmarshalYAML

func (c *WebhookOutputConfig) UnmarshalYAML(unmarshal func(any) error) error

UnmarshalYAML implements the yaml.Unmarshaler interface, validating the webhook output configuration.

Note: SecretURL.UnmarshalYAML delegates to ParseURL, which already enforces a non-empty host and an http(s) scheme. The only way an otherwise-valid config reaches this function with a degenerate URL is via the "<secret>" placeholder shortcut in SecretURL.UnmarshalYAML, which sets URL to an empty url.URL{}. We catch that case here.

Directories

Path Synopsis
events
v2

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL