insights

package
v0.22.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package insights records and queries a structured, human-oriented history of what the Slack Manager does: alert outcomes, issue lifecycle, user commands, Slack posts and periodic channel snapshots.

Insights complement logs and metrics. Metrics remain the operational view; insights are the queryable per-channel and per-issue view, intended for dashboards.

Recording

Recorder writes types.InsightsEvent values asynchronously to a types.InsightsSink, typically provided by the database plugin. Recording is designed to never affect alert processing: Recorder.Record never blocks, writes happen in batches on a background goroutine, and sink errors and panics are contained. The manager and the API server create and manage their recorder internally; most hosts never use Recorder directly.

Reading

Service answers dashboard questions (summaries, time series, breakdowns, current state, lifecycle statistics, issue timelines) on top of a types.InsightsReader.

Index

Constants

View Source
const (
	DefaultBufferSize       = 10000
	DefaultBatchSize        = 500
	DefaultFlushInterval    = time.Second
	DefaultWriteTimeout     = 5 * time.Second
	DefaultInitialBackoff   = time.Second
	DefaultMaxBackoff       = time.Minute
	DefaultErrorLogInterval = time.Minute
)

Default recorder settings. See the corresponding RecorderOption functions.

View Source
const (
	DefaultCurrentStateCacheTTL = 10 * time.Second
	DefaultStaleAfter           = 24 * time.Hour
	DefaultTopN                 = 10
	DefaultTimelineRange        = 90 * 24 * time.Hour
)

Default service settings. See the corresponding ServiceOption functions.

Variables

This section is empty.

Functions

This section is empty.

Types

type AlertSummary

type AlertSummary struct {
	// Accepted is the number of alerts fully processed by the manager (NewIssue + Grouped + Ignored).
	Accepted int64 `json:"accepted"`
	NewIssue int64 `json:"newIssue"`
	Grouped  int64 `json:"grouped"`
	Ignored  int64 `json:"ignored"`
	Dropped  int64 `json:"dropped"`
	Retried  int64 `json:"retried"`

	// Queued, Filtered and Rejected are only recorded when the API server has an insights sink.
	Queued   int64 `json:"queued"`
	Filtered int64 `json:"filtered"`
	Rejected int64 `json:"rejected"`

	IgnoredByReason  map[string]int64 `json:"ignoredByReason"`
	DroppedByReason  map[string]int64 `json:"droppedByReason"`
	RetriedByReason  map[string]int64 `json:"retriedByReason"`
	FilteredByReason map[string]int64 `json:"filteredByReason"`
	RejectedByReason map[string]int64 `json:"rejectedByReason"`
}

AlertSummary counts alert outcomes. See the types.InsightsEventAlert* constants for precise definitions.

type ChannelState

type ChannelState struct {
	StateCounts

	ChannelID   string `json:"channelId"`
	ChannelName string `json:"channelName"`
}

ChannelState holds the open issue counts of one channel.

type CommandCounts

type CommandCounts struct {
	Executed int64 `json:"executed"`
	Failed   int64 `json:"failed"`
	Skipped  int64 `json:"skipped"`
}

CommandCounts counts the outcomes of one command action.

type Count

type Count struct {
	Key   string `json:"key"`
	Count int64  `json:"count"`
}

Count is a value with its count, returned by breakdowns.

type CurrentState

type CurrentState struct {
	GeneratedAt time.Time `json:"generatedAt"`

	// Totals counts all open issues in the selected channels.
	Totals StateCounts `json:"totals"`

	// Channels holds the counts per active channel (channels with open issues), ordered by alerting issues, then open issues.
	Channels []ChannelState `json:"channels"`

	// Oldest lists the oldest alerting issues, Stale the alerting issues longest without new alerts,
	// and MostAlerts the open issues with the most alerts.
	Oldest     []OpenIssue `json:"oldest"`
	Stale      []OpenIssue `json:"stale"`
	MostAlerts []OpenIssue `json:"mostAlerts"`

	// Issues holds all open issues, if requested with [CurrentStateQuery.IncludeIssues].
	Issues []OpenIssue `json:"issues,omitempty"`
}

CurrentState is a snapshot of all open issues.

type CurrentStateQuery

type CurrentStateQuery struct {
	// ChannelIDs limits the result to these channels. Empty means all channels.
	ChannelIDs []string

	// StaleAfter is the time without new alerts after which an unresolved open issue is considered stale.
	// Zero means [DefaultStaleAfter].
	StaleAfter time.Duration

	// TopN is the length of the Oldest, Stale and MostAlerts lists. Zero means [DefaultTopN].
	TopN int

	// IncludeIssues includes all open issues in [CurrentState.Issues].
	IncludeIssues bool
}

CurrentStateQuery selects the current state of open issues.

type Filter

type Filter struct {
	From       time.Time `json:"from"`
	To         time.Time `json:"to"`
	ChannelIDs []string  `json:"channelIds,omitempty"`
}

Filter selects the time range [From, To) and the channels of a query. Empty ChannelIDs means all channels.

type IssueCount

type IssueCount struct {
	IssueID       string `json:"issueId"`
	ChannelID     string `json:"channelId"`
	CorrelationID string `json:"correlationId"`
	Count         int64  `json:"count"`
}

IssueCount is an issue with a count, e.g. the number of reopens.

type IssueSummary

type IssueSummary struct {
	Created int64 `json:"created"`

	// Updates is the number of alert-driven issue updates; UpdatedIssues the number of distinct issues updated.
	Updates       int64 `json:"updates"`
	UpdatedIssues int64 `json:"updatedIssues"`

	Resolved  int64 `json:"resolved"`
	Reopened  int64 `json:"reopened"`
	Escalated int64 `json:"escalated"`
	Moved     int64 `json:"moved"`
	Archived  int64 `json:"archived"`

	ResolvedByReason map[string]int64 `json:"resolvedByReason"`
	ReopenedByReason map[string]int64 `json:"reopenedByReason"`
	MovedByReason    map[string]int64 `json:"movedByReason"`
	ArchivedByReason map[string]int64 `json:"archivedByReason"`
}

IssueSummary counts issue lifecycle events. See the types.InsightsEventIssue* constants for precise definitions.

type LifecycleStats

type LifecycleStats struct {
	Filter Filter `json:"filter"`

	// Archived is the number of archived issues, and ByResolution how they ended (see [types.InsightsEventIssueArchived]).
	Archived     int64            `json:"archived"`
	ByResolution map[string]int64 `json:"byResolution"`

	// OpenSeconds is the time from creation to archiving.
	OpenSeconds ValueStats `json:"openSeconds"`

	// TimeToResolveSeconds is the time from creation to resolution (issues with follow-up, excluding terminated).
	TimeToResolveSeconds ValueStats `json:"timeToResolveSeconds"`

	// TimeToFirstInvestigationSeconds is the time from creation to the first investigation (investigated issues only).
	TimeToFirstInvestigationSeconds ValueStats `json:"timeToFirstInvestigationSeconds"`

	// AlertsPerIssue is the number of alerts per issue.
	AlertsPerIssue ValueStats `json:"alertsPerIssue"`

	// ReopensPerIssue is the number of reopens per issue.
	ReopensPerIssue ValueStats `json:"reopensPerIssue"`

	// Notified is the number of archived issues for which an alert Slack post was sent. Archived - Notified
	// includes issues whose notification was delayed and never sent.
	Notified int64 `json:"notified"`
}

LifecycleStats describes issues archived in the time range.

type Metric

type Metric struct {
	Type   types.InsightsEventType `json:"type"`
	Reason string                  `json:"reason,omitempty"`
}

Metric identifies a series of events: an event type, optionally restricted to one reason.

func (Metric) String

func (m Metric) String() string

String returns "type" or "type/reason".

type MoveFlow

type MoveFlow struct {
	SourceChannelID string `json:"sourceChannelId"`
	TargetChannelID string `json:"targetChannelId"`
	Reason          string `json:"reason"`
	Count           int64  `json:"count"`
}

MoveFlow counts issue moves from one channel to another.

type OpenIssue

type OpenIssue struct {
	ID            string              `json:"id"`
	ChannelID     string              `json:"channelId"`
	ChannelName   string              `json:"channelName"`
	CorrelationID string              `json:"correlationId"`
	Header        string              `json:"header"`
	Severity      types.AlertSeverity `json:"severity"`
	RouteKey      string              `json:"routeKey,omitempty"`
	Host          string              `json:"host,omitempty"`
	Author        string              `json:"author,omitempty"`
	Created       time.Time           `json:"created"`
	LastAlertAt   time.Time           `json:"lastAlertAt"`
	AlertCount    int                 `json:"alertCount"`
	ReopenCount   int                 `json:"reopenCount"`
	FollowUp      bool                `json:"followUp"`

	// Resolved is true if the issue is resolved (by alert, user or auto-resolve) but not yet archived.
	Resolved   bool      `json:"resolved"`
	ResolvedBy string    `json:"resolvedBy,omitempty"`
	ResolveAt  time.Time `json:"resolveAt,omitzero"`
	ArchiveAt  time.Time `json:"archiveAt,omitzero"`

	Muted             bool      `json:"muted"`
	MutedBy           string    `json:"mutedBy,omitempty"`
	Investigated      bool      `json:"investigated"`
	InvestigatedBy    string    `json:"investigatedBy,omitempty"`
	InvestigatedSince time.Time `json:"investigatedSince,omitzero"`
	Escalated         bool      `json:"escalated"`
	Moved             bool      `json:"moved"`
	Terminated        bool      `json:"terminated"`

	// SlackPostID is the timestamp of the current Slack post, empty if none (e.g. a delayed notification).
	SlackPostID string `json:"slackPostId,omitempty"`
}

OpenIssue describes an open (non-archived) issue.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder records types.InsightsEvent values asynchronously to a types.InsightsSink.

The Recorder is designed to never affect the code that records events:

  • Recorder.Record never blocks. Events are put in a bounded buffer, and dropped (and counted) when it is full.
  • Events are written in batches by a single background goroutine, never by the caller.
  • Sink errors and panics are recovered, counted and logged (rate-limited). They never propagate.
  • After a failed write, the recorder pauses with exponential backoff, to avoid adding load to a struggling database.
  • Recorder.Shutdown flushes remaining events with a hard deadline.

A nil *Recorder is valid and records nothing, so callers can use it without checks. Use Recorder.Enabled to skip building events entirely when recording is disabled.

Lost events are reported in the event stream itself, as an types.InsightsEventGap event written after the next successful write.

func NewRecorder

func NewRecorder(sink types.InsightsSink, logger types.Logger, metrics types.Metrics, opts ...RecorderOption) *Recorder

NewRecorder creates a Recorder writing to sink. It returns nil (a valid, disabled recorder) if sink is nil. Call Recorder.Start to start writing, and Recorder.Shutdown to stop.

func (*Recorder) Enabled

func (r *Recorder) Enabled() bool

Enabled returns true if the recorder records events (i.e. it is not nil).

func (*Recorder) Record

func (r *Recorder) Record(e types.InsightsEvent)

Record queues an event for writing. It never blocks: if the buffer is full, the event is dropped and counted. If e.Time is zero, it is set to the current time. Record is safe for concurrent use, and may be called before Recorder.Start and after Recorder.Shutdown (events are then buffered or dropped).

func (*Recorder) Shutdown

func (r *Recorder) Shutdown(timeout time.Duration)

Shutdown stops the recorder. It writes the events remaining in the buffer until the buffer is empty or the timeout has passed, and then returns. It never blocks longer than the timeout (plus scheduling overhead): an in-flight sink write is cancelled at the deadline. A zero or negative timeout skips flushing. Events not written by the deadline are dropped and counted. It is safe to call multiple times.

func (*Recorder) Start

func (r *Recorder) Start()

Start starts the background writer goroutine. It is safe to call multiple times.

type RecorderOption

type RecorderOption func(*recorderOptions)

RecorderOption configures a Recorder.

func WithBackoff

func WithBackoff(initial, maxBackoff time.Duration) RecorderOption

WithBackoff sets the initial and maximum pause after a failed sink write. The pause doubles after each consecutive failure. While paused, events accumulate in the buffer, and are dropped when it is full. Values below 1ms are ignored. Defaults DefaultInitialBackoff and DefaultMaxBackoff.

func WithBatchSize

func WithBatchSize(n int) RecorderOption

WithBatchSize sets the maximum number of events per sink write. Values below 1 are ignored. Default DefaultBatchSize.

func WithBufferSize

func WithBufferSize(n int) RecorderOption

WithBufferSize sets the capacity of the in-memory event buffer. When the buffer is full, new events are dropped. Values below 1 are ignored. Default DefaultBufferSize.

func WithErrorLogInterval

func WithErrorLogInterval(d time.Duration) RecorderOption

WithErrorLogInterval sets the minimum interval between error log entries. Errors in between are counted and summarized in the next entry. Values below 1ms are ignored. Default DefaultErrorLogInterval.

func WithFlushInterval

func WithFlushInterval(d time.Duration) RecorderOption

WithFlushInterval sets the maximum time an event waits in a partial batch before it is written. Values below 1ms are ignored. Default DefaultFlushInterval.

func WithWriteTimeout

func WithWriteTimeout(d time.Duration) RecorderOption

WithWriteTimeout sets the deadline of each sink write. Values below 1ms are ignored. Default DefaultWriteTimeout.

type Series

type Series struct {
	Bucket time.Duration `json:"bucket"`

	// Buckets holds the start time of each bucket, in ascending order.
	Buckets []time.Time `json:"buckets"`

	// Values holds one value per bucket for each series. Keys are the metric (see [Metric.String]), or
	// "<metric>@<channel ID>" for per-channel series.
	Values map[string][]float64 `json:"values"`
}

Series holds zero-filled time series sharing the same buckets.

type SeriesQuery

type SeriesQuery struct {
	Filter

	// Bucket is the time bucket size. Buckets are aligned to the Unix epoch (UTC).
	Bucket time.Duration

	// Metrics lists the series to return. Required.
	Metrics []Metric

	// ByChannel splits each metric into one series per channel.
	ByChannel bool
}

SeriesQuery selects time series of event counts.

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service answers dashboard-style questions about the Slack Manager, on top of a types.InsightsReader.

Activity methods (summaries, series, breakdowns, lifecycle statistics, timelines) require an insights reader, and return types.ErrInsightsNotSupported without one. Service.CurrentState also works with any types.DB set with WithDB. A Service is safe for concurrent use.

func New

func New(reader types.InsightsReader, opts ...ServiceOption) *Service

New creates a Service on top of reader. The reader may be nil, in which case only Service.CurrentState works, and only if a database is set with WithDB.

func NewFromDB

func NewFromDB(db types.DB, opts ...ServiceOption) *Service

NewFromDB creates a Service for a database. If db implements types.InsightsReaderProvider and returns a non-nil reader, all methods are supported; otherwise only Service.CurrentState is.

func (*Service) Breakdown

func (s *Service) Breakdown(ctx context.Context, f Filter, metric Metric, dimension types.InsightsDimension, limit int) ([]Count, error)

Breakdown returns the top values of a dimension for events of a metric, highest count first. For example, the noisiest route keys for new issues, or the most common reasons for dropped alerts. A limit of zero means DefaultTopN.

func (*Service) CurrentState

func (s *Service) CurrentState(ctx context.Context, q CurrentStateQuery) (*CurrentState, error)

CurrentState returns a snapshot of the open issues. It works with an insights reader, or with any database set with WithDB. Results are cached for the configured cache TTL.

func (*Service) Events

Events returns a page of events matching the query.

func (*Service) Flapping

func (s *Service) Flapping(ctx context.Context, f Filter, minReopens int64, limit int) ([]IssueCount, error)

Flapping returns the issues reopened at least minReopens times in the time range, most reopens first. A minReopens below 1 is treated as 1, and a limit of zero means DefaultTopN.

func (*Service) IssueTimeline

func (s *Service) IssueTimeline(ctx context.Context, q TimelineQuery) (*types.InsightsEventPage, error)

IssueTimeline returns the events of one issue, oldest first, for a drill-down view.

func (*Service) Lifecycle

func (s *Service) Lifecycle(ctx context.Context, f Filter) (*LifecycleStats, error)

Lifecycle returns statistics about issues archived in the time range.

func (*Service) MoveFlows

func (s *Service) MoveFlows(ctx context.Context, f Filter) ([]MoveFlow, error)

MoveFlows returns issue moves between channels, most frequent first. If the filter has channels, moves both from and to those channels are included.

func (*Service) OpenIssuesSeries

func (s *Service) OpenIssuesSeries(ctx context.Context, f Filter, bucket time.Duration) (*Series, error)

OpenIssuesSeries returns the number of open issues over time, per channel, from channel snapshots. Series keys are channel IDs. The value of a bucket is the highest snapshot in it; buckets without a snapshot are zero, so choose a bucket at least as large as the channel stats interval (default 5 minutes). If the number of buckets times the number of channels exceeds types.MaxInsightsRows, it fails with types.ErrInsightsTooManyRows; use a shorter time range, a larger bucket or fewer channels.

func (*Service) Series

func (s *Service) Series(ctx context.Context, q SeriesQuery) (*Series, error)

Series returns time series of event counts.

The underlying query returns one row per bucket, event type, reason (only if a metric has a reason) and channel (only with ByChannel). If that exceeds types.MaxInsightsRows, it fails with types.ErrInsightsTooManyRows; use a shorter time range, a larger bucket or fewer channels.

func (*Service) Summary

func (s *Service) Summary(ctx context.Context, f Filter) (*Summary, error)

Summary returns all activity counts for the filter.

func (*Service) Supported

func (s *Service) Supported() bool

Supported returns true if activity queries are supported, i.e. the service has an insights reader.

type ServiceOption

type ServiceOption func(*Service)

ServiceOption configures a Service.

func WithClock

func WithClock(now func() time.Time) ServiceOption

WithClock sets the clock used to evaluate time-dependent issue state and default time ranges. Default time.Now.

func WithCurrentStateCacheTTL

func WithCurrentStateCacheTTL(d time.Duration) ServiceOption

WithCurrentStateCacheTTL sets how long Service.CurrentState results are cached, to protect the database from frequent dashboard refreshes. Zero disables caching. Default DefaultCurrentStateCacheTTL.

func WithDB

func WithDB(db types.DB) ServiceOption

WithDB sets a database used for current state queries when no types.InsightsReader is available (any types.DB implementation works, via FindActiveChannels and LoadOpenIssuesInChannel).

type SlackPostSummary

type SlackPostSummary struct {
	Created        int64            `json:"created"`
	Updated        int64            `json:"updated"`
	Deleted        int64            `json:"deleted"`
	Throttled      int64            `json:"throttled"`
	Failed         int64            `json:"failed"`
	FailedByReason map[string]int64 `json:"failedByReason"`
}

SlackPostSummary counts Slack post operations.

type StateCounts

type StateCounts struct {
	// Open is the number of open (non-archived) issues.
	Open int `json:"open"`

	// Alerting is the number of open issues that are neither resolved nor info, and BySeverity splits them by severity.
	Alerting   int                         `json:"alerting"`
	BySeverity map[types.AlertSeverity]int `json:"bySeverity"`

	Resolved     int `json:"resolved"`
	Muted        int `json:"muted"`
	Investigated int `json:"investigated"`
	Escalated    int `json:"escalated"`
	Moved        int `json:"moved"`

	// Stale is the number of alerting issues without new alerts for longer than StaleAfter.
	Stale int `json:"stale"`
}

StateCounts counts open issues by state.

type Summary

type Summary struct {
	Filter     Filter                   `json:"filter"`
	Alerts     AlertSummary             `json:"alerts"`
	Issues     IssueSummary             `json:"issues"`
	Commands   map[string]CommandCounts `json:"commands"`
	SlackPosts SlackPostSummary         `json:"slackPosts"`
	Webhooks   WebhookSummary           `json:"webhooks"`

	// LostEvents is the number of insights events lost in the time range (from gap events, all channels).
	// A non-zero value means the counts may be incomplete.
	LostEvents int64 `json:"lostEvents"`
}

Summary holds all activity counts for a time range and set of channels.

type TimelineQuery

type TimelineQuery struct {
	// IssueID is the issue to return events for. Required.
	IssueID string

	// From and To limit the time range. Defaults: the last [DefaultTimelineRange] up to now.
	From time.Time
	To   time.Time

	// Cursor continues a previous page. Limit is the page size (zero means [types.DefaultInsightsPageSize]).
	Cursor string
	Limit  int
}

TimelineQuery selects the events of one issue.

type ValueStats

type ValueStats struct {
	Count int64   `json:"count"`
	Avg   float64 `json:"avg"`
	P50   float64 `json:"p50"`
	P90   float64 `json:"p90"`
	P99   float64 `json:"p99"`
	Max   float64 `json:"max"`
}

ValueStats holds aggregates of a numeric value over a set of issues.

type WebhookSummary

type WebhookSummary struct {
	Succeeded int64 `json:"succeeded"`
	Failed    int64 `json:"failed"`
}

WebhookSummary counts webhook deliveries.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL