Documentation
¶
Overview ¶
Package insights records and queries a structured, human-oriented history of what the Slack Manager does: alert outcomes, issue lifecycle, user commands, Slack posts and periodic channel snapshots.
Insights complement logs and metrics. Metrics remain the operational view; insights are the queryable per-channel and per-issue view, intended for dashboards.
Recording ¶
Recorder writes types.InsightsEvent values asynchronously to a types.InsightsSink, typically provided by the database plugin. Recording is designed to never affect alert processing: Recorder.Record never blocks, writes happen in batches on a background goroutine, and sink errors and panics are contained. The manager and the API server create and manage their recorder internally; most hosts never use Recorder directly.
Reading ¶
Service answers dashboard questions (summaries, time series, breakdowns, current state, lifecycle statistics, issue timelines) on top of a types.InsightsReader.
Index ¶
- Constants
- type AlertSummary
- type ChannelState
- type CommandCounts
- type Count
- type CurrentState
- type CurrentStateQuery
- type Filter
- type IssueCount
- type IssueSummary
- type LifecycleStats
- type Metric
- type MoveFlow
- type OpenIssue
- type Recorder
- type RecorderOption
- func WithBackoff(initial, maxBackoff time.Duration) RecorderOption
- func WithBatchSize(n int) RecorderOption
- func WithBufferSize(n int) RecorderOption
- func WithErrorLogInterval(d time.Duration) RecorderOption
- func WithFlushInterval(d time.Duration) RecorderOption
- func WithWriteTimeout(d time.Duration) RecorderOption
- type Series
- type SeriesQuery
- type Service
- func (s *Service) Breakdown(ctx context.Context, f Filter, metric Metric, ...) ([]Count, error)
- func (s *Service) CurrentState(ctx context.Context, q CurrentStateQuery) (*CurrentState, error)
- func (s *Service) Events(ctx context.Context, q *types.InsightsListQuery) (*types.InsightsEventPage, error)
- func (s *Service) Flapping(ctx context.Context, f Filter, minReopens int64, limit int) ([]IssueCount, error)
- func (s *Service) IssueTimeline(ctx context.Context, q TimelineQuery) (*types.InsightsEventPage, error)
- func (s *Service) Lifecycle(ctx context.Context, f Filter) (*LifecycleStats, error)
- func (s *Service) MoveFlows(ctx context.Context, f Filter) ([]MoveFlow, error)
- func (s *Service) OpenIssuesSeries(ctx context.Context, f Filter, bucket time.Duration) (*Series, error)
- func (s *Service) Series(ctx context.Context, q SeriesQuery) (*Series, error)
- func (s *Service) Summary(ctx context.Context, f Filter) (*Summary, error)
- func (s *Service) Supported() bool
- type ServiceOption
- type SlackPostSummary
- type StateCounts
- type Summary
- type TimelineQuery
- type ValueStats
- type WebhookSummary
Constants ¶
const ( DefaultBufferSize = 10000 DefaultBatchSize = 500 DefaultFlushInterval = time.Second DefaultWriteTimeout = 5 * time.Second DefaultInitialBackoff = time.Second DefaultMaxBackoff = time.Minute DefaultErrorLogInterval = time.Minute )
Default recorder settings. See the corresponding RecorderOption functions.
const ( DefaultCurrentStateCacheTTL = 10 * time.Second DefaultStaleAfter = 24 * time.Hour DefaultTopN = 10 DefaultTimelineRange = 90 * 24 * time.Hour )
Default service settings. See the corresponding ServiceOption functions.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AlertSummary ¶
type AlertSummary struct {
// Accepted is the number of alerts fully processed by the manager (NewIssue + Grouped + Ignored).
Accepted int64 `json:"accepted"`
NewIssue int64 `json:"newIssue"`
Grouped int64 `json:"grouped"`
Ignored int64 `json:"ignored"`
Dropped int64 `json:"dropped"`
Retried int64 `json:"retried"`
// Queued, Filtered and Rejected are only recorded when the API server has an insights sink.
Queued int64 `json:"queued"`
Filtered int64 `json:"filtered"`
Rejected int64 `json:"rejected"`
IgnoredByReason map[string]int64 `json:"ignoredByReason"`
DroppedByReason map[string]int64 `json:"droppedByReason"`
RetriedByReason map[string]int64 `json:"retriedByReason"`
FilteredByReason map[string]int64 `json:"filteredByReason"`
RejectedByReason map[string]int64 `json:"rejectedByReason"`
}
AlertSummary counts alert outcomes. See the types.InsightsEventAlert* constants for precise definitions.
type ChannelState ¶
type ChannelState struct {
StateCounts
ChannelID string `json:"channelId"`
ChannelName string `json:"channelName"`
}
ChannelState holds the open issue counts of one channel.
type CommandCounts ¶
type CommandCounts struct {
Executed int64 `json:"executed"`
Failed int64 `json:"failed"`
Skipped int64 `json:"skipped"`
}
CommandCounts counts the outcomes of one command action.
type CurrentState ¶
type CurrentState struct {
GeneratedAt time.Time `json:"generatedAt"`
// Totals counts all open issues in the selected channels.
Totals StateCounts `json:"totals"`
// Channels holds the counts per active channel (channels with open issues), ordered by alerting issues, then open issues.
Channels []ChannelState `json:"channels"`
// Oldest lists the oldest alerting issues, Stale the alerting issues longest without new alerts,
// and MostAlerts the open issues with the most alerts.
Oldest []OpenIssue `json:"oldest"`
Stale []OpenIssue `json:"stale"`
MostAlerts []OpenIssue `json:"mostAlerts"`
// Issues holds all open issues, if requested with [CurrentStateQuery.IncludeIssues].
Issues []OpenIssue `json:"issues,omitempty"`
}
CurrentState is a snapshot of all open issues.
type CurrentStateQuery ¶
type CurrentStateQuery struct {
// ChannelIDs limits the result to these channels. Empty means all channels.
ChannelIDs []string
// Clients limits the result to open issues whose effective client ([OpenIssue.Client]) is one of these values
// (exact match). Empty means all clients. Channels without matching issues are left out of Channels.
Clients []string
// StaleAfter is the time without new alerts after which an unresolved open issue is considered stale.
// Zero means [DefaultStaleAfter].
StaleAfter time.Duration
// TopN is the length of the Oldest, Stale and MostAlerts lists. Zero means [DefaultTopN].
TopN int
// IncludeIssues includes all open issues in [CurrentState.Issues].
IncludeIssues bool
}
CurrentStateQuery selects the current state of open issues.
type Filter ¶
type Filter struct {
From time.Time `json:"from"`
To time.Time `json:"to"`
ChannelIDs []string `json:"channelIds,omitempty"`
Clients []string `json:"clients,omitempty"`
}
Filter selects the time range [From, To), the channels and the clients of a query. Empty ChannelIDs means all channels.
Clients selects events by their effective client (see types.Alert.EffectiveClient). Empty means all clients. Events that do not relate to an alert or issue (e.g. commands without an issue) have no client, so they never match a non-empty Clients filter.
type IssueCount ¶
type IssueCount struct {
IssueID string `json:"issueId"`
ChannelID string `json:"channelId"`
CorrelationID string `json:"correlationId"`
Count int64 `json:"count"`
}
IssueCount is an issue with a count, e.g. the number of reopens.
type IssueSummary ¶
type IssueSummary struct {
Created int64 `json:"created"`
// Updates is the number of alert-driven issue updates; UpdatedIssues the number of distinct issues updated.
Updates int64 `json:"updates"`
UpdatedIssues int64 `json:"updatedIssues"`
Resolved int64 `json:"resolved"`
Reopened int64 `json:"reopened"`
Escalated int64 `json:"escalated"`
Moved int64 `json:"moved"`
Archived int64 `json:"archived"`
ResolvedByReason map[string]int64 `json:"resolvedByReason"`
ReopenedByReason map[string]int64 `json:"reopenedByReason"`
MovedByReason map[string]int64 `json:"movedByReason"`
ArchivedByReason map[string]int64 `json:"archivedByReason"`
}
IssueSummary counts issue lifecycle events. See the types.InsightsEventIssue* constants for precise definitions.
type LifecycleStats ¶
type LifecycleStats struct {
Filter Filter `json:"filter"`
// Archived is the number of archived issues, and ByResolution how they ended (see [types.InsightsEventIssueArchived]).
Archived int64 `json:"archived"`
ByResolution map[string]int64 `json:"byResolution"`
// OpenSeconds is the time from creation to archiving.
OpenSeconds ValueStats `json:"openSeconds"`
// TimeToResolveSeconds is the time from creation to resolution (issues with follow-up, excluding terminated).
TimeToResolveSeconds ValueStats `json:"timeToResolveSeconds"`
// TimeToFirstInvestigationSeconds is the time from creation to the first investigation (investigated issues only).
TimeToFirstInvestigationSeconds ValueStats `json:"timeToFirstInvestigationSeconds"`
// AlertsPerIssue is the number of alerts per issue.
AlertsPerIssue ValueStats `json:"alertsPerIssue"`
// ReopensPerIssue is the number of reopens per issue.
ReopensPerIssue ValueStats `json:"reopensPerIssue"`
// Notified is the number of archived issues for which an alert Slack post was sent. Archived - Notified
// includes issues whose notification was delayed and never sent.
Notified int64 `json:"notified"`
}
LifecycleStats describes issues archived in the time range.
type Metric ¶
type Metric struct {
Type types.InsightsEventType `json:"type"`
Reason string `json:"reason,omitempty"`
}
Metric identifies a series of events: an event type, optionally restricted to one reason.
type MoveFlow ¶
type MoveFlow struct {
SourceChannelID string `json:"sourceChannelId"`
TargetChannelID string `json:"targetChannelId"`
Reason string `json:"reason"`
Count int64 `json:"count"`
}
MoveFlow counts issue moves from one channel to another.
type OpenIssue ¶
type OpenIssue struct {
ID string `json:"id"`
ChannelID string `json:"channelId"`
ChannelName string `json:"channelName"`
CorrelationID string `json:"correlationId"`
Header string `json:"header"`
Severity types.AlertSeverity `json:"severity"`
RouteKey string `json:"routeKey,omitempty"`
Host string `json:"host,omitempty"`
Author string `json:"author,omitempty"`
// Client is the effective client of the last alert (see [types.Alert.EffectiveClient]): the alert client,
// else the username, else [types.UnknownClient]. It is never empty.
Client string `json:"client"`
// Username is the Slack username of the last alert, i.e. the bot persona the issue is posted as.
Username string `json:"username,omitempty"`
Created time.Time `json:"created"`
LastAlertAt time.Time `json:"lastAlertAt"`
AlertCount int `json:"alertCount"`
ReopenCount int `json:"reopenCount"`
FollowUp bool `json:"followUp"`
// Resolved is true if the issue is resolved (by alert, user or auto-resolve) but not yet archived.
Resolved bool `json:"resolved"`
ResolvedBy string `json:"resolvedBy,omitempty"`
ResolveAt time.Time `json:"resolveAt,omitzero"`
ArchiveAt time.Time `json:"archiveAt,omitzero"`
Muted bool `json:"muted"`
MutedBy string `json:"mutedBy,omitempty"`
Investigated bool `json:"investigated"`
InvestigatedBy string `json:"investigatedBy,omitempty"`
InvestigatedSince time.Time `json:"investigatedSince,omitzero"`
Escalated bool `json:"escalated"`
Moved bool `json:"moved"`
Terminated bool `json:"terminated"`
// SlackPostID is the timestamp of the current Slack post, empty if none (e.g. a delayed notification).
SlackPostID string `json:"slackPostId,omitempty"`
}
OpenIssue describes an open (non-archived) issue.
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder records types.InsightsEvent values asynchronously to a types.InsightsSink.
The Recorder is designed to never affect the code that records events:
- Recorder.Record never blocks. Events are put in a bounded buffer, and dropped (and counted) when it is full.
- Events are written in batches by a single background goroutine, never by the caller.
- Sink errors and panics are recovered, counted and logged (rate-limited). They never propagate.
- After a failed write, the recorder pauses with exponential backoff, to avoid adding load to a struggling database.
- Recorder.Shutdown flushes remaining events with a hard deadline.
A nil *Recorder is valid and records nothing, so callers can use it without checks. Use Recorder.Enabled to skip building events entirely when recording is disabled.
Lost events are reported in the event stream itself, as an types.InsightsEventGap event written after the next successful write.
func NewRecorder ¶
func NewRecorder(sink types.InsightsSink, logger types.Logger, metrics types.Metrics, opts ...RecorderOption) *Recorder
NewRecorder creates a Recorder writing to sink. It returns nil (a valid, disabled recorder) if sink is nil. Call Recorder.Start to start writing, and Recorder.Shutdown to stop.
func (*Recorder) Enabled ¶
Enabled returns true if the recorder records events (i.e. it is not nil).
func (*Recorder) Record ¶
func (r *Recorder) Record(e types.InsightsEvent)
Record queues an event for writing. It never blocks: if the buffer is full, the event is dropped and counted. If e.Time is zero, it is set to the current time. Record is safe for concurrent use, and may be called before Recorder.Start and after Recorder.Shutdown (events are then buffered or dropped).
func (*Recorder) Shutdown ¶
Shutdown stops the recorder. It writes the events remaining in the buffer until the buffer is empty or the timeout has passed, and then returns. It never blocks longer than the timeout (plus scheduling overhead): an in-flight sink write is cancelled at the deadline. A zero or negative timeout skips flushing. Events not written by the deadline are dropped and counted. It is safe to call multiple times.
type RecorderOption ¶
type RecorderOption func(*recorderOptions)
RecorderOption configures a Recorder.
func WithBackoff ¶
func WithBackoff(initial, maxBackoff time.Duration) RecorderOption
WithBackoff sets the initial and maximum pause after a failed sink write. The pause doubles after each consecutive failure. While paused, events accumulate in the buffer, and are dropped when it is full. Values below 1ms are ignored. Defaults DefaultInitialBackoff and DefaultMaxBackoff.
func WithBatchSize ¶
func WithBatchSize(n int) RecorderOption
WithBatchSize sets the maximum number of events per sink write. Values below 1 are ignored. Default DefaultBatchSize.
func WithBufferSize ¶
func WithBufferSize(n int) RecorderOption
WithBufferSize sets the capacity of the in-memory event buffer. When the buffer is full, new events are dropped. Values below 1 are ignored. Default DefaultBufferSize.
func WithErrorLogInterval ¶
func WithErrorLogInterval(d time.Duration) RecorderOption
WithErrorLogInterval sets the minimum interval between error log entries. Errors in between are counted and summarized in the next entry. Values below 1ms are ignored. Default DefaultErrorLogInterval.
func WithFlushInterval ¶
func WithFlushInterval(d time.Duration) RecorderOption
WithFlushInterval sets the maximum time an event waits in a partial batch before it is written. Values below 1ms are ignored. Default DefaultFlushInterval.
func WithWriteTimeout ¶
func WithWriteTimeout(d time.Duration) RecorderOption
WithWriteTimeout sets the deadline of each sink write. Values below 1ms are ignored. Default DefaultWriteTimeout.
type Series ¶
type Series struct {
Bucket time.Duration `json:"bucket"`
// Buckets holds the start time of each bucket, in ascending order.
Buckets []time.Time `json:"buckets"`
// Values holds one value per bucket for each series. Keys are the metric (see [Metric.String]), or
// "<metric>@<channel ID>" for per-channel series.
Values map[string][]float64 `json:"values"`
}
Series holds zero-filled time series sharing the same buckets.
type SeriesQuery ¶
type SeriesQuery struct {
Filter
// Bucket is the time bucket size. Buckets are aligned to the Unix epoch (UTC).
Bucket time.Duration
// Metrics lists the series to return. Required.
Metrics []Metric
// ByChannel splits each metric into one series per channel.
ByChannel bool
}
SeriesQuery selects time series of event counts.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service answers dashboard-style questions about the Slack Manager, on top of a types.InsightsReader.
Activity methods (summaries, series, breakdowns, lifecycle statistics, timelines) require an insights reader, and return types.ErrInsightsNotSupported without one. Service.CurrentState also works with any types.DB set with WithDB. A Service is safe for concurrent use.
func New ¶
func New(reader types.InsightsReader, opts ...ServiceOption) *Service
New creates a Service on top of reader. The reader may be nil, in which case only Service.CurrentState works, and only if a database is set with WithDB.
func NewFromDB ¶
func NewFromDB(db types.DB, opts ...ServiceOption) *Service
NewFromDB creates a Service for a database. If db implements types.InsightsReaderProvider and returns a non-nil reader, all methods are supported; otherwise only Service.CurrentState is.
func (*Service) Breakdown ¶
func (s *Service) Breakdown(ctx context.Context, f Filter, metric Metric, dimension types.InsightsDimension, limit int) ([]Count, error)
Breakdown returns the top values of a dimension for events of a metric, highest count first. For example, the noisiest route keys for new issues, or the most common reasons for dropped alerts. A limit of zero means DefaultTopN.
func (*Service) CurrentState ¶
func (s *Service) CurrentState(ctx context.Context, q CurrentStateQuery) (*CurrentState, error)
CurrentState returns a snapshot of the open issues. It works with an insights reader, or with any database set with WithDB. Results are cached for the configured cache TTL.
func (*Service) Events ¶
func (s *Service) Events(ctx context.Context, q *types.InsightsListQuery) (*types.InsightsEventPage, error)
Events returns a page of events matching the query.
func (*Service) Flapping ¶
func (s *Service) Flapping(ctx context.Context, f Filter, minReopens int64, limit int) ([]IssueCount, error)
Flapping returns the issues reopened at least minReopens times in the time range, most reopens first. A minReopens below 1 is treated as 1, and a limit of zero means DefaultTopN.
func (*Service) IssueTimeline ¶
func (s *Service) IssueTimeline(ctx context.Context, q TimelineQuery) (*types.InsightsEventPage, error)
IssueTimeline returns the events of one issue, oldest first, for a drill-down view.
func (*Service) MoveFlows ¶
MoveFlows returns issue moves between channels, most frequent first. If the filter has channels, moves both from and to those channels are included.
func (*Service) OpenIssuesSeries ¶
func (s *Service) OpenIssuesSeries(ctx context.Context, f Filter, bucket time.Duration) (*Series, error)
OpenIssuesSeries returns the number of open issues over time, per channel, from channel snapshots. Series keys are channel IDs. The value of a bucket is the highest snapshot in it; buckets without a snapshot are zero, so choose a bucket at least as large as the channel stats interval (default 5 minutes). If the number of buckets times the number of channels exceeds types.MaxInsightsRows, it fails with types.ErrInsightsTooManyRows; use a shorter time range, a larger bucket or fewer channels. Channel snapshots have no client, so a filter with Clients is rejected.
func (*Service) Series ¶
Series returns time series of event counts.
The underlying query returns one row per bucket, event type, reason (only if a metric has a reason) and channel (only with ByChannel). If that exceeds types.MaxInsightsRows, it fails with types.ErrInsightsTooManyRows; use a shorter time range, a larger bucket or fewer channels.
type ServiceOption ¶
type ServiceOption func(*Service)
ServiceOption configures a Service.
func WithClock ¶
func WithClock(now func() time.Time) ServiceOption
WithClock sets the clock used to evaluate time-dependent issue state and default time ranges. Default time.Now.
func WithCurrentStateCacheTTL ¶
func WithCurrentStateCacheTTL(d time.Duration) ServiceOption
WithCurrentStateCacheTTL sets how long Service.CurrentState results are cached, to protect the database from frequent dashboard refreshes. Zero disables caching. Default DefaultCurrentStateCacheTTL.
func WithDB ¶
func WithDB(db types.DB) ServiceOption
WithDB sets a database used for current state queries when no types.InsightsReader is available (any types.DB implementation works, via FindActiveChannels and LoadOpenIssuesInChannel).
type SlackPostSummary ¶
type SlackPostSummary struct {
Created int64 `json:"created"`
Updated int64 `json:"updated"`
Deleted int64 `json:"deleted"`
Throttled int64 `json:"throttled"`
Failed int64 `json:"failed"`
FailedByReason map[string]int64 `json:"failedByReason"`
}
SlackPostSummary counts Slack post operations.
type StateCounts ¶
type StateCounts struct {
// Open is the number of open (non-archived) issues.
Open int `json:"open"`
// Alerting is the number of open issues that are neither resolved nor info, and BySeverity splits them by severity.
Alerting int `json:"alerting"`
BySeverity map[types.AlertSeverity]int `json:"bySeverity"`
Resolved int `json:"resolved"`
Muted int `json:"muted"`
Investigated int `json:"investigated"`
Escalated int `json:"escalated"`
Moved int `json:"moved"`
// Stale is the number of alerting issues without new alerts for longer than StaleAfter.
Stale int `json:"stale"`
}
StateCounts counts open issues by state.
type Summary ¶
type Summary struct {
Filter Filter `json:"filter"`
Alerts AlertSummary `json:"alerts"`
Issues IssueSummary `json:"issues"`
Commands map[string]CommandCounts `json:"commands"`
SlackPosts SlackPostSummary `json:"slackPosts"`
Webhooks WebhookSummary `json:"webhooks"`
// LostEvents is the number of insights events lost in the time range (from gap events, all channels).
// A non-zero value means the counts may be incomplete.
LostEvents int64 `json:"lostEvents"`
}
Summary holds all activity counts for a time range and set of channels.
type TimelineQuery ¶
type TimelineQuery struct {
// IssueID is the issue to return events for. Required.
IssueID string
// From and To limit the time range. Defaults: the last [DefaultTimelineRange] up to now.
From time.Time
To time.Time
// Cursor continues a previous page. Limit is the page size (zero means [types.DefaultInsightsPageSize]).
Cursor string
Limit int
}
TimelineQuery selects the events of one issue.
type ValueStats ¶
type ValueStats struct {
Count int64 `json:"count"`
Avg float64 `json:"avg"`
P50 float64 `json:"p50"`
P90 float64 `json:"p90"`
P99 float64 `json:"p99"`
Max float64 `json:"max"`
}
ValueStats holds aggregates of a numeric value over a set of issues.
type WebhookSummary ¶
WebhookSummary counts webhook deliveries.