moira

package module
v2.5.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 12, 2019 License: MIT Imports: 10 Imported by: 124

README

Moira 2.0 Documentation Status Telegram Gitter Go Report Card

Moira is a real-time alerting tool, based on Graphite data.

Build status

branch status
master Build Status Coverage Status
develop Build Status Coverage Status

Installation

Docker Compose is the easiest way to try:

git clone https://github.com/moira-alert/docker-compose.git
cd docker-compose
docker-compose pull
docker-compose up

Feed data in Graphite format to localhost:2003:

echo "local.random.diceroll 4 `date +%s`" | nc localhost 2003

Configure triggers at localhost:8080 using your browser.

Other installation methods are available, see documentation.

Getting Started

See our user guide that is based on a number of real-life scenarios, from simple and universal to complicated and specific.

Why 2.0?

Moira 2.0 is different from the first version in two important ways:

  1. We got rid of Python, because it was slow. Checker and API services are now written in Go, based on carbonapi implementation of Graphite functions.
  2. We got rid of Angular, because our main stack is React now. We just don't know how to do Angular anymore. We also revamped the UI.

What is in the other repositories?

Code in this repository is the backend part of Moira monitoring application.

Contact us

If you have any questions, you can ask us on Telegram.

Thanks

SKB Kontur

Moira was originally developed and is supported by SKB Kontur, a B2G company based in Ekaterinburg, Russia. We express gratitude to our company for encouraging us to opensource Moira and for giving back to the community that created Graphite and many other useful DevOps tools.

Documentation

Index

Constants

View Source 
const (
	// VariableContactID is used to render template with contact.ID
	VariableContactID = "${contact_id}"
	// VariableContactValue is used to render template with contact.Value
	VariableContactValue = "${contact_value}"
	// VariableContactType is used to render template with contact.Type
	VariableContactType = "${contact_type}"
	// VariableTriggerID is used to render template with trigger.ID
	VariableTriggerID = "${trigger_id}"
	// VariableTriggerName is used to render template with trigger.Name
	VariableTriggerName = "${trigger_name}"
)
View Source 
const (
	// FallingTrigger represents falling trigger type, in which OK > WARN > ERROR
	FallingTrigger = "falling"
	// RisingTrigger represents rising trigger type, in which OK < WARN < ERROR
	RisingTrigger = "rising"
	// ExpressionTrigger represents trigger type with custom user expression
	ExpressionTrigger = "expression"
)
View Source 
const (
	SelfStateOK    = "OK"    // OK means notifier is healthy
	SelfStateERROR = "ERROR" // ERROR means notifier is stopped, admin intervention is required
)

Moira notifier self-states

Variables

This section is empty.

Functions

func ChunkSlice 

func ChunkSlice(original []string, chunkSize int) (divided [][]string)

ChunkSlice gets slice of strings and chunks it to a given size. It returns a batch of chunked lists

func GetStringListsDiff 

func GetStringListsDiff(stringLists ...[]string) []string

GetStringListsDiff returns the members of the set resulting from the difference between the first set and all the successive lists.

func GetStringListsUnion 

func GetStringListsUnion(stringLists ...[]string) []string

GetStringListsUnion returns the union set of stringLists

func Int64ToTime 

func Int64ToTime(timeStamp int64) time.Time

Int64ToTime returns time.Time from int64

func IsValidFloat64 

func IsValidFloat64(val float64) bool

IsValidFloat64 checks float64 for Inf and NaN. If it is then float64 is not valid

func SetMaintenanceUserAndTime 

func SetMaintenanceUserAndTime(maintenanceCheck MaintenanceCheck, maintenance int64, user string, callMaintenance int64)

SetMaintenanceUserAndTime set startuser and starttime or stopuser and stoptime for MaintenanceInfo

func Subset 

func Subset(first, second []string) bool

Subset return whether first is a subset of second

func UnsafeBytesToString 

func UnsafeBytesToString(b []byte) string

UnsafeBytesToString converts source to string without copying

func UnsafeStringToBytes 

func UnsafeStringToBytes(s string) []byte

UnsafeStringToBytes converts string to source without copying

func UseFloat64 

func UseFloat64(f *float64) float64

UseFloat64 gets pointer value of float64 or default float64 if pointer is nil

func UseString 

func UseString(str *string) string

UseString gets pointer value of string or default string if pointer is nil

Types

type BytesScanner 

type BytesScanner struct {
	// contains filtered or unexported fields
}

BytesScanner allows to scan for subslices separated by separator

func NewBytesScanner 

func NewBytesScanner(bytes []byte, separator byte) *BytesScanner

NewBytesScanner slices bytes into all subslices separated by separator and returns a scanner which allows to scan for these subslices

func (*BytesScanner) HasNext 

func (it *BytesScanner) HasNext() bool

HasNext checks if next subslice available or not

func (*BytesScanner) Next 

func (it *BytesScanner) Next() (result []byte)

Next returns available subslice and advances the scanner to next slice

type CheckData 

type CheckData struct {
	Metrics                      map[string]MetricState `json:"metrics"`
	Score                        int64                  `json:"score"`
	State                        State                  `json:"state"`
	Maintenance                  int64                  `json:"maintenance,omitempty"`
	MaintenanceInfo              MaintenanceInfo        `json:"maintenance_info"`
	Timestamp                    int64                  `json:"timestamp,omitempty"`
	EventTimestamp               int64                  `json:"event_timestamp,omitempty"`
	LastSuccessfulCheckTimestamp int64                  `json:"last_successful_check_timestamp"`
	Suppressed                   bool                   `json:"suppressed,omitempty"`
	SuppressedState              State                  `json:"suppressed_state,omitempty"`
	Message                      string                 `json:"msg,omitempty"`
}

CheckData represents last trigger check data

func (CheckData) GetEventTimestamp 

func (checkData CheckData) GetEventTimestamp() int64

GetEventTimestamp gets event timestamp for given check

func (*CheckData) GetMaintenance 

func (checkData *CheckData) GetMaintenance() (MaintenanceInfo, int64)

GetMaintenance return metricState MaintenanceInfo

func (*CheckData) GetOrCreateMetricState 

func (checkData *CheckData) GetOrCreateMetricState(metric string, emptyTimestampValue int64, muteNewMetric bool) MetricState

GetOrCreateMetricState gets metric state from check data or create new if CheckData has no state for given metric

func (*CheckData) SetMaintenance 

func (checkData *CheckData) SetMaintenance(maintenanceInfo *MaintenanceInfo, maintenance int64)

SetMaintenance set maintenance user, time for CheckData

func (*CheckData) UpdateScore 

func (checkData *CheckData) UpdateScore() int64

UpdateScore update and return checkData score, based on metric states and checkData state

type ContactData 

type ContactData struct {
	Type  string `json:"type"`
	Value string `json:"value"`
	ID    string `json:"id"`
	User  string `json:"user"`
}

ContactData represents contact object

type Database 

type Database interface {
	// SelfState
	UpdateMetricsHeartbeat() error
	GetMetricsUpdatesCount() (int64, error)
	GetChecksUpdatesCount() (int64, error)
	GetRemoteChecksUpdatesCount() (int64, error)
	GetNotifierState() (string, error)
	SetNotifierState(string) error

	// Tag storing
	GetTagNames() ([]string, error)
	RemoveTag(tagName string) error
	GetTagTriggerIDs(tagName string) ([]string, error)

	// LastCheck storing
	GetTriggerLastCheck(triggerID string) (CheckData, error)
	SetTriggerLastCheck(triggerID string, checkData *CheckData, isRemote bool) error
	RemoveTriggerLastCheck(triggerID string) error
	SetTriggerCheckMaintenance(triggerID string, metrics map[string]int64, triggerMaintenance *int64, userLogin string, timeCallMaintenance int64) error

	// Trigger storing
	GetLocalTriggerIDs() ([]string, error)
	GetAllTriggerIDs() ([]string, error)
	GetRemoteTriggerIDs() ([]string, error)
	GetTrigger(triggerID string) (Trigger, error)
	GetTriggers(triggerIDs []string) ([]*Trigger, error)
	GetTriggerChecks(triggerIDs []string) ([]*TriggerCheck, error)
	SaveTrigger(triggerID string, trigger *Trigger) error
	RemoveTrigger(triggerID string) error
	GetPatternTriggerIDs(pattern string) ([]string, error)
	RemovePatternTriggerIDs(pattern string) error

	// Throttling
	GetTriggerThrottling(triggerID string) (time.Time, time.Time)
	SetTriggerThrottling(triggerID string, next time.Time) error
	DeleteTriggerThrottling(triggerID string) error

	// NotificationEvent storing
	GetNotificationEvents(triggerID string, start, size int64) ([]*NotificationEvent, error)
	PushNotificationEvent(event *NotificationEvent, ui bool) error
	GetNotificationEventCount(triggerID string, from int64) int64
	FetchNotificationEvent() (NotificationEvent, error)
	RemoveAllNotificationEvents() error

	// ContactData storing
	GetContact(contactID string) (ContactData, error)
	GetContacts(contactIDs []string) ([]*ContactData, error)
	GetAllContacts() ([]*ContactData, error)
	RemoveContact(contactID string) error
	SaveContact(contact *ContactData) error
	GetUserContactIDs(userLogin string) ([]string, error)

	// SubscriptionData storing
	GetSubscription(id string) (SubscriptionData, error)
	GetSubscriptions(subscriptionIDs []string) ([]*SubscriptionData, error)
	SaveSubscription(subscription *SubscriptionData) error
	SaveSubscriptions(subscriptions []*SubscriptionData) error
	RemoveSubscription(subscriptionID string) error
	GetUserSubscriptionIDs(userLogin string) ([]string, error)
	GetTagsSubscriptions(tags []string) ([]*SubscriptionData, error)

	// ScheduledNotification storing
	GetNotifications(start, end int64) ([]*ScheduledNotification, int64, error)
	RemoveNotification(notificationKey string) (int64, error)
	RemoveAllNotifications() error
	FetchNotifications(to int64) ([]*ScheduledNotification, error)
	AddNotification(notification *ScheduledNotification) error
	AddNotifications(notification []*ScheduledNotification, timestamp int64) error

	// Patterns and metrics storing
	GetPatterns() ([]string, error)
	AddPatternMetric(pattern, metric string) error
	GetPatternMetrics(pattern string) ([]string, error)
	RemovePattern(pattern string) error
	RemovePatternsMetrics(pattern []string) error
	RemovePatternWithMetrics(pattern string) error

	SubscribeMetricEvents(tomb *tomb.Tomb) (<-chan *MetricEvent, error)
	SaveMetrics(buffer map[string]*MatchedMetric) error
	GetMetricRetention(metric string) (int64, error)
	GetMetricsValues(metrics []string, from int64, until int64) (map[string][]*MetricValue, error)
	RemoveMetricValues(metric string, toTime int64) error
	RemoveMetricsValues(metrics []string, toTime int64) error

	AddLocalTriggersToCheck(triggerIDs []string) error
	GetLocalTriggersToCheck(count int) ([]string, error)
	GetLocalTriggersToCheckCount() (int64, error)

	AddRemoteTriggersToCheck(triggerIDs []string) error
	GetRemoteTriggersToCheck(count int) ([]string, error)
	GetRemoteTriggersToCheckCount() (int64, error)

	// TriggerCheckLock storing
	AcquireTriggerCheckLock(triggerID string, timeout int) error
	DeleteTriggerCheckLock(triggerID string) error
	SetTriggerCheckLock(triggerID string) (bool, error)

	// Bot data storing
	GetIDByUsername(messenger, username string) (string, error)
	SetUsernameID(messenger, username, id string) error
	RemoveUser(messenger, username string) error

	// Triggers without subscription manipulation
	MarkTriggersAsUnused(triggerIDs ...string) error
	GetUnusedTriggerIDs() ([]string, error)
	MarkTriggersAsUsed(triggerIDs ...string) error

	// Triggers to reindex in full-text search index
	FetchTriggersToReindex(from int64) ([]string, error)
	RemoveTriggersToReindex(to int64) error

	// Creates Lock
	NewLock(name string, ttl time.Duration) Lock
}

Database implements DB functionality

type Lock 

type Lock interface {
	Acquire(stop <-chan struct{}) (lost <-chan struct{}, error error)
	Release()
}

Lock implements lock abstraction

type Logger 

type Logger interface {
	Debug(args ...interface{})
	Debugf(format string, args ...interface{})
	Info(args ...interface{})
	Infof(format string, args ...interface{})
	Error(args ...interface{})
	Errorf(format string, args ...interface{})
	Fatal(args ...interface{})
	Fatalf(format string, args ...interface{})
	Warning(args ...interface{})
	Warningf(format string, args ...interface{})
}

Logger implements logger abstraction

type MaintenanceCheck 

type MaintenanceCheck interface {
	SetMaintenance(maintenanceInfo *MaintenanceInfo, maintenance int64)
	GetMaintenance() (MaintenanceInfo, int64)
}

MaintenanceCheck set maintenance user, time

type MaintenanceInfo 

type MaintenanceInfo struct {
	StartUser *string `json:"setup_user"`
	StartTime *int64  `json:"setup_time"`
	StopUser  *string `json:"remove_user"`
	StopTime  *int64  `json:"remove_time"`
}

MaintenanceInfo represents user and time set/unset maintenance

func (*MaintenanceInfo) Set 

func (maintenanceInfo *MaintenanceInfo) Set(startUser *string, startTime *int64, stopUser *string, stopTime *int64)

Set maintanace start and stop users and times

type MatchedMetric 

type MatchedMetric struct {
	Metric             string
	Patterns           []string
	Value              float64
	Timestamp          int64
	RetentionTimestamp int64
	Retention          int
}

MatchedMetric represents parsed and matched metric data

type MetricEvent 

type MetricEvent struct {
	Metric  string `json:"metric"`
	Pattern string `json:"pattern"`
}

MetricEvent represents filter metric event

type MetricState 

type MetricState struct {
	EventTimestamp  int64           `json:"event_timestamp"`
	State           State           `json:"state"`
	Suppressed      bool            `json:"suppressed"`
	SuppressedState State           `json:"suppressed_state,omitempty"`
	Timestamp       int64           `json:"timestamp"`
	Value           *float64        `json:"value,omitempty"`
	Maintenance     int64           `json:"maintenance,omitempty"`
	MaintenanceInfo MaintenanceInfo `json:"maintenance_info"`
}

MetricState represents metric state data for given timestamp

func (*MetricState) GetCheckPoint 

func (metricState *MetricState) GetCheckPoint(checkPointGap int64) int64

GetCheckPoint gets check point for given MetricState CheckPoint is the timestamp from which to start checking the current state of the metric

func (MetricState) GetEventTimestamp 

func (metricState MetricState) GetEventTimestamp() int64

GetEventTimestamp gets event timestamp for given metric

func (*MetricState) GetMaintenance 

func (metricState *MetricState) GetMaintenance() (MaintenanceInfo, int64)

GetMaintenance return metricState MaintenanceInfo

func (*MetricState) SetMaintenance 

func (metricState *MetricState) SetMaintenance(maintenanceInfo *MaintenanceInfo, maintenance int64)

SetMaintenance set maintenance user, time for MetricState

type MetricValue 

type MetricValue struct {
	RetentionTimestamp int64   `json:"step,omitempty"`
	Timestamp          int64   `json:"ts"`
	Value              float64 `json:"value"`
}

MetricValue represents metric data

type NotificationEvent 

type NotificationEvent struct {
	IsTriggerEvent bool     `json:"trigger_event,omitempty"`
	Timestamp      int64    `json:"timestamp"`
	Metric         string   `json:"metric"`
	Value          *float64 `json:"value,omitempty"`
	State          State    `json:"state"`
	TriggerID      string   `json:"trigger_id"`
	SubscriptionID *string  `json:"sub_id,omitempty"`
	ContactID      string   `json:"contactId,omitempty"`
	OldState       State    `json:"old_state"`
	Message        *string  `json:"msg,omitempty"`
}

NotificationEvent represents trigger state changes event

func (NotificationEvent) FormatTimestamp 

func (event NotificationEvent) FormatTimestamp(location *time.Location) string

FormatTimestamp gets event timestamp and format it using given location to human readable presentation

func (NotificationEvent) GetMetricValue 

func (event NotificationEvent) GetMetricValue() string

GetMetricValue gets event metric value and format it to human readable presentation

func (NotificationEvent) String 

func (event NotificationEvent) String() string

type NotificationEvents 

type NotificationEvents []NotificationEvent

NotificationEvents represents slice of NotificationEvent

func (NotificationEvents) GetSubjectState 

func (events NotificationEvents) GetSubjectState() State

GetSubjectState returns the most critical state of events

type PlotTheme 

type PlotTheme interface {
	GetTitleStyle() chart.Style
	GetGridStyle() chart.Style
	GetCanvasStyle() chart.Style
	GetBackgroundStyle(maxMarkLen int) chart.Style
	GetThresholdStyle(thresholdType string) chart.Style
	GetAnnotationStyle(thresholdType string) chart.Style
	GetSerieStyles(curveInd int) (curveStyle, pointStyle chart.Style)
	GetLegendStyle() chart.Style
	GetXAxisStyle() chart.Style
	GetYAxisStyle() chart.Style
}

PlotTheme is an interface to access plot theme styles

type PlottingData 

type PlottingData struct {
	Enabled bool   `json:"enabled"`
	Theme   string `json:"theme"`
}

PlottingData represents plotting settings

type ScheduleData 

type ScheduleData struct {
	Days           []ScheduleDataDay `json:"days"`
	TimezoneOffset int64             `json:"tzOffset"`
	StartOffset    int64             `json:"startOffset"`
	EndOffset      int64             `json:"endOffset"`
}

ScheduleData represents subscription schedule

func (*ScheduleData) IsScheduleAllows 

func (schedule *ScheduleData) IsScheduleAllows(ts int64) bool

IsScheduleAllows check if the time is in the allowed schedule interval

type ScheduleDataDay 

type ScheduleDataDay struct {
	Enabled bool   `json:"enabled"`
	Name    string `json:"name,omitempty"`
}

ScheduleDataDay represents week day of schedule

type ScheduledNotification 

type ScheduledNotification struct {
	Event     NotificationEvent `json:"event"`
	Trigger   TriggerData       `json:"trigger"`
	Contact   ContactData       `json:"contact"`
	Plotting  PlottingData      `json:"plotting"`
	Throttled bool              `json:"throttled"`
	SendFail  int               `json:"send_fail"`
	Timestamp int64             `json:"timestamp"`
}

ScheduledNotification represent notification object

func (*ScheduledNotification) GetKey 

func (notification *ScheduledNotification) GetKey() string

GetKey return notification key to prevent duplication to the same contact

type SearchHighlight 

type SearchHighlight struct {
	Field, Value string
}

SearchHighlight represents highlight

type SearchResult 

type SearchResult struct {
	ObjectID   string
	Highlights []SearchHighlight
}

SearchResult represents fulltext search result

type Searcher 

type Searcher interface {
	Start() error
	Stop() error
	IsReady() bool
	SearchTriggers(filterTags []string, searchString string, onlyErrors bool,
		page int64, size int64) (searchResults []*SearchResult, total int64, err error)
}

Searcher interface implements full-text search index functionality

type Sender 

type Sender interface {
	SendEvents(events NotificationEvents, contact ContactData, trigger TriggerData, plot []byte, throttled bool) error
	Init(senderSettings map[string]string, logger Logger, location *time.Location, dateTimeFormat string) error
}

Sender interface for implementing specified contact type sender

type State 

type State string

State type describe all default moira triggers or metrics states 

var (
	StateOK        State = "OK"
	StateWARN      State = "WARN"
	StateERROR     State = "ERROR"
	StateNODATA    State = "NODATA"
	StateEXCEPTION State = "EXCEPTION" // Use this for trigger check unexpected errors
	StateTEST      State = "TEST"      // Use this only for test notifications
)

Moira trigger and metric states

type SubscriptionData 

type SubscriptionData struct {
	Contacts          []string     `json:"contacts"`
	Tags              []string     `json:"tags"`
	Schedule          ScheduleData `json:"sched"`
	Plotting          PlottingData `json:"plotting"`
	ID                string       `json:"id"`
	Enabled           bool         `json:"enabled"`
	IgnoreWarnings    bool         `json:"ignore_warnings,omitempty"`
	IgnoreRecoverings bool         `json:"ignore_recoverings,omitempty"`
	ThrottlingEnabled bool         `json:"throttling"`
	User              string       `json:"user"`
}

SubscriptionData represents user subscription

func (*SubscriptionData) MustIgnore 

func (subscription *SubscriptionData) MustIgnore(eventData *NotificationEvent) bool

MustIgnore returns true if given state transition must be ignored

type TTLState 

type TTLState string

TTLState declares all ttl (NODATA) states, used if metric has no values for given interval (ttl) 

var (
	TTLStateOK     TTLState = "OK"
	TTLStateWARN   TTLState = "WARN"
	TTLStateERROR  TTLState = "ERROR"
	TTLStateNODATA TTLState = "NODATA"
	TTLStateDEL    TTLState = "DEL"
)

Moira ttl states

func (TTLState) ToMetricState 

func (state TTLState) ToMetricState() State

ToMetricState is an auxiliary function to handle metric state properly.

func (TTLState) ToTriggerState 

func (state TTLState) ToTriggerState() State

ToTriggerState is an auxiliary function to handle trigger state properly.

type Trigger 

type Trigger struct {
	ID               string        `json:"id"`
	Name             string        `json:"name"`
	Desc             *string       `json:"desc,omitempty"`
	Targets          []string      `json:"targets"`
	WarnValue        *float64      `json:"warn_value"`
	ErrorValue       *float64      `json:"error_value"`
	TriggerType      string        `json:"trigger_type"`
	Tags             []string      `json:"tags"`
	TTLState         *TTLState     `json:"ttl_state,omitempty"`
	TTL              int64         `json:"ttl,omitempty"`
	Schedule         *ScheduleData `json:"sched,omitempty"`
	Expression       *string       `json:"expression,omitempty"`
	PythonExpression *string       `json:"python_expression,omitempty"`
	Patterns         []string      `json:"patterns"`
	IsRemote         bool          `json:"is_remote"`
	MuteNewMetrics   bool          `json:"mute_new_metrics"`
}

Trigger represents trigger data object

func GetTriggerListsDiff 

func GetTriggerListsDiff(triggerLists ...[]*Trigger) []*Trigger

GetTriggerListsDiff returns the members of the set resulting from the difference between the first set and all the successive lists.

func (*Trigger) IsSimple 

func (trigger *Trigger) IsSimple() bool

IsSimple checks triggers patterns If patterns more than one or it contains standard graphite wildcard symbols, when this target can contain more then one metrics, and is it not simple trigger

type TriggerCheck 

type TriggerCheck struct {
	Trigger
	Throttling int64             `json:"throttling"`
	LastCheck  CheckData         `json:"last_check"`
	Highlights map[string]string `json:"highlights"`
}

TriggerCheck represents trigger data with last check data and check timestamp

type TriggerData 

type TriggerData struct {
	ID         string   `json:"id"`
	Name       string   `json:"name"`
	Desc       string   `json:"desc"`
	Targets    []string `json:"targets"`
	WarnValue  float64  `json:"warn_value"`
	ErrorValue float64  `json:"error_value"`
	IsRemote   bool     `json:"is_remote"`
	Tags       []string `json:"__notifier_trigger_tags"`
}

TriggerData represents trigger object

func (*TriggerData) GetTags 

func (trigger *TriggerData) GetTags() string

GetTags returns "[tag1][tag2]...[tagN]" string

func (TriggerData) GetTriggerURI 

func (trigger TriggerData) GetTriggerURI(frontURI string) string

GetTriggerURI gets frontUri and returns triggerUrl, returns empty string on selfcheck and test notifications

Source Files

View all Source files

Directories

Path Synopsis
api
controller
dto
nolint nolint nolint nolint nolint nolint nolint nolint nolint
 nolint nolint nolint nolint nolint nolint nolint nolint nolint
handler
middleware
worker
cmd
api command
checker command
cli command
filter command
notifier command
redis
redis/reply
connection
heartbeat
matched_metrics
patterns
bleve
fixtures
mapping
logging
go-logging
local
remote
metrics
graphite
graphite/go-metrics
nolint nolint nolint nolint nolint nolint
 nolint nolint nolint nolint nolint nolint
mock
metric_source
Package mock_metric_source is a generated GoMock package.
 Package mock_metric_source is a generated GoMock package.
moira-alert
Package mock_moira_alert is a generated GoMock package.
 Package mock_moira_alert is a generated GoMock package.
notifier
Package mock_notifier is a generated GoMock package.
 Package mock_notifier is a generated GoMock package.
scheduler
Package mock_scheduler is a generated GoMock package.
 Package mock_scheduler is a generated GoMock package.
events
notifications
selfstate
fonts
themes/dark
themes/light
senders
mail
pushover
script
slack
telegram
twilio
webhook

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.