prepalert

package module
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 24, 2023 License: MIT Imports: 42 Imported by: 0

README

prepalert

Latest GitHub release Github Actions test License

Toil reduction tool to prepare before responding to Mackerel alerts

preplert consists of two parts: a webhook server that receives Mackerel webhooks and sends the payload to Amazon SQS, and a worker that queries various data based on the webhooks and pastes information for alert response as a GraphAnnotation.

Install

Homebrew (macOS and Linux)
$ brew install mashiike/tap/prepalert
Binary packages

Releases

QuickStart

Set your Mackerel API key to the environment variable MACKEREL_APIKEY.
and Execute the following command:

$ prepalert init 

Or the following command:

$ prepalert --coinfig <output config path> init

Usage

Usage: prepalert <command>

A webhook server for prepare alert memo

Flags:
  -h, --help                      Show context-sensitive help.
      --log-level="info"          output log-level ($PREPALERT_LOG_LEVEL)
      --mackerel-apikey=STRING    for access mackerel API ($MACKEREL_APIKEY)
      --config="."                config path ($PREPALERT_CONFIG)

Commands:
  run
    run server (default command)

  init
    create initial config

  validate
    validate the configuration

  exec <alert-id>
    Generate a virtual webhook from past alert to execute the rule

  version
    Show version

Run "prepalert <command> --help" for more information on a command.

If the command is omitted, the run command is executed.

Configurations

Configuration file is HCL (HashiCorp Configuration Language) format. prepalert init can generate a initial configuration file.

The most small configuration file is as follows:

prepalert {
    required_version = ">=v0.12.0"
    sqs_queue_name   = "prepalert"
}

locals {
    default_message =  <<EOF
How do you respond to alerts?
Describe information about your alert response here.
EOF
}

rule "simple" {
    // rule execute when org_name is "Macker..." and alert id is "4gx..."
    when = [
        webhook.org_name == "Macker...",
        get_monitor(webhook.alert).id == "4gx...",
    ]
    update_alert {
        memo = local.default_message
    }
}

Usage with AWS Lambda (serverless)

prepalert works with AWS Lambda and Amazon SQS.

Lambda Function requires a webhook and a worker

sequenceDiagram
  autonumber
  Mackerel->>+http lambda function : POST /
  http lambda function ->>+Amazon SQS: SendMessage
  Amazon SQS-->- http lambda function: 200 Ok
  http lambda function-->- Mackerel: 200 Ok
  Amazon SQS ->>+ worker lambda function: trigger by AWS Lambda
  worker lambda function ->>+ Data Source: query
  Data Source -->- worker lambda function: query results
  worker lambda function  ->>+ Mackerel: Create Graph Annotation
  Mackerel-->- worker lambda function : 200 Ok
  worker lambda function ->>-  Amazon SQS: Success Delete message

Let's solidify the Lambda package with the following zip arcive (runtime provided.al2)

lambda.zip
├── bootstrap    # build binary
└── config.hcl   # configuration file

A related document is https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

for example.

deploy lambda function, prepalert in lambda directory
The example of lambda directory uses lambroll for deployment.

Advanced Usage

Plugin System

prepalert has a plugin system. you can add custom provider. plugin is gRPC Server program. protocol buffer definition is here

in Go language, following code is example plugin.

package main

import (
	"context"

	"github.com/mashiike/prepalert/plugin"
	"github.com/mashiike/prepalert/provider"
)

type Provider struct {
	// ...
}

func (p *Provider) ValidateProviderParameter(ctx context.Context, pp *provider.ProviderParameter) error {
	/*
		Your custom provider parameter validation
	*/
	return nil
}

func (p *Provider) GetQuerySchema(ctx context.Context) (*plugin.Schema, error) {
	/*
		Your custom provider's query paramete schema
	*/
	return &plugin.Schema{}, nil
}

func (p *Provider) RunQuery(ctx context.Context, req *plugin.RunQueryRequest) (*plugin.RunQueryResponse, error) {
	/*
		Your custom provider's query execution
	*/
	return &plugin.RunQueryResponse{
		Name:    req.QueryName,
		Query:   "<your query string>",
		Columns: []string{"<column name>", "<column name>", "<column name>"},
		Rows: [][]string{
			{"<row value>", "<row value>", "<row value>"},
			{"<row value>", "<row value>", "<row value>"},
			{"<row value>", "<row value>", "<row value>"},
		},
	}, nil
}

func main() {
	plugin.ServePlugin(plugin.WithProviderPlugin(&Provider{}))
}

example plugin is cmd/example-http-csv-plugin

configuration is as follows:

prepalert {
  required_version = ">=v0.12.0"
  sqs_queue_name   = "prepalert"

  plugins {
    http = {
      cmd         = "go run github.com/mashiike/prepalert/cmd/example-http-csv-plugin@latest" // your plugin execution command
      sync_output = true  // sync plugin output to prepalert log
    }
  }
}

provider "http" {
  endpoint = "<your csv server endpoint>"
}

query "http" "csv" {}

rule "always" {
  when = true
  update_alert {
    memo = "${query.http.csv.result.query}\n${result_to_table(query.http.csv)}"
  }
}

LICENSE

MIT License

Copyright (c) 2022 IKEDA Masashi

Documentation

Index

Constants

View Source
const (
	FindGraphAnnotationOffset = int64(15 * time.Minute / time.Second)
)
View Source
const (
	HeaderRequestID = "Prepalert-Request-ID"
)

Variables

View Source
var (
	GraphAnnotationDescriptionMaxSize = 1024
	AlertMemoMaxSize                  = 80 * 1000
	CacheDuration                     = 1 * time.Minute
)
View Source
var Version = "current"

Functions

func ExpressionToString added in v0.12.0

func ExpressionToString(expr hcl.Expression, evalCtx *hcl.EvalContext) (string, error)

func GenerateInitialConfig added in v0.12.0

func GenerateInitialConfig(sqsQueueName string, orgName string) ([]byte, error)

func RunCLI added in v0.10.0

func RunCLI(ctx context.Context, args []string, setLogLevel func(string)) error

Types

type Alert

type Alert struct {
	OpenedAt          int64    `json:"openedAt" cty:"opened_at"`
	ClosedAt          *int64   `json:"closedAt" cty:"closed_at"`
	CreatedAt         int64    `json:"createdAt" cty:"created_at"`
	CriticalThreshold *float64 `json:"criticalThreshold,omitempty" cty:"critical_threshold,omitempty"`
	Duration          int64    `json:"duration" cty:"duration"`
	IsOpen            bool     `json:"isOpen" cty:"is_open"`
	MetricLabel       string   `json:"metricLabel" cty:"metric_label"`
	MetricValue       float64  `json:"metricValue" cty:"metric_value"`
	MonitorName       string   `json:"monitorName" cty:"monitor_name"`
	MonitorOperator   string   `json:"monitorOperator" cty:"monitor_operator"`
	Status            string   `json:"status" cty:"status"`
	Trigger           string   `json:"trigger" cty:"trigger"`
	ID                string   `json:"id" cty:"id"`
	URL               string   `json:"url" cty:"url"`
	WarningThreshold  *float64 `json:"warningThreshold,omitempty" cty:"warning_threshold,omitempty"`
}

type App

type App struct {
	// contains filtered or unexported fields
}

func New

func New(apikey string) *App

func (*App) Backend added in v0.12.0

func (app *App) Backend() Backend

func (*App) CheckBasicAuth

func (app *App) CheckBasicAuth(r *http.Request) bool

func (*App) Close added in v0.12.0

func (app *App) Close() error

func (*App) DecodeBody added in v0.12.0

func (app *App) DecodeBody(body hcl.Body, evalCtx *hcl.EvalContext) hcl.Diagnostics

func (*App) EnableBasicAuth

func (app *App) EnableBasicAuth() bool

func (*App) EnableWebhookServer added in v0.12.0

func (app *App) EnableWebhookServer() bool

func (*App) Exec added in v0.5.0

func (app *App) Exec(ctx context.Context, opts *ExecOptions) error

func (*App) ExecuteRules added in v0.12.0

func (app *App) ExecuteRules(ctx context.Context, body *WebhookBody) error

func (*App) Init added in v0.12.0

func (app *App) Init(ctx context.Context, outputPath string) error

func (*App) LoadConfig added in v0.12.0

func (app *App) LoadConfig(dir string, optFns ...func(*LoadConfigOptions)) error

func (*App) LoadPlugin added in v0.12.0

func (app *App) LoadPlugin(ctx context.Context, cfg *LoadPluginConfig) error

func (*App) MackerelService added in v0.12.0

func (app *App) MackerelService() *MackerelService

func (*App) NewEvalContext added in v0.12.0

func (app *App) NewEvalContext(body *WebhookBody) (*hcl.EvalContext, error)

func (*App) NewRule added in v0.12.0

func (app *App) NewRule(ruleName string) *Rule

func (*App) ProviderList added in v0.12.0

func (app *App) ProviderList() []string

func (*App) QueryList added in v0.12.0

func (app *App) QueryList() []string

func (*App) Rules added in v0.12.0

func (app *App) Rules() []*Rule

func (*App) Run

func (app *App) Run(ctx context.Context, opts *RunOptions) error

func (*App) SQSQueueName added in v0.12.0

func (app *App) SQSQueueName() string

func (*App) ServeHTTP

func (app *App) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*App) SetMackerelClient added in v0.12.0

func (app *App) SetMackerelClient(client MackerelClient) *App

func (*App) SetupS3Buckend added in v0.12.0

func (app *App) SetupS3Buckend(body hcl.Body) hcl.Diagnostics

func (*App) UnwrapAndDumpDiagnoctics added in v0.12.0

func (app *App) UnwrapAndDumpDiagnoctics(err error) error

func (*App) WebhookServerIsReady added in v0.12.0

func (app *App) WebhookServerIsReady() bool

func (*App) WithPrepalertFunctions added in v0.12.0

func (app *App) WithPrepalertFunctions(evalCtx *hcl.EvalContext) *hcl.EvalContext

func (*App) WorkerIsReady added in v0.12.0

func (app *App) WorkerIsReady() bool

type Backend added in v0.12.0

type Backend interface {
	http.Handler
	fmt.Stringer
	Upload(ctx context.Context, evalCtx *hcl.EvalContext, name string, body io.Reader) (string, bool, error)
}

type CLI added in v0.10.0

type CLI struct {
	LogLevel       string        `help:"output log-level" env:"PREPALERT_LOG_LEVEL" default:"info"`
	MackerelAPIKey string        `name:"mackerel-apikey" help:"for access mackerel API" env:"MACKEREL_APIKEY"`
	ErrorHandling  ErrorHandling `help:"error handling" env:"PREPALERT_ERROR_HANDLING" default:"continue" enum:"continue,return"`
	Config         string        `help:"config path" env:"PREPALERT_CONFIG" default:"."`
	Run            *RunOptions   `cmd:"" help:"run server (default command)" default:""`
	Init           struct{}      `cmd:"" help:"create initial config"`
	Validate       struct{}      `cmd:"" help:"validate the configuration"`
	Exec           *ExecOptions  `cmd:"" help:"Generate a virtual webhook from past alert to execute the rule"`
	Version        struct{}      `cmd:"" help:"Show version"`
}

func ParseCLI added in v0.10.0

func ParseCLI(ctx context.Context, args []string, opts ...kong.Option) (string, *CLI, error)

type DiscardBackend added in v0.12.0

type DiscardBackend struct{}

func NewDiscardBackend added in v0.12.0

func NewDiscardBackend() *DiscardBackend

func (*DiscardBackend) ServeHTTP added in v0.12.0

func (b *DiscardBackend) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*DiscardBackend) String added in v0.12.0

func (b *DiscardBackend) String() string

func (*DiscardBackend) Upload added in v0.12.0

func (b *DiscardBackend) Upload(ctx context.Context, evalCtx *hcl.EvalContext, name string, body io.Reader) (string, bool, error)

type ErrorHandling added in v0.12.0

type ErrorHandling int
const (
	ContinueOnError ErrorHandling = iota // if load config on error, continue run
	ReturnOnError                        // if load config on error, return error
)

func (ErrorHandling) String added in v0.12.0

func (e ErrorHandling) String() string

func (*ErrorHandling) UnmarshalText added in v0.12.0

func (e *ErrorHandling) UnmarshalText(text []byte) error

type ExecOptions added in v0.10.0

type ExecOptions struct {
	AlertID string `arg:"" name:"alert-id" help:"Mackerel AlertID" required:""`
}

type Host

type Host struct {
	ID        string  `json:"id" cty:"id"`
	Name      string  `json:"name" cty:"name"`
	URL       string  `json:"url" cty:"url"`
	Type      string  `json:"type,omitempty" cty:"type"`
	Status    string  `json:"status" cty:"status"`
	Memo      string  `json:"memo" cty:"memo"`
	IsRetired bool    `json:"isRetired" cty:"is_retired"`
	Roles     []*Role `json:"roles" cty:"roles,omitempty"`
}

type LoadConfigOptions added in v0.12.0

type LoadConfigOptions struct {
	DiagnosticDestination io.Writer
	Color                 *bool
	Width                 *uint
}

type LoadPluginConfig added in v0.12.0

type LoadPluginConfig struct {
	PluginName string `cty:"-"`
	Command    string `cty:"cmd"`
	SyncOutput bool   `cty:"sync_output"`
}

type MackerelClient added in v0.12.0

type MackerelClient interface {
	UpdateAlert(alertID string, param mackerel.UpdateAlertParam) (*mackerel.UpdateAlertResponse, error)
	FindGraphAnnotations(service string, from int64, to int64) ([]mackerel.GraphAnnotation, error)
	UpdateGraphAnnotation(annotationID string, annotation *mackerel.GraphAnnotation) (*mackerel.GraphAnnotation, error)
	CreateGraphAnnotation(annotation *mackerel.GraphAnnotation) (*mackerel.GraphAnnotation, error)
	GetOrg() (*mackerel.Org, error)
	GetAlert(string) (*mackerel.Alert, error)
	GetMonitor(string) (mackerel.Monitor, error)
	FindHost(id string) (*mackerel.Host, error)
}

type MackerelService added in v0.12.0

type MackerelService struct {
	// contains filtered or unexported fields
}

func NewMackerelService added in v0.12.0

func NewMackerelService(client MackerelClient) *MackerelService

func (*MackerelService) GetAlertWithCache added in v0.12.0

func (svc *MackerelService) GetAlertWithCache(ctx context.Context, alertID string) (*mackerel.Alert, error)

func (*MackerelService) GetMonitorByAlertID added in v0.12.0

func (svc *MackerelService) GetMonitorByAlertID(ctx context.Context, alertID string) (mackerel.Monitor, error)

func (*MackerelService) GetMonitorWithCache added in v0.12.0

func (svc *MackerelService) GetMonitorWithCache(ctx context.Context, monitorID string) (mackerel.Monitor, error)

func (*MackerelService) NewEmulatedWebhookBody added in v0.12.0

func (svc *MackerelService) NewEmulatedWebhookBody(ctx context.Context, alertID string) (*WebhookBody, error)

func (*MackerelService) NewExampleWebhookBody added in v0.12.0

func (svc *MackerelService) NewExampleWebhookBody() *WebhookBody

func (*MackerelService) NewMackerelUpdater added in v0.12.0

func (svc *MackerelService) NewMackerelUpdater(body *WebhookBody, backend Backend) *MackerelUpdater

func (*MackerelService) PostGraphAnnotation added in v0.12.0

func (svc *MackerelService) PostGraphAnnotation(ctx context.Context, params *mackerel.GraphAnnotation) error

func (*MackerelService) UpdateAlertMemo added in v0.12.0

func (svc *MackerelService) UpdateAlertMemo(ctx context.Context, alertID string, memo string) error

type MackerelUpdater added in v0.12.0

type MackerelUpdater struct {
	// contains filtered or unexported fields
}

func (*MackerelUpdater) AddAdditionalDescription added in v0.12.0

func (u *MackerelUpdater) AddAdditionalDescription(service string, text string)

func (*MackerelUpdater) AddMemoSectionText added in v0.12.0

func (u *MackerelUpdater) AddMemoSectionText(text string, sizeLimit *int)

func (*MackerelUpdater) AddService added in v0.12.0

func (u *MackerelUpdater) AddService(service string)

func (*MackerelUpdater) Flush added in v0.12.0

func (u *MackerelUpdater) Flush(ctx context.Context, evalCtx *hcl.EvalContext) error

type PostGraphAnnotationAction added in v0.12.0

type PostGraphAnnotationAction struct {
	// contains filtered or unexported fields
}

func (*PostGraphAnnotationAction) DecodeBody added in v0.12.0

func (action *PostGraphAnnotationAction) DecodeBody(body hcl.Body, evalCtx *hcl.EvalContext) hcl.Diagnostics

func (*PostGraphAnnotationAction) DependsOnQueries added in v0.12.0

func (action *PostGraphAnnotationAction) DependsOnQueries() []string

func (*PostGraphAnnotationAction) Enable added in v0.12.0

func (action *PostGraphAnnotationAction) Enable() bool

func (*PostGraphAnnotationAction) Execute added in v0.12.0

func (action *PostGraphAnnotationAction) Execute(ctx context.Context, evalCtx *hcl.EvalContext, u *MackerelUpdater) error

type RequestIDGenerator added in v0.12.0

type RequestIDGenerator interface {
	NextID() (uint64, error)
}
var DefaultRequestIDGeneartor RequestIDGenerator = must(katsubushi.NewGenerator(1))

type Role

type Role struct {
	Fullname    string `json:"fullname" cty:"fullname"`
	ServiceName string `json:"serviceName" cty:"service_name"`
	ServiceURL  string `json:"serviceUrl" cty:"service_url"`
	RoleName    string `json:"roleName" cty:"role_name"`
	RoleURL     string `json:"roleUrl" cty:"role_url"`
}

type Rule

type Rule struct {
	// contains filtered or unexported fields
}

func (*Rule) DecodeBody added in v0.12.0

func (rule *Rule) DecodeBody(body hcl.Body, evalCtx *hcl.EvalContext) hcl.Diagnostics

func (*Rule) DependsOnQueries added in v0.12.0

func (rule *Rule) DependsOnQueries() []string

func (*Rule) Execute added in v0.12.0

func (rule *Rule) Execute(ctx context.Context, evalCtx *hcl.EvalContext, u *MackerelUpdater) error

func (*Rule) Match

func (rule *Rule) Match(evalCtx *hcl.EvalContext) bool

func (*Rule) Name added in v0.5.0

func (rule *Rule) Name() string

func (*Rule) PostGraphAnnotationAction added in v0.12.0

func (rule *Rule) PostGraphAnnotationAction() *PostGraphAnnotationAction

func (*Rule) Priority added in v0.12.0

func (rule *Rule) Priority() int

func (*Rule) UpdateAlertAction added in v0.12.0

func (rule *Rule) UpdateAlertAction() *UpdateAlertAction

type RunOptions

type RunOptions struct {
	Mode      string `help:"run mode" env:"PREPALERT_MODE" default:"all" enum:"all,http,worker,webhook"`
	Address   string `help:"run local address" env:"PREPALERT_ADDRESS" default:":8080"`
	Prefix    string `help:"run server prefix" env:"PREPALERT_PREFIX" default:"/"`
	BatchSize int    `help:"run local sqs batch size" env:"PREPALERT_BATCH_SIZE" default:"1"`
}

type S3Backend added in v0.12.0

type S3Backend struct {
	BucketName                    string
	ObjectKeyPrefix               *string
	ObjectKeyTemplate             *hcl.Expression
	ViewerBaseURLString           string
	ViewerGoogleClientID          *string
	ViewerGoogleClientSecret      *string
	ViewerSessionEncryptKeyString *string
	Allowed                       []string
	Denied                        []string

	ViewerBaseURL           *url.URL
	ViewerSessionEncryptKey []byte
	// contains filtered or unexported fields
}

func (*S3Backend) EnableGoogleAuth added in v0.12.0

func (b *S3Backend) EnableGoogleAuth() bool

func (*S3Backend) IsEmpty added in v0.12.0

func (b *S3Backend) IsEmpty() bool

func (*S3Backend) ServeHTTP added in v0.12.0

func (b *S3Backend) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*S3Backend) String added in v0.12.0

func (b *S3Backend) String() string

func (*S3Backend) Upload added in v0.12.0

func (b *S3Backend) Upload(ctx context.Context, evalCtx *hcl.EvalContext, name string, body io.Reader) (string, bool, error)

type S3Client added in v0.12.0

type S3Client interface {
	manager.UploadAPIClient
	ls3viewer.S3Client
}
var GlobalS3Client S3Client

type Service

type Service struct {
	ID    string  `json:"id" cty:"id"`
	Memo  string  `json:"memo" cty:"memo"`
	Name  string  `json:"name" cty:"name"`
	OrgID string  `json:"orgId" cty:"org_id"`
	Roles []*Role `json:"roles" cty:"roles,omitempty"`
}

type UpdateAlertAction added in v0.12.0

type UpdateAlertAction struct {
	// contains filtered or unexported fields
}

func (*UpdateAlertAction) DecodeBody added in v0.12.0

func (action *UpdateAlertAction) DecodeBody(body hcl.Body, evalCtx *hcl.EvalContext) hcl.Diagnostics

func (*UpdateAlertAction) DependsOnQueries added in v0.12.0

func (action *UpdateAlertAction) DependsOnQueries() []string

func (*UpdateAlertAction) Enable added in v0.12.0

func (action *UpdateAlertAction) Enable() bool

func (*UpdateAlertAction) Execute added in v0.12.0

func (action *UpdateAlertAction) Execute(ctx context.Context, evalCtx *hcl.EvalContext, u *MackerelUpdater) error

type WebhookBody

type WebhookBody struct {
	OrgName  string   `json:"orgName" cty:"org_name"`
	Text     string   `json:"text" cty:"-"`
	Event    string   `json:"event" cty:"event"`
	ImageURL *string  `json:"imageUrl" cty:"image_url"`
	Memo     string   `json:"memo" cty:"memo"`
	Host     *Host    `json:"host,omitempty" cty:"host,omitempty"`
	Service  *Service `json:"service,omitempty" cty:"service,omitempty"`
	Alert    *Alert   `json:"alert" cty:"alert,omitempty"`
}

func WebhookFromEvalContext added in v0.12.0

func WebhookFromEvalContext(evalCtx *hcl.EvalContext) (*WebhookBody, error)

Directories

Path Synopsis
cmd
prepalert command
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL