havoc

package module
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 18, 2024 License: MIT Imports: 27 Imported by: 0

README

Havoc

Havoc is a tool that introspects your k8s namespace and generates a ChaosMesh CRDs suite for you

You can use havoc as a CLI to quickly test hypothesis or run it in "monkey" mode with your load tests and have Grafana annotations

Goals

  • Make chaos testing easy by generating most of the things automatically just by looking at your namespace
  • Easy integration with Grafana to understand how chaos affects your services
  • Be easy to use both programmatically and as a CLI

How it works

Havoc generates groups of experiments based just on your pods and labels found in namespace

Single pod experiments:

  • PodFailure
  • NetworkChaos (Pod latency)
  • Stress (Memory)
  • Stress (CPU)
  • External service failure (Network partition)
  • Blockchain specific experiments

Group experiments:

  • Group failure
  • Group latency
  • Group CPU
  • Group memory
  • Group network partition

You can generate default chaos suite by configuring havoc then set dir param and add your custom experiments, then run monkey to test your services

Install

Please use GitHub releases of this repo Download latest release

You need kubectl to available on your machine

If you wish Grafana integration, please set env variables (optional)

HAVOC_LOG_LEVEL={warn,info,debug,trace}
GRAFANA_URL="..."
GRAFANA_TOKEN="..."
DASHBOARD_NAME="..."

Manual usage

Generate default experiments for your namespace

havoc -c havoc.toml generate [namespace]

Check this section for ignore_pods and ignore_group_labels, default settings should be reasonable, however, you can tweak them

This will create havoc-experiments dir, then you can choose from recommended experiments

havoc -c havoc.toml apply

You can also apply your experiment directly, using absolute or relative path to experiment file

havoc -c havoc.toml apply ${experiment_file_path}

Monkey mode

You can run havoc as an automated sequential or randomized suite

havoc -c havoc.toml run [namespace]

See [havoc.monkey] config here

Programmatic usage

See how you can use recommended experiments from code in examples

Custom experiments

Havoc is just a generator and a module that reads your dir = $mydir from config

If you wish to add custom experiments written by hand create your custom directory and add experiments

Experiments will be executed in lexicographic order, however, for custom experiments there are 2 simple rules:

  • directory names must be in ["external", "failure", "latency", "cpu", "memory", "group-failure", "group-latency"]
  • metadata.name should be equal to your experiment filename

When you are using run monkey command, if directory is not empty havoc won't automatically generate experiments, so you can extend generated experiments with your custom modifications

Developing

We are using nix

Enter the shell

nix develop

Why not to use ChaosMesh UI/API instead of CRDs?

ChaosMesh UI/API is great, but it has some downsides:

  • No OpenAPI spec, hard to integrate
  • No dynamic generation for a namespace, you need to rely on labels that might change
  • Writing chaos experiments is tedious, in most cases you just copy-paste a lot, or you can forget something
  • Workflows validation is broken
  • Can't mix chaos experiments and API calls
  • No straightforward integration with load testing tools, it's easy to run an experiment, but it's hard to validate it right away without additional code
  • Can't check chaos experiments statuses through API and fail the test, need to use k8s
  • Experiments created from YAML and UI and not always compatible

Documentation

Index

Constants

View Source
const (
	DefaultCMDTimeout = "3m"

	ErrNoSelection       = "no selection, exiting"
	ErrInvalidNamespace  = "first argument must be a valid k8s namespace"
	ErrAutocompleteError = "autocomplete file walk errored"
)
View Source
const (
	ErrReadSethConfig      = "failed to read TOML config for havoc"
	ErrUnmarshalSethConfig = "failed to unmarshal TOML config for havoc"

	ErrFailureGroupIsNil      = "failure group must be specified in config"
	ErrLatencyGroupIsNil      = "latency group must be specified in config"
	ErrStressCPUGroupIsNil    = "stress cpu group must be specified in config"
	ErrStressMemoryGroupIsNil = "stress memory group must be specified in config"
	ErrFormat                 = "format error"
)
View Source
const (
	DefaultExperimentsDir           = "havoc-experiments"
	DefaultPodFailureDuration       = "1m"
	DefaultNetworkLatencyDuration   = "1m"
	DefaultNetworkPartitionDuration = "1m"
	DefaultNetworkPartitionLabel    = "network-partition-group"
	DefaultStressMemoryDuration     = "1m"
	DefaultStressMemoryWorkers      = 1
	DefaultStressMemoryAmount       = "512MB"
	DefaultStressCPUDuration        = "1m"
	DefaultStressCPUWorkers         = 1
	DefaultStressCPULoad            = 100
	DefaultNetworkLatency           = "300ms"
	DefaultMonkeyDuration           = "24h"
	DefaultMonkeyMode               = "seq"
	DefaultMonkeyCooldown           = "30s"
)
View Source
const (
	ErrParsingTemplate = "failed to parse Go text template"

	ErrExperimentTimeout = "waiting for experiment to finish timed out"
	ErrExperimentApply   = "error applying experiment manifest"
	ErrInvalidCustomKind = "invalid custom Kind of experiment"
)
View Source
const (
	ChaosTypeBlockchainSetHead = "blockchain_rewind_head"
	ChaosTypeFailure           = "failure"
	ChaosTypeGroupFailure      = "group-failure"
	ChaosTypeLatency           = "latency"
	ChaosTypeGroupLatency      = "group-latency"
	ChaosTypeStressMemory      = "memory"
	ChaosTypeStressGroupMemory = "group-memory"
	ChaosTypeStressCPU         = "cpu"
	ChaosTypeStressGroupCPU    = "group-cpu"
	ChaosTypePartitionExternal = "external"
	ChaosTypePartitionGroup    = "group-partition"
	PodChaosKind               = "PodChaos"
	NetworkChaosKind           = "NetworkChaos"
)
View Source
const (
	MonkeyModeSeq    = "seq"
	MonkeyModeRandom = "rand"

	ErrInvalidMode = "monkey mode is invalid, should be either \"seq\" or \"rand\""
)
View Source
const (
	ErrNoNamespace    = "no namespace found"
	ErrEmptyNamespace = "no pods found inside namespace, namespace is empty or check your filter"
)

Variables

View Source
var (
	DefaultGroupPercentage                 = []string{"10", "20", "30"}
	DefaultGroupFixed                      = []string{"1", "2", "3"}
	DefaultNetworkPartitionGroupPercentage = []string{"100"}
)
View Source
var (
	DefaultIgnoreGroupLabels = []string{
		"mainnet",
		"release",
		"intents.otterize.com",
		"pod-template-hash",
		"rollouts-pod-template-hash",
		"chain.link/app",
		"chain.link/cost-center",
		"chain.link/env",
		"chain.link/project",
		"chain.link/team",
		"app.kubernetes.io/part-of",
		"app.kubernetes.io/managed-by",
		"app.chain.link/product",
		"app.kubernetes.io/version",
		"app.chain.link/blockchain",
		"app.kubernetes.io/instance",
		"app.kubernetes.io/name",
	}
)
View Source
var (
	ExperimentsToCRDs = map[string]string{
		ChaosTypeFailure:           "podchaos.chaos-mesh.org",
		ChaosTypeGroupFailure:      "podchaos.chaos-mesh.org",
		PodChaosKind:               "podchaos.chaos-mesh.org",
		ChaosTypeLatency:           "networkchaos.chaos-mesh.org",
		ChaosTypeGroupLatency:      "networkchaos.chaos-mesh.org",
		NetworkChaosKind:           "networkchaos.chaos-mesh.org",
		ChaosTypeStressMemory:      "stresschaos.chaos-mesh.org",
		ChaosTypeStressGroupMemory: "stresschaos.chaos-mesh.org",
		ChaosTypeStressCPU:         "stresschaos.chaos-mesh.org",
		ChaosTypeStressGroupCPU:    "stresschaos.chaos-mesh.org",
		ChaosTypePartitionExternal: "networkchaos.chaos-mesh.org",
		ChaosTypePartitionGroup:    "networkchaos.chaos-mesh.org",
	}
)

Functions

func ExecCmd

func ExecCmd(command string) (string, error)

func InitDefaultLogging

func InitDefaultLogging()

func MarshalTemplate

func MarshalTemplate(jobSpec interface{}, name, templateString string) (string, error)

MarshalTemplate Helper to marshal templates

func RunCLI

func RunCLI(args []string) error

func SetGlobalLogger

func SetGlobalLogger(l zerolog.Logger)

Types

type ActionablePodInfo

type ActionablePodInfo struct {
	PodName  string
	Labels   []string
	HasGroup bool
}

ActionablePodInfo info about pod and labels for which we can generate a chaos experiment

type BlockchainRewindHead added in v0.1.3

type BlockchainRewindHead struct {
	Duration            string  `toml:"duration"`
	ExecutorPodPrefix   string  `toml:"executor_pod_prefix"`
	NodeInternalHTTPURL string  `toml:"node_internal_http_url"`
	Blocks              []int64 `toml:"blocks"`
}

type BlockchainRewindHeadExperiment added in v0.1.3

type BlockchainRewindHeadExperiment struct {
	ExperimentName      string    `yaml:"experimentName"`
	Metadata            *Metadata `yaml:"metadata"`
	Namespace           string    `yaml:"namespace"`
	PodName             string    `yaml:"podName"`
	ExecutorPodPrefix   string    `yaml:"executorPodPrefix"`
	NodeInternalHTTPURL string    `yaml:"nodeInternalHTTPURL"`
	Blocks              int64     `yaml:"blocks"`
}

func (BlockchainRewindHeadExperiment) String added in v0.1.3

type CRD added in v0.1.4

type CRD struct {
	Kind       string `yaml:"kind"`
	APIVersion string `yaml:"apiVersion"`
	Metadata   struct {
		Name      string `yaml:"name"`
		Namespace string `yaml:"namespace"`
	} `yaml:"metadata"`
	Spec interface{} `yaml:"spec"` // Use interface{} if the spec can have various structures
}

type ChaosSpecs

type ChaosSpecs struct {
	ExperimentsByType map[string]map[string]string
}

func (*ChaosSpecs) Dump

func (m *ChaosSpecs) Dump(dir string) error

type CommonExperimentMeta

type CommonExperimentMeta struct {
	Kind     string `yaml:"kind"`
	Metadata struct {
		Name      string `yaml:"name"`
		Namespace string `yaml:"namespace"`
	} `yaml:"metadata"`
}

type Config

type Config struct {
	Havoc *Havoc `toml:"havoc"`
}

func DefaultConfig

func DefaultConfig() *Config

func ReadConfig

func ReadConfig(path string) (*Config, error)

func (*Config) Validate

func (c *Config) Validate() []error

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

func NewController

func NewController(cfg *Config) (*Controller, error)

func (*Controller) AnnotateExperiment

func (m *Controller) AnnotateExperiment(a *ExperimentAction) error

AnnotateExperiment sends annotation marker to Grafana dashboard

func (*Controller) ApplyAndAnnotate

func (m *Controller) ApplyAndAnnotate(exp *NamedExperiment) error

func (*Controller) ApplyCustomKindChaosFile added in v0.1.3

func (m *Controller) ApplyCustomKindChaosFile(exp *NamedExperiment, chaosType string, wait bool) error

func (*Controller) ApplyExperiment added in v0.1.4

func (m *Controller) ApplyExperiment(exp *NamedExperiment, wait bool) error

func (*Controller) GenerateSpecs

func (m *Controller) GenerateSpecs(ns string) error

GenerateSpecs generates specs from namespace, should be used programmatically in tests

func (*Controller) GetPodsInfo

func (m *Controller) GetPodsInfo(namespace string) (*PodsListResponse, error)

GetPodsInfo gets info about all the pods in the namespace

func (*Controller) ReadExperimentsFromDir

func (m *Controller) ReadExperimentsFromDir(expTypes []string, dir string) ([]*NamedExperiment, error)

func (*Controller) Run

func (m *Controller) Run() error

func (*Controller) Stop

func (m *Controller) Stop() []error

func (*Controller) Wait

func (m *Controller) Wait() []error

type CurrentBlockResponse added in v0.1.3

type CurrentBlockResponse struct {
	Result string `json:"result"`
}

type EventJSONItemResponse

type EventJSONItemResponse struct {
	APIVersion     string    `json:"apiVersion"`
	Count          int       `json:"count"`
	EventTime      any       `json:"eventTime"`
	FirstTimestamp time.Time `json:"firstTimestamp"`
	InvolvedObject struct {
		APIVersion      string `json:"apiVersion"`
		Kind            string `json:"kind"`
		Name            string `json:"name"`
		Namespace       string `json:"namespace"`
		ResourceVersion string `json:"resourceVersion"`
		UID             string `json:"uid"`
	} `json:"involvedObject"`
	Kind          string    `json:"kind"`
	LastTimestamp time.Time `json:"lastTimestamp"`
	Message       string    `json:"message"`
	Metadata      struct {
		Annotations struct {
			ChaosMeshOrgType string `json:"chaos-mesh.org/type"`
		} `json:"annotations"`
		CreationTimestamp time.Time `json:"creationTimestamp"`
		Name              string    `json:"name"`
		Namespace         string    `json:"namespace"`
		ResourceVersion   string    `json:"resourceVersion"`
		UID               string    `json:"uid"`
	} `json:"metadata"`
	Reason             string `json:"reason"`
	ReportingComponent string `json:"reportingComponent"`
	ReportingInstance  string `json:"reportingInstance"`
	Source             struct {
		Component string `json:"component"`
	} `json:"source"`
	Type string `json:"type"`
}

type EventsJSONResponse

type EventsJSONResponse struct {
	APIVersion string                   `json:"apiVersion"`
	Items      []*EventJSONItemResponse `json:"items"`
	Kind       string                   `json:"kind"`
	Metadata   struct {
		ResourceVersion string `json:"resourceVersion"`
	} `json:"metadata"`
}

type ExperimentAction

type ExperimentAction struct {
	Name           string
	ExperimentKind string
	ExperimentSpec string
	TimeStart      int64
	TimeEnd        int64
}

type ExperimentAnnotationBody

type ExperimentAnnotationBody struct {
	DashboardUID string   `json:"dashboardUID"`
	Time         int64    `json:"time"`
	TimeEnd      int64    `json:"timeEnd"`
	Tags         []string `json:"tags"`
	Text         string   `json:"text"`
}

type ExternalTargets

type ExternalTargets struct {
	Duration string   `toml:"duration"`
	URLs     []string `toml:"urls"`
}

type Failure

type Failure struct {
	Duration        string   `toml:"duration"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type Grafana

type Grafana struct {
	URL           string   `toml:"grafana_url"`
	Token         string   `toml:"grafana_token"`
	DashboardUIDs []string `toml:"dashboard_uids"`
}

type Havoc

type Havoc struct {
	Dir                  string                `toml:"dir"`
	ExperimentTypes      []string              `toml:"experiment_types"`
	NamespaceLabelFilter string                `toml:"namespace_label_filter"`
	IgnoredPods          []string              `toml:"ignore_pods"`
	IgnoreGroupLabels    []string              `toml:"ignore_group_labels"`
	Failure              *Failure              `toml:"failure"`
	Latency              *Latency              `toml:"latency"`
	NetworkPartition     *NetworkPartition     `toml:"network_partition"`
	StressMemory         *StressMemory         `toml:"stress_memory"`
	StressCPU            *StressCPU            `toml:"stress_cpu"`
	ExternalTargets      *ExternalTargets      `toml:"external_targets"`
	BlockchainRewindHead *BlockchainRewindHead `toml:"blockchain_rewind_head"`
	Monkey               *Monkey               `toml:"monkey"`
	Grafana              *Grafana              `toml:"grafana"`
}

type Latency

type Latency struct {
	Duration        string   `toml:"duration"`
	Latency         string   `toml:"latency"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type ManifestPart

type ManifestPart struct {
	Kind                string
	Name                string
	LabelSelectors      []*ActionablePodInfo
	AnnotationSelectors []*ActionablePodInfo
	FlattenedManifest   map[string]interface{}
}

type Metadata added in v0.1.4

type Metadata struct {
	Name   string            `json:"name"`
	Labels map[string]string `json:"labels"`
}

type Monkey

type Monkey struct {
	Duration string `toml:"duration"`
	Cooldown string `toml:"cooldown"`
	Mode     string `toml:"mode"`
}

type NamedExperiment

type NamedExperiment struct {
	CRD
	Name     string
	Path     string
	CRDBytes []byte
}

func NewNamedExperiment added in v0.1.4

func NewNamedExperiment(expPath string) (*NamedExperiment, error)

type NetworkChaosExperiment

type NetworkChaosExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Duration       string
	Latency        string
	PodName        string
	Selector       string
}

func (NetworkChaosExperiment) String

func (m NetworkChaosExperiment) String() (string, error)

type NetworkChaosExternalPartitionExperiment

type NetworkChaosExternalPartitionExperiment struct {
	ExperimentName string
	Namespace      string
	Duration       string
	PodName        string
	ExternalURL    string
}

func (NetworkChaosExternalPartitionExperiment) String

type NetworkChaosGroupPartitionExperiment

type NetworkChaosGroupPartitionExperiment struct {
	ExperimentName string
	ModeTo         string
	ModeToValue    string
	ModeFrom       string
	ModeFromValue  string
	Direction      string
	Namespace      string
	Duration       string
	SelectorFrom   string
	SelectorTo     string
}

func (NetworkChaosGroupPartitionExperiment) String

type NetworkPartition

type NetworkPartition struct {
	Duration        string   `toml:"duration"`
	Label           string   `toml:"label"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type PodFailureExperiment

type PodFailureExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Duration       string
	PodName        string
	Selector       string
}

func (PodFailureExperiment) String

func (m PodFailureExperiment) String() (string, error)

type PodResponse

type PodResponse struct {
	Metadata struct {
		Name   string            `json:"name"`
		Labels map[string]string `json:"labels"`
	} `json:"metadata"`
}

PodResponse pod info response from kubectl in JSON

type PodStressCPUExperiment

type PodStressCPUExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Workers        int
	Load           int
	Duration       string
	PodName        string
	Selector       string
}

func (PodStressCPUExperiment) String

func (m PodStressCPUExperiment) String() (string, error)

type PodStressMemoryExperiment

type PodStressMemoryExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Workers        int
	Memory         string
	Duration       string
	PodName        string
	Selector       string
}

func (PodStressMemoryExperiment) String

func (m PodStressMemoryExperiment) String() (string, error)

type PodsListResponse

type PodsListResponse struct {
	Items []*PodResponse `json:"items"`
}

PodsListResponse pod list response from kubectl in JSON

type StressCPU

type StressCPU struct {
	Duration        string   `toml:"duration"`
	Workers         int      `toml:"workers"`
	Load            int      `toml:"load"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type StressMemory

type StressMemory struct {
	Duration        string   `toml:"duration"`
	Workers         int      `toml:"workers"`
	Memory          string   `toml:"memory"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

Directories

Path Synopsis
k8schaos module
lib module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL