havoc

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 14, 2024 License: MIT Imports: 26 Imported by: 0

README

Havoc

Havoc is a tool that introspects your k8s namespace and generates a ChaosMesh CRDs suite for you

You can use havoc as a CLI to quickly test hypothesis or run it in "monkey" mode with your load tests and have Grafana annotations

Goals

  • Make chaos testing easy by generating most of the things automatically just by looking at your namespace
  • Easy integration with Grafana to understand how chaos affects your services
  • Be easy to use both programmatically and as a CLI

How it works

Havoc generates groups of experiments based just on your pods and labels found in namespace

Single pod experiments:

  • PodFailure
  • NetworkChaos (Pod latency)
  • Stress (Memory)
  • Stress (CPU)
  • External service failure (Network partition)

Group experiments:

  • Group failure
  • Group latency
  • Group CPU
  • Group memory

You can generate default chaos suite by configuring havoc then set dir param and add your custom experiments, then run monkey to test your services

Install

Please use GitHub releases of this repo Download latest release

You need kubectl to available on your machine

If you wish Grafana integration, please set env variables (optional)

HAVOC_LOG_LEVEL={warn,info,debug,trace}
GRAFANA_URL="..."
GRAFANA_TOKEN="..."
DASHBOARD_NAME="..."

Manual usage

Generate default experiments for your namespace

havoc -c havoc.toml generate [namespace]

Check this section for ignore_pods and ignore_group_labels, default settings should be reasonable, however, you can tweak them

This will create havoc-experiments dir, then you can choose from recommended experiments

havoc -c havoc.toml apply

Monkey mode

You can run havoc as an automated sequential or randomized suite

havoc -c havoc.toml run [namespace]

See [havoc.monkey] config here

Programmatic usage

See how you can use recommended experiments from code in examples

Custom experiments

Havoc is just a generator and a module that reads your dir = $mydir from config

If you wish to add custom experiments written by hand create your custom directory and add experiments

Experiments will be executed in lexicographic order, however, for custom experiments there are 2 simple rules:

  • directory names must be in ["external", "failure", "latency", "cpu", "memory", "group-failure", "group-latency"]
  • metadata.name should be equal to your experiment filename

When you are using run monkey command, if directory is not empty havoc won't automatically generate experiments, so you can extend generated experiments with your custom modifications

Developing

We are using nix

Enter the shell

nix develop

Why not to use ChaosMesh UI/API instead of CRDs?

ChaosMesh UI/API is great, but it has some downsides:

  • No OpenAPI spec, hard to integrate
  • No dynamic generation for a namespace, you need to rely on labels that might change
  • Writing chaos experiments is tedious, in most cases you just copy-paste a lot, or you can forget something
  • Workflows validation is broken
  • Can't mix chaos experiments and API calls
  • No straightforward integration with load testing tools, it's easy to run an experiment, but it's hard to validate it right away without additional code
  • Can't check chaos experiments statuses through API and fail the test, need to use k8s
  • Experiments created from YAML and UI and not always compatible

Disclaimer

This software represents an educational example of how we break a Chainlink system, product, or service and is provided to demonstrate how to interact with Chainlink’s systems. This code is provided “AS IS” and “AS AVAILABLE” without warranties of any kind, it has not been audited, and it may be missing key checks or error handling to make the usage of the system, product or service more clear. Do not use the code in this example in a production environment without completing your own audits and application of best practices. Neither Chainlink Labs, the Chainlink Foundation, nor Chainlink node operators are responsible for unintended outputs that are generated due to errors in code.

Documentation

Index

Constants

View Source
const (
	DefaultCMDTimeout = "3m"

	ErrNoSelection       = "no selection, exiting"
	ErrInvalidNamespace  = "first argument must be a valid k8s namespace"
	ErrAutocompleteError = "autocomplete file walk errored"
)
View Source
const (
	ErrReadSethConfig      = "failed to read TOML config for havoc"
	ErrUnmarshalSethConfig = "failed to unmarshal TOML config for havoc"

	ErrFailureGroupIsNil      = "failure group must be specified in config"
	ErrLatencyGroupIsNil      = "latency group must be specified in config"
	ErrStressCPUGroupIsNil    = "stress cpu group must be specified in config"
	ErrStressMemoryGroupIsNil = "stress memory group must be specified in config"
	ErrFormat                 = "format error"
)
View Source
const (
	DefaultExperimentsDir           = "havoc-experiments"
	DefaultPodFailureDuration       = "1m"
	DefaultNetworkLatencyDuration   = "1m"
	DefaultNetworkPartitionDuration = "1m"
	DefaultNetworkPartitionLabel    = "network-partition-group"
	DefaultStressMemoryDuration     = "1m"
	DefaultStressMemoryWorkers      = 1
	DefaultStressMemoryAmount       = "512MB"
	DefaultStressCPUDuration        = "1m"
	DefaultStressCPUWorkers         = 1
	DefaultStressCPULoad            = 100
	DefaultNetworkLatency           = "300ms"
	DefaultMonkeyDuration           = "24h"
	DefaultMonkeyMode               = "seq"
	DefaultMonkeyCooldown           = "30s"
)
View Source
const (
	ErrParsingTemplate = "failed to parse Go text template"

	ErrExperimentTimeout   = "waiting for experiment to finish timed out"
	ErrExperimentApply     = "error applying experiment manifest"
	ErrEmptyExperimentsDir = "no experiments have been found, dir is empty, check 'experiment_types' setting in your config, they should match"
)
View Source
const (
	ChaosTypeFailure           = "failure"
	ChaosTypeGroupFailure      = "group-failure"
	ChaosTypeLatency           = "latency"
	ChaosTypeGroupLatency      = "group-latency"
	ChaosTypeStressMemory      = "memory"
	ChaosTypeStressGroupMemory = "group-memory"
	ChaosTypeStressCPU         = "cpu"
	ChaosTypeStressGroupCPU    = "group-cpu"
	ChaosTypePartitionExternal = "external"
	ChaosTypePartitionGroup    = "group-partition"
)
View Source
const (
	MonkeyModeSeq    = "seq"
	MonkeyModeRandom = "rand"

	ErrInvalidMode = "monkey mode is invalid, should be either \"seq\" or \"rand\""
)
View Source
const (
	ErrNoNamespace    = "no namespace found"
	ErrEmptyNamespace = "no pods found inside namespace, namespace is empty or check your filter"
)

Variables

View Source
var (
	DefaultGroupPercentage                 = []string{"10", "20", "30"}
	DefaultGroupFixed                      = []string{"1", "2", "3"}
	DefaultNetworkPartitionGroupPercentage = []string{"100"}
)
View Source
var (
	DefaultIgnoreGroupLabels = []string{
		"mainnet",
		"release",
		"intents.otterize.com",
		"pod-template-hash",
		"rollouts-pod-template-hash",
		"chain.link/app",
		"chain.link/cost-center",
		"chain.link/env",
		"chain.link/project",
		"chain.link/team",
		"app.kubernetes.io/part-of",
		"app.kubernetes.io/managed-by",
		"app.chain.link/product",
		"app.kubernetes.io/version",
		"app.chain.link/blockchain",
		"app.kubernetes.io/instance",
		"app.kubernetes.io/name",
	}
)
View Source
var (
	ExperimentsToCRDs = map[string]string{
		ChaosTypeFailure:           "podchaos.chaos-mesh.org",
		ChaosTypeGroupFailure:      "podchaos.chaos-mesh.org",
		ChaosTypeLatency:           "networkchaos.chaos-mesh.org",
		ChaosTypeGroupLatency:      "networkchaos.chaos-mesh.org",
		ChaosTypeStressMemory:      "stresschaos.chaos-mesh.org",
		ChaosTypeStressGroupMemory: "stresschaos.chaos-mesh.org",
		ChaosTypeStressCPU:         "stresschaos.chaos-mesh.org",
		ChaosTypeStressGroupCPU:    "stresschaos.chaos-mesh.org",
		ChaosTypePartitionExternal: "networkchaos.chaos-mesh.org",
		ChaosTypePartitionGroup:    "networkchaos.chaos-mesh.org",
	}
)

Functions

func ExecCmd

func ExecCmd(command string) (string, error)

func InitDefaultLogging

func InitDefaultLogging()

func MarshalTemplate

func MarshalTemplate(jobSpec interface{}, name, templateString string) (string, error)

MarshalTemplate Helper to marshal templates

func RunCLI

func RunCLI(args []string) error

func SetGlobalLogger

func SetGlobalLogger(l zerolog.Logger)

Types

type ActionablePodInfo

type ActionablePodInfo struct {
	PodName  string
	Labels   []string
	HasGroup bool
}

ActionablePodInfo info about pod and labels for which we can generate a chaos experiment

type ChaosSpecs

type ChaosSpecs struct {
	ExperimentsByType map[string]map[string]string
}

func (*ChaosSpecs) Dump

func (m *ChaosSpecs) Dump(dir string) error

type CommonExperimentMeta

type CommonExperimentMeta struct {
	Kind     string `yaml:"string"`
	Metadata struct {
		Name      string `yaml:"name"`
		Namespace string `yaml:"namespace"`
	} `yaml:"metadata"`
}

type Config

type Config struct {
	Havoc *Havoc `toml:"havoc"`
}

func DefaultConfig

func DefaultConfig() *Config

func ReadConfig

func ReadConfig(path string) (*Config, error)

func (*Config) Validate

func (c *Config) Validate() []error

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

func NewController

func NewController(cfg *Config) (*Controller, error)

func (*Controller) AnnotateExperiment

func (m *Controller) AnnotateExperiment(a *ExperimentAction) error

AnnotateExperiment sends annotation marker to Grafana dashboard

func (*Controller) ApplyAndAnnotate

func (m *Controller) ApplyAndAnnotate(exp *NamedExperiment) error

func (*Controller) ApplyChaosFile

func (m *Controller) ApplyChaosFile(chaosType string, expName string, wait bool) error

func (*Controller) GenerateSpecs

func (m *Controller) GenerateSpecs(ns string) error

GenerateSpecs generates specs from namespace, should be used programmatically in tests

func (*Controller) GetPodsInfo

func (m *Controller) GetPodsInfo(namespace string) (*PodsListResponse, error)

GetPodsInfo gets info about all the pods in the namespace

func (*Controller) ReadExperimentsFromDir

func (m *Controller) ReadExperimentsFromDir(expTypes []string, dir string) ([]*NamedExperiment, error)

func (*Controller) Run

func (m *Controller) Run() error

func (*Controller) Stop

func (m *Controller) Stop() []error

func (*Controller) Wait

func (m *Controller) Wait() []error

type EventJSONItemResponse

type EventJSONItemResponse struct {
	APIVersion     string    `json:"apiVersion"`
	Count          int       `json:"count"`
	EventTime      any       `json:"eventTime"`
	FirstTimestamp time.Time `json:"firstTimestamp"`
	InvolvedObject struct {
		APIVersion      string `json:"apiVersion"`
		Kind            string `json:"kind"`
		Name            string `json:"name"`
		Namespace       string `json:"namespace"`
		ResourceVersion string `json:"resourceVersion"`
		UID             string `json:"uid"`
	} `json:"involvedObject"`
	Kind          string    `json:"kind"`
	LastTimestamp time.Time `json:"lastTimestamp"`
	Message       string    `json:"message"`
	Metadata      struct {
		Annotations struct {
			ChaosMeshOrgType string `json:"chaos-mesh.org/type"`
		} `json:"annotations"`
		CreationTimestamp time.Time `json:"creationTimestamp"`
		Name              string    `json:"name"`
		Namespace         string    `json:"namespace"`
		ResourceVersion   string    `json:"resourceVersion"`
		UID               string    `json:"uid"`
	} `json:"metadata"`
	Reason             string `json:"reason"`
	ReportingComponent string `json:"reportingComponent"`
	ReportingInstance  string `json:"reportingInstance"`
	Source             struct {
		Component string `json:"component"`
	} `json:"source"`
	Type string `json:"type"`
}

type EventsJSONResponse

type EventsJSONResponse struct {
	APIVersion string                   `json:"apiVersion"`
	Items      []*EventJSONItemResponse `json:"items"`
	Kind       string                   `json:"kind"`
	Metadata   struct {
		ResourceVersion string `json:"resourceVersion"`
	} `json:"metadata"`
}

type ExperimentAction

type ExperimentAction struct {
	Name           string
	ExperimentType string
	ExperimentSpec string
	TimeStart      int64
	TimeEnd        int64
}

type ExperimentAnnotationBody

type ExperimentAnnotationBody struct {
	DashboardUID string   `json:"dashboardUID"`
	Time         int64    `json:"time"`
	TimeEnd      int64    `json:"timeEnd"`
	Tags         []string `json:"tags"`
	Text         string   `json:"text"`
}

type ExternalTargets

type ExternalTargets struct {
	Duration string   `toml:"duration"`
	URLs     []string `toml:"urls"`
}

type Failure

type Failure struct {
	Duration        string   `toml:"duration"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type Grafana

type Grafana struct {
	URL           string `toml:"grafana_url"`
	Token         string `toml:"grafana_token"`
	DashboardName string `toml:"dashboard_name"`
}

type Havoc

type Havoc struct {
	Dir                  string            `toml:"dir"`
	ExperimentTypes      []string          `toml:"experiment_types"`
	NamespaceLabelFilter string            `toml:"namespace_label_filter"`
	IgnoredPods          []string          `toml:"ignore_pods"`
	IgnoreGroupLabels    []string          `toml:"ignore_group_labels"`
	Failure              *Failure          `toml:"failure"`
	Latency              *Latency          `toml:"latency"`
	NetworkPartition     *NetworkPartition `toml:"network_partition"`
	StressMemory         *StressMemory     `toml:"stress_memory"`
	StressCPU            *StressCPU        `toml:"stress_cpu"`
	ExternalTargets      *ExternalTargets  `toml:"external_targets"`
	Monkey               *Monkey           `toml:"monkey"`
	Grafana              *Grafana          `toml:"grafana"`
}

type Latency

type Latency struct {
	Duration        string   `toml:"duration"`
	Latency         string   `toml:"latency"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type ManifestPart

type ManifestPart struct {
	Kind                string
	Name                string
	LabelSelectors      []*ActionablePodInfo
	AnnotationSelectors []*ActionablePodInfo
	FlattenedManifest   map[string]interface{}
}

type Monkey

type Monkey struct {
	Duration string `toml:"duration"`
	Cooldown string `toml:"cooldown"`
	Mode     string `toml:"mode"`
}

type NamedExperiment

type NamedExperiment struct {
	Name     string
	Type     string
	Manifest string
}

type NetworkChaosExperiment

type NetworkChaosExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Duration       string
	Latency        string
	PodName        string
	Selector       string
}

func (NetworkChaosExperiment) String

func (m NetworkChaosExperiment) String() (string, error)

type NetworkChaosExternalPartitionExperiment

type NetworkChaosExternalPartitionExperiment struct {
	ExperimentName string
	Namespace      string
	Duration       string
	PodName        string
	ExternalURL    string
}

func (NetworkChaosExternalPartitionExperiment) String

type NetworkChaosGroupPartitionExperiment

type NetworkChaosGroupPartitionExperiment struct {
	ExperimentName string
	ModeTo         string
	ModeToValue    string
	ModeFrom       string
	ModeFromValue  string
	Direction      string
	Namespace      string
	Duration       string
	SelectorFrom   string
	SelectorTo     string
}

func (NetworkChaosGroupPartitionExperiment) String

type NetworkPartition

type NetworkPartition struct {
	Duration        string   `toml:"duration"`
	Label           string   `toml:"label"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type PodFailureExperiment

type PodFailureExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Duration       string
	PodName        string
	Selector       string
}

func (PodFailureExperiment) String

func (m PodFailureExperiment) String() (string, error)

type PodResponse

type PodResponse struct {
	Metadata struct {
		Name   string            `json:"name"`
		Labels map[string]string `json:"labels"`
	} `json:"metadata"`
}

PodResponse pod info response from kubectl in JSON

type PodStressCPUExperiment

type PodStressCPUExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Workers        int
	Load           int
	Duration       string
	PodName        string
	Selector       string
}

func (PodStressCPUExperiment) String

func (m PodStressCPUExperiment) String() (string, error)

type PodStressMemoryExperiment

type PodStressMemoryExperiment struct {
	ExperimentName string
	Mode           string
	ModeValue      string
	Namespace      string
	Workers        int
	Memory         string
	Duration       string
	PodName        string
	Selector       string
}

func (PodStressMemoryExperiment) String

func (m PodStressMemoryExperiment) String() (string, error)

type PodsListResponse

type PodsListResponse struct {
	Items []*PodResponse `json:"items"`
}

PodsListResponse pod list response from kubectl in JSON

type StressCPU

type StressCPU struct {
	Duration        string   `toml:"duration"`
	Workers         int      `toml:"workers"`
	Load            int      `toml:"load"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

type StressMemory

type StressMemory struct {
	Duration        string   `toml:"duration"`
	Workers         int      `toml:"workers"`
	Memory          string   `toml:"memory"`
	GroupPercentage []string `toml:"group_percentage"`
	GroupFixed      []string `toml:"group_fixed"`
}

Directories

Path Synopsis
k8schaos module
lib module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL