loadtest

Load testing framework for golang

Disclaimer: This is still in initial development. It should be pretty stable at this point, but API changes may still occur before a 1.0.0 release is made.

Synopsis

Provides a framework for writing and running load test simulations that allow testing a system (say a web service) against a configurable load that consists of multiple individual tests being run sequentially or in sequence.

Installation

go get github.com/cjbearman/loadtest

Writing tests

A test would typically perform a single action against the system under test, such as performing a REST call. The test is automatically measured for time taken and must indicate a pass/fail outcome.

A test is any valid go struct that conforms to the Test interface.

type Test interface {
	// Run will be called to execute the test
	// The test should populate the rest object and, if needed, utilize testContext
	// for parameters necessary for the operation of the specific iteration of the test
	// If it returns a test context, a copy will be sent to any delegate runners attached
	// to the current runner to trigger next tests
	Run(result *Result, testContext TestContext) *TestContext
}

The Run method is called, with a Result object. The simplest test, which does nothing but pass, would look like this:

type DoesNothingTest struct {}
func (*DoesNothingTest) Run(result *Result, testContext TestContext) *TestContext {
  result.Pass = true
  return nil
}

The test context may contain information necessary to run the tests which would typically have been provided by prior tests.

Running tests

To run tests, you use a runner. All runners are configured with the following parameters

• The test to run (anything implementing the Test interface)
• The rate at which the test will run. Rate is expressed as the BHCA parameter, which is the target rate of executions over one hour. For example 3,600 BHCA would be one test per second.
• The maximum number of simultaneous tests that may be in progress at any one time (unlimited if configured at 0). If this number is exceeded, the test is throttled and another instance will not be started until the next scheduled interval is reached (if sufficient space is then available).

There are two types of runner.

• Initial runners run a test a pre-determined number of times, using the previously described parameters. Each test iteration receives an empty test context.
• Delegate runners are triggered by a completed test from any prior runner. Any runner may be given a list of "next" runners that should be triggered by a test from that runner that returns a context. When a test triggers delegate runners, a copy of the context from that test (along with any values written to the context) is passed to the test in the delegate runner.

Sequencing runners

Whilst tests and runners can be created and run entirely in code, it is usually prefferable to describe tests and runners using the sequencer, which accepts a YAML definition. This allows a single executable to run a variety of easily configurable test scenarios using the same set of tests, or subset thereof.

Since golang lacks the ability to identify all Test implementations via reflection, in order to use the sequencer it is necessary to register all tests you create with the registry by giving them a name with which they can be referenced from the sequencer.

import "github.com/cjbearman/loadtest/registry"

...
  registry.Register("does-nothing", &DoesNothingTest{})

For a test to be registered in the registry, it must also implement TestCreator which is called to create a new unique copy of the test:

type TestCreator interface {
        // NewInstance creates a new instance of a test
        NewInstance() Test
}

The TestCreator implementation does not have to be the Test implementation, but that is usually most convenient. For example, see nop.go for an exmaple test that also defines this interface.

Bootstrapping your executable

All you need to do is to have your main() method register your tests and then create and execute a sequence using a YAML file. Here is how you would do that.

package main

import (
	"log"
	"os"

	"github.com/cjbearman/loadtest/registry"
	"github.com/cjbearman/loadtest/sequence"
)

func main() {
	if len(os.Args) != 2 {
		log.Fatalf("Please provide the name of a single yaml file on the command line")
	}

	// Register all test cases first
	registry.Register("test-1", &TestOneImpl{})
	registry.Register("test-2", &TestTwoImpl{})
	registry.Register("test-3", &TestThreeImpl{})

	// Now we can load the sequence (tests must be registered before this)
	seq, err := sequence.NewSequenceFromFile(os.Args[1])
	if err != nil {
		log.Fatalf("Failed to process YAML sequence: %v", err)
	}

	// And run the sequence
	seq.Execute()
}

Test initialization

Each runner will receive a single instance of a test. The Run method will be called once for each iteration of the test within that runner.

You can provide parameters to the test from your YAML by implementing the InitializableTest interface on your test:

type InitializableTest interface {
	// Init is called after test creation and before the runner is executed
	// if the test supports InitializableTest
	Init(params map[string]interface{})
}

You can then store those parameters in the test struct and use them in the Run method. How to do this is described below

Whilst you can store other data in the struct from the Run method, be careful. There may be multiple instances of the Run method for the same struct simultaneously executing, so take care to use sync.Mutex to protect things.

Writing the YAML sequence

The YAML sequence consists of several sections:

Monitor flag

The monitor flag can be set to true to have a live monitor display of all executors as the sequence runs. If false, the sequence runs silently.

Common parameters

The common section accepts key/value configuration pairs that will be passed to the Init method of all InitializableTest implementations.

These are defined as:

• Key (mandatory) - The key for the configuration element
• Value (optional) - The value for the configuration element
• Env (optional) - The name of an environment variable from which the configuration element may be taken

You must provide either Value or Env, or both.

If you provide Env only, then the value of the key will be whatever is defined by that environment variable. If the environment variable is not present, the test will panic and the sequence will not run.

If you provide Value only, the value is used exclusively.

If you provide Env and Value, the value of the env var is used, if defined in the environment, else value is used.

Definitions section

The definition section defines the runners you want to use.

You must define a name for each runner.

Each runner is defined along with it's BHCA and max in progress (optional) settings. The test (as provided by the registry) name for the runner must be provided.

Optionally you can provide a params section which is formatted the same as the common section to provide parameters specific to this runner. If a parameter here has the same key as a common parameter, it's value will override the common parameter.

You may optionaly provide a next section which gives the name of the "next" executors to which test contexts will be passed for tests that return a context.

You may also provide filenames for three types of reports:

• Txt: A textual report of the runner including details of all tests. Any values in the test context at the conclusion of the text will be included.
• Csv: A CSV report containing the iteration number, start (ns/epoch) and end (ns/epoch) execution time of each test
• Svg: An SVG (scalable vector graphics) histogram representing the latency of tests run by the executor.

For Txt and Csv you may provide "console" instead of a filename, and the report will be output to the console at the conclusion of the sequence.

Execution section

The execution section defines how to run the executors.

All executors must be "run", at which point they start processing work. Multiple executors can be run concurrently.

All executors must be "wait"ed. Waiting on an executor will prevent the execution sequence from moving forward until all tests scheduled by that executor have completed. When waiting a delegate executor, the delegate executor will no longer accept new work once the wait state is entered.

For example, the following execution sequence:

- run runner1
- run runner2
- wait runner1
- wait runner2
- run runner3
- wait runner3

With the above sequence, runner1 and runner2 run simultaneously to completion. Once both have completed, runner3 will run.

Note that when using delegate runners, work may be sent to it before it starts to run and whilst it's running, work is queued and executed in FIFO order.

YAML example

# We want to see live monitoring
monitor: true

# Definition of common parameters that will be passed to ALL tests
common:
  # KEY1 will have the fixed "value1"
  - key: KEY1
    value: value1

  # KEY2 will have "value2" unless ENV_FOR_KEY_2 is defined, in which case the value is taken from that ENV VAR
  - key: KEY2
    value: value2
    env: ENV_FOR_KEY_2

# Runner definitions
definitions:

# runner1 will operate at 3600 ops per hour (1 per sec)
# As an initial runner, it will run it's test 10 times (iterations)
# If the test (nop) returns true, then it's test context is handed off to runner2
- name: runner1
  type: initial
  bhca: 3600
  iterations: 10
  test: nop
  next: 
    - runner2

  # These parameters will only be passed to the test for this runner
  params:
    - key: KEY3
      value: value3

# runner2 is a delegate, it is accepting tests from runner1
# It will permit only a single instance of the test to be running at any time
- name: runner2
  type: delegate
  bhca: 3600
  max: 1
  test: seq-test

# runner3 runs 10 more iterations of a test
- name: runner3
  type: initial
  bhca: 3600
  iterations: 10
  test: nop
  reports:
    # CSV report to file
    csv: runner3.csv
    # TXT report to console
    txt: console
    # SVG Histogram to file
    svg: runner3.svg

# Executions will have runner1 and runner2 run concurrently.
# Runner3 will run once both runner1 and runner2 have finished
executions:
  - run runner1
  - run runner2
  - wait runner1
  - wait runner2
  - run runner3
  - wait runner3

HTTP Testing

The test package includes a test named HttpTest. This can be used for simple load generation against HTTP endpoints.

Make sure and register it in your executable

registry.Register("http", &test.HttpTest{})

You must provide the URL to be tested as "URL" parameter in the definition for the runner. It will assume a GET operation, and assert a pass condition provided the URL responds with 200 OK within 5 seconds.

There are various other parmaeters you can tweak through parameters to change the operation.

This is an example sequence using this test, to concurrently run operations against two endpoints, one expecting 200 OK, the other 404 Not Found:

monitor: true
definitions:
  - name: page-found
    type: initial
    bhca: 360000
    max: 10
    iterations: 100
    test: http
    next: 
      - page-nf
    params:
    - key: URL
      value: https://www.chrisjb.com/index.html
  - name: page-nf
    type: delegate
    bhca: 360000
    max: 10
    test: http
    params:
    - key: URL
      value: https://www.chrisjb.com/nf.html
    - key: EXPECT
      value: 404

executions:
  - run page-found
  - run page-nf
  - wait page-found
  - wait page-nf

Full example

Using the above YAML, you can look in example for a complete example.

> cd example
> go build loadtest.go
> ./loadtest http.yaml
      Name :    TOTAL     PASS     FAIL INPROG    HWM  Throt.    Avg. Lat    Max. Lat      BHCA [COMPLETE]
----------------------------------------------------------------------------------------------------------
page-found :      100      100        0      0     10       4      41.150     289.847    275356 [COMPLETE]
   page-nf :      100      100        0      0      1       0      27.263      62.437     36248 [COMPLETE]

The monitoring information (as shown above) will be displayed throughout the test. If your terminal supports it, colors will be used and it will be updated in place. If your terminal does not support it, it is repeatedly displayed at a slower rate.

The fields in the report are as follows:

• Name : The name of the runner
• TOTAL : Total tests run (blue)
• PASS : Total tests passed (green)
• FAIL : Total tests failed (turns red if >1)
• INPROG : Number of tests currently simultaneously in progress
• HWM : High water mark (maximum number of tests that have been in progress simultaneously at any time). Green, turns red if tests get throttled.
• Throt.: The count of throttles (test starts delayed because max iterations was reached)
• Avg. Lat: Average latency of tests
• Max. Lat: The latency of the longest running test
• BHCA: The busy hour call attempt rate being achieved
• [COMPLETE]: Flags [COMPLETE] when the runner completes