oasis-test-runner

Oasis Test Runner

The Oasis Test Runner initializes and executes different types of tests (e.g. end-to-end, remote signer) also referred to as scenarios.

To list all available scenarios and their corresponding parameters, use:

oasis-test-runner list

If no flags are provided, all scenarios are executed.

To run a specific scenario, e.g e2e/runtime/runtime-dynamic, pass the --scenario flag (or its short version, -s) followed by scenario's name:

oasis-test-runner --scenario e2e/runtime/runtime-dynamic

Benchmarking

To benchmark scenarios, set the --metrics.address flag to the address of the Prometheus push gateway along with the --metrics.interval flag. Additionally, you can set scenario-specific parameters and the number of runs of each scenario with the --num_runs flag.

Benchmark analysis with oasis-test-runner cmp command

The cmp sub-command connects to the Prometheus server instance containing benchmark results generated by Oasis Test Runner and the corresponding Oasis Node workers. It compares the benchmark results of the last (source) and pre-last (target) scenario batch (also called instance). You need to pass the address of the Prometheus query server using --metrics.address flag, e.g.:

oasis-test-runner cmp \
  --metrics.address http://prometheus.myorg.com:9090

By default, the oasis-test-runner cmp command will fetch the results of all scenarios and metrics supported by the Oasis Test Runner. If you want to compare specific metric(s) and scenario(s), set the --metrics and --scenario flags appropriately, e.g.:

oasis-test-runner cmp \
  --metrics time \
  --scenario e2e/runtime/multiple-runtimes

The cmp sub-command takes thresholds for each metric into account. Currently, avg_ratio and max_ratio are supported which correspond to the average and maximum ratio of metric values of all scenario runs in the benchmark batch.

For each ratio, you can set the max_threshold and min_threshold. The former specifies which ratio should not be exceeded and the latter which ratio must be reached. If a value is not within these thresholds, the oasis-test-runner cmp command will exit with an error code.

This is useful for integration into CI pipelines. Thresholds can be set using the following flag specification:

--{min|max}_threshold.<metric name>.{avg|max}_ratio

For example:

oasis-test-runner cmp \
  --metrics time \
  --scenario e2e/runtime/multiple-runtimes \
  --max_threshold.time.avg_ratio 1.1

will require that the average duration of the scenario in the last benchmark batch should be at most 10% slower than from the pre-last benchmark batch.

If you are developing or improving a feature in a separate git branch, you will want to perform benchmarks and compare the results of your branch to the ones from the master branch. The Oasis Test Runner automatically sends information of the git branch it was compiled on to Prometheus, so all you need to do is perform benchmarks on both, the master and a feature branch, using the same Prometheus instance.

Afterwards, pass the name of source and target branch names to the cmp sub-command and it will compare the last benchmark batch from each git branch (respectively):

oasis-test-runner cmp \
  --metrics.source.git_branch=feature_branch \
  --metrics.target.git_branch=master

For detailed help, run:

oasis-test-runner cmp --help