chronos

chronos is a continuous benchmarking tool that tracks performance metrics over time

Features

Here are the features that chronos provides:

• A UI with benchmark grouping, filtering, point comparison, and chart screenshots;
• Summarization of pull request benchmarks (and comparison with the latest one) in GitHub Job Summary;

Rationale

There is at least one GitHub Action that serves the same purpose (github-action-benchmark), but I was not satisfied with it for several reasons:

• Inconvenient UI with little to no interactivity;
• The parser of benchmark output does not always work correctly and introduces weird units (e.g., B/op 437 ns/op);
• The way of configuring the GitHub Action seemed a little cumbersome (but that's very subjective);

Setting chronos up

Preconfiguration

Since chronos operates on two Git branches (by default these are chronos-storage and gh-pages), you need to create and push these branches to the remote:

git checkout --orphan chronos-storage \
  && git commit --allow-empty -m "[chronos] init" \
  && git push origin chronos-storage

You must run the same commands for your GitHub Pages branch (if it has not been created yet).

git checkout --orphan gh-pages \
  && git commit --allow-empty -m "[chronos] init" \
  && git push origin gh-pages

Action Configuration

Please refer to this workflow file to understand how to incorporate chronos into your GitHub Workflows. Here you can find what inputs chronos expects.

Don't worry - it's dead simple! Just don't forget to pin the release version.

Notes

I should mention that I used AI assistance to build the UI for data visualization (I am far from frontend development), but I performed common-sense checks of the code that was generated by the LLM - it's perfectly manageable.

Roadmap

• Support benchstat format to mitigate noisy results (https://github.com/dkharms/chronos/issues/13);
• Support compare (?) action which will fail when incoming benchmark results are significantly worse;
• Support option to configure maximum count of data points on charts (https://github.com/dkharms/chronos/pull/17);