antithesis

Antithesis Testing

This package supports testing with Antithesis, a SaaS offering that enables deployment of distributed systems (such as Lux) to a deterministic and simulated environment that enables discovery and reproduction of anomalous behavior.

Package details

Instrumentation

Software running in Antithesis's environment must be instrumented to take full advantage of the supported traceability. Since the Antithesis Go SDK only supports the amd64/x86_64 architecture as of this writing, running of instrumented binaries on Macs (arm64) is not possible without emulation (which would be very slow). To support test development on Macs, a local build will not be instrumented.

Defining a new test setup

When defining a new test setup - whether in the node repo or for a VM in another repo - following the example of an existing test setup is suggested. The following table enumerates the files defining a test setup:

In addition, github workflows are suggested to ensure scripts/tests.build_antithesis_images.sh runs against PRs and scripts/build_antithesis_images.sh runs against pushes.

Use of a builder image

To simplify building instrumented (for running in CI) and non-instrumented (for running locally) versions of the workload and node images, a common builder image is used. If on an amd64 host, tests/antithesis/node/Dockerfile.builder-instrumented is used to create an instrumented builder. On an arm64 host, tests/antithesis/node/Dockerfile.builder-uninstrumented is used to create an uninstrumented builder. In both cases, the builder image is based on the default golang image and will include the source code necessary to build the node and workload binaries. The alternative would require duplicating builder setup for instrumented and non-instrumented builds for the workload and node images of each test setup.

Troubleshooting a test setup

Running a workload directly

The workload of the 'node' test setup can be invoked against an arbitrary network:

$ AVAWL_URIS="http://10.0.20.3:9630 http://10.0.20.4:9630" go run ./tests/antithesis/node

The workload of a subnet test setup like 'xsvm' additionally requires a network with a configured chain for the xsvm VM and the ID for that chain needs to be provided to the workload:

$ AVAWL_URIS=... CHAIN_IDS="2S9ypz...AzMj9" go run ./tests/antithesis/xsvm

Running a workload with docker-compose

Running the test script for a given test setup with the DEBUG flag set will avoid cleaning up the the temporary directory where the docker-compose setup is written to. This will allow manual invocation of docker-compose to see the log output of the workload.

$ DEBUG=1 ./scripts/tests.build_antithesis_images.sh

After the test script has terminated, the name of the temporary directory will appear in the output of the script:

...
using temporary directory /tmp/tmp.E6eHdDr4ln as the docker-compose path"
...

Running compose from the temporary directory will ensure the workload output appears on stdout for inspection:

$ cd [temporary directory]

# Start the compose project
$ docker-compose up

# Cleanup the compose project
$ docker-compose down --volumes

Manually triggering an Antithesis test run

When making changes to a test setup, it may be useful to manually trigger an Antithesis test run outside of the normal schedule. This can be performed against master or an arbitrary branch:

• Navigate to the 'Actions' tab of the node repo.
• Select the Publish Antithesis Images workflow on the left.
• Find the 'Run workflow' drop-down on the right and trigger the workflow against the desired branch. The default value for image_tag (latest) is used by scheduled test runs, so consider supplying a different value to avoid interferring with the results of the scheduled runs.
• Wait for the publication job to complete successfully so that the images are available to be tested against.
• Select the Trigger Antithesis Test Runs workflow on the left.
• Find the 'Run workflow' drop-down on the right and trigger the workflow against the desired branch. The branch only determines the CI configuration (the images have already been built), so master is probably fine. Make sure to supply the same image_tag that was provided to the publishing workflow and consider setting recipients to your own email rather than sending the test report to everyone on the regular distribution list.