e2e

Cifuzz E2E tests

Cifuzz supports a growing list of behaviors. Multiplied by the number of supported operating systems, environment setups etc. it's a lot of combinations.

• Command
• Arguments and their combinations
• Operating system
• Environment and its setup. E.g., CIFUZZ_PRERELEASE=true, tooling flags or config files and their state.
• Tooling versions. E.g., Gradle versions we rely on. Or Jazzer version.
• State of SaaS Auth
• [...]

This end-to-end matrix test is not meant to replace our Integration tests! Our Integration tests cover complex scenarios in detail. The E2E matrix is a way for us to have a high-level, but wide test coverage for the CLI.

Note Tests are currently not blocking PRs or Releases.

They run after every release and on a nightly schedule.

• Nightly tests are using make install and code that's available in the repository.
• The release tests are run once, on Release publish, using the latest curl | sh installation method, to simulate the user installing it.

Sample

When I call cifuzz bundle --help Then the exit code is 0 And there is expected help output

func TestHelpArgsWithSubcommands(t *testing.T) {
	e2e.RunTest(t, &e2e.Test{
		Command: "bundle",
		Args:    []string{"--help"},
		Assert: func(t *testing.T, output e2e.CommandOutput) {
			assert.EqualValues(t, 0, output.ExitCode)
			assert.Contains(t, output.Stdout, "This command bundles all runtime artifacts")
		},
	})
}

See tests related to cifuzz help command in the /help folder for more examples.