e2e

E2E tests

The purpose of this docs is to describe KICS' E2E test suite

Getting Started

There are several ways to execute the E2E tests.

TLDR

This steps will build the kics and then run the E2E using the built binary (placed by default under ${PWD}/bin/kics)

make test-e2e

Step by Step

These steps will build the kics and then run the E2E using the built binary.

go build -o ./bin/kics cmd/console/main.go

If you want to provide a version:

go build -o ./bin/kics -ldflags "-X github.com/Checkmarx/kics/internal/constants.Version=$(git rev-parse --short HEAD) cmd/console/main.go

and then:

E2E_KICS_BINARY=./bin/kics go test "github.com/Checkmarx/kics/e2e" -v

Test Structure

Test case main structure

var tests = []struct {
	name          string // name of the test case
	args          args // args structure of the test
	wantStatus    int // expected output status code
	removeFiles []string // path for removing files created during the test
	validation    Validation // custom validation function for the test case (usually regex)
}

Test case args structure

type cmdArgs []string

type args struct {
	args            []cmdArgs // args to pass to kics binary
	expectedOut     []string  // path to file with expected output
	expectedPayload []string  // path to file with expected payload
	expectedResult  []string // path to file with expected result
}

Each test can use more or less keys within its structure, depending on the complexity of the test.

A single test can contain verification based on:

• status code only (required for all tests)
• status code + custom regex verification
• status code + CLI output (file check)
• status code + payload content (file check)
• status code + result content (file check)
• status code + result content (file check) + payload content (file check)
• status code + CLI output + payload content (file check) + result content (file check) + custom regex verification ....

Test Scenarios

Example 1: A test case that executes a command and checks only the status code output

This type of test is simple and does not require the creation of files.

// E2E-CLI-020 - KICS scan with --exclude-queries flag
// should not run queries that was provided in this flag.
{
	name: "E2E-CLI-020",
	args: args{
		args: []cmdArgs{
			[]string{"scan", "--exclude-queries", "15ffbacc-fa42-4f6f-a57d-2feac7365caa", "-s",
				"-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"},
		},
	},
	wantStatus: []int{0},
}

Example 2: A test case that executes a command and checks:

• Status code output
• Custom validation function (regex)

The custom validation function must perform a validation based on the CLI output generated by the test. The function must return true if the validation is correct and false otherwise.

	// E2E-CLI-006 - KICS generate-id should exhibit
	// a valid UUID in the CLI and return exit code 0
	{
		name: "E2E-CLI-006",
		args: args{
			args: []cmdArgs{
				[]string{"generate-id"},
			},
		},
		wantStatus: 0,
		// When the validation function is used,
		// the expectedOut function can be suppressed

		// The validator function allows the creation of customized functions to
		// perform actions on the output generated by the test.
		validation: func(outputText string) bool {
			// Usually, a validation check for elements present in the output using regex
			uuidRegex := "[a-f0-9]{8}-[a-f0-9]{4}-4{1}[a-f0-9]{3}-[89ab]{1}[a-f0-9]{3}-[a-f0-9]{12}"
			match, _ := regexp.MatchString(uuidRegex, outputText)
			
			// After the custom check, the function should return
			// a boolean indicating if the test has passed or failed.
			return match
		},
	},
}

Example 3: A test case that executes a command and checks:

• Status code output
• Kics CLI output
• Generated payload file content

The Tests that check CLI output, payload or results, need a comparison file for each output you want to compare (except for the status code).

The example below contains 2 files for comparing the outputs of CLI and the payload file: "E2E_CLI_005" and "E2E_CLI_005_PAYLOAD.json". Files used for comparing outputs must have the same name as the test and must be added to the "fixture" folder.

In addition, it is necessary to remove the files that will be generated during the test by adding them to removeFiles (required only for test cases that generates files from results and/or payloads). Files created by Kics during testing should always be created in the 'outputs' folder

// E2E-CLI-005 - KICS scan with -- payload-path flag should create a file with the
// passed name containing the payload of the files scanned
{
	name: "E2E-CLI-005",
	args: args{
		args: []cmdArgs{
			[]string{"scan", "--silent", "-q", "../assets/queries", "-p", "fixtures/samples/terraform.tf",
				"--payload-path", "output/E2E_CLI_005_PAYLOAD.json"},
		},
		expectedOut: []string{
			"E2E_CLI_005",
		},
		expectedPayload: []string{
			"E2E_CLI_005_PAYLOAD.json",
		},
	},
	wantStatus:  []int{50},
	removeFiles: []string{"E2E_CLI_005_PAYLOAD.json"},
}

E2E tests are skiped in short mode:

func Test_E2E_CLI(t *testing.T) {
	kicsPath := getKICSBinaryPath("")

	if testing.Short() {
		t.Skip("skipping E2E tests in short mode.")
	}
//...
}

Test Functions (utils.go)

getKICSBinaryPath: This function gets the path of the kics executable.

runCommand: This function is intended to run kics with the input arguments sent by the test

readFixture & readFile: These functions reads a file (from its folder and name). The folders used are always: fixtures or output.

prepareExpected: This function prepares the file to be compared, avoiding errors in reading and formatting.

checkLine & checkJSONLog: These functions are used to check the Kics output generated in the CLI and compare it with the expectedOutput file.

fileCheck & setFields: TThese functions read and compare the expected files with the files provided during the test. This function compares files from expectedPayload and expectedResult.