testfilter

Test Filter Package

The test_filter package provides unified test filtering functionality for SystemQuest course testers. It allows selective execution of test cases using command-line flags, eliminating code duplication across multiple tester implementations.

Features

• Stage Filtering: Run tests from specific stages by slug (e.g., --stage=hw1,jr2)
• Test ID Filtering: Run specific tests by slug (e.g., --test=hw1-health-check,jr2-echo)
• Exclusion Filtering: Exclude tests by prefix (e.g., --exclude=jr2-)
• Unified API: Single RegisterFlags() call eliminates flag parsing duplication
• Validation: Built-in validation for filter conflicts and invalid values
• Migration Support: Provides clear error messages if numeric stage format is used

Usage

In a Course Tester (e.g., rate-limiter-v2-tester)

package main

import (
	"flag"
	"github.com/SystemQuest/tester-utils/test_filter"
	// ... other imports
)

func main() {
	// Register test filter flags (just 2 lines!)
	toFilter := testfilter.RegisterFlags()
	flag.Parse()
	
	// Convert flags to filter
	filter, err := toFilter()
	if err != nil {
		fmt.Printf("Invalid filter: %s\n", err)
		os.Exit(1)
	}
	
	// Load test cases
	testCases := loadTestCases()
	
	// Run tests with filter
	exitCode := tester_utils.RunHTTPCLIWithFilter(testCases, definition, isDebug, filter)
	os.Exit(exitCode)
}

Command-Line Examples

# Run all tests
./tester

# Run only stage hw1 tests
./tester --stage=hw1

# Run tests from multiple stages
./tester --stage=hw1,jr2

# Run specific tests by slug
./tester --test=hw1-health-check,jr2-echo

# Run stage jr2 excluding specific test prefix
./tester --stage=jr2 --exclude=jr2-calculator

# Combine filters
./tester --stage=hw1,jr2 --exclude=hw1-not-found

Migration from Numeric Stage Format

Old format (no longer supported):

./tester --stage=1,2  # ❌ Will show error

New format (slug-based):

./tester --stage=hw1,jr2  # ✅ Correct

If you try to use numeric stages, you'll see a helpful error message:

invalid --stage value: numeric stage numbers are no longer supported. 
Please use stage slugs instead (e.g., --stage=hw1,jr2 instead of --stage=1,2)

API Reference

TestFilter

The main filter type holding filtering criteria:

type TestFilter struct {
	Stages  []string   // Stage slugs to include (e.g., []string{"hw1", "jr2"})
	Tests   []string   // Test slugs to include (e.g., []string{"hw1-health-check"})
	Exclude []string   // Test slug prefixes to exclude (e.g., []string{"jr2-"})
}

RegisterFlags()

Registers test filter flags and returns a converter function:

func RegisterFlags() func() (*TestFilter, error)

Usage:

toFilter := testfilter.RegisterFlags()
flag.Parse()
filter, err := toFilter()

Apply()

Filters test cases based on configured criteria:

func (f *TestFilter) Apply(testCases []httpcontext.HTTPTestCase) []httpcontext.HTTPTestCase

Validate()

Checks if filter configuration is valid:

func (f *TestFilter) Validate() error

IsEmpty()

Returns true if no filtering criteria is set:

func (f *TestFilter) IsEmpty() bool

String()

Returns human-readable filter description:

func (f *TestFilter) String() string

Filter Logic

The filter applies criteria in this priority order:

1. Exclude (highest priority): Tests matching exclude prefixes are removed first
2. Tests: If specified, only tests with matching slugs are included
3. Stages: If specified, only tests from matching stages are included

All criteria use AND logic - a test must pass all specified filters to be included.

Examples

Filter by Stage

filter := &testfilter.TestFilter{
	Stages: []int{1, 2},
}
filtered := filter.Apply(allTestCases)

Filter by Test ID

filter := &testfilter.TestFilter{
	Tests: []string{"bt1", "fw2-global"},
}
filtered := filter.Apply(allTestCases)

Exclude by Prefix

filter := &testfilter.TestFilter{
	Exclude: []string{"fw2-per", "sw3-"},
}
filtered := filter.Apply(allTestCases)

Combined Filters

filter := &testfilter.TestFilter{
	Stages:  []int{2},
	Exclude: []string{"fw2-per"},
}
filtered := filter.Apply(allTestCases) // Stage 2, excluding fw2-per* tests

Integration with tester-utils

The package is designed to work seamlessly with tester-utils:

// Use RunHTTPCLIWithFilter instead of RunHTTPCLI
exitCode := tester_utils.RunHTTPCLIWithFilter(testCases, definition, isDebug, filter)

The RunHTTPCLIWithFilter function:

• Validates the filter
• Prints filter information if active
• Applies filtering before running tests
• Falls back to running all tests if filter is nil or empty

Testing

The package includes comprehensive unit tests:

cd sdk/tester-utils-go/test_filter
go test -v

All tests cover:

• Filter creation and validation
• Stage normalization (stage1, Stage2, STAGE_3, etc.)
• CSV parsing with whitespace handling
• Apply logic with various filter combinations
• Flag conversion and error handling

Benefits

Before (Duplicated Code)

Each tester needed ~30 lines of flag parsing:

// In rate-limiter-v2-tester/main.go
stageFlag := flag.String("stage", "", "...")
testFlag := flag.String("test", "", "...")
excludeFlag := flag.String("exclude", "", "...")
// ... 20+ more lines of parsing logic

// Repeated in lru-cache-tester/main.go
// Repeated in mini-numpy-tester/main.go
// ... etc

After (Unified API)

Now just 2 lines per tester:

toFilter := testfilter.RegisterFlags()
flag.Parse()
filter, _ := toFilter()

Result: ~90% reduction in code duplication across 3+ testers!

Version History

• v0.7.0: Initial release with test filtering support
  • RegisterFlags() for unified flag parsing
  • Apply() for filtering test cases
  • Validate() for filter validation
  • Support for stage, test, and exclude filters

License

MIT License - See LICENSE file for details