e2e

End-to-End Tests

This directory contains end-to-end tests for ToolHive, including both CLI and HTTP API tests.

Overview

These tests validate ToolHive functionality by exercising the full application stack:

• CLI Tests: Test command-line interface operations (run, list, stop, restart, etc.)
• API Tests: Test HTTP API endpoints with a real API server instance
• Integration Tests: Test interactions between different components

Structure

Test Files

• *_test.go - Individual test files organized by feature
• e2e_suite_test.go - Ginkgo test suite setup
• api_helpers.go - Helper functions for starting API server and making HTTP requests
• helpers.go - General helper functions for e2e tests
• mcp_client_helpers.go - MCP client helper utilities
• oidc_mock.go - Mock OIDC server for authentication tests
• run_tests.sh - Test runner script

Test Categories

Tests are organized using Ginkgo labels for parallelization and filtering:

Core CLI Tests (Label: core)

• Client management (client_test.go)
• Group operations (group_*.go)
• Server restart (restart_test.go)
• Export functionality (export_test.go)
• THVIgnore support (thvignore_test.go)

MCP Run Tests (Label: mcp-run)

• MCP server operations (fetch_mcp_server_test.go, osv_mcp_server_test.go)

MCP Protocol Tests (Label: mcp-protocol)

• Streamable HTTP (osv_streamable_http_mcp_server_test.go)
• Remote MCP servers (remote_mcp_server_test.go)
• Protocol builds (protocol_builds_e2e_test.go)
• Inspector functionality (inspector_test.go, inspector_autocleanup_test.go)

Note: Both mcp-run and mcp-protocol tests also carry the parent mcp label, so LABEL_FILTER=mcp still runs all MCP tests locally.

Proxy Tests (Label: proxy)

• Stdio proxy (proxy_stdio_test.go)
• OAuth authentication (proxy_oauth_test.go)
• Tunnel functionality (proxy_tunnel_e2e_test.go)
• Streamable HTTP proxy (stdio_proxy_over_streamable_http_mcp_server_test.go)
• SSE endpoint rewriting (sse_endpoint_rewrite_test.go)
• Network isolation (network_isolation_test.go)

Middleware & Stability Tests (Label: middleware || stability)

• Audit middleware (audit_middleware_e2e_test.go)
• Authorization (osv_authz_test.go, http_pdp_authz_test.go)
• Telemetry middleware (telemetry_middleware_e2e_test.go, telemetry_metrics_validation_e2e_test.go)
• Stability tests (unhealthy_workload_test.go, health_check_zombie_test.go)

API Registry Tests (Label: api-registry)

• Registry CRUD operations (api_registry_test.go)

API Workloads Tests (Label: api-workloads)

• Workload endpoints (api_workloads_test.go)
• Workload lifecycle (api_workload_lifecycle_test.go)

API Clients Tests (Label: api-clients)

• Client management (api_clients_test.go, api_clients_validation_test.go)
• Skills API (api_skills_test.go)

API Misc Tests (Label: api-misc)

• Discovery API (api_discovery_test.go)
• Groups API (api_groups_test.go)
• Health check API (api_healthcheck_test.go)
• Version API (api_version_test.go)
• Secrets API (api_secrets_test.go)

Note: All api-* tests also carry the parent api label, so LABEL_FILTER=api still runs all API tests locally.

Running Tests

Prerequisites

• Go installed
• Ginkgo CLI installed: go install github.com/onsi/ginkgo/v2/ginkgo@latest
• Docker, Podman, or Colima container runtime
• ToolHive binary built (for CLI tests): task build

Run All Tests

cd test/e2e
./run_tests.sh

Run Tests by Label

cd test/e2e

# Run only core CLI tests
E2E_LABEL_FILTER=core ./run_tests.sh

# Run only API tests
E2E_LABEL_FILTER=api ./run_tests.sh

# Run only MCP protocol tests
E2E_LABEL_FILTER=mcp ./run_tests.sh

# Run proxy tests
E2E_LABEL_FILTER=proxy ./run_tests.sh

# Run middleware and stability tests
E2E_LABEL_FILTER='middleware || stability' ./run_tests.sh

Run with Ginkgo Directly

cd test/e2e

# Run all tests
ginkgo run --vv .

# Run specific label
ginkgo run --label-filter="api" .

# Run specific test file
ginkgo run --focus-file="api_healthcheck_test.go" .

Run from Project Root

# Run all e2e tests
task test-e2e

# Run with custom label filter
E2E_LABEL_FILTER=api task test-e2e

GitHub Actions Integration

The e2e tests run in parallel in GitHub Actions using label filters. The workflow:

1. Builds the ToolHive binary once and shares it across jobs
2. Runs tests in parallel using matrix strategy with 9 label-based buckets:
   • core: Core CLI functionality (~57 specs)
   • mcp-run: MCP server run tests (~33 specs)
   • mcp-protocol: MCP protocol & inspector tests (~37 specs)
   • proxy: Proxy tests (~25 specs)
   • middleware: Middleware & stability tests (~28 specs)
   • api-registry: Registry API tests (~41 specs)
   • api-workloads: Workloads API tests (~56 specs)
   • api-clients: Clients & skills API tests (~44 specs)
   • api-misc: Discovery, groups, health, version, secrets API tests (~50 specs)
3. Uploads test results as artifacts

See .github/workflows/e2e-tests.yml for the full configuration.

Writing Tests

Adding New CLI Tests

1. Create a new test file (e.g., feature_test.go)
2. Add appropriate labels for categorization
3. Use existing helper functions from helpers.go
4. Follow the pattern of existing tests

Example:

var _ = Describe("Feature Name", Label("core", "e2e"), func() {
    It("should do something", func() {
        // Test implementation
    })
})

Adding New API Tests

1. Create a new test file (e.g., api_workloads_test.go)
2. Use the api label along with specific labels
3. Use e2e.StartServer() helper to start the API server
4. Make HTTP requests using the server's methods

Example:

var _ = Describe("Workloads API", Label("api", "workloads"), func() {
    var apiServer *e2e.Server

    BeforeEach(func() {
        config := e2e.NewServerConfig()
        apiServer = e2e.StartServer(config)
    })

    It("should list workloads", func() {
        resp, err := apiServer.Get("/api/v1beta/workloads")
        Expect(err).ToNot(HaveOccurred())
        defer resp.Body.Close()
        Expect(resp.StatusCode).To(Equal(http.StatusOK))
    })
})

Troubleshooting

Container Runtime Not Available

Ensure Docker, Podman, or Colima is running:

docker ps
# or
podman ps
# or
colima status

Binary Not Found (CLI Tests)

Build the ToolHive binary:

task build
# Binary will be at ./bin/thv

Set the binary path if needed:

export THV_BINARY=/path/to/thv

Test Timeouts

Increase the timeout:

TEST_TIMEOUT=20m ./run_tests.sh

Port Conflicts (API Tests)

API tests use random available ports by default. If you encounter port binding issues, the system will automatically find an available port.

Test Best Practices

1. Use descriptive labels - Make it easy to filter and run related tests
2. Clean up resources - Use DeferCleanup or AfterEach to clean up
3. Use unique names - Use GenerateUniqueServerName() for server names
4. Avoid hardcoded ports - Use random ports for API tests
5. Test isolation - Ensure tests can run independently
6. Meaningful assertions - Add context messages to assertions
7. Use Serial when needed - Mark tests as Serial if they can't run in parallel