integration

Prism Integration Tests

Overview

This directory contains comprehensive integration tests for the Prism platform with 60+ test functions covering all major features:

• Profile Management (7 tests, ~390 lines) - Profile lifecycle, export/import, validation
• Storage & Volume (7 tests, ~475 lines) - EFS volumes and EBS storage operations
• Template System (13 tests, ~510 lines) - Template validation, inheritance, discovery
• Instance Lifecycle (5 tests) - Launch, management, multi-instance operations
• Hibernation (3 tests) - Hibernation operations and non-capable instances
• Idle Detection (7 tests) - Idle policies, history, savings reports
• Backup Operations (6 tests) - Backup lifecycle, validation, incremental backups
• Snapshot Operations (4 tests) - Snapshot management and cloning
• User Management (1 test) - Research user operations
• Project Management (1 test) - Enterprise project features
• Core CLI (4 tests) - Help, output formats, error handling, daemon operations
• Persona Workflows (5 tests) - End-to-end user scenarios

Total: ~4,000+ lines of integration test code providing comprehensive coverage.

---

Prerequisites

1. Build Prism

# From repository root
make build

# Verify binaries exist
ls -lh bin/prism bin/prismd

2. AWS Credentials

Most integration tests require valid AWS credentials:

# Option 1: AWS CLI default credentials
aws configure

# Option 2: Environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-west-2"

# Option 3: AWS profile
export AWS_PROFILE="your-profile-name"

3. Environment Variables

Set test environment variables (optional, have defaults):

# AWS profile for testing (default: "default")
export CWS_TEST_AWS_PROFILE="default"

# AWS region for testing (default: "us-west-2")
export CWS_TEST_AWS_REGION="us-west-2"

# Test timeout multiplier (default: 1.0)
export CWS_TEST_TIMEOUT_MULTIPLIER="1.5"

---

Running Tests

Quick Start

# Run all integration tests
cd test/integration
go test -tags=integration -v

# Run specific test function
go test -tags=integration -v -run TestCLIProfileOperations

# Run specific subtest
go test -tags=integration -v -run TestCLIProfileOperations/AddProfile

By Category

Profile Tests (~2 minutes, no AWS required)

go test -tags=integration -v -run TestCLIProfile

Tests:

• TestCLIProfileOperations (8 subtests)
• TestCLIProfileExportImport (5 subtests)
• TestCLIProfileErrorHandling (7 subtests)
• TestCLIProfileSwitchingAffectsOperations (4 subtests)
• TestCLIProfileRegionVariations (3 subtests)
• TestCLIProfileValidation (2 subtests)

Storage/Volume Tests (~5 minutes, AWS required)

go test -tags=integration -v -run TestCLIStorage -timeout 10m
go test -tags=integration -v -run TestCLIEFS -timeout 10m

Tests:

• TestCLIEFSVolumeLifecycle
• TestCLIEBSStorageLifecycle
• TestCLIStorageAttachmentPersistence
• TestCLIStorageErrorHandling
• (+ additional storage tests)

Template Tests (~3 minutes, no AWS required)

go test -tags=integration -v -run TestCLITemplate

Tests:

• TestCLITemplateValidation
• TestCLITemplateList
• TestCLITemplateInfo
• TestCLITemplateSearch
• TestCLITemplateDiscover
• TestCLITemplateInheritance
• (+ 7 additional template tests)

Instance Lifecycle Tests (~15 minutes, AWS required, INCURS COSTS)

go test -tags=integration -v -run TestCLIInstance -timeout 20m

Warning: These tests launch real EC2 instances and incur AWS charges (~$0.10-0.50 per test run).

Tests:

• TestCLIInstanceLaunch
• TestCLIInstanceLifecycle
• TestCLIMultipleInstances
• TestCLILaunchOptions
• TestCLIRealAWSIntegration

Hibernation Tests (~6 minutes, AWS required)

go test -tags=integration -v -run TestCLIHibernation -timeout 10m

Idle Detection Tests (~8 minutes, AWS required)

go test -tags=integration -v -run TestCLIIdle -timeout 10m

Backup/Snapshot Tests (~5 minutes, AWS required)

go test -tags=integration -v -run TestCLIBackup -timeout 10m
go test -tags=integration -v -run TestCLISnapshot -timeout 10m

Persona Workflow Tests (~20 minutes, AWS required, INCURS COSTS)

go test -tags=integration -v -run TestPersona -timeout 30m

Tests:

• TestSoloResearcherPersona
• TestLabEnvironmentPersona
• TestUniversityClassPersona
• TestCrossInstitutionalPersona
• TestConferenceWorkshopPersona

---

Test Options

Timeout Configuration

# Default timeout (2 minutes)
go test -tags=integration -v -run TestCLIProfile

# Extended timeout for AWS tests
go test -tags=integration -v -run TestCLIInstance -timeout 20m

# Very long timeout for persona tests
go test -tags=integration -v -run TestPersona -timeout 30m

Parallel Execution

# Run tests in parallel (default)
go test -tags=integration -v -parallel 4

# Run tests sequentially (safer for AWS rate limits)
go test -tags=integration -v -parallel 1

Verbose Output

# Standard verbose output
go test -tags=integration -v

# Very verbose with test logs
go test -tags=integration -v -count=1

# JSON output for CI/CD
go test -tags=integration -json

Selective Execution

# Run only fast tests (no AWS)
go test -tags=integration -v -run "TestCLIProfile|TestCLITemplate"

# Skip specific tests
go test -tags=integration -v -skip "TestPersona"

# Run single test multiple times (check for flakiness)
go test -tags=integration -v -run TestCLIProfileOperations -count 5

---

Understanding Test Output

Successful Test

=== RUN   TestCLIProfileOperations
=== RUN   TestCLIProfileOperations/ListProfiles
=== RUN   TestCLIProfileOperations/AddProfile
=== RUN   TestCLIProfileOperations/SwitchProfile
--- PASS: TestCLIProfileOperations (0.68s)
    --- PASS: TestCLIProfileOperations/ListProfiles (0.05s)
    --- PASS: TestCLIProfileOperations/AddProfile (0.02s)
    --- PASS: TestCLIProfileOperations/SwitchProfile (0.01s)
PASS
ok      github.com/scttfrdmn/prism/test/integration    0.713s

Failed Test

=== RUN   TestCLIProfileOperations
=== RUN   TestCLIProfileOperations/AddProfile
    cli_profile_test.go:43: Command failed: exit status 1
    cli_profile_test.go:43: Stdout: 
    cli_profile_test.go:43: Stderr: Error: unknown command "add" for "prism profiles"
--- FAIL: TestCLIProfileOperations (0.10s)
    --- FAIL: TestCLIProfileOperations/AddProfile (0.02s)
FAIL
FAIL    github.com/scttfrdmn/prism/test/integration    0.123s

Skipped Test

=== RUN   TestCLIProfileSetupWizard
    cli_profile_test.go:284: Interactive setup wizard requires input automation - implement with expect or similar
--- SKIP: TestCLIProfileSetupWizard (0.00s)
PASS
ok      github.com/scttfrdmn/prism/test/integration    0.015s

---

Debugging Failed Tests

Step 1: Run with Verbose Output

go test -tags=integration -v -run TestCLIProfileOperations/AddProfile

Look for:

• Command that failed
• Exit code
• Stdout/Stderr output
• Error messages

Step 2: Check Binary Exists

ls -lh ../../bin/prism
# Should show: -rwxr-xr-x  ...  ../../bin/prism

# If missing, rebuild
cd ../..
make build
cd test/integration

Step 3: Verify CLI Command Structure

# Test command manually
../../bin/prism profiles add --help

# Check actual CLI structure matches test expectations

Step 4: Check AWS Credentials

# Verify AWS credentials work
aws sts get-caller-identity

# Check Prism can access AWS
../../bin/prism list

Step 5: Examine Test Artifacts

# Check test state directory (created during test)
ls -la /tmp/cws-test-*

# Examine test logs if present
cat /tmp/cws-test-*/daemon.log

Step 6: Run Test in Isolation

# Clean state before test
../../bin/prism daemon stop
rm -rf /tmp/cws-test-*

# Run single test
go test -tags=integration -v -run TestCLIProfileOperations/AddProfile -count=1

---

Common Issues and Solutions

Issue: "Command not found: prism"

Problem: Test can't find prism binary

Solution:

# Rebuild binaries
cd ../..
make build
cd test/integration

# Verify binary exists
ls -lh ../../bin/prism

---

Issue: "AWS credentials not configured"

Problem: Tests requiring AWS fail with credential errors

Solution:

# Configure AWS credentials
aws configure

# Or set environment variables
export AWS_PROFILE="your-profile"
export AWS_DEFAULT_REGION="us-west-2"

---

Issue: "Timeout exceeded"

Problem: Tests timeout before completion

Solution:

# Increase timeout for AWS tests
go test -tags=integration -v -run TestCLIInstance -timeout 30m

# Or set timeout multiplier
export CWS_TEST_TIMEOUT_MULTIPLIER="2.0"

---

Issue: "Profile validation fails"

Problem: TestCLIProfileValidation/ValidateInvalidProfile fails

Reason: Environment-dependent test - expects validation to fail for nonexistent AWS profile, but succeeds if default credentials work

Solution: This is expected behavior in some environments, not a test failure

---

Issue: "Port already in use"

Problem: Daemon port 8947 already in use

Solution:

# Stop any running daemon
../../bin/prism daemon stop

# Or kill process manually
lsof -ti:8947 | xargs kill -9

---

Issue: "Test leaves resources behind"

Problem: AWS resources not cleaned up after test failure

Solution:

# List remaining instances
../../bin/prism list

# Terminate instances
../../bin/prism terminate <instance-name>

# List volumes
../../bin/prism volume list

# Delete volumes
../../bin/prism volume delete <volume-name>

---

Writing New Tests

Test Structure

//go:build integration
// +build integration

package integration

import (
    "testing"
)

// TestCLIFeatureOperations tests feature lifecycle
func TestCLIFeatureOperations(t *testing.T) {
    ctx := NewCLITestContext(t)
    defer ctx.Cleanup()
    
    resourceName := GenerateTestName("test-resource")
    
    t.Run("CreateResource", func(t *testing.T) {
        result := ctx.Prism("feature", "create", resourceName)
        result.AssertSuccess(t, "feature create should succeed")
        result.AssertContains(t, resourceName, "should show resource name")
    })
    
    t.Run("ListResource", func(t *testing.T) {
        result := ctx.Prism("feature", "list")
        result.AssertSuccess(t, "feature list should succeed")
        result.AssertContains(t, resourceName, "should list created resource")
    })
    
    t.Run("DeleteResource", func(t *testing.T) {
        result := ctx.Prism("feature", "delete", resourceName)
        result.AssertSuccess(t, "feature delete should succeed")
    })
}

Best Practices

1. Use CLITestContext: Provides isolated test environment

   ctx := NewCLITestContext(t)
   defer ctx.Cleanup()
   

2. Generate Unique Names: Avoid conflicts with concurrent tests

   resourceName := GenerateTestName("my-resource")
   

3. Use Subtests: Organize related test steps

   t.Run("SubtestName", func(t *testing.T) { ... })
   

4. Verify Commands: Use assertion methods

   result.AssertSuccess(t, "operation should succeed")
   result.AssertFailure(t, "invalid operation should fail")
   result.AssertContains(t, "expected text", "should contain text")
   

5. Track Resources: Let context track resources for cleanup

   ctx.TrackVolume(volumeName)
   ctx.TrackInstance(instanceName)
   

6. Handle Cleanup: Always defer cleanup

   defer ctx.Cleanup()
   defer os.Remove(tempFile)
   

7. Set Timeouts: For AWS operations

   // Add to test function comment:
   // Timeout: 10 minutes (AWS operations)
   

8. Document Costs: For tests that incur AWS charges

   // Cost: ~$0.10-0.50 per test run (EC2 instances)
   

---

CI/CD Integration

GitHub Actions Example

name: Integration Tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  integration-tests:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Go
        uses: actions/setup-go@v4
        with:
          go-version: '1.21'
      
      - name: Build Prism
        run: make build
      
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-west-2
      
      - name: Run Integration Tests (No AWS)
        run: |
          cd test/integration
          go test -tags=integration -v -run "TestCLIProfile|TestCLITemplate" -timeout 5m
      
      - name: Run Integration Tests (AWS Required)
        run: |
          cd test/integration
          go test -tags=integration -v -run "TestCLIStorage|TestCLIIdle" -timeout 15m

Required Secrets

• AWS_ACCESS_KEY_ID: AWS access key for test account
• AWS_SECRET_ACCESS_KEY: AWS secret key for test account

Test Selection for CI

Fast tests (no AWS, <5 min):

• Profile tests
• Template tests
• Core CLI tests

Medium tests (AWS, 5-10 min):

• Storage tests
• Hibernation tests
• Idle detection tests

Slow tests (AWS, >10 min):

• Instance lifecycle tests
• Backup/snapshot tests
• Persona workflow tests

Recommendation: Run fast + medium tests on every PR, slow tests on main branch only.

---

Test Coverage by Feature

Feature Area	Tests	Lines	Status	Notes
Profile Management	7	390	✅ 96.4%	27/28 subtests passing
Storage/Volume	7	475	✅ Ready	Verified correct
Template System	13	510	✅ 100%	All passing
Instance Lifecycle	5	~500	✅ Active	AWS required
Hibernation	3	~300	✅ Active	AWS required
Idle Detection	7	~400	✅ Active	AWS required
Backup/Snapshot	10	~650	✅ Ready	AWS required
User Management	1	~150	✅ Ready	Research users
Project Management	1	~200	✅ Ready	Enterprise features
Core CLI	4	~300	✅ Ready	Help, formats, errors
Persona Workflows	5	~800	✅ Ready	End-to-end scenarios
S3 Integration	2	~250	✅ Active	Storage transfer

Total: 60+ test functions, 4,000+ lines of integration test code

---

Quick Reference

# Build Prism
make build

# Run all integration tests
cd test/integration && go test -tags=integration -v

# Run fast tests only (no AWS)
go test -tags=integration -v -run "TestCLIProfile|TestCLITemplate"

# Run specific test
go test -tags=integration -v -run TestCLIProfileOperations

# Run with extended timeout
go test -tags=integration -v -timeout 20m

# Run in parallel
go test -tags=integration -v -parallel 4

# Check test coverage
go test -tags=integration -v -cover

# Generate coverage report
go test -tags=integration -coverprofile=coverage.out
go tool cover -html=coverage.out

# Clean test artifacts
rm -rf /tmp/cws-test-*
../../bin/prism daemon stop

---

Additional Resources

• Test Implementation: test_helpers.go - Test utilities and helpers
• Coverage Analysis: /tmp/integration_test_coverage_analysis.md - Comprehensive coverage report
• Remediation History: /tmp/final_remediation_summary.md - Recent test fixes and improvements

---

Last Updated: November 2025
Status: Integration test suite production-ready with comprehensive coverage