apptest

package module
v0.0.0-...-7856175 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 16, 2026 License: MIT Imports: 6 Imported by: 0

README

App Test Spec

Go CI Go Lint Go SAST Go Report Card Docs Visualization License

Universal test specification format for QA, SDET, and Security testing across API and Web applications.

Features

  • Unified Format: Single schema supporting API functional, API security, web functional, and web security tests
  • Multiple Execution Modes: Deterministic (scripted), LLM-driven (agent-based), and Hybrid (record and replay)
  • Built-in Payloads: Comprehensive SQL injection, XSS, and IDOR payload libraries
  • OpenAPI Integration: Auto-generate tests from OpenAPI specifications
  • Agent Support: Integration with multi-agent-spec for LLM-driven testing

Installation

go get github.com/plexusone/app-test-spec

Quick Start

Define a Test Case
id: login-sqli-test
name: SQL Injection on Login Endpoint
kind: api-security
category: security
execution: deterministic
securityType: sqli
severity: critical

api:
  request:
    method: POST
    url: "{{baseUrl}}/api/auth/login"
    headers:
      Content-Type: application/json
    body:
      email: "admin@example.com"
      password: "test"
  injection:
    parameter: email
    location: body
    payloadSet: sqli/generic

assertions:
  - type: status
    operator: in
    value: [400, 401]
    message: "SQLi should not bypass authentication"
Load and Use in Go
package main

import (
    "fmt"
    "log"

    apptest "github.com/plexusone/app-test-spec"
)

func main() {
    // Load a test case
    tc, err := apptest.LoadTestCase("tests/login-sqli.yaml")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Test: %s (%s)\n", tc.Name, tc.Kind)

    // Load payloads
    ps, err := apptest.LoadPayloadSet("payloads/sqli/mysql.yaml")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Loaded %d payloads\n", len(ps.Payloads))
}

Test Kinds

Kind Description
api-functional API endpoint functionality tests
api-security API security vulnerability tests
web-functional Browser-based functionality tests
web-security Browser-based security tests

Execution Modes

Mode Description Use Case
deterministic Scripted, reproducible Regression testing, CI/CD
llm-driven Agent-based exploration Exploratory testing, fuzzing
hybrid LLM explores, then replay Recording for evidence

Security Types

Type Description
sqli SQL Injection
xss Cross-Site Scripting
idor Insecure Direct Object Reference
bola Broken Object Level Authorization
bfla Broken Function Level Authorization
csrf Cross-Site Request Forgery
ssrf Server-Side Request Forgery
auth-bypass Authentication Bypass
mass-assignment Mass Assignment

Built-in Payloads

payloads/
├── sqli/
│   ├── generic.yaml    # Database-agnostic
│   ├── mysql.yaml      # MySQL-specific
│   ├── postgres.yaml   # PostgreSQL-specific
│   ├── mssql.yaml      # SQL Server-specific
│   └── sqlite.yaml     # SQLite-specific
├── xss/
│   ├── reflected.yaml  # Reflected XSS
│   ├── stored.yaml     # Stored/Persistent XSS
│   └── dom.yaml        # DOM-based XSS
└── idor/
    └── common.yaml     # IDOR/BOLA payloads

Directory Structure

app-test-spec/
├── types.go          # Core types (TestCase, TestSuite, TestPlan)
├── api.go            # API test types
├── web.go            # Web test types
├── agent.go          # Agent test types
├── payload.go        # PayloadSet types
├── auth.go           # Authentication types
├── loader.go         # YAML/JSON loader
├── payloads/         # Built-in payloads
├── agents/           # Built-in agents
└── examples/         # Example test specs

Documentation

  • PRD.md - Product Requirements Document
  • TRD.md - Technical Requirements Document
  • PLAN.md - Implementation Plan

Integration

OpenAPI
testPlan:
  target:
    baseUrl: https://api.example.com
    openapi: ./openapi.yaml
  suites:
    - generate:
        from: openapi
        securityTests: [sqli, idor]
multi-agent-spec
agent:
  agentRef: security/sqli-tester
  goal: "Find SQL injection vulnerabilities"
  tools: [http, browser]
vibium-go
web:
  mode: mcp
  agent:
    tools: [navigate, click, fill, screenshot]

License

MIT

Documentation

Overview

Package apptest provides a universal test specification format for QA, SDET, and Security testing across API and Web applications.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func LoadPayloadSetsFromDir 

func LoadPayloadSetsFromDir(dir string) (map[string]*PayloadSet, error)

LoadPayloadSetsFromDir loads all payload sets from a directory.

func LoadPayloads 

func LoadPayloads(path string) ([]string, error)

LoadPayloads loads payloads from a file (simplified format - just values).

func SavePayloadSet 

func SavePayloadSet(ps *PayloadSet, path string) error

SavePayloadSet saves a PayloadSet to a file.

func SaveTestCase 

func SaveTestCase(tc *TestCase, path string) error

SaveTestCase saves a TestCase to a file.

func SaveTestPlan 

func SaveTestPlan(tp *TestPlan, path string) error

SaveTestPlan saves a TestPlan to a file.

func SaveTestSuite 

func SaveTestSuite(ts *TestSuite, path string) error

SaveTestSuite saves a TestSuite to a file.

Types

type APITestSpec 

type APITestSpec struct {
	// OpenAPI reference (optional - derive from spec)
	OpenAPIRef *OpenAPIReference `json:"openapi,omitempty" yaml:"openapi,omitempty"`

	// Direct specification
	Request *HTTPRequest `json:"request,omitempty" yaml:"request,omitempty"`

	// Payload injection (for security tests)
	Injection *InjectionSpec `json:"injection,omitempty" yaml:"injection,omitempty"`

	// Expected response (for functional tests)
	Expected *ExpectedResponse `json:"expected,omitempty" yaml:"expected,omitempty"`

	// Detection (for security tests)
	Detection *DetectionSpec `json:"detection,omitempty" yaml:"detection,omitempty"`
}

APITestSpec defines an API test (compatible with OpenAPI operations).

type AgentAction 

type AgentAction struct {
	Timestamp string                 `json:"timestamp" yaml:"timestamp"`
	Tool      string                 `json:"tool" yaml:"tool"`
	Input     map[string]interface{} `json:"input" yaml:"input"`
	Output    interface{}            `json:"output,omitempty" yaml:"output,omitempty"`
	Error     string                 `json:"error,omitempty" yaml:"error,omitempty"`
	Reasoning string                 `json:"reasoning,omitempty" yaml:"reasoning,omitempty"`
}

AgentAction represents an action taken by an agent (for recording).

type AgentReference 

type AgentReference struct {
	// Path to agent definition file (multi-agent-spec format)
	Path string `json:"path,omitempty" yaml:"path,omitempty"`

	// Or built-in agent name
	BuiltIn string `json:"builtIn,omitempty" yaml:"builtIn,omitempty"`

	// Parameter overrides
	Parameters map[string]string `json:"parameters,omitempty" yaml:"parameters,omitempty"`

	// Tool restrictions (subset of agent's available tools)
	Tools []string `json:"tools,omitempty" yaml:"tools,omitempty"`
}

AgentReference references an agent definition.

type AgentResult 

type AgentResult struct {
	Status   AgentStatus `json:"status" yaml:"status"`
	Summary  string      `json:"summary,omitempty" yaml:"summary,omitempty"`
	Findings []Finding   `json:"findings,omitempty" yaml:"findings,omitempty"`
}

AgentResult represents the outcome of an agent session.

type AgentSession 

type AgentSession struct {
	ID        string        `json:"id" yaml:"id"`
	TestCase  string        `json:"testCase" yaml:"testCase"` // TestCase ID
	Agent     string        `json:"agent" yaml:"agent"`       // Agent reference
	Goal      string        `json:"goal" yaml:"goal"`
	Actions   []AgentAction `json:"actions" yaml:"actions"`
	Result    AgentResult   `json:"result" yaml:"result"`
	StartTime string        `json:"startTime" yaml:"startTime"`
	EndTime   string        `json:"endTime" yaml:"endTime"`
}

AgentSession represents a recorded agent session.

type AgentStatus 

type AgentStatus string

AgentStatus represents the status of an agent session. 

const (
	AgentStatusSuccess    AgentStatus = "success"     // Goal achieved
	AgentStatusFailure    AgentStatus = "failure"     // Goal not achieved
	AgentStatusVulnFound  AgentStatus = "vuln_found"  // Vulnerability found (for security tests)
	AgentStatusError      AgentStatus = "error"       // Agent encountered an error
	AgentStatusMaxRetries AgentStatus = "max_retries" // Max iterations reached
)

type AgentTestSpec 

type AgentTestSpec struct {
	// Reference to multi-agent-spec agent (e.g., "security/sqli-tester")
	AgentRef string `json:"agentRef,omitempty" yaml:"agentRef,omitempty"`

	// Or inline agent definition
	InlineAgent *InlineAgent `json:"inlineAgent,omitempty" yaml:"inlineAgent,omitempty"`

	// Goal/objective for the agent
	Goal string `json:"goal" yaml:"goal"`

	// Context to provide to the agent
	Context map[string]interface{} `json:"context,omitempty" yaml:"context,omitempty"`

	// Tools available to the agent (MCP tools from vibium-go, HTTP tools, etc.)
	Tools []string `json:"tools,omitempty" yaml:"tools,omitempty"`

	// Success criteria (conditions that indicate the test passed)
	SuccessCriteria []string `json:"successCriteria,omitempty" yaml:"successCriteria,omitempty"`

	// Failure criteria (conditions that indicate vulnerability found - for security tests)
	FailureCriteria []string `json:"failureCriteria,omitempty" yaml:"failureCriteria,omitempty"`

	// Max iterations before stopping
	MaxIterations int `json:"maxIterations,omitempty" yaml:"maxIterations,omitempty"`

	// Recording: capture LLM actions for deterministic replay
	RecordActions bool `json:"recordActions,omitempty" yaml:"recordActions,omitempty"`

	// Model to use
	Model string `json:"model,omitempty" yaml:"model,omitempty"` // haiku, sonnet, opus
}

AgentTestSpec defines an LLM-driven test using multi-agent-spec.

type AgentTool 

type AgentTool struct {
	Name        string                 `json:"name" yaml:"name"`
	Description string                 `json:"description,omitempty" yaml:"description,omitempty"`
	Parameters  map[string]interface{} `json:"parameters,omitempty" yaml:"parameters,omitempty"`
}

AgentTool represents a tool available to an agent.

type Assertion 

type Assertion struct {
	Type     AssertionType `json:"type" yaml:"type"`
	Target   string        `json:"target,omitempty" yaml:"target,omitempty"`     // What to assert on (response.status, response.body, etc.)
	Operator string        `json:"operator,omitempty" yaml:"operator,omitempty"` // equals, contains, matches, etc.
	Value    interface{}   `json:"value,omitempty" yaml:"value,omitempty"`       // Expected value
	Message  string        `json:"message,omitempty" yaml:"message,omitempty"`   // Custom failure message
}

Assertion defines a condition that must be true for a test to pass.

type AssertionType 

type AssertionType string

AssertionType identifies the type of assertion. 

const (
	AssertStatus       AssertionType = "status"       // HTTP status code
	AssertHeader       AssertionType = "header"       // HTTP header value
	AssertBody         AssertionType = "body"         // Response body content
	AssertJSON         AssertionType = "json"         // JSON path assertion
	AssertResponseTime AssertionType = "responseTime" // Response time threshold
	AssertSchema       AssertionType = "schema"       // JSON schema validation
	AssertNotContains  AssertionType = "notContains"  // Content should NOT be present
	AssertRegex        AssertionType = "regex"        // Regex match
)

type AuthProfile 

type AuthProfile struct {
	ID          string   `json:"id" yaml:"id"`
	Name        string   `json:"name" yaml:"name"`
	Description string   `json:"description,omitempty" yaml:"description,omitempty"`
	Auth        AuthSpec `json:"auth" yaml:"auth"`

	// Role for RBAC testing
	Role string `json:"role,omitempty" yaml:"role,omitempty"`

	// Permissions for authorization testing
	Permissions []string `json:"permissions,omitempty" yaml:"permissions,omitempty"`
}

AuthProfile represents a named authentication profile.

type AuthSpec 

type AuthSpec struct {
	Type AuthType `json:"type" yaml:"type"`

	// For basic auth
	Username string `json:"username,omitempty" yaml:"username,omitempty"`
	Password string `json:"password,omitempty" yaml:"password,omitempty"`

	// For bearer token
	Token string `json:"token,omitempty" yaml:"token,omitempty"`

	// For API key
	Key      string `json:"key,omitempty" yaml:"key,omitempty"`
	Value    string `json:"value,omitempty" yaml:"value,omitempty"`
	Location string `json:"location,omitempty" yaml:"location,omitempty"` // header, query, cookie

	// For OAuth2
	OAuth2 *OAuth2Config `json:"oauth2,omitempty" yaml:"oauth2,omitempty"`

	// For form-based auth (e.g., Spring Security, Django, etc.)
	LoginURL     string            `json:"loginUrl,omitempty" yaml:"loginUrl,omitempty"`
	FormFields   map[string]string `json:"formFields,omitempty" yaml:"formFields,omitempty"`
	SuccessCheck string            `json:"successCheck,omitempty" yaml:"successCheck,omitempty"` // status:302, cookie:JSESSIONID, body:contains:Welcome, header:Location:/dashboard

	// For custom auth
	Headers map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"`
	Cookies map[string]string `json:"cookies,omitempty" yaml:"cookies,omitempty"`

	// Reference to environment variable or secret
	EnvVar string `json:"envVar,omitempty" yaml:"envVar,omitempty"`
}

AuthSpec defines authentication configuration.

type AuthTest 

type AuthTest struct {
	// Profile to use
	Profile string `json:"profile" yaml:"profile"`

	// Expected behavior
	ShouldSucceed bool `json:"shouldSucceed" yaml:"shouldSucceed"`

	// For IDOR/BOLA testing - try to access another user's resources
	TargetUser string `json:"targetUser,omitempty" yaml:"targetUser,omitempty"`

	// For privilege escalation testing
	EscalateTo string `json:"escalateTo,omitempty" yaml:"escalateTo,omitempty"`
}

AuthTest defines an authentication/authorization test scenario.

type AuthType 

type AuthType string

AuthType identifies the authentication method. 

const (
	AuthNone    AuthType = "none"
	AuthBasic   AuthType = "basic"
	AuthBearer  AuthType = "bearer"
	AuthAPIKey  AuthType = "apikey"
	AuthOAuth2  AuthType = "oauth2"
	AuthForm    AuthType = "form"
	AuthDigest  AuthType = "digest"
	AuthNTLM    AuthType = "ntlm"
	AuthAWSSig4 AuthType = "aws-sig4"
	AuthCustom  AuthType = "custom"
)

type BrowserConfig 

type BrowserConfig struct {
	Type       BrowserType `json:"type,omitempty" yaml:"type,omitempty"` // chromium, firefox, webkit
	Headless   bool        `json:"headless,omitempty" yaml:"headless,omitempty"`
	Viewport   *Viewport   `json:"viewport,omitempty" yaml:"viewport,omitempty"`
	UserAgent  string      `json:"userAgent,omitempty" yaml:"userAgent,omitempty"`
	Timeout    Duration    `json:"timeout,omitempty" yaml:"timeout,omitempty"`
	Screenshot bool        `json:"screenshot,omitempty" yaml:"screenshot,omitempty"` // Take screenshots on failure
	Video      bool        `json:"video,omitempty" yaml:"video,omitempty"`           // Record video
	Trace      bool        `json:"trace,omitempty" yaml:"trace,omitempty"`           // Record trace
}

BrowserConfig configures browser settings for web tests.

type BrowserType 

type BrowserType string

BrowserType identifies the browser engine. 

const (
	BrowserChromium BrowserType = "chromium"
	BrowserFirefox  BrowserType = "firefox"
	BrowserWebkit   BrowserType = "webkit"
)

type DetectionMethod 

type DetectionMethod string

DetectionMethod identifies the vulnerability detection technique. 

const (
	DetectError   DetectionMethod = "error"   // Error-based (look for error messages)
	DetectBoolean DetectionMethod = "boolean" // Boolean-based (compare true/false responses)
	DetectTime    DetectionMethod = "time"    // Time-based (measure response delay)
	DetectUnion   DetectionMethod = "union"   // Union-based (look for injected data)
	DetectOOB     DetectionMethod = "oob"     // Out-of-band (external callback)
	DetectDiff    DetectionMethod = "diff"    // Differential (compare to baseline)
	DetectContent DetectionMethod = "content" // Content-based (look for specific content)
)

type DetectionSpec 

type DetectionSpec struct {
	Method   DetectionMethod `json:"method" yaml:"method"`                         // error, boolean, time, union, oob, diff
	Patterns []string        `json:"patterns,omitempty" yaml:"patterns,omitempty"` // Regex patterns to match

	// Time-based detection
	ExpectedDelay *Duration `json:"expectedDelay,omitempty" yaml:"expectedDelay,omitempty"`

	// Boolean-based detection
	TrueCondition  *HTTPRequest `json:"trueCondition,omitempty" yaml:"trueCondition,omitempty"`
	FalseCondition *HTTPRequest `json:"falseCondition,omitempty" yaml:"falseCondition,omitempty"`

	// Differential detection
	BaselineRequest *HTTPRequest `json:"baseline,omitempty" yaml:"baseline,omitempty"`

	// OOB detection
	CallbackURL string `json:"callbackUrl,omitempty" yaml:"callbackUrl,omitempty"`
}

DetectionSpec defines how to detect successful exploitation.

type Duration 

type Duration string

Duration is a wrapper for time duration that supports YAML/JSON.

type Evidence 

type Evidence struct {
	Type    EvidenceType `json:"type" yaml:"type"`
	Content string       `json:"content" yaml:"content"`
	URL     string       `json:"url,omitempty" yaml:"url,omitempty"`
}

Evidence represents evidence for a security finding.

type EvidenceType 

type EvidenceType string

EvidenceType identifies the type of evidence. 

const (
	EvidenceRequest    EvidenceType = "request"    // HTTP request
	EvidenceResponse   EvidenceType = "response"   // HTTP response
	EvidenceScreenshot EvidenceType = "screenshot" // Screenshot
	EvidenceLog        EvidenceType = "log"        // Log output
	EvidencePayload    EvidenceType = "payload"    // Payload that triggered the finding
)

type ExecutionConfig 

type ExecutionConfig struct {
	Timeout     Duration `json:"timeout,omitempty" yaml:"timeout,omitempty"`
	Retries     int      `json:"retries,omitempty" yaml:"retries,omitempty"`
	Parallel    bool     `json:"parallel,omitempty" yaml:"parallel,omitempty"`
	MaxWorkers  int      `json:"maxWorkers,omitempty" yaml:"maxWorkers,omitempty"`
	StopOnFail  bool     `json:"stopOnFail,omitempty" yaml:"stopOnFail,omitempty"`
	Environment string   `json:"environment,omitempty" yaml:"environment,omitempty"` // dev, staging, prod
}

ExecutionConfig contains runtime configuration.

type ExecutionMode 

type ExecutionMode string

ExecutionMode determines how the test is executed. 

const (
	ExecDeterministic ExecutionMode = "deterministic" // Scripted, reproducible
	ExecLLMDriven     ExecutionMode = "llm-driven"    // Agent-based exploration
	ExecHybrid        ExecutionMode = "hybrid"        // LLM generates, then deterministic replay
)

type ExpectedResponse 

type ExpectedResponse struct {
	Status  int               `json:"status,omitempty" yaml:"status,omitempty"`
	Headers map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"`
	Body    interface{}       `json:"body,omitempty" yaml:"body,omitempty"`
	Schema  string            `json:"schema,omitempty" yaml:"schema,omitempty"` // JSON schema reference
}

ExpectedResponse defines expected response for functional tests.

type FileFormat 

type FileFormat string

FileFormat represents the format of a specification file. 

const (
	FormatYAML FileFormat = "yaml"
	FormatJSON FileFormat = "json"
)

type Finding 

type Finding struct {
	ID           string       `json:"id" yaml:"id"`
	Title        string       `json:"title" yaml:"title"`
	Description  string       `json:"description" yaml:"description"`
	SecurityType SecurityType `json:"securityType" yaml:"securityType"`
	Severity     Severity     `json:"severity" yaml:"severity"`
	Evidence     []Evidence   `json:"evidence" yaml:"evidence"`
	Remediation  string       `json:"remediation,omitempty" yaml:"remediation,omitempty"`
}

Finding represents a security finding discovered by an agent.

type HTTPRequest 

type HTTPRequest struct {
	Method  string            `json:"method" yaml:"method"`
	URL     string            `json:"url" yaml:"url"`
	Headers map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"`
	Query   map[string]string `json:"query,omitempty" yaml:"query,omitempty"`
	Body    interface{}       `json:"body,omitempty" yaml:"body,omitempty"`
	Auth    *AuthSpec         `json:"auth,omitempty" yaml:"auth,omitempty"`
}

HTTPRequest defines an HTTP request.

type InjectionContext 

type InjectionContext struct {
	// SQL context
	SQLContext *SQLContext `json:"sql,omitempty" yaml:"sql,omitempty"`

	// Encoding to apply
	Encoding []string `json:"encoding,omitempty" yaml:"encoding,omitempty"` // url, base64, html, etc.

	// Prefix/suffix to wrap payload
	Prefix string `json:"prefix,omitempty" yaml:"prefix,omitempty"`
	Suffix string `json:"suffix,omitempty" yaml:"suffix,omitempty"`
}

InjectionContext provides context for payload injection.

type InjectionSpec 

type InjectionSpec struct {
	// Target parameter for injection
	Parameter string            `json:"parameter" yaml:"parameter"`
	Location  ParameterLocation `json:"location" yaml:"location"` // query, path, body, header

	// Payload source (one of)
	Payloads    []string `json:"payloads,omitempty" yaml:"payloads,omitempty"`       // Inline payloads
	PayloadSet  string   `json:"payloadSet,omitempty" yaml:"payloadSet,omitempty"`   // Reference to PayloadSet
	PayloadFile string   `json:"payloadFile,omitempty" yaml:"payloadFile,omitempty"` // Path to payload file

	// Injection context
	Context *InjectionContext `json:"context,omitempty" yaml:"context,omitempty"`
}

InjectionSpec defines payload injection for security tests.

type InlineAgent 

type InlineAgent struct {
	Name        string            `json:"name" yaml:"name"`
	Description string            `json:"description,omitempty" yaml:"description,omitempty"`
	Model       string            `json:"model" yaml:"model"` // haiku, sonnet, opus
	Tools       []string          `json:"tools" yaml:"tools"`
	Prompt      string            `json:"prompt" yaml:"prompt"` // System prompt / instructions
	Parameters  map[string]string `json:"parameters,omitempty" yaml:"parameters,omitempty"`
}

InlineAgent defines an agent inline (rather than referencing multi-agent-spec).

type OAuth2Config 

type OAuth2Config struct {
	GrantType    OAuth2GrantType `json:"grantType" yaml:"grantType"`
	TokenURL     string          `json:"tokenUrl" yaml:"tokenUrl"`
	ClientID     string          `json:"clientId" yaml:"clientId"`
	ClientSecret string          `json:"clientSecret,omitempty" yaml:"clientSecret,omitempty"`
	Scope        string          `json:"scope,omitempty" yaml:"scope,omitempty"`
	Username     string          `json:"username,omitempty" yaml:"username,omitempty"` // For password grant
	Password     string          `json:"password,omitempty" yaml:"password,omitempty"` // For password grant
	RefreshToken string          `json:"refreshToken,omitempty" yaml:"refreshToken,omitempty"`
	AuthURL      string          `json:"authUrl,omitempty" yaml:"authUrl,omitempty"` // For authorization code
	RedirectURI  string          `json:"redirectUri,omitempty" yaml:"redirectUri,omitempty"`
}

OAuth2Config defines OAuth2 authentication configuration.

type OAuth2GrantType 

type OAuth2GrantType string

OAuth2GrantType identifies the OAuth2 grant type. 

const (
	GrantClientCredentials OAuth2GrantType = "client_credentials"
	GrantPassword          OAuth2GrantType = "password"
	GrantAuthorizationCode OAuth2GrantType = "authorization_code"
	GrantRefreshToken      OAuth2GrantType = "refresh_token"
)

type OpenAPIReference 

type OpenAPIReference struct {
	SpecFile    string                 `json:"specFile" yaml:"specFile"`       // Path to OpenAPI spec
	OperationID string                 `json:"operationId" yaml:"operationId"` // Operation to test
	Parameters  map[string]interface{} `json:"parameters,omitempty" yaml:"parameters,omitempty"`
}

OpenAPIReference references an operation in an OpenAPI spec.

type ParameterLocation 

type ParameterLocation string

ParameterLocation identifies where a parameter is located in a request. 

const (
	LocationQuery  ParameterLocation = "query"
	LocationPath   ParameterLocation = "path"
	LocationBody   ParameterLocation = "body"
	LocationHeader ParameterLocation = "header"
	LocationCookie ParameterLocation = "cookie"
)

type Payload 

type Payload struct {
	// The payload value
	Value string `json:"value" yaml:"value"`

	// Description of what this payload tests
	Description string `json:"description,omitempty" yaml:"description,omitempty"`

	// For boolean-based detection
	TrueValue  string `json:"trueValue,omitempty" yaml:"trueValue,omitempty"`
	FalseValue string `json:"falseValue,omitempty" yaml:"falseValue,omitempty"`

	// For time-based detection
	ExpectedDelay string `json:"expectedDelay,omitempty" yaml:"expectedDelay,omitempty"`

	// Detection hints
	DetectionMethod DetectionMethod `json:"detectionMethod,omitempty" yaml:"detectionMethod,omitempty"`
	Patterns        []string        `json:"patterns,omitempty" yaml:"patterns,omitempty"` // Regex patterns indicating success

	// Context requirements
	RequiresContext *PayloadContext `json:"requiresContext,omitempty" yaml:"requiresContext,omitempty"`

	// Tags for filtering
	Tags []string `json:"tags,omitempty" yaml:"tags,omitempty"`

	// Risk level (some payloads are more dangerous)
	Risk PayloadRisk `json:"risk,omitempty" yaml:"risk,omitempty"`
}

Payload represents a single test payload.

type PayloadContext 

type PayloadContext struct {
	QuoteChar    string `json:"quoteChar,omitempty" yaml:"quoteChar,omitempty"`       // Required quote character
	IsNumeric    bool   `json:"isNumeric,omitempty" yaml:"isNumeric,omitempty"`       // Numeric context
	IsInClause   bool   `json:"isInClause,omitempty" yaml:"isInClause,omitempty"`     // IN clause context
	IsOrderBy    bool   `json:"isOrderBy,omitempty" yaml:"isOrderBy,omitempty"`       // ORDER BY context
	DatabaseType string `json:"databaseType,omitempty" yaml:"databaseType,omitempty"` // Required database type
}

PayloadContext defines context requirements for a payload.

type PayloadGenerator 

type PayloadGenerator struct {
	Type       PayloadGeneratorType   `json:"type" yaml:"type"`
	Parameters map[string]interface{} `json:"parameters,omitempty" yaml:"parameters,omitempty"`
}

PayloadGenerator defines how to generate payloads dynamically.

type PayloadGeneratorType 

type PayloadGeneratorType string

PayloadGeneratorType identifies the type of payload generator. 

const (
	GeneratorWordlist PayloadGeneratorType = "wordlist" // Load from wordlist file
	GeneratorRange    PayloadGeneratorType = "range"    // Numeric range
	GeneratorPermute  PayloadGeneratorType = "permute"  // Permutations of characters
	GeneratorEncode   PayloadGeneratorType = "encode"   // Apply encoding to existing payloads
	GeneratorFuzz     PayloadGeneratorType = "fuzz"     // Fuzzing mutations
	GeneratorLLM      PayloadGeneratorType = "llm"      // LLM-generated payloads
)

type PayloadRisk 

type PayloadRisk string

PayloadRisk indicates the risk level of using a payload. 

const (
	RiskSafe        PayloadRisk = "safe"        // Safe to use, no side effects
	RiskLow         PayloadRisk = "low"         // Minor side effects possible
	RiskMedium      PayloadRisk = "medium"      // Moderate side effects
	RiskHigh        PayloadRisk = "high"        // Significant side effects possible
	RiskDestructive PayloadRisk = "destructive" // May cause data loss or system damage
)

type PayloadSet 

type PayloadSet struct {
	ID          string `json:"id" yaml:"id"`
	Name        string `json:"name" yaml:"name"`
	Description string `json:"description,omitempty" yaml:"description,omitempty"`

	// Classification
	SecurityType SecurityType `json:"securityType" yaml:"securityType"`
	DatabaseType string       `json:"databaseType,omitempty" yaml:"databaseType,omitempty"` // mysql, postgres, mssql, sqlite

	// Payloads
	Payloads []Payload `json:"payloads" yaml:"payloads"`

	// Metadata
	Source  string   `json:"source,omitempty" yaml:"source,omitempty"` // Where these payloads came from
	Tags    []string `json:"tags,omitempty" yaml:"tags,omitempty"`
	Version string   `json:"version,omitempty" yaml:"version,omitempty"`
}

PayloadSet is a reusable collection of payloads.

func LoadPayloadSet 

func LoadPayloadSet(path string) (*PayloadSet, error)

LoadPayloadSet loads a PayloadSet from a file.

type PayloadTransform 

type PayloadTransform struct {
	Type       TransformType `json:"type" yaml:"type"`
	Parameters interface{}   `json:"parameters,omitempty" yaml:"parameters,omitempty"`
}

PayloadTransform defines a transformation to apply to payloads.

type RecordingConfig 

type RecordingConfig struct {
	Enabled      bool   `json:"enabled" yaml:"enabled"`
	ExportFormat string `json:"exportFormat,omitempty" yaml:"exportFormat,omitempty"` // yaml, json
	OutputFile   string `json:"outputFile,omitempty" yaml:"outputFile,omitempty"`
}

RecordingConfig configures LLM action recording.

type ReporterConfig 

type ReporterConfig struct {
	Type    ReporterType `json:"type" yaml:"type"`
	Output  string       `json:"output,omitempty" yaml:"output,omitempty"` // Output path
	Options interface{}  `json:"options,omitempty" yaml:"options,omitempty"`
}

ReporterConfig configures output reporters.

type ReporterType 

type ReporterType string

ReporterType identifies the output format. 

const (
	ReporterSARIF     ReporterType = "sarif"      // SARIF for security findings
	ReporterJUnit     ReporterType = "junit"      // JUnit XML for test results
	ReporterJSON      ReporterType = "json"       // JSON for detailed evidence
	ReporterHTML      ReporterType = "html"       // HTML for human-readable reports
	ReporterConsole   ReporterType = "console"    // Console output
	ReporterTrafficIR ReporterType = "traffic-ir" // traffic2openapi IR for OpenAPI spec generation
)

type SQLContext 

type SQLContext struct {
	DatabaseType string `json:"databaseType,omitempty" yaml:"databaseType,omitempty"` // mysql, postgres, mssql, sqlite
	QuoteChar    string `json:"quoteChar,omitempty" yaml:"quoteChar,omitempty"`       // ', ", `
	IsNumeric    bool   `json:"isNumeric,omitempty" yaml:"isNumeric,omitempty"`
	IsInClause   bool   `json:"isInClause,omitempty" yaml:"isInClause,omitempty"`
	IsOrderBy    bool   `json:"isOrderBy,omitempty" yaml:"isOrderBy,omitempty"`
}

SQLContext provides context for SQL injection testing.

type SecurityType 

type SecurityType string

SecurityType identifies the type of security vulnerability being tested. 

const (
	SecSQLi       SecurityType = "sqli"
	SecXSS        SecurityType = "xss"
	SecIDOR       SecurityType = "idor"
	SecRCE        SecurityType = "rce"
	SecCSRF       SecurityType = "csrf"
	SecAuthBypass SecurityType = "auth-bypass"
	SecBOLA       SecurityType = "bola"            // Broken Object Level Authorization
	SecBFLA       SecurityType = "bfla"            // Broken Function Level Authorization
	SecMassAssign SecurityType = "mass-assignment" // Mass Assignment
	SecInjection  SecurityType = "injection"       // Generic injection
	SecSSRF       SecurityType = "ssrf"            // Server-Side Request Forgery
	SecXXE        SecurityType = "xxe"             // XML External Entity
	SecPathTrav   SecurityType = "path-traversal"  // Path Traversal
)

type Severity 

type Severity string

Severity represents the severity level of a security finding. 

const (
	SeverityCritical Severity = "critical"
	SeverityHigh     Severity = "high"
	SeverityMedium   Severity = "medium"
	SeverityLow      Severity = "low"
	SeverityInfo     Severity = "info"
)

type TargetSpec 

type TargetSpec struct {
	BaseURL string    `json:"baseUrl" yaml:"baseUrl"`
	OpenAPI string    `json:"openapi,omitempty" yaml:"openapi,omitempty"` // Path to OpenAPI spec
	Auth    *AuthSpec `json:"auth,omitempty" yaml:"auth,omitempty"`

	// For web testing
	Browser *BrowserConfig `json:"browser,omitempty" yaml:"browser,omitempty"`
}

TargetSpec defines the application under test.

type TestCase 

type TestCase struct {
	// Metadata
	ID          string   `json:"id" yaml:"id"`
	Name        string   `json:"name" yaml:"name"`
	Description string   `json:"description,omitempty" yaml:"description,omitempty"`
	Tags        []string `json:"tags,omitempty" yaml:"tags,omitempty"`

	// Classification
	Kind      TestKind      `json:"kind" yaml:"kind"`           // api-functional, api-security, web-functional, web-security
	Category  TestCategory  `json:"category" yaml:"category"`   // functional, security, performance, accessibility
	Execution ExecutionMode `json:"execution" yaml:"execution"` // deterministic, llm-driven, hybrid

	// Security-specific (optional)
	SecurityType *SecurityType `json:"securityType,omitempty" yaml:"securityType,omitempty"`
	Severity     *Severity     `json:"severity,omitempty" yaml:"severity,omitempty"`

	// Execution specification (one of)
	API   *APITestSpec   `json:"api,omitempty" yaml:"api,omitempty"`
	Web   *WebTestSpec   `json:"web,omitempty" yaml:"web,omitempty"`
	Agent *AgentTestSpec `json:"agent,omitempty" yaml:"agent,omitempty"`

	// Assertions
	Assertions []Assertion `json:"assertions,omitempty" yaml:"assertions,omitempty"`

	// Dependencies
	DependsOn []string `json:"dependsOn,omitempty" yaml:"dependsOn,omitempty"`
	Setup     []string `json:"setup,omitempty" yaml:"setup,omitempty"`       // TestCase IDs to run first
	Teardown  []string `json:"teardown,omitempty" yaml:"teardown,omitempty"` // TestCase IDs to run after
}

TestCase represents a single test scenario.

func LoadTestCase 

func LoadTestCase(path string) (*TestCase, error)

LoadTestCase loads a TestCase from a file.

func LoadTestCasesFromDir 

func LoadTestCasesFromDir(dir string) ([]*TestCase, error)

LoadTestCasesFromDir loads all test cases from a directory.

type TestCategory 

type TestCategory string

TestCategory represents the high-level category of testing. 

const (
	CategoryFunctional    TestCategory = "functional"
	CategorySecurity      TestCategory = "security"
	CategoryPerformance   TestCategory = "performance"
	CategoryAccessibility TestCategory = "accessibility"
)

type TestKind 

type TestKind string

TestKind identifies the type of test. 

const (
	KindAPIFunctional TestKind = "api-functional"
	KindAPISecurity   TestKind = "api-security"
	KindWebFunctional TestKind = "web-functional"
	KindWebSecurity   TestKind = "web-security"
)

type TestPlan 

type TestPlan struct {
	Version     string `json:"version" yaml:"version"` // Schema version
	ID          string `json:"id" yaml:"id"`
	Name        string `json:"name" yaml:"name"`
	Description string `json:"description,omitempty" yaml:"description,omitempty"`

	// Target application
	Target TargetSpec `json:"target" yaml:"target"`

	// Test suites to execute
	Suites    []TestSuite `json:"suites,omitempty" yaml:"suites,omitempty"`
	SuiteRefs []string    `json:"suiteRefs,omitempty" yaml:"suiteRefs,omitempty"`

	// Shared resources
	PayloadSets map[string]PayloadSet `json:"payloadSets,omitempty" yaml:"payloadSets,omitempty"`
	Variables   map[string]string     `json:"variables,omitempty" yaml:"variables,omitempty"`

	// Agent definitions (or references to multi-agent-spec)
	Agents map[string]AgentReference `json:"agents,omitempty" yaml:"agents,omitempty"`

	// Execution configuration
	Config ExecutionConfig `json:"config,omitempty" yaml:"config,omitempty"`

	// Reporting
	Reporters []ReporterConfig `json:"reporters,omitempty" yaml:"reporters,omitempty"`
}

TestPlan is the top-level orchestration document.

func LoadTestPlan 

func LoadTestPlan(path string) (*TestPlan, error)

LoadTestPlan loads a TestPlan from a file.

type TestSuite 

type TestSuite struct {
	ID          string   `json:"id" yaml:"id"`
	Name        string   `json:"name" yaml:"name"`
	Description string   `json:"description,omitempty" yaml:"description,omitempty"`
	Tags        []string `json:"tags,omitempty" yaml:"tags,omitempty"`

	// Test cases (inline or references)
	Tests    []TestCase `json:"tests,omitempty" yaml:"tests,omitempty"`
	TestRefs []string   `json:"testRefs,omitempty" yaml:"testRefs,omitempty"` // Paths to test case files

	// Suite-level setup/teardown
	BeforeAll  []TestCase `json:"beforeAll,omitempty" yaml:"beforeAll,omitempty"`
	AfterAll   []TestCase `json:"afterAll,omitempty" yaml:"afterAll,omitempty"`
	BeforeEach []TestCase `json:"beforeEach,omitempty" yaml:"beforeEach,omitempty"`
	AfterEach  []TestCase `json:"afterEach,omitempty" yaml:"afterEach,omitempty"`

	// Execution config
	Parallel   bool `json:"parallel,omitempty" yaml:"parallel,omitempty"`
	MaxWorkers int  `json:"maxWorkers,omitempty" yaml:"maxWorkers,omitempty"`
	StopOnFail bool `json:"stopOnFail,omitempty" yaml:"stopOnFail,omitempty"`
}

TestSuite groups related test cases.

func LoadTestSuite 

func LoadTestSuite(path string) (*TestSuite, error)

LoadTestSuite loads a TestSuite from a file.

type TransformType 

type TransformType string

TransformType identifies the type of payload transformation. 

const (
	TransformURLEncode     TransformType = "url-encode"
	TransformURLEncodeFull TransformType = "url-encode-full"
	TransformBase64        TransformType = "base64"
	TransformHTMLEncode    TransformType = "html-encode"
	TransformHex           TransformType = "hex"
	TransformUnicode       TransformType = "unicode"
	TransformDoubleEncode  TransformType = "double-encode"
	TransformCase          TransformType = "case" // upper, lower, mixed
	TransformPadding       TransformType = "padding"
	TransformComment       TransformType = "comment" // SQL comment insertion
	TransformSpace         TransformType = "space"   // Space alternatives
)

type Viewport 

type Viewport struct {
	Width  int `json:"width" yaml:"width"`
	Height int `json:"height" yaml:"height"`
}

Viewport defines browser viewport dimensions.

type WaitCondition 

type WaitCondition struct {
	Type     WaitType `json:"type" yaml:"type"`
	Selector string   `json:"selector,omitempty" yaml:"selector,omitempty"`
	Timeout  Duration `json:"timeout,omitempty" yaml:"timeout,omitempty"`
	Value    string   `json:"value,omitempty" yaml:"value,omitempty"`
}

WaitCondition defines what to wait for before proceeding.

type WaitType 

type WaitType string

WaitType identifies what condition to wait for. 

const (
	WaitElement    WaitType = "element"    // Wait for element to appear
	WaitVisible    WaitType = "visible"    // Wait for element to be visible
	WaitHidden     WaitType = "hidden"     // Wait for element to be hidden
	WaitNavigation WaitType = "navigation" // Wait for navigation to complete
	WaitNetwork    WaitType = "network"    // Wait for network to be idle
	WaitTimeout    WaitType = "timeout"    // Fixed timeout
	WaitFunction   WaitType = "function"   // Wait for JS function to return true
)

type WebAction 

type WebAction string

WebAction identifies the action to perform in a web step. 

const (
	ActionNavigate   WebAction = "navigate"
	ActionClick      WebAction = "click"
	ActionFill       WebAction = "fill"       // Clear and type
	ActionType       WebAction = "type"       // Type without clearing
	ActionPress      WebAction = "press"      // Press key
	ActionSelect     WebAction = "select"     // Select dropdown option
	ActionCheck      WebAction = "check"      // Check checkbox
	ActionUncheck    WebAction = "uncheck"    // Uncheck checkbox
	ActionHover      WebAction = "hover"      // Hover over element
	ActionScroll     WebAction = "scroll"     // Scroll to element
	ActionWait       WebAction = "wait"       // Wait for condition
	ActionScreenshot WebAction = "screenshot" // Take screenshot
	ActionEval       WebAction = "eval"       // Evaluate JavaScript
	ActionUpload     WebAction = "upload"     // Upload file
	ActionDownload   WebAction = "download"   // Download file
)

type WebAssertion 

type WebAssertion struct {
	Type     WebAssertionType `json:"type" yaml:"type"`
	Selector string           `json:"selector,omitempty" yaml:"selector,omitempty"`
	Value    string           `json:"value,omitempty" yaml:"value,omitempty"`
	Operator string           `json:"operator,omitempty" yaml:"operator,omitempty"` // equals, contains, matches, visible, hidden
	Message  string           `json:"message,omitempty" yaml:"message,omitempty"`
}

WebAssertion defines an assertion on web page state.

type WebAssertionType 

type WebAssertionType string

WebAssertionType identifies the type of web assertion. 

const (
	WebAssertText      WebAssertionType = "text"      // Element text content
	WebAssertValue     WebAssertionType = "value"     // Input value
	WebAssertAttribute WebAssertionType = "attribute" // Element attribute
	WebAssertVisible   WebAssertionType = "visible"   // Element is visible
	WebAssertHidden    WebAssertionType = "hidden"    // Element is hidden
	WebAssertExists    WebAssertionType = "exists"    // Element exists in DOM
	WebAssertNotExists WebAssertionType = "notExists" // Element does not exist
	WebAssertURL       WebAssertionType = "url"       // Current URL
	WebAssertTitle     WebAssertionType = "title"     // Page title
	WebAssertCookie    WebAssertionType = "cookie"    // Cookie value
	WebAssertStorage   WebAssertionType = "storage"   // Local/session storage
	WebAssertConsole   WebAssertionType = "console"   // Console output (for XSS detection)
	WebAssertDialog    WebAssertionType = "dialog"    // Alert/confirm/prompt dialog
	WebAssertNoErrors  WebAssertionType = "noErrors"  // No JavaScript errors
)

type WebExecutionMode 

type WebExecutionMode string

WebExecutionMode identifies how a web test is executed. 

const (
	WebModeScript WebExecutionMode = "script" // Deterministic script execution
	WebModeMCP    WebExecutionMode = "mcp"    // MCP tool-based (vibium-go)
	WebModeRecord WebExecutionMode = "record" // Record LLM actions for replay
)

type WebInjection 

type WebInjection struct {
	// This step contains payload
	PayloadMarker bool `json:"payloadMarker" yaml:"payloadMarker"`

	// Payload source
	Payloads   []string `json:"payloads,omitempty" yaml:"payloads,omitempty"`
	PayloadSet string   `json:"payloadSet,omitempty" yaml:"payloadSet,omitempty"`

	// Security type being tested
	SecurityType SecurityType `json:"securityType,omitempty" yaml:"securityType,omitempty"`
}

WebInjection marks a web step as an injection point.

type WebStep 

type WebStep struct {
	// Action to perform
	Action WebAction `json:"action" yaml:"action"` // navigate, click, fill, type, wait, screenshot, etc.

	// Target element
	Selector string `json:"selector,omitempty" yaml:"selector,omitempty"`

	// Value for input actions
	Value string `json:"value,omitempty" yaml:"value,omitempty"`

	// Additional options
	Options map[string]interface{} `json:"options,omitempty" yaml:"options,omitempty"`

	// For security tests - mark injection points
	Injection *WebInjection `json:"injection,omitempty" yaml:"injection,omitempty"`

	// Assertions at this step
	Assert []WebAssertion `json:"assert,omitempty" yaml:"assert,omitempty"`

	// Store result for later use
	Store string `json:"store,omitempty" yaml:"store,omitempty"`

	// Wait condition before proceeding
	WaitFor *WaitCondition `json:"waitFor,omitempty" yaml:"waitFor,omitempty"`
}

WebStep represents a single action in a web test (compatible with vibium-go).

type WebTestSpec 

type WebTestSpec struct {
	// Target URL
	URL string `json:"url" yaml:"url"`

	// Execution mode
	Mode WebExecutionMode `json:"mode" yaml:"mode"` // script, mcp, record

	// For deterministic execution (vibium-go script format)
	Steps []WebStep `json:"steps,omitempty" yaml:"steps,omitempty"`

	// For LLM-driven execution (reference to agent)
	Agent *AgentReference `json:"agent,omitempty" yaml:"agent,omitempty"`

	// For recording mode (LLM explores, then exports deterministic)
	Recording *RecordingConfig `json:"recording,omitempty" yaml:"recording,omitempty"`

	// Browser configuration
	Browser *BrowserConfig `json:"browser,omitempty" yaml:"browser,omitempty"`
}

WebTestSpec defines a browser-based test (compatible with vibium-go).

Source Files

View all Source files

Directories

Path Synopsis
Package executor provides test execution with HTTP traffic capture.
 Package executor provides test execution with HTTP traffic capture.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.