Documentation
¶
Overview ¶
Package testcase defines test cases — individual evaluation scenarios within a suite. Cases can be traditional (input/output) or persona-aware (targeting specific human-like dimensions).
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Case ¶
type Case struct {
sentinel.Entity
ID id.CaseID `json:"id" bun:"id,pk"`
SuiteID id.SuiteID `json:"suite_id" bun:"suite_id,notnull"`
Name string `json:"name" bun:"name,notnull"`
Input string `json:"input" bun:"input,notnull"`
Expected string `json:"expected,omitempty" bun:"expected"`
ScenarioType ScenarioType `json:"scenario_type" bun:"scenario_type,notnull,default:'standard'"`
Scorers []ScorerConfig `json:"scorers" bun:"scorers,type:jsonb,default:'[]'"`
Tags []string `json:"tags" bun:"tags,type:jsonb,default:'[]'"`
Context map[string]any `json:"context,omitempty" bun:"context,type:jsonb,default:'{}'"`
Metadata map[string]any `json:"metadata" bun:"metadata,type:jsonb,default:'{}'"`
}
Case is a single test case within an evaluation suite.
func ImportCSV ¶
ImportCSV parses test cases from CSV data. Expected columns: name, input, expected.
func ImportJSON ¶
ImportJSON parses test cases from a JSON array.
type ScenarioType ¶
type ScenarioType string
ScenarioType defines what aspect of the agent is being tested.
const ( ScenarioStandard ScenarioType = "standard" // Traditional input/output eval ScenarioSkillChallenge ScenarioType = "skill_challenge" // Test tool selection & proficiency ScenarioTraitProbe ScenarioType = "trait_probe" // Test personality consistency ScenarioBehaviorTrigger ScenarioType = "behavior_trigger" // Test condition-action patterns ScenarioCognitiveStress ScenarioType = "cognitive_stress" // Test thinking strategy transitions ScenarioCommsAdaptation ScenarioType = "comms_adaptation" // Test communication style ScenarioPerceptionTest ScenarioType = "perception_test" // Test attention & focus ScenarioPersonaCoherence ScenarioType = "persona_coherence" // Test end-to-end identity )
type ScorerConfig ¶
type ScorerConfig struct {
Name string `json:"name"`
Config map[string]any `json:"config,omitempty"`
}
ScorerConfig defines which scorer to apply to a case and its configuration.
type Store ¶
type Store interface {
CreateCase(ctx context.Context, tc *Case) error
CreateCaseBatch(ctx context.Context, cases []*Case) error
GetCase(ctx context.Context, caseID id.CaseID) (*Case, error)
UpdateCase(ctx context.Context, tc *Case) error
DeleteCase(ctx context.Context, caseID id.CaseID) error
ListCases(ctx context.Context, suiteID id.SuiteID) ([]*Case, error)
CountCases(ctx context.Context, suiteID id.SuiteID) (int64, error)
// ImportCases is unused. Every backend implemented it as a stub, and
// engine.ImportCases now parses and writes cases itself. It stays on the
// interface so custom stores keep compiling.
ImportCases(ctx context.Context, suiteID id.SuiteID, format string, data []byte) (int64, error)
}
Store defines persistence operations for test cases.
Source Files
Click to show internal directories.
Click to hide internal directories.