Documentation
¶
Overview ¶
Package corpus holds the command-safety classifier's versioned Codex-parity evaluation corpus: independently worded synthetic scenarios, each paired with the assessment a correct classifier should produce and the resulting end-to-end eligibility, plus a short record of how the case compares to the corresponding Codex Guardian scenario (design §22.6/§22.7).
This package builds real github.com/looprig/harness/pkg/gate values from each fixture (a fully digest-stamped gate.PermissionReviewSubject) so the deterministic evaluation runner in pkg/commandsafety can exercise the classifier's actual MarshalInput/ValidateResult pipeline and Harness's own unmodified gate.EvaluatePermissionAssessment, never a reimplementation of either.
Index ¶
Constants ¶
const Revision = "command-safety-corpus/v1"
Revision is the corpus's own content revision label, independent of the classifier's prompt/policy/schema/wire revisions (design §19.2). Bump it whenever a case's scenario, expected assessment, or parity record changes meaning, per docs/evaluations/README.md.
Variables ¶
This section is empty.
Functions ¶
func ValidDesignCategory ¶
func ValidDesignCategory(category DesignCategory) bool
ValidDesignCategory reports whether category belongs to the closed §22.6 domain.
Types ¶
type Case ¶
type Case struct {
ID string `json:"id"`
DesignCategories []DesignCategory `json:"design_categories"`
Description string `json:"description"`
Request RequestFixture `json:"request"`
ContextEntries []ContextEntryFixture `json:"context_entries"`
Truncation TruncationFixture `json:"truncation"`
Expected ExpectedAssessmentFixture `json:"expected"`
ExpectedEligible bool `json:"expected_eligible"`
Parity CodexParity `json:"codex_parity"`
}
Case is one versioned, independently worded corpus fixture.
func Load ¶
Load decodes and validates every case in testdata, in a stable order (sorted by ID, independent of file layout), and rejects a duplicate ID. Every returned case has already passed Case.Validate.
func (Case) ParseExpected ¶
func (c Case) ParseExpected() (ParsedExpectation, error)
ParseExpected decodes Expected into gate's closed types, rejecting any unsupported enum value or invalid category list.
func (Case) Subject ¶
func (c Case) Subject() (gate.PermissionReviewSubject, error)
Subject builds the real, digest-stamped gate.PermissionReviewSubject this case reviews. The review context is always built through Harness's own gate.BuildReviewContext: ordinary cases use a generous policy under which nothing truncates, while a case with Truncation.Force uses a deliberately tightened one so any resulting material truncation is genuinely computed by Harness, never hand-asserted.
func (Case) Validate ¶
Validate checks a case's own structural invariants: required text fields, a closed and non-empty DesignCategories list, a parseable Expected assessment, and a parity record whose Comparison is one of the two closed values with a non-empty Justification whenever it claims "stricter".
type CodexParity ¶
type CodexParity struct {
GuardianScenario string `json:"guardian_scenario"`
Comparison CodexParityComparison `json:"comparison"`
Justification string `json:"justification"`
}
CodexParity records, in the corpus author's own words, which Guardian scenario or category a case corresponds to and whether this classifier's expected result is equal to or stricter than what that Guardian scenario would produce.
type CodexParityComparison ¶
type CodexParityComparison string
CodexParityComparison is the closed comparison between this classifier's expected result for a case and the corresponding Guardian scenario. There is deliberately no "looser" value: the type system makes an undocumented looser result unrepresentable, and a case whose comparison is neither of these two values fails Case.Validate.
const ( ParityEqual CodexParityComparison = "equal" ParityStricter CodexParityComparison = "stricter" )
type ContextEntryFixture ¶
type ContextEntryFixture struct {
Origin string `json:"origin"`
Kind string `json:"kind"`
Content string `json:"content"`
}
ContextEntryFixture is one live review-context entry. Origin and Kind must form one of gate's closed pairs (e.g. "user"/"user_message", "tool"/"tool_result").
type DesignCategory ¶
type DesignCategory string
DesignCategory is one closed scenario category from design §22.6 the corpus is required to cover at least once. Values are deliberately granular (e.g. deletion_narrow and deletion_broad are distinct) so completeness checking is genuinely load-bearing rather than a coarse per-bullet checklist.
const ( DesignRoutineBuild DesignCategory = "routine_build" DesignRoutineTest DesignCategory = "routine_test" DesignRoutineFormat DesignCategory = "routine_format" DesignRoutineLint DesignCategory = "routine_lint" DesignPackageInstallation DesignCategory = "package_installation" DesignToolInstallation DesignCategory = "tool_installation" DesignSandboxRetry DesignCategory = "sandbox_retry" DesignSandboxEscalation DesignCategory = "sandbox_escalation" DesignDeletionNarrow DesignCategory = "deletion_narrow" DesignDeletionBroad DesignCategory = "deletion_broad" DesignDeletionTargetMissing DesignCategory = "deletion_target_missing" DesignDeletionTargetEmpty DesignCategory = "deletion_target_empty" DesignDeletionTargetSymlinked DesignCategory = "deletion_target_symlinked" DesignDeletionTargetChanged DesignCategory = "deletion_target_changed" DesignGitStatus DesignCategory = "git_status" DesignGitCheckout DesignCategory = "git_checkout" DesignGitReset DesignCategory = "git_reset" DesignGitClean DesignCategory = "git_clean" DesignGitForcePush DesignCategory = "git_force_push" DesignGitBranchDeletion DesignCategory = "git_branch_deletion" DesignGitProtectedBranch DesignCategory = "git_protected_branch" DesignCredentialUse DesignCategory = "credential_use" // #nosec G101 -- Closed taxonomy label, not a credential. DesignCredentialReading DesignCategory = "credential_reading" // #nosec G101 -- Closed taxonomy label, not a credential. DesignCredentialExfiltration DesignCategory = "credential_exfiltration" // #nosec G101 -- Closed taxonomy label, not a credential. DesignRemotePublic DesignCategory = "remote_public" DesignRemotePrivate DesignCategory = "remote_private" DesignRemoteInternal DesignCategory = "remote_internal" DesignRemoteUnknown DesignCategory = "remote_unknown" DesignUploadDataMovement DesignCategory = "upload_data_movement" DesignProductionMutation DesignCategory = "production_mutation" DesignSecurityControlChange DesignCategory = "security_control_change" DesignArbitraryCodeExecution DesignCategory = "arbitrary_code_execution" DesignAuthorizationExplicit DesignCategory = "authorization_explicit" DesignAuthorizationImplied DesignCategory = "authorization_implied" DesignAuthorizationAbsent DesignCategory = "authorization_absent" DesignAuthorizationConflicting DesignCategory = "authorization_conflicting" DesignAuthorizationPostWarning DesignCategory = "authorization_post_warning" DesignInjectionInFile DesignCategory = "injection_in_file" DesignInjectionInToolResult DesignCategory = "injection_in_tool_result" DesignInjectionInRepoInstructions DesignCategory = "injection_in_repo_instructions" DesignInjectionInCommandArguments DesignCategory = "injection_in_command_arguments" DesignInjectionInFetchedContent DesignCategory = "injection_in_fetched_content" DesignTruncatedMaterialContext DesignCategory = "truncated_material_context" DesignEvidenceFailure DesignCategory = "evidence_failure" )
func AllDesignCategories ¶
func AllDesignCategories() []DesignCategory
AllDesignCategories returns the closed, complete design §22.6 coverage domain the corpus completeness test requires at least one case for.
type ExpectedAssessmentFixture ¶
type ExpectedAssessmentFixture struct {
Risk string `json:"risk"`
Authorization string `json:"authorization"`
Categories []string `json:"categories"`
Recommendation string `json:"recommendation"`
Rationale string `json:"rationale"`
}
ExpectedAssessmentFixture is the assessment a correct classifier should produce for the case, in the wire string form of gate's closed enums.
type ParsedExpectation ¶
type ParsedExpectation struct {
Risk gate.ReviewRisk
Authorization gate.ReviewAuthorization
Categories []gate.ReviewRiskCategory
Recommendation gate.ReviewRecommendation
Rationale string
}
ParsedExpectation is Case.Expected decoded into gate's closed types.
type RequestFixture ¶
type RequestFixture struct {
ToolName string `json:"tool_name"`
Summary string `json:"summary"`
Command string `json:"command"`
WorkingDirectory string `json:"working_directory"`
}
RequestFixture is the synthetic prepared-command shape a case reviews. Its single command.execute requirement is derived automatically by Case.Subject; a fixture never lists requirements directly, so the requirement can never drift out of sync with Command (see tool.ValidateRequest's command-grant invariant).
type TruncationFixture ¶
type TruncationFixture struct {
Force bool `json:"force"`
TargetKind string `json:"target_kind"`
LimitBytes int `json:"limit_bytes"`
}
TruncationFixture configures whether Case.Subject builds the review context under a deliberately tightened per-kind byte budget. When Force is true, Case.Subject uses TargetKind/LimitBytes instead of the generous default policy, through Harness's own gate.BuildReviewContext, so the resulting material truncation is genuinely computed by Harness rather than hand-asserted.
Source Files