Documentation
¶
Overview ¶
Package llm provides LLM-as-a-Judge functionality for accessibility evaluation.
Index ¶
- type Evaluation
- type Finding
- type Judge
- func (j *Judge) EvaluateFinding(ctx context.Context, finding Finding, pageContext PageContext) (*Evaluation, error)
- func (j *Judge) EvaluateFindings(ctx context.Context, findings []Finding, pageContext PageContext) ([]Evaluation, error)
- func (j *Judge) ReviewCategories() []string
- func (j *Judge) ShouldEvaluate(finding Finding) bool
- type JudgeConfig
- type PageContext
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Evaluation ¶
type Evaluation struct {
FindingID string `json:"findingId"`
Confirmed bool `json:"confirmed"`
Confidence float64 `json:"confidence"`
Severity string `json:"severity"`
Reasoning string `json:"reasoning"`
Remediation string `json:"remediation"`
// For items needing manual review
NeedsManualReview bool `json:"needsManualReview"`
ReviewGuidance string `json:"reviewGuidance,omitempty"`
// Metadata
Model string `json:"model"`
EvalTime time.Time `json:"evalTime"`
TokensIn int `json:"tokensIn"`
TokensOut int `json:"tokensOut"`
}
Evaluation represents the judge's evaluation of a finding.
type Finding ¶
type Finding struct {
ID string
RuleID string
Description string
SuccessCriteria []string
Level string
Impact string
Selector string
HTML string
Help string
}
Finding represents an accessibility finding to be evaluated. This is a simplified type to avoid import cycles with the audit package.
type Judge ¶
type Judge struct {
// contains filtered or unexported fields
}
Judge evaluates accessibility findings using LLM reasoning.
func NewJudge ¶
func NewJudge(client *omnillm.ChatClient, model string, logger *slog.Logger, config JudgeConfig) *Judge
NewJudge creates a new LLM judge using an omnillm client.
func (*Judge) EvaluateFinding ¶
func (j *Judge) EvaluateFinding(ctx context.Context, finding Finding, pageContext PageContext) (*Evaluation, error)
EvaluateFinding evaluates a single accessibility finding.
func (*Judge) EvaluateFindings ¶
func (j *Judge) EvaluateFindings(ctx context.Context, findings []Finding, pageContext PageContext) ([]Evaluation, error)
EvaluateFindings evaluates multiple findings concurrently.
func (*Judge) ReviewCategories ¶
ReviewCategories returns the categories this judge evaluates.
func (*Judge) ShouldEvaluate ¶
ShouldEvaluate checks if a finding should be evaluated by the LLM.
type JudgeConfig ¶
type JudgeConfig struct {
// Categories to evaluate
Categories []string
// Confidence threshold for automated decisions
ConfidenceThreshold float64
// Maximum concurrent evaluations
Concurrency int
// System prompt override
SystemPrompt string
}
JudgeConfig configures the LLM judge.
func DefaultJudgeConfig ¶
func DefaultJudgeConfig() JudgeConfig
DefaultJudgeConfig returns default configuration.
type PageContext ¶
type PageContext struct {
URL string `json:"url"`
Title string `json:"title"`
IsSPA bool `json:"isSPA"`
Framework string `json:"framework,omitempty"`
Language string `json:"language"`
Screenshot string `json:"screenshot,omitempty"` // Base64
HTMLSnippet string `json:"htmlSnippet,omitempty"`
}
PageContext provides context about the page being evaluated.
Source Files
Directories