Documentation
¶
Overview ¶
Package decisiontest reads and scores external System One evaluation corpora.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Answer ¶
type Answer struct {
Type string `json:"type"`
Choice string `json:"choice"`
Noul *float64 `json:"noul"`
Score *float64 `json:"score"`
Confidence *float64 `json:"confidence"`
Probabilities map[string]*float64 `json:"probabilities"`
Legend map[string]any `json:"legend"`
}
Pointers distinguish missing/null numbers from valid zero values.
type CacheExpectation ¶
type CacheExpectation struct {
Min *float64 `json:"min,omitempty"`
Max *float64 `json:"max,omitempty"`
}
CacheExpectation bounds the fraction of input tokens reused in an ordered run. Omitted telemetry is an error whenever a case requests a cache assertion.
type Case ¶
type Case struct {
ID string `json:"id"`
Dataset string `json:"dataset"`
Request decision.Request `json:"request"`
ImageFiles []string `json:"image_files,omitempty"`
Expected map[string]Expected `json:"expected"`
Cache *CacheExpectation `json:"cache,omitempty"`
ImageBytes int `json:"-"`
ImagePixels int64 `json:"-"`
}
func (Case) CheckCache ¶
func (Case) Mismatches ¶
Mismatches compares labels, never the model's confidence in those labels. result must have passed response validation in Do.
type Corpus ¶
type Response ¶
type Tally ¶
type Tally struct {
Requests int `json:"attempted_requests"`
Errors int `json:"errors"`
CorrectCases int `json:"correct_cases"`
Questions int `json:"questions"`
CorrectQuestions int `json:"correct_questions"`
}
Tally keeps failed requests in the denominator, including malformed responses.
func (Tally) CaseAccuracy ¶
func (Tally) QuestionAccuracy ¶
Click to show internal directories.
Click to hide internal directories.