decisiontest

package
v0.40.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Overview

Package decisiontest reads and scores external System One evaluation corpora.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Mode

func Mode(value string) (string, error)

Mode validates the benchmark's label policy.

Types

type Answer

type Answer struct {
	Type          string              `json:"type"`
	Choice        string              `json:"choice"`
	Noul          *float64            `json:"noul"`
	Score         *float64            `json:"score"`
	Confidence    *float64            `json:"confidence"`
	Probabilities map[string]*float64 `json:"probabilities"`
	Legend        map[string]any      `json:"legend"`
}

Pointers distinguish missing/null numbers from valid zero values.

type CacheExpectation

type CacheExpectation struct {
	Min *float64 `json:"min,omitempty"`
	Max *float64 `json:"max,omitempty"`
}

CacheExpectation bounds the fraction of input tokens reused in an ordered run. Omitted telemetry is an error whenever a case requests a cache assertion.

type Case

type Case struct {
	ID          string              `json:"id"`
	Dataset     string              `json:"dataset"`
	Request     decision.Request    `json:"request"`
	ImageFiles  []string            `json:"image_files,omitempty"`
	Expected    map[string]Expected `json:"expected"`
	Cache       *CacheExpectation   `json:"cache,omitempty"`
	ImageBytes  int                 `json:"-"`
	ImagePixels int64               `json:"-"`
}

func (Case) CheckCache

func (c Case) CheckCache(response Response) error

func (Case) Mismatches

func (c Case) Mismatches(result Response) []string

Mismatches compares labels, never the model's confidence in those labels. result must have passed response validation in Do.

func (Case) Options

func (c Case) Options() int

type Corpus

type Corpus struct {
	Cases  []Case
	SHA256 string
}

func Load

func Load(path string) (Corpus, error)

Load resolves image_files relative to the JSONL file, before any timed requests. The digest covers both the JSONL bytes and the referenced image contents.

func (Corpus) HasImages

func (c Corpus) HasImages() bool

type Expected

type Expected struct {
	Choice *string  `json:"choice,omitempty"`
	Noul   *bool    `json:"noul,omitempty"`
	Min    *float64 `json:"min,omitempty"`
	Max    *float64 `json:"max,omitempty"`
	Level  *int     `json:"level,omitempty"`
}

type Response

type Response struct {
	Model                 string            `json:"model"`
	Answers               map[string]Answer `json:"answers"`
	Usage                 decision.Usage    `json:"usage"`
	PromptEvalCachedCount *int              `json:"prompt_eval_cached_count"`
}

func Do

func Do(ctx context.Context, endpoint string, input decision.Request) (Response, time.Duration, error)

Do measures the complete HTTP round trip, excluding request encoding and response validation. The elapsed time includes network and server queueing.

type Tally

type Tally struct {
	Requests         int `json:"attempted_requests"`
	Errors           int `json:"errors"`
	CorrectCases     int `json:"correct_cases"`
	Questions        int `json:"questions"`
	CorrectQuestions int `json:"correct_questions"`
}

Tally keeps failed requests in the denominator, including malformed responses.

func (*Tally) Add

func (t *Tally) Add(c Case, wrong []string, err error)

func (Tally) CaseAccuracy

func (t Tally) CaseAccuracy() float64

func (Tally) QuestionAccuracy

func (t Tally) QuestionAccuracy() float64

func (Tally) String

func (t Tally) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL