Documentation
¶
Overview ¶
Package qual qualifies a model or model configuration for enterprise use.
Pluto is a product and test-pack layer over github.com/looprig/eval. It contributes versioned packs, run manifests, bounded scorecards, and organization qualification profiles. It does not fork the eval runner and takes no runtime action against live sessions.
Index ¶
Constants ¶
const MaxManifestStringBytes = 256
MaxManifestStringBytes bounds every free-form manifest string, in bytes.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Capability ¶
type Capability string
Capability names a target feature a pack table may require. There is no valid zero value.
const ( CapabilityTools Capability = "tools" CapabilityStructuredOutput Capability = "structured_output" CapabilityImages Capability = "images" CapabilityThinking Capability = "thinking" )
func (Capability) Validate ¶
func (c Capability) Validate() error
Validate reports whether c is a known capability.
type DimensionScore ¶
type DimensionScore struct {
Dimension eval.Name
Score float64
Coverage float64
Verdicts int
Assessments int
SkippedTables int
Undecided bool
}
DimensionScore is a bounded [0,100] quality score with separately reported coverage. Score is the mean over verdict-bearing assessments only (pass=1, fail=0). Unverified, error, and skipped assessments contribute no quality value; they reduce Coverage (verdicts / non-skipped assessments). A dimension with zero verdicts is Undecided, never a silent zero or a silent pass.
type EndpointClass ¶
type EndpointClass string
EndpointClass records where the target executes, which bounds what Pluto can claim to have observed. There is no valid zero value.
const ( // EndpointRemote is hosted inference: only requests, responses, tool calls, // usage, latency, and errors are observable. EndpointRemote EndpointClass = "remote" // EndpointLocal is a local inference server on this host. EndpointLocal EndpointClass = "local" // EndpointProcess is a foreign process executed under sandbox control. EndpointProcess EndpointClass = "process" )
func (EndpointClass) Validate ¶
func (c EndpointClass) Validate() error
Validate reports whether c is a known endpoint class.
type Manifest ¶
type Manifest struct {
TargetID string
Role ModelRole
Provider string
Model string
APIFormat string
BaseURL string
Effort string
Revision eval.Revision
EndpointClass EndpointClass
Capabilities []Capability
}
Manifest is the secret-free identity of one model configuration under test. It deliberately has no credential field: authentication is resolved outside Pluto and never becomes part of a report. Fingerprint gives the manifest a stable reproducibility identity.
func (Manifest) Fingerprint ¶
Fingerprint returns a deterministic sha256 identity over the manifest's canonical JSON form. It validates first so an ill-formed manifest can never acquire an identity. Capabilities is order-independent (Validate only rejects duplicates, not reordering), so the hash input sorts a copy of Capabilities before marshaling; the receiver and any caller-owned slice are left untouched.
type ModelRole ¶
type ModelRole string
ModelRole distinguishes the candidate under qualification from the incumbent it is compared against. There is no valid zero value.
type Pack ¶
Pack is a versioned set of tables. Scenario IDs are unique across the whole pack so results remain unambiguous when tables are rolled up.
type Scorecard ¶
type Scorecard struct {
Manifest Manifest
Results []TableResult
}
Scorecard is the objective result of one Pluto run for one manifest. It carries no policy: dispositions are derived later by a profile.
func (Scorecard) Dimensions ¶
func (s Scorecard) Dimensions() ([]DimensionScore, error)
Dimensions rolls every table result up by dimension, in dimension name order. It fails on an empty scorecard: no evidence is not a score.
func (Scorecard) FindingCount ¶
func (s Scorecard) FindingCount(code eval.FindingCode) int
FindingCount counts findings carrying code across every executed (non skipped) table's assessments. This, together with SeverityCount, lets Scorecard satisfy profile.Card.
func (Scorecard) SeverityCount ¶
SeverityCount counts findings carrying severity across every executed (non skipped) table's assessments.
func (Scorecard) StatusRollup ¶
func (s Scorecard) StatusRollup() (StatusRollup, error)
StatusRollup computes the report-wide status rollup.
type StatSummary ¶
type StatSummary struct {
Count int
Mean float64
Median float64
Quantile float64
Min float64
Max float64
Variance float64
}
StatSummary is a deterministic numeric rollup. Quantile uses linear interpolation between closest ranks (the "linear" / type-7 method: the quantile q over n sorted values is taken at index q*(n-1), interpolating between neighbors). Variance is the unbiased sample variance (n-1 divisor), 0 for a single observation.
type StatusRollup ¶
type StatusRollup struct {
Samples int
TargetErrors int
ByStatus map[eval.AssessmentStatus]int
}
StatusRollup aggregates raw assessment status counts and sample counts over every executed table. Status counts are diagnostics, never quality values.
type Table ¶
type Table struct {
Name eval.Name
Revision eval.Revision
Dimension eval.Name
Requires []Capability
Scenarios []eval.Scenario
Evaluators []eval.Evaluator
}
Table is Pluto's unit of execution: a named, versioned scenario family that shares one evaluator set and contributes to one score dimension. A runnable table expands to exactly one eval.Suite; Pluto never adds a second runner.
type TablePlan ¶
type TablePlan struct {
Pack eval.Name
Table eval.Name
Dimension eval.Name
Runnable bool
Missing []Capability
Suite eval.Suite
Evaluators []eval.Evaluator
}
TablePlan is the preflight result for one table against one manifest. A non-runnable plan retains the table identity and the missing capabilities so the scorecard can report skipped coverage instead of silently dropping work.
type TableResult ¶
type TableResult struct {
Pack eval.Name
Table eval.Name
Dimension eval.Name
Skipped bool
Missing []Capability
Report eval.Report
}
TableResult is one table's outcome: either a full eval.Report or a capability skip retained from preflight. The raw report is preserved intact behind every rollup.
type ValidationError ¶
ValidationError reports a structurally invalid Pluto value. Following eval's convention, it names the field and reason but never echoes the offending value.
func (*ValidationError) Error ¶
func (e *ValidationError) Error() string
Source Files
Directories