Documentation
¶
Overview ¶
Package compare rolls up an eval/compare diff between a candidate and an incumbent MPQT scorecard, aligned by (Pack, Table). It performs no trial pairing of its own: github.com/looprig/eval/compare already classifies each (scenario, evaluator) case, retaining every per-trial result; this package only aligns tables across the two scorecards and rolls the retained case classification up per table.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Comparison ¶
type Comparison struct {
Candidate qual.Manifest
Incumbent qual.Manifest
Tables []TableComparison
UnmatchedTables []UnmatchedTable
}
Comparison is the full candidate-vs-incumbent diff: matched tables plus every table that could not be matched.
func Compare ¶
func Compare(candidate, incumbent qual.Scorecard) (Comparison, error)
Compare validates both manifests (structurally, and that candidate carries qual.RoleCandidate and incumbent carries qual.RoleIncumbent) before any comparison work happens, then aligns tables by (Pack, Table) key. A table present on only one side, or skipped on either side, surfaces in UnmatchedTables rather than being dropped. Every remaining matched pair is diffed with evalcompare.Compare (baseline=incumbent, candidate=candidate) and rolled up per table.
type RoleMismatchError ¶
RoleMismatchError reports that a manifest's role did not match the role required of its position in the comparison (candidate or incumbent).
func (*RoleMismatchError) Error ¶
func (e *RoleMismatchError) Error() string
type Side ¶
type Side string
Side names why a table failed to align between the two scorecards.
const ( // SideCandidateOnly: the table appears in the candidate scorecard but not // the incumbent's. SideCandidateOnly Side = "candidate-only" // SideIncumbentOnly: the table appears in the incumbent scorecard but not // the candidate's. SideIncumbentOnly Side = "incumbent-only" // SideSkipped: the table's (Pack, Table) key is present on both sides but // at least one side skipped it (missing capability), so there is no // executed report to compare. SideSkipped Side = "skipped" )
type TableComparison ¶
type TableComparison struct {
Pack, Table, Dimension eval.Name
Result evalcompare.Comparison
Regressed int
Improved int
Unchanged int
Incompatible int
}
TableComparison is one matched (Pack, Table) pair's rolled-up diff. Result is the full github.com/looprig/eval/compare output, retained intact. Regressed, Improved, Unchanged, and Incompatible are per-case counts over Result.Cases; see classifyCounts for the exact mapping from CaseClass (and, for Regressed/Improved, the per-side trial outcome) to these buckets. A case that is CaseAdded, CaseRemoved, or a CaseChanged/CaseFailed/ CaseErrored/CaseUnverified case that doesn't fit the Regressed/Improved definition below is not tallied into any of the four counts; it remains visible in Result.Cases.
type UnmatchedTable ¶
UnmatchedTable is a (Pack, Table) that could not be compared, with the reason it was excluded. It is never silently dropped from a Comparison.
Source Files