Documentation
¶
Overview ¶
Package run executes MPQT packs against a target and rolls the results up into a qual.Scorecard. It is the shared execution core behind both pkg/mpqttest (offline, go-test-driven runs) and the CLI (live runs against a real inference client): both re-express their own Spec in terms of this package's Execute and never re-implement the plan -> eval.Run -> scorecard pipeline themselves.
pkg/run never constructs an inference.Client itself ("dependency confinement"): BuildTarget accepts a caller-supplied client and turns a table's environment template plus the run's manifest into a live eval.Target. Offline runs never call BuildTarget at all; they hand Execute a pre-built fixture target directly.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildTarget ¶
func BuildTarget(client inference.Client, m qual.Manifest, env *packfile.Environment, tableRevision eval.Revision) (eval.Target, error)
BuildTarget constructs the live inference target for one table: it expands env into the provider-neutral request template (Environment.Template, pkg/packfile), stamps the manifest's model onto that template, and wraps both in an eval.Target revisioned to tableRevision so Sample.Validate accepts observations against that table's suite (ground rule 2). client is supplied by the caller (the CLI module) — BuildTarget never constructs one itself.
A nil env is legal (a table without an environment yields a zero request template, per Environment.Template). m is validated before use, so an unknown or malformed manifest field — including an unknown capability — is rejected here rather than silently producing an under- or over-permissioned model descriptor.
func DecodeManifest ¶
DecodeManifest strictly decodes r (via packfile.StrictDecode, so an unknown field is rejected the same way the rest of the packfile YAML corpus rejects one) into a qual.Manifest and validates it before returning. This package never imports gopkg.in/yaml.v3 itself, reusing packfile's strict-decode logic instead; pkg/gen imports yaml.v3 directly for an unrelated concern (encode-side yaml.Node surgery).
func DecodeProfile ¶
DecodeProfile strictly decodes r (via packfile.StrictDecode) into a profile.Profile and validates it before returning.
func ManifestModel ¶
ManifestModel maps a secret-free qual.Manifest onto the inference module's model.Model descriptor used to build a live target: Provider, Model (as Name), APIFormat, and BaseURL carry across directly, Effort becomes the model's default Sampling.Effort, and each qual.Capability sets the matching model.Capabilities field (Tools, StructuredOutput, Images -> AcceptsImages, Thinking -> Thinking). ManifestModel assumes m is already valid (m.Capabilities holds only known values); BuildTarget validates m first, so an unknown capability is rejected there rather than silently dropped here.
Types ¶
type Result ¶
Result binds the rolled-up scorecard to its per-table eval reports and the skipped-table plans (visible coverage, never silent).
func Execute ¶
Execute plans every pack in s against s.Manifest and runs each runnable table plan through eval.Run using the plan's own Suite and Evaluators. A non-runnable plan (missing capability) contributes a Skipped TableResult and is retained verbatim in Result.Skipped; it is never executed and never silently dropped. Execute returns an error from Spec validation, from qual.Plan (an ill-formed pack or manifest), from Spec.TargetForTable, or from eval.Run's own preflight check. Only Spec validation failure (before any pack is planned or any target invoked) yields a zero-value Result: once any table has actually been planned or run, a later failure returns the Result accumulated from every pack/table processed successfully so far ALONGSIDE the error, rather than discarding it. This extends the same "visible coverage, never silent" philosophy behind Result.Skipped to the error path — an operator running many tables against a live target should not lose every already-executed (and potentially paid) table's report just because a later table hit a transient error. Callers must check the error to know whether the run as a whole completed; a non-nil error always means something failed, but the accompanying Result may still carry partial, trustworthy coverage worth reporting.
type Spec ¶
type Spec struct {
Manifest qual.Manifest
Packs []qual.Pack
Target eval.Target
TargetForTable func(qual.TablePlan) (eval.Target, error)
Config eval.RunConfig // zero value = eval defaults
}
Spec is one qualification execution. Target may be any eval.Target (scripted fixture or live inference target); pkg/run never constructs clients (design: "Dependency confinement").
Exactly one of Target and TargetForTable must be set. Target is used as-is for every table (the offline case: one fixture target answers every scenario in every pack). TargetForTable builds one target per table plan (the live case: the CLI constructs a fresh live target per table from that table's own environment via BuildTarget). Setting both, or neither, is a configuration error rejected by Execute before any pack is planned or any target is invoked.
Source Files