Documentation
¶
Overview ¶
Package run executes Pluto packs against a target and rolls the results up into a qual.Scorecard. It is the shared execution core behind both pkg/plutotest (offline, go-test-driven runs) and the CLI (live runs against a real inference client): both re-express their own Spec in terms of this package's Execute and never re-implement the plan -> eval.Run -> scorecard pipeline themselves.
pkg/run never constructs an inference.Client itself ("dependency confinement"): BuildTarget accepts a caller-supplied client and turns a table's environment template plus the run's manifest into a live eval.Target. Offline runs never call BuildTarget at all; they hand Execute a pre-built fixture target directly.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildTarget ¶
func BuildTarget(client inference.Client, m qual.Manifest, env *packfile.Environment, tableRevision eval.Revision) (eval.Target, error)
BuildTarget constructs the live inference target for one table: it expands env into the provider-neutral request template (Environment.Template, pkg/packfile), stamps the manifest's model onto that template, and wraps both in an eval.Target revisioned to tableRevision so Sample.Validate accepts observations against that table's suite (ground rule 2). client is supplied by the caller (the CLI module) — BuildTarget never constructs one itself.
A nil env is legal (a table without an environment yields a zero request template, per Environment.Template). m is validated before use, so an unknown or malformed manifest field — including an unknown capability — is rejected here rather than silently producing an under- or over-permissioned model descriptor.
func DecodeManifest ¶
DecodeManifest strictly decodes r (via packfile.StrictDecode, so an unknown field is rejected the same way the rest of the packfile YAML corpus rejects one) into a qual.Manifest and validates it before returning. This package never imports gopkg.in/yaml.v3 itself, reusing packfile's strict-decode logic instead; pkg/gen imports yaml.v3 directly for an unrelated concern (encode-side yaml.Node surgery).
func DecodeProfile ¶
DecodeProfile strictly decodes r (via packfile.StrictDecode) into a profile.Profile and validates it before returning.
func ManifestModel ¶
ManifestModel maps a secret-free qual.Manifest onto the inference module's model.Model descriptor used to build a live target: Provider, Model (as Name), APIFormat, and BaseURL carry across directly, Effort becomes the model's default Sampling.Effort, and each qual.Capability sets the matching model.Capabilities field (Tools, StructuredOutput, Images -> AcceptsImages, Thinking -> Thinking). ManifestModel assumes m is already valid (m.Capabilities holds only known values); BuildTarget validates m first, so an unknown capability is rejected there rather than silently dropped here.
Types ¶
type Result ¶
Result binds the rolled-up scorecard to its per-table eval reports and the skipped-table plans (visible coverage, never silent).
func Execute ¶
Execute plans every pack in s against s.Manifest and runs each runnable table plan through eval.Run using the plan's own Suite and Evaluators. A non-runnable plan (missing capability) contributes a Skipped TableResult and is retained verbatim in Result.Skipped; it is never executed and never silently dropped. Execute returns an error from Spec validation, from qual.Plan (an ill-formed pack or manifest), from Spec.TargetForTable, or from eval.Run's own preflight check. Only Spec validation failure (before any pack is planned or any target invoked) yields a zero-value Result: once any table has actually been planned or run, a later failure returns the Result accumulated from every pack/table processed successfully so far ALONGSIDE the error, rather than discarding it. This extends the same "visible coverage, never silent" philosophy behind Result.Skipped to the error path — an operator running many tables against a live target should not lose every already-executed (and potentially paid) table's report just because a later table hit a transient error. Callers must check the error to know whether the run as a whole completed; a non-nil error always means something failed, but the accompanying Result may still carry partial, trustworthy coverage worth reporting.
type Spec ¶
type Spec struct {
Manifest qual.Manifest
Packs []qual.Pack
Target eval.Target
TargetForTable func(qual.TablePlan) (eval.Target, error)
Config eval.RunConfig // zero value = eval defaults
// Progress, if non-nil, is called once per table plan in pack/plan order,
// just before that table is executed or recorded as skipped. It is a UI
// hook for live per-table feedback during a long live run (a run of many
// tables against a real model emits nothing else until the final report);
// offline callers like plutotest leave it nil. It must not mutate the plan.
Progress func(qual.TablePlan)
// OnResult, if non-nil, is called once per RUNNABLE table immediately after
// it executes, with that table's eval.Report — the companion to Progress
// for a live UI that shows each table's pass/fail outcome as it completes.
// It never fires for a skipped table (those have no report) and must not
// mutate the report. Under table concurrency it is called from a worker
// goroutine, possibly for several tables at once, so an implementation must
// be safe for concurrent use.
OnResult func(qual.TablePlan, eval.Report)
// TableConcurrency is how many tables execute in parallel. 0 or 1 runs them
// sequentially (the default, preserving strict pack/table order and
// stop-at-first-error). A value >1 runs up to that many tables at once
// through a worker pool — the throughput win for a corpus of many
// single-scenario tables, where eval's own per-sample Config.Concurrency
// cannot help. Per-provider request load is bounded independently by the
// caller's rate-limited client (pluto's --max-concurrent-requests/--max-rpm).
TableConcurrency int
}
Spec is one qualification execution. Target may be any eval.Target (scripted fixture or live inference target); pkg/run never constructs clients (design: "Dependency confinement").
Exactly one of Target and TargetForTable must be set. Target is used as-is for every table (the offline case: one fixture target answers every scenario in every pack). TargetForTable builds one target per table plan (the live case: the CLI constructs a fresh live target per table from that table's own environment via BuildTarget). Setting both, or neither, is a configuration error rejected by Execute before any pack is planned or any target is invoked.
Source Files