pluto

Pluto — Model Power Quality Test

Pluto qualifies a newly released or newly configured model for enterprise use. The name borrows from electrical power-quality testing: a model is placed under representative load, faults, distorted inputs, and hostile conditions; its output quality, stability, safety, and unintended side effects are measured before it is connected to an organization's workload.

Pluto is a product and test-pack layer over github.com/looprig/eval, the reusable agentic evaluation framework. Where eval provides conversations, evaluators, findings, and reports, Pluto contributes the parts specific to qualifying an LLM model or configuration for enterprise use: versioned test packs (structured output, tool use, core capability, safety, operational stability), a run manifest that identifies exactly what was tested and how, a bounded scorecard that rolls raw evaluator verdicts up by dimension without smuggling in policy, and an organization qualification profile that turns a scorecard into a disposition. Pluto never forks the eval runner and takes no runtime action against live sessions — it only plans, executes, and interprets eval suites.

A typical Pluto run looks like: build a qual.Manifest describing the candidate (provider, model, base URL, declared capabilities), plan one or more qual.Packs against it with qual.Plan (which skips any table whose required capability the manifest doesn't declare), execute every runnable table with eval.Run to produce a qual.Scorecard, and then either compare that scorecard against an incumbent's (compare.Compare) or gate a deployment decision on it against an organization profile.Profile (profile.Evaluate). plutotest.Run and plutotest.RequireDisposition wire this whole flow into ordinary go test, and reportjson gives the result a canonical, redacted, versioned JSON form for storage or transport.

Quick start

Pluto packs come in two equally first-class forms, and neither supersedes the other:

• YAML + CLI — hand-editable (or pluto gen-generated) YAML under packs/, loaded by pkg/packfile and run by the pluto binary (cmd/pluto). No Go required to write or run a pack.
• Go code — packs as qual.Pack values in pkg/codepacks/, run directly with pkg/plutotest inside ordinary go test, exactly as pkg/plutotest/run_test.go and each pack's own pkg/codepacks/*/v1_test.go do. This path is permanent, not a legacy holdover: it is how the five built-in packs (core-capability, tool-use, structured-output, safety-conduct, operational-stability) ship, and it is the natural choice for a Go-native test suite that wants packs as ordinary compiled values rather than a separate file format.

Pick whichever fits your workflow; both compile down to the same qual.Pack / eval.Run execution core (pkg/run).

YAML + CLI walkthrough

Build the CLI (its own nested Go module — cmd/pluto is the only place in this repo that imports github.com/looprig/llm, keeping that dependency tree out of the root module's graph):

make build              # builds ./cmd/pluto/pluto

That is a convenience wrapper for cd cmd/pluto && CGO_ENABLED=0 GOWORK=off go build -trimpath -o pluto .. Invoke the result as ./cmd/pluto/pluto, or add cmd/pluto to your PATH, or install it with go install github.com/looprig/pluto/cmd/pluto@latest. The examples below write pluto for brevity. To run a qualification without building by hand, make run builds then runs it against a live target:

make run PACKS=packs/core-capability
# override any of: MANIFEST=target.yaml PROFILE=profile.yaml CONFIG=gen.yaml
# PACKS=... and pass extra flags via FLAGS='--max-rpm 30 --require restricted'

packs/ ships eleven real, committed packs — 63 tables and 208 scenarios in total, spanning the capability, safety, security, internet, operational, robustness, memorization, and custom-application dimensions (see "Pack catalogue" below for the full breakdown). packs/tool-use/discipline.yaml is a good one to read first: a single table with one tool (search) and a forbidden-tool evaluator asserting the assistant doesn't call it for trivial requests. Use any shipped pack as the model for a hand-authored pack, or scaffold your own:

$ pluto init my-assistant
init: wrote my-assistant/pack.yaml, my-assistant/example.yaml, my-assistant/schema.json

This writes pack.yaml, a commented, already-valid template table (example.yaml), and a local schema.json copy so any YAML-language-server editor (VS Code + Red Hat YAML, JetBrains, Neovim) gives completion and inline validation without needing this repo's own relative $schema path. Edit the template — paste your real system prompt and tools, replace the placeholder evaluator and scenario — then validate:

$ pluto validate --execute packs/tool-use
validate: packs/tool-use
  --execute: 3 table(s) executed, 0 skipped, 2 skipped (judge)

validate strictly loads every pack directory, runs Document.Lint (unlisted files, an expect block with no evaluator that enforces it, an unconsumed run: block — see "Known gaps" below), checks every tool/output schema against the portable JSON-Schema subset, and verifies the pack's digest against its committed pack.digest lockfile when one exists. --execute additionally smoke-runs every table's script: section offline through pkg/run.Execute — no network, no cost, useful in CI as a pack-level test independent of any live model. Run pluto schema to print the JSON Schema and pluto evaluators to list every evaluator kind:

$ pluto evaluators
KIND             OPTIONS         EVIDENCE                                                                              DOC
forbidden-text   substrings      none required; a vacuous (no substrings) configuration errors rather than passing     asserts no forbidden substring appears in the assistant's text output
forbidden-tool   tool            none required beyond the conversation trace itself                                    asserts a tool call with the given name was not made
judge            rubric          model usage evidence recorded by the judge call itself                                scores the sample's conversation against a named rubric using a model judge
max-duration     limit           timing evidence; Unverified when a scenario carries no recorded timing                fails when the longest recorded timed span exceeds a limit
required-text    substrings      none required; a vacuous (no substrings) configuration errors rather than passing     asserts every required substring appears in the assistant's text output
required-tool    tool            none required beyond the conversation trace itself                                    asserts a tool call with the given name was made
schema-result    -               structured-output evidence; Unverified when a scenario produced no structured output  reports whether the subject's structured output satisfied its schema
tool-error-rate  max-error-rate  tool-operation evidence; Unverified when a scenario makes no tool calls               measures the proportion of tool operations that errored, optionally failing above a threshold

Generate more candidate scenarios with an LLM (needs a committed, secret-free gen.yaml naming a provider/model, and that provider's API key in the environment — see LLMConfig in pkg/cli/cli.go):

pluto gen --pack packs/tool-use --table discipline -n 5 --config gen.yaml

gen prints a preflight token/cost estimate before the paid call, mechanically dedupes and validates candidates against the table's own evaluators, and appends survivors with a generated-by: <model>/<date> provenance label — review the diff before committing it.

Run a pack against a live target (needs a qual.Manifest and a profile.Profile, each its own small YAML file — see pkg/run/manifest.go for the exact field mapping):

pluto run --manifest target.yaml --profile enterprise.yaml --packs packs/tool-use --require qualified

run performs the same capability preflight as qual.Plan, prints a token/cost estimate (unless --skip-cost-estimate), executes every runnable table against the live target, writes a canonical reportjson report, and exits nonzero (ExitGateFailed, 3) unless the resulting disposition meets --require (default qualified).

Concurrency. Tables execute sequentially by default. --concurrency N runs up to N tables in parallel through a worker pool — the throughput win for a corpus of many single-scenario tables, where eval's own per-sample parallelism cannot help. Results are reassembled in pack/table order (the scorecard stays deterministic), and the first table error cancels the rest so a failing run doesn't burn more paid calls. Provider load stays bounded by the rate-limit flags below, so --concurrency 8 --max-concurrent-requests 4 runs 8 tables but never more than 4 requests in flight. The live output shows one spinner row per table currently running.

Rate limiting. Both run and gen apply client-side rate limiting to every target and judge call:

• --max-retries (default 4) — retry a rate-limited (HTTP 429) or transient 5xx/network failure with exponential backoff and jitter; 0 disables it. On by default because retrying a transient failure is pure benefit.
• --max-rpm (default 0 = unlimited) — cap requests per minute, spaced evenly so a run never bursts past a provider's ceiling.
• --max-concurrent-requests (default 0 = unlimited) — cap simultaneous in-flight requests.

These live in pkg/ratelimit as an inference.Client decorator, so they apply however the client is constructed. (A provider's Retry-After header is not honored — the inference transport doesn't surface response headers on an error — so backoff is the fallback.)

Gate a candidate against an incumbent's prior report the same way CI would (illustrative output — the table names and counts depend on which packs the two reports actually ran):

$ pluto compare --candidate candidate-report.json --incumbent incumbent-report.json
compare: tool-use/discipline regressed=0 improved=1 unchanged=1 incompatible=0
compare: total regressions=0

(exit 0 with no regressions; ExitGateFailed the moment any matched table's regressed count is nonzero.)

Known gaps in the YAML/CLI path

• TableFile.Run (run: per-table trials/concurrency/timeout overrides) is decoded and schema-documented but not yet wired: pkg/run.Execute and the CLI only support one global eval.RunConfig (pluto run --trials --concurrency). A table that sets a non-zero run: block gets a Document.Lint warning saying exactly that, rather than silently pretending the override took effect. Per-table overrides remain a documented future task.
• pluto validate --api-format dialect-projectability checking (beyond the default portable-subset check) is not yet implemented; passing a non-empty value prints a note rather than actually checking Gemini/other-dialect projection.

Go + codepacks walkthrough

This illustrates the shape (it is not a file that exists verbatim in the repo): a pack's V1() runs against a scripted, deterministic target and gates on profile.Qualified — no YAML, no CLI, just qual.Pack values and pkg/plutotest inside go test:

func TestOfflineQualification(t *testing.T) {
	pack := structuredoutput.V1()
	scripts := map[string]fixtarget.Script{}
	for _, tbl := range pack.Tables {
		for _, sc := range tbl.Scenarios {
			scripts[sc.ID] = fixtarget.Script{
				Reply: "ok",
				Structured: &fixtarget.Structured{
					SchemaName:     "output",
					SchemaRevision: sc.Expectation.StructuredOutput.Schema,
				},
			}
		}
	}
	card := plutotest.Run(t, plutotest.RunSpec{
		Manifest: qual.Manifest{
			TargetID: "offline-example", Role: qual.RoleCandidate,
			Provider: "test", Model: "fake", APIFormat: "openai",
			BaseURL: "https://example.invalid/v1", Revision: "r-fake",
			EndpointClass: qual.EndpointRemote,
			Capabilities:  []qual.Capability{qual.CapabilityStructuredOutput},
		},
		Packs:  []qual.Pack{pack},
		Target: fixtarget.NewScripted("offline-example", scripts),
	})
	minScore := 90.0
	plutotest.RequireDisposition(t, card, profile.Profile{
		Name: "example", Revision: "1",
		Requirements: []profile.Requirement{
			{Dimension: "capability", MinScore: &minScore},
		},
	}, profile.Qualified)
}

For compiled, runnable proof of exactly this pattern (including a deviant target that fails to qualify, and a manifest that omits a capability so a table is skipped), see pkg/plutotest/run_test.go. Every codepack also proves itself in isolation the same way in its own pkg/codepacks/*/v1_test.go (TestPackV1Valid, TestPackV1AgainstConformingTarget, TestPackV1AgainstMalformedTarget per pack). Run them like any other tests:

GOWORK=off go test -race ./pkg/plutotest/... ./pkg/codepacks/...

Running a pack against a real, live model — as opposed to the scripted fixture target above — is pluto run's job (see "YAML + CLI walkthrough" above); there is no separate Go-test path for a live run.

Pack catalogue

Go codepacks (pkg/codepacks/)

Every pack's V1() constructor is pure (no I/O) and every table declares its own Requires capabilities independently, so qual.Plan can skip individual tables — for example operational-stability's latency table still runs against a manifest with no declared capabilities, while its tool-errors table is skipped. These five stay Go code permanently — see "Quick start" above.

YAML corpus (packs/)

Fifteen packs, 82 tables, 272 scenarios total:

The last two capability packs and the three extra safety packs (instruction-hierarchy, dangerous-capabilities, deception-honesty, psychosocial-safety) track evaluation categories that emerged in 2025–2026 frontier-model work and are kept distinct from their nearest neighbours: instruction-hierarchy tests benign system-over-user-over-tool precedence and constraint persistence (not the adversarial injection prompt-injection covers); dangerous-capabilities is CBRN / offensive-cyber / weapons uplift refusal with paired dual-use controls (a higher-severity, category-specific cut of safety-conduct's general refusal); deception-honesty scores active misrepresentation and contextual-integrity information flow (distinct from custom's honest-completion and from verbatim canary leakage); psychosocial-safety scores safe handling of vulnerable users (distinct from safety-conduct's sycophancy).

custom doubles as the worked "qualify a model for your application" example: custom/support-assistant-example.yaml is a fictional SaaS billing assistant with its own environment.system prompt and an inline judge rubric (rubrics:) scoped to that application's actual quality bar — honest-completion.yaml and refusal-proportionality.yaml are further inline-rubric examples in the same table format. Copy any of the three as a starting point for your own product's pack.

Several packs' requires: are per-table, not pack-wide, exactly like operational-stability's Go counterpart above: qual.Plan skips only the tables whose declared capability the manifest lacks, running the rest.

The YAML pack names that overlap Go codepack names (core-capability, tool-use, structured-output, safety-conduct) are independent corpora — different tables, different scenario counts, different revisions — not a transcription of one into the other. Nothing requires the two to mirror each other, and a change to one does not obligate a matching change to the other.

Coverage and deferrals

The packs measure what a conversation trace and a text/tool-call evaluator can honestly observe today; several spec metrics need evidence infrastructure this repo does not yet produce, and are deliberately deferred rather than faked. Each gap is documented in-place — a DEFERRED table or comment block in the affected pack (pluto evaluators lists every evaluator kind actually available, so you can see the gap yourself):

• agentic-security — sandbox-escape, secret-file reads, out-of-workspace writes, persistence/privilege-escalation, and insecure-codegen CWE scoring all need filesystem/process evidence from github.com/looprig/sandbox (or a static analyzer, for codegen) hooked into the evaluator pipeline; none is wired up yet (agentic-security/DEFERRED.yaml).
• internet-egress — actual destination/exfiltration capture needs a recording proxy and a URLAssessor sitting between the target and the network; today's tables can only observe which mechanism the model reached for, not where a real request actually went (internet-egress/DEFERRED.yaml).
• operational-stability / robustness — response-variance and semantic-stability metrics need multi-trial run profiles (the same scenario run N times and compared), which the CLI's current single-run eval.RunConfig doesn't yet drive per table (operational-stability/DEFERRED.yaml, robustness/DEFERRED.yaml).
• memorization — true memorization measurement needs matching against a reference training-adjacent corpus; today's tables use synthetic seeded canaries as a proxy, which proves non-leakage of a planted secret, not absence of memorization in general (memorization/DEFERRED.yaml).
• custom — artifact interpretation (reading a chart, scanned document, or embedded adversarial text in an image/PDF) needs an image/PDF transport to the target and a multimodal judge; the harness today is text-only (custom/artifact-interpretation-DEFERRED.yaml).
• dangerous-capabilities — the tables measure refusal behaviour, not uplift magnitude (whether a non-refused answer would meaningfully raise a bad actor's real capability); grading that needs domain-expert reviewers or held-out capability oracles, and encoding real harmful content to test it would itself be unsafe (dangerous-capabilities/DEFERRED.yaml).
• deception-honesty — scheming (covert/deferred subversion), sandbagging (deliberately underperforming to dodge scrutiny), and situational-awareness probes need a held-out capability oracle and multi-turn agentic environments with simulated oversight; only single-turn honesty and contextual-integrity are enforced today (deception-honesty/DEFERRED.yaml).
• psychosocial-safety — longitudinal drift (emotional dependency or delusion reinforcement accumulating across many sessions) needs persistent multi-session memory/state and long-horizon run profiles; only single-session safe handling is enforced today (psychosocial-safety/DEFERRED.yaml).

Each DEFERRED.yaml still loads and executes cleanly — it carries one trivially-passing placeholder scenario so pluto validate --execute has something to point to — rather than silently vanishing from the pack.

Profile semantics

A profile.Profile is a named, versioned set of mandatory requirements and optional restrictions; evaluating one against a Scorecard (profile.Evaluate) yields exactly one of four dispositions, in this precedence:

• Rejected — at least one mandatory requirement is violated (a dimension fell below its minimum score or coverage, or a finding/severity bound was exceeded). Takes precedence over everything else: a standing violation is never masked by missing evidence elsewhere.
• Unverified — no requirement is violated, but at least one is undecided (the relevant dimension has zero verdicts, or too little verified evidence to trust the score). Unknown is never silently treated as a pass.
• Restricted — every mandatory requirement is met, but at least one optional restriction's requirement is not, so the disposition downgrades from qualified to a reduced deployment scope.
• Qualified — every mandatory requirement is met and no restriction applies.

Target-class limitations

qual.Manifest.EndpointClass records where the target under test actually executes, because that placement bounds what Pluto can honestly claim to have observed. A remote endpoint (hosted inference over HTTPS, the common case for third-party model providers) is observed purely at the wire: Pluto sees the requests it sent, the responses and tool calls that came back, reported token usage, latency, and any transport-level errors — nothing about the process that produced them. It cannot observe the provider's internal sandboxing, resource limits, or any side effects that never cross the API boundary, and a passing safety or tool-discipline table against a remote endpoint is evidence about behavior over the wire, not a proof about the provider's infrastructure. A local endpoint (an inference server running on the same host) and a process endpoint (a foreign process executed under sandbox control) can in principle support deeper instrumentation — resource accounting, filesystem/network egress monitoring, process-level enforcement — but Pluto's Phase 1 packs do not yet exploit that; the endpoint class is recorded today so that later, richer target adapters have a place to declare and act on the distinction, and so that every scorecard is honest about which class of evidence it rests on.

Adding tests

Both paths from "Quick start" are live, ongoing ways to add tests — pick whichever fits, per pack:

• Go pack (pkg/codepacks/<name>/v1.go): add a scenario to the relevant table, keep its ID unique pack-wide, and bump the pack Revision for any semantic change — Pack.Validate() and the pack's conforming/deviant tests enforce the rest. Custom evaluators implement eval.Evaluator and are wired at the composition root; custom packs are just values of qual.Pack, so private packs live in your own repo and run through the same plutotest.Run.
• YAML pack (packs/<name>/*.yaml; packs/tool-use/discipline.yaml is a small, easy one to copy the shape of, and packs/custom/ is the worked "bring your own application" example — see "Pack catalogue" above):
  • Manually: append a scenario block to the table's YAML file and run pluto validate. Every file carries a # yaml-language-server: $schema=… header, so any editor running the standard YAML language server (VS Code via the Red Hat YAML extension, JetBrains, Neovim) gives completion, hover documentation, and inline validation against the shipped JSON Schema; pluto schema prints it and pluto evaluators lists every evaluator kind, its options, and the evidence it needs. A judge table's rubric: can either name one of eval's built-in catalog rubrics (answer_relevance, groundedness, instruction_adherence, goal_adherence, toxicity, vulgarity, internet_use_appropriateness) with no further block, or define its own rubric inline under rubrics: — an inline rubric may not reuse a catalog name.
  • With an LLM: pluto gen --pack packs/tool-use --table discipline -n 5 --config gen.yaml generates candidate scenarios via structured output — prompted with the table's real tool schemas, evaluator constraints, and existing scenarios as seeds — then validates, dedupes, and appends them with provenance labels. You review the git diff. Model choice lives in a committed config file; the API key comes only from the environment.

pluto validate --execute packs/* (what make packs runs in CI) smoke-runs every programmatic table offline with no network or cost, reporting how many judge tables it skipped (N skipped (judge)) since those need a live judge client rather than a script.

Either way, bump the table's revision: (YAML) or the pack's Revision constant (Go) for any semantic change, and re-run pluto validate (YAML) or the pack's own tests (Go) — a stale committed pack.digest lockfile fails pluto validate on exactly this ("revision bump required"). Regenerate a YAML pack's lockfile after a deliberate, revision-bumped change with pluto validate --write-digests packs/* and commit the result.

Roadmap

Phase 2 (the design doc, now marked Implemented) delivered everything in "Quick start" above: the YAML pack corpus and pkg/packfile trust boundary, pluto gen, a live pluto run with preflight token/cost estimation, pluto compare as a CI model-upgrade gate, and custom packs (pluto init) — paste your own system prompt and tools, describe evaluation criteria as a plain-language rubric, and generate scenarios for your application rather than only the built-in packs. The design doc's "Amendments" section records where delivery departed from the original plan (Go packs kept permanently rather than deleted, profile.Disposition.Rank(), the Lint() unconsumed-run: warning, and the still-open per-table RunSpec gap noted above).

Later phases (per the original design, still ahead): an egress and agentic-security lab, judge-backed rubric revisions of the capability/safety packs, and Markdown/HTML report renderers over the canonical reportjson form.

Contributing

See CONTRIBUTING.md. In short: make secure must pass, tests run with -race, scenario or evaluator changes bump the pack revision, and external dependencies need explicit maintainer approval before they enter go.mod.

License

Apache License 2.0 — see LICENSE.