Documentation
¶
Overview ¶
Package modeltest is test support for the tiers that call a real model endpoint: the provider live-contract tests and the end-to-end eval suite. It owns the opt-in contract those tiers share — consent to spend money is a tier variable in the environment, never the presence of a configured .env — and resolves the one endpoint they drive. Production code must never import it.
The tiers, and what each costs:
RUN_LIVE_MODEL_TESTS=1 one real turn against the configured endpoint (cents) RUN_EVALS=1 the end-to-end eval suite (minutes, dollars)
Two variables rather than one because their costs differ by an order of magnitude: opting into the cheap smoke must not silently buy the suite.
Index ¶
Constants ¶
const ( // LiveEnv opts into the live-model contract tier: the provider adapters' // single real turn against the configured endpoint. LiveEnv = "RUN_LIVE_MODEL_TESTS" // EvalsEnv opts into the live-system eval suite: whole sessions driven // through the public API against a real model and real sandboxes. EvalsEnv = "RUN_EVALS" )
The tier opt-in variables. Any non-empty value opts in.
Variables ¶
This section is empty.
Functions ¶
func TierEnabled ¶
TierEnabled reports whether tierEnv opts into its tier, for the one caller that cannot use Endpoint: a TestMain, which has no *testing.T to skip.
It exists so that caller asks the same question Endpoint asks instead of re-spelling the rule — two copies of "what counts as opted in" drift, and the drift is silent in the worst direction: a TestMain that reads "off" while Endpoint reads "on" skips the suite's setup and then runs its tests against the setup that never happened.
A TestMain needs this because expensive setup (a database container, an image pull) runs before m.Run and so before any test can skip itself. Tests themselves must still call Endpoint — this only answers whether to build the world, never whether a given test may run.
Types ¶
type Config ¶
type Config struct {
Protocol string // "anthropic" | "openai"
BaseURL string
APIKey string
Model string
}
Config is the model endpoint a live tier drives, read from MODEL_PROTOCOL / MODEL_BASE_URL / MODEL_API_KEY / MODEL_ID. It mirrors the four fields a provider.Config needs.
func Endpoint ¶
Endpoint gates a live tier and returns the endpoint it should drive.
Not opted in: the test skips, no model is called, and the credential file is never even opened — an ordinary `go test ./...` costs nothing and touches nothing, whatever the .env holds. Opted in but misconfigured: the test FAILS, because a safety net that skips itself when its credentials rot is not a safety net. Gate before any `testing.Short()` check, or short mode becomes a way to opt in and still not be told the configuration is broken.
Naming protocols restricts the test to an endpoint speaking one of them, skipping otherwise: one .env holds one endpoint, and the adapter it does not belong to has nothing to prove against it. Name none to accept any endpoint the registry can route.
func (Config) Format ¶
Format carries the redaction to every verb fmt will let it. String alone would not: fmt consults it for %v/%s and their kin, but reaches past it for %#v, and for a mismatched verb like %d it falls back to a diagnostic that prints the raw fields — precisely the debugging accident this type exists to survive. Unexporting the credential would not help either; fmt prints unexported fields too.
%p is the one that gets through: fmt resolves %p and %T before consulting any method ("we always do them first" — fmt/print.go, printArg), so %p on a Config prints a %!p diagnostic carrying the fields. Nothing here can intercept it; pointing %p at a struct value is already a mistake fmt shouts about, so it is documented rather than defended against.
Source Files