Documentation
¶
Overview ¶
Package benchmark is one honest score for any model, on the tests everyone quotes.
It is the native benchmark ARENA — run the top-N canonical public benchmarks against any model or endpoint, under ONE standardized harness, measure Hanzo's own models (enso, zen), and reconcile any external provider-reported claim against that measurement. Sibling to /v1/evals (evals = YOUR data + YOUR judge; benchmark = the canonical public tests, comparable + provenance-first + leaderboard).
Provenance-first, never blended: a `published_claim` (what a vendor reports) and a `hanzo-measured` attempt (what OUR harness gets) are separate planes — the gap is the signal (some provider-reported claims run 3-13pp hot vs one standardized harness). The store is append-only; a re-scored label is a new score_event, never an overwrite.
Its own binary (plugin/benchmark) states Name/Price/Mount; the host learns the prefix from its manifest row. The Python enso-bench harness is the research prototype, THIS is the product surface.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AttemptStore ¶
type AttemptStore interface {
// Attempts returns every stored attempt for a benchmark (the measured plane).
Attempts(benchmark string) []attempt
// Has reports whether (benchmark, item, model) is already attempted — the
// cross-run cache the worker checks before spending.
Has(benchmark, item, model string) bool
// Append records one attempt idempotently (same key ⇒ no duplicate spend).
Append(a attempt) error
}
type Benchmark ¶
type Benchmark struct {
ID string `json:"id"`
Title string `json:"title"`
Axis string `json:"axis"`
Items int `json:"items,omitempty"`
Native bool `json:"native"`
Source string `json:"source"`
}
Benchmark is one canonical, versioned public test. `Native` marks harness support today; the rest are adapter-pending on the same registry + provenance.
type LeaderRow ¶
type LeaderRow struct {
Model string `json:"model"`
Measured *float64 `json:"measured"` // hanzo-measured accuracy % (nil if unrun)
N int `json:"n"` // coverage — NEVER compare across different n
Published *float64 `json:"published"` // provider-claimed % (nil if none)
Gap *float64 `json:"gap"` // published − measured (the arena signal)
Protocol string `json:"protocol,omitempty"`
}
LeaderRow layers the two planes for one model, coverage-aware, never blended.
type Preset ¶
type Preset struct {
Name string `json:"name"` // served as enso-<name>
Owner string `json:"owner"` // scoping org (never cross-tenant)
Arms []string `json:"arms"` // the blend — model ids from the arena
Rank []string `json:"rank"` // escalation order over arms
Panel int `json:"panel"` // fan-out width (>=1)
Note string `json:"note,omitempty"` // why this blend (audit)
}
Preset is one user-authored router blend. Arms are catalog model ids (or BYO); rank is the escalation order (probe = rank[0], panel = rank[:panel]); panel bounds fan-out.
type RunRequest ¶
type RunRequest struct {
Benchmarks []string `json:"benchmarks"`
Model string `json:"model"`
Endpoint string `json:"endpoint,omitempty"`
Attempts int `json:"attempts,omitempty"`
}
RunRequest: run a benchmark against a model/endpoint. target is a catalog model id OR your own chat-completions endpoint+key (the cloud offering: benchmark YOUR model). The runner caches before spend (skip any (item, model) already attempted) and records provenance. Execution is the async worker (follow-on); this admits + queues.
Source Files