eval

Overview ¶

Package evalsvc mounts the Hanzo Cloud /v1/evals/* surface: a thin facade that composes two systems that ALREADY work — the console eval engine (the Langfuse v3 fork at console.hanzo.svc, whose public REST API owns Datasets, DatasetItems, Evaluators, DatasetRuns and Scores) and the in-process model gateway (the AI subsystem's OpenAI-compatible /v1/chat/completions) — into one OpenAI-style evals API.

No eval logic is reimplemented here. Datasets, dataset-items, evaluators and score reads are proxied verbatim to the console's public API (HTTP Basic auth, a public/secret key pair that is itself project-scoped — so the key pair IS the IAM-org → console-project binding; there is no separate projectId to thread). POST /v1/evals/runs orchestrates a REAL run: for each dataset item it calls the in-process gateway for the model-under-test output, records a trace + dataset-run-item in the console, then (synchronously) calls the gateway again as LLM-as-judge and posts the score against that trace.

Auth, two domains, kept orthogonal:

• console (datasets/scores/...) ← HTTP Basic, key pair from config/KMS.
• model gateway (chat) ← the CALLER's own Authorization bearer, forwarded to the loopback /v1/chat/completions. The run therefore executes with the caller's identity and model entitlements; no privilege escalation, and the gateway authenticates it exactly as a direct call.

Order 145: binds /v1/evals/* BEFORE the AI subsystem's /v1/* catch-all (150), the same slot productsvc uses. The composition root auto-registers GET /v1/evals/health (serve.go) for every subsystem in the registry.