eval

type Board ¶ added in v1.786.165

type Board struct {
	Scope   BoardScope   `json:"scope"`
	Range   BoardRange   `json:"range"`
	Totals  BoardTotals  `json:"totals"`
	Series  []BoardPoint `json:"series"`
	ByModel []ModelStat  `json:"byModel"`
	Other   *ModelStat   `json:"other,omitempty"`
	Latency LatencyStat  `json:"latency"`
}

Board is the full AI-overview dashboard payload.

type BoardPoint ¶ added in v1.786.165

type BoardPoint struct {
	T           string `json:"t"` // RFC3339 (UTC) bucket start
	Generations int64  `json:"generations"`
	CostCents   int64  `json:"costCents"`
	TotalTokens int64  `json:"totalTokens"`
	Errors      int64  `json:"errors"`
}

BoardPoint is one gap-filled time bucket for the volume/cost/token/error series.

type BoardRange ¶ added in v1.786.165

type BoardRange struct {
	Range    string `json:"range"`    // echoed label (24h | 7d | 30d | custom)
	Start    string `json:"start"`    // RFC3339 (UTC)
	End      string `json:"end"`      // RFC3339 (UTC)
	Interval string `json:"interval"` // hour | day
}

type BoardScope ¶ added in v1.786.165

type BoardScope struct {
	Org     string `json:"org"` // "" when AllOrgs
	Project string `json:"project"`
	AllOrgs bool   `json:"allOrgs"`
}

type BoardTotals ¶ added in v1.786.165

type BoardTotals struct {
	Generations      int64   `json:"generations"`
	PromptTokens     int64   `json:"promptTokens"`
	CompletionTokens int64   `json:"completionTokens"`
	TotalTokens      int64   `json:"totalTokens"`
	CostCents        int64   `json:"costCents"`
	Errors           int64   `json:"errors"`
	SuccessRate      float64 `json:"successRate"` // 0..1
	Models           int64   `json:"models"`
	Users            int64   `json:"users"`
}

type Dataset ¶ added in v1.786.12

type Dataset struct {
	ID          string
	Org         string
	Name        string
	Description string
	Metadata    string // opaque JSON object, stored verbatim (bounded at the edge)
	CreatedAt   int64
	UpdatedAt   int64
}

Dataset is an org-scoped named collection of eval items. (org,name) is unique.

type DatasetItem ¶ added in v1.786.12

type DatasetItem struct {
	ID        string
	Org       string
	Dataset   string
	Input     string // opaque JSON, verbatim
	Expected  string // opaque JSON, verbatim ("expectedOutput" on the wire)
	Metadata  string // opaque JSON, verbatim
	Status    string // ACTIVE | ARCHIVED
	CreatedAt int64
	UpdatedAt int64
}

DatasetItem is one input/expected pair inside a dataset. Items are addressed by their own id; (org,dataset,id) scopes every lookup. Status ACTIVE|ARCHIVED mirrors the observation item lifecycle (a run consumes only ACTIVE items).

type DatasetRun ¶ added in v1.786.12

type DatasetRun struct {
	ID         string
	Org        string
	Dataset    string
	Name       string
	Model      string
	JudgeModel string
	Items      int
	Scored     int
	AvgScore   float64
	CreatedAt  int64
	UpdatedAt  int64
}

DatasetRun is the DEFINITION of a run (its metadata): which dataset+model, the run name, and rollup counters. The per-item scores/traces are telemetry (datastore); this row is the durable, listable run record. (org,dataset,name) is unique so a run name is stable per dataset.

type EvalRunner ¶ added in v1.786.12

type EvalRunner interface {
	// Complete runs one item's input through the model-under-test and returns its
	// raw text output.
	Complete(ctx context.Context, authz, model string, input any) (string, error)

	// Judge scores one item's output against the rubric using the judge model,
	// returning a numeric score in [0,1] and a one-line reasoning. It never
	// invents a score: an unparseable judge reply is an error the caller records
	// as the item's failure, not a fabricated 0 or 1.
	Judge(ctx context.Context, authz string, judge judgeSpec, input, expected any, output string) (float64, string, error)
}

EvalRunner is the PLUGGABLE execution seam (P3): the two independent steps of an evaluation, kept orthogonal so a DigitalOcean-backed runner and the in-process gateway runner both satisfy the same contract. The store, API and FE stay native and runner-agnostic regardless of which runner is wired.

The two steps are deliberately separate — Complete (produce the model-under-test output) and Judge (score that output against a rubric) — because DO's Agent Evaluations fuse run+judge into one async job with a FIXED (OpenAI) judge, whereas the gateway path needs them independent and needs the judge to be ANY model the caller can reach. A future DO adapter implements the same two methods (internally mapping a batch → evaluation_datasets upload → test_cases → evaluation_runs → poll → per-item results); nothing else changes.

Both methods take the caller's own Authorization bearer (authz): the run executes with the CALLER's identity and model entitlements — no privilege escalation, no service identity, the gateway authenticates it exactly as a direct call would.

type Evaluator ¶ added in v1.786.12

type Evaluator struct {
	ID        string
	Org       string
	Name      string
	Model     string
	Criteria  string
	ScoreName string // the score name this evaluator emits (defaults to Name)
	CreatedAt int64
	UpdatedAt int64
}

Evaluator is an org-scoped judge definition: a model + rubric that scores a run's items. (org,name) is unique. It carries NO secret — the model key is the caller's own bearer at run time, never stored here.

type LatencyStat ¶ added in v1.786.165

type LatencyStat struct {
	Available bool     `json:"available"`
	P50Ms     *float64 `json:"p50Ms"`
	P95Ms     *float64 `json:"p95Ms"`
	P99Ms     *float64 `json:"p99Ms"`
}

LatencyStat is the board's overall latency. Available=false with nil percentiles is the honest "no GenAI spans" state.

type MetricsFilter ¶ added in v1.786.165

type MetricsFilter struct {
	Org      string
	Project  string
	AllOrgs  bool
	Since    time.Time
	Until    time.Time
	Interval string
	TopN     int
}

MetricsFilter is the resolved, already-authorized dashboard query. Org is the validated tenant; AllOrgs (SuperAdmin) drops the org predicate; the window [Since, Until) is closed; Interval is the server-chosen bucket ("hour" | "day"); TopN bounds the by-model table (the rest fold into "other").

Project is the org SUB-SCOPE (server-minted, "" for the org's default project == whole-org view). When non-empty it ANDs a project predicate on the ledger and the latency spans, so the board narrows to that project WITHIN the org — org stays the hard tenant boundary regardless.

type ModelStat ¶ added in v1.786.165

type ModelStat struct {
	Model            string   `json:"model"`
	Provider         string   `json:"provider"`
	Requests         int64    `json:"requests"`
	PromptTokens     int64    `json:"promptTokens"`
	CompletionTokens int64    `json:"completionTokens"`
	TotalTokens      int64    `json:"totalTokens"`
	CostCents        int64    `json:"costCents"`
	Errors           int64    `json:"errors"`
	ErrorRate        float64  `json:"errorRate"` // 0..1
	CostPct          float64  `json:"costPct"`   // share of total spend, 0..100
	P50Ms            *float64 `json:"p50Ms"`
	P95Ms            *float64 `json:"p95Ms"`
	P99Ms            *float64 `json:"p99Ms"`
	ModelCount       int      `json:"modelCount,omitempty"` // >0 only on the "other" fold
}

ModelStat is one row of the model-usage table (or the folded "other" bucket). The latency percentiles are pointers so "no latency data for this model" is a null the console renders as "—", never a fabricated 0.

type ScoreConfig ¶ added in v1.786.12

type ScoreConfig struct {
	ID         string
	Org        string
	Name       string
	DataType   string   // NUMERIC | CATEGORICAL | BOOLEAN
	MinValue   *float64 // NUMERIC lower bound (nil = unbounded)
	MaxValue   *float64 // NUMERIC upper bound (nil = unbounded)
	Categories []string // CATEGORICAL allowed labels
	CreatedAt  int64
	UpdatedAt  int64
}

ScoreConfig is an org-scoped definition of a score's shape: NUMERIC (with optional min/max), CATEGORICAL (with an allowed value set), or BOOLEAN. Scores written for this name are validated against it (Red: score integrity).

type ScoreEvent ¶ added in v1.786.12

type ScoreEvent struct {
	ID          string
	Org         string
	Name        string
	TraceID     string
	RunName     string
	Dataset     string
	ItemID      string
	DataType    string
	Value       float64
	StringValue string
	Comment     string
	Timestamp   time.Time
}

ScoreEvent is one recorded score (append-only). Value is the numeric score; StringValue is the categorical/boolean label (empty for pure numeric). DataType is NUMERIC|CATEGORICAL|BOOLEAN. TraceID links it to its trace/run.

type ScoreFilter ¶ added in v1.786.12

type ScoreFilter struct {
	Org     string
	Name    string
	RunName string
	TraceID string
	Limit   int
}

ScoreFilter bounds a scores read. Org is MANDATORY (the caller passes the authoritative org); the others narrow within the org. Limit is always applied.

type Store ¶ added in v1.786.12

type Store struct {
	// contains filtered or unexported fields
}

Store is the eval metastore over one SQLite file ({DataDir}/evals.db). Tenancy is the org column; MaxOpenConns(1) serializes writes against the file lock (same discipline as prompts/projects).

func (s *Store) Close() error

func (s *Store) CountItems(ctx context.Context, org, dataset string) (int, error)

func (s *Store) DeleteDataset(ctx context.Context, org, name string) (bool, error)

func (s *Store) GetDataset(ctx context.Context, org, name string) (Dataset, error)

func (s *Store) GetEvaluator(ctx context.Context, org, name string) (Evaluator, error)

func (s *Store) GetItem(ctx context.Context, org, id string) (DatasetItem, error)

func (s *Store) GetScoreConfig(ctx context.Context, org, name string) (ScoreConfig, error)

func (s *Store) ListDatasets(ctx context.Context, org string, limit int) ([]Dataset, error)

func (s *Store) ListEvaluators(ctx context.Context, org string, limit int) ([]Evaluator, error)

func (s *Store) ListItems(ctx context.Context, org, dataset string, activeOnly bool, limit int) ([]DatasetItem, error)

func (s *Store) ListRuns(ctx context.Context, org, dataset string, limit int) ([]DatasetRun, error)

func (s *Store) ListScoreConfigs(ctx context.Context, org string, limit int) ([]ScoreConfig, error)

func (s *Store) PutItem(ctx context.Context, it DatasetItem) (DatasetItem, error)

func (s *Store) UpsertDataset(ctx context.Context, d Dataset) (Dataset, error)

func (s *Store) UpsertEvaluator(ctx context.Context, e Evaluator) (Evaluator, error)

func (s *Store) UpsertRun(ctx context.Context, r DatasetRun) (DatasetRun, error)

func (s *Store) UpsertScoreConfig(ctx context.Context, c ScoreConfig) (ScoreConfig, error)

type Telemetry ¶ added in v1.786.12

type Telemetry interface {
	RecordTrace(ctx context.Context, t Trace) error
	RecordScore(ctx context.Context, sc ScoreEvent) error
	ListScores(ctx context.Context, f ScoreFilter) ([]ScoreEvent, error)
	ListTraces(ctx context.Context, f TraceFilter) ([]Trace, error)
	Metrics(ctx context.Context, f MetricsFilter) (Board, error)
	Close() error
}

Telemetry is the append-only event store for eval traces + scores. Every method takes an authoritative org (via the value's Org field or the filter); no method can read across orgs.

Production LLM generations ("observations") are NOT read here: the observation of record is the o11y gen_ai span plane (/v1/o11y/observations), not a second projection of hanzo.cloud_usage. cloud_usage stays the metering warehouse (read by /v1/usage + billing), never an obs store. eval owns only its two eval-specific tables (eval_traces, eval_scores).

type Trace ¶ added in v1.786.12

type Trace struct {
	ID         string
	Org        string
	ProjectID  string
	Name       string
	Dataset    string
	ItemID     string
	RunName    string
	SessionID  string
	APIKeyHash string
	Model      string
	Input      string
	Output     string
	StartTime  time.Time
	EndTime    time.Time
	Timestamp  time.Time
}

Trace is one model-under-test invocation recorded during a run. Input/Output are opaque JSON/text; the run/dataset/item fields carry the linkage.

Attribution (HIP-0106): a trace records WHO/WHERE/WHEN alongside WHAT, so the eval trace list carries the same observability dimensions the production gen_ai span plane does:

• ProjectID — the org SUB-SCOPE (principal.Project); "" / "default" is the org's default project. Org stays the tenant-isolation key; project narrows.
• SessionID — groups the traces of one logical session; an eval run stamps its RunName so a run's item-traces read back as one session.
• APIKeyHash — a NON-reversible ref (SHA-256 hex) of the caller credential the run drove the model with. Never the plaintext key: the ref correlates a trace to a key without the store ever holding a secret.
• StartTime/EndTime — the model-under-test call window; EndTime-StartTime is the trace latency the view surfaces (LatencyMs).

type TraceFilter ¶ added in v1.786.12

type TraceFilter struct {
	Org       string
	ProjectID string
	SessionID string
	RunName   string
	Dataset   string
	Limit     int
}

TraceFilter bounds a traces read. Org is MANDATORY; ProjectID/SessionID/RunName/ Dataset narrow WITHIN the org. ProjectID is the caller's server-minted project (principal.Project), so a project-scoped caller sees only its project's traces.