eval

type Dataset ¶ added in v1.786.12

type Dataset struct {
	ID          string
	Org         string
	Name        string
	Description string
	Metadata    string // opaque JSON object, stored verbatim (bounded at the edge)
	CreatedAt   int64
	UpdatedAt   int64
}

Dataset is an org-scoped named collection of eval items. (org,name) is unique.

type DatasetItem ¶ added in v1.786.12

type DatasetItem struct {
	ID        string
	Org       string
	Dataset   string
	Input     string // opaque JSON, verbatim
	Expected  string // opaque JSON, verbatim ("expectedOutput" on the wire)
	Metadata  string // opaque JSON, verbatim
	Status    string // ACTIVE | ARCHIVED
	CreatedAt int64
	UpdatedAt int64
}

DatasetItem is one input/expected pair inside a dataset. Items are addressed by their own id; (org,dataset,id) scopes every lookup. Status ACTIVE|ARCHIVED mirrors the observation item lifecycle (a run consumes only ACTIVE items).

type DatasetRun ¶ added in v1.786.12

type DatasetRun struct {
	ID         string
	Org        string
	Dataset    string
	Name       string
	Model      string
	JudgeModel string
	Items      int
	Scored     int
	AvgScore   float64
	CreatedAt  int64
	UpdatedAt  int64
}

DatasetRun is the DEFINITION of a run (its metadata): which dataset+model, the run name, and rollup counters. The per-item scores/traces are telemetry (ClickHouse); this row is the durable, listable run record. (org,dataset,name) is unique so a run name is stable per dataset.

type EvalRunner ¶ added in v1.786.12

type EvalRunner interface {
	// Complete runs one item's input through the model-under-test and returns its
	// raw text output.
	Complete(ctx context.Context, authz, model string, input any) (string, error)

	// Judge scores one item's output against the rubric using the judge model,
	// returning a numeric score in [0,1] and a one-line reasoning. It never
	// invents a score: an unparseable judge reply is an error the caller records
	// as the item's failure, not a fabricated 0 or 1.
	Judge(ctx context.Context, authz string, judge judgeSpec, input, expected any, output string) (float64, string, error)
}

EvalRunner is the PLUGGABLE execution seam (P3): the two independent steps of an evaluation, kept orthogonal so a DigitalOcean-backed runner and the in-process gateway runner both satisfy the same contract. The store, API and FE stay native and runner-agnostic regardless of which runner is wired.

The two steps are deliberately separate — Complete (produce the model-under-test output) and Judge (score that output against a rubric) — because DO's Agent Evaluations fuse run+judge into one async job with a FIXED (OpenAI) judge, whereas the gateway path needs them independent and needs the judge to be ANY model the caller can reach. A future DO adapter implements the same two methods (internally mapping a batch → evaluation_datasets upload → test_cases → evaluation_runs → poll → per-item results); nothing else changes.

Both methods take the caller's own Authorization bearer (authz): the run executes with the CALLER's identity and model entitlements — no privilege escalation, no service identity, the gateway authenticates it exactly as a direct call would.

type Evaluator ¶ added in v1.786.12

type Evaluator struct {
	ID        string
	Org       string
	Name      string
	Model     string
	Criteria  string
	ScoreName string // the score name this evaluator emits (defaults to Name)
	CreatedAt int64
	UpdatedAt int64
}

Evaluator is an org-scoped judge definition: a model + rubric that scores a run's items. (org,name) is unique. It carries NO secret — the model key is the caller's own bearer at run time, never stored here.

type Observation ¶ added in v1.786.50

type Observation struct {
	ID               string
	TraceID          string
	Org              string
	UserID           string
	Model            string
	Provider         string
	PromptTokens     int64
	CompletionTokens int64
	TotalTokens      int64
	CostCents        int64
	Status           string
	ErrorMsg         string
	Timestamp        time.Time
}

Telemetry is the append-only event store for eval traces + scores. Every method takes an authoritative org (via the value's Org field or the filter); no method can read across orgs. Observation is ONE production LLM generation — the read projection of the proven `hanzo.cloud_usage` ledger (written on every AI call by ai/object's zapWriteUsage, the same funnel that emits the OTel GenAI span). It is the v3 "observation of type GENERATION" shape the console's Observe surface renders: model/provider, in/out/total tokens, cost, status, attribution. This is a READ over an ai/object-owned table — eval never writes it (its own tables stay eval_traces/eval_scores) — so there is exactly ONE emission path (recordTrace), fanned to o11y (span), the observability ingest (span), and this ledger; no parallel write.

type ObservationFilter ¶ added in v1.786.50

type ObservationFilter struct {
	Org    string
	Model  string
	UserID string
	Limit  int
}

ObservationFilter scopes a generations read. Org is authoritative (bound, never interpolated); Model/UserID are optional narrowers; Limit is always bounded.

type ScoreConfig ¶ added in v1.786.12

type ScoreConfig struct {
	ID         string
	Org        string
	Name       string
	DataType   string   // NUMERIC | CATEGORICAL | BOOLEAN
	MinValue   *float64 // NUMERIC lower bound (nil = unbounded)
	MaxValue   *float64 // NUMERIC upper bound (nil = unbounded)
	Categories []string // CATEGORICAL allowed labels
	CreatedAt  int64
	UpdatedAt  int64
}

ScoreConfig is an org-scoped definition of a score's shape: NUMERIC (with optional min/max), CATEGORICAL (with an allowed value set), or BOOLEAN. Scores written for this name are validated against it (Red: score integrity).

type ScoreEvent ¶ added in v1.786.12

type ScoreEvent struct {
	ID          string
	Org         string
	Name        string
	TraceID     string
	RunName     string
	Dataset     string
	ItemID      string
	DataType    string
	Value       float64
	StringValue string
	Comment     string
	Timestamp   time.Time
}

ScoreEvent is one recorded score (append-only). Value is the numeric score; StringValue is the categorical/boolean label (empty for pure numeric). DataType is NUMERIC|CATEGORICAL|BOOLEAN. TraceID links it to its trace/run.

type ScoreFilter ¶ added in v1.786.12

type ScoreFilter struct {
	Org     string
	Name    string
	RunName string
	TraceID string
	Limit   int
}

ScoreFilter bounds a scores read. Org is MANDATORY (the caller passes the authoritative org); the others narrow within the org. Limit is always applied.

type Store ¶ added in v1.786.12

type Store struct {
	// contains filtered or unexported fields
}

Store is the eval metastore over one SQLite file ({DataDir}/evals.db). Tenancy is the org column; MaxOpenConns(1) serializes writes against the file lock (same discipline as prompts/projects).

func (s *Store) Close() error

func (s *Store) CountItems(ctx context.Context, org, dataset string) (int, error)

func (s *Store) DeleteDataset(ctx context.Context, org, name string) (bool, error)

func (s *Store) GetDataset(ctx context.Context, org, name string) (Dataset, error)

func (s *Store) GetEvaluator(ctx context.Context, org, name string) (Evaluator, error)

func (s *Store) GetItem(ctx context.Context, org, id string) (DatasetItem, error)

func (s *Store) GetScoreConfig(ctx context.Context, org, name string) (ScoreConfig, error)

func (s *Store) ListDatasets(ctx context.Context, org string, limit int) ([]Dataset, error)

func (s *Store) ListEvaluators(ctx context.Context, org string, limit int) ([]Evaluator, error)

func (s *Store) ListItems(ctx context.Context, org, dataset string, activeOnly bool, limit int) ([]DatasetItem, error)

func (s *Store) ListRuns(ctx context.Context, org, dataset string, limit int) ([]DatasetRun, error)

func (s *Store) ListScoreConfigs(ctx context.Context, org string, limit int) ([]ScoreConfig, error)

func (s *Store) PutItem(ctx context.Context, it DatasetItem) (DatasetItem, error)

func (s *Store) UpsertDataset(ctx context.Context, d Dataset) (Dataset, error)

func (s *Store) UpsertEvaluator(ctx context.Context, e Evaluator) (Evaluator, error)

func (s *Store) UpsertRun(ctx context.Context, r DatasetRun) (DatasetRun, error)

func (s *Store) UpsertScoreConfig(ctx context.Context, c ScoreConfig) (ScoreConfig, error)

type Telemetry ¶ added in v1.786.12

type Telemetry interface {
	RecordTrace(ctx context.Context, t Trace) error
	RecordScore(ctx context.Context, sc ScoreEvent) error
	ListScores(ctx context.Context, f ScoreFilter) ([]ScoreEvent, error)
	ListTraces(ctx context.Context, f TraceFilter) ([]Trace, error)
	ListObservations(ctx context.Context, f ObservationFilter) ([]Observation, error)
	Close() error
}

type Trace ¶ added in v1.786.12

type Trace struct {
	ID        string
	Org       string
	Name      string
	Dataset   string
	ItemID    string
	RunName   string
	Model     string
	Input     string
	Output    string
	Timestamp time.Time
}

Trace is one model-under-test invocation recorded during a run. Input/Output are opaque JSON/text; metadata carries the run/dataset/item linkage.

type TraceFilter ¶ added in v1.786.12

type TraceFilter struct {
	Org     string
	RunName string
	Dataset string
	Limit   int
}

TraceFilter bounds a traces read. Org is MANDATORY.