Documentation
¶
Overview ¶
Package eval is a small LLM-judged eval harness: a Suite of Cases (input + judge criteria), each dispatched through an agentloop.Loop and scored by a judge llm.Client on a 0-10 scale.
This is a port of studio-apps' core/evals — extracted the same way the rest of this module was, with two simplifications specific to agentloop rather than a multi-tenant platform:
- No OwnerID scoping. The original's Store methods took an ownerID to enforce tenant isolation across products sharing one harness. agentloop has no tenancy concept baked into its own stores (see agentloop.SessionStore) — an application that needs scoping can add it to its own Store implementation.
- No agents.Runner / llm.Registry indirection. The original dispatched cases through a Runner interface and looked up a judge client from a Registry, because the platform runs many named agents behind one harness. Here, Service just takes an agentloop.Loop directly — its Run method already has exactly the shape a Runner provided — and an llm.Client for judging. SessionStore.Get's documented auto-create-on-unknown-ID behavior means RunSuite needs no separate session-creation step: each case gets a fresh session ID, and the Loop's own SessionStore provisions it.
Service.RunSuite dispatches every Case in a Suite through the Loop, has the judge llm.Client rate each response against the case's criteria (either a single rubric with PassThreshold, or a per-criterion CriteriaItems breakdown scored and rolled up separately), and persists the run via Store. A case that errors (agent failure or judge failure) records the error inline rather than aborting the suite — RunSuite always finishes and returns a Run.
Store is a persistence seam, not an opinion: package evalmem ships an in-memory implementation for local dev and CI; back it with whatever your application already uses for anything that needs to survive a process restart.
Index ¶
- Constants
- type Case
- type CaseResult
- type CriterionItem
- type CriterionResult
- type Run
- type RunSummary
- type Service
- func (s *Service) AddCase(ctx context.Context, suiteID, name, input, criteria string, ...) (Case, error)
- func (s *Service) CreateSuite(ctx context.Context, name, judgeModel string) (Suite, error)
- func (s *Service) DeleteCase(ctx context.Context, id string) error
- func (s *Service) DeleteSuite(ctx context.Context, id string) error
- func (s *Service) GetRun(ctx context.Context, runID string) (Run, error)
- func (s *Service) ListCases(ctx context.Context, suiteID string) ([]Case, error)
- func (s *Service) ListRuns(ctx context.Context, suiteID string) ([]Run, error)
- func (s *Service) ListSuites(ctx context.Context) ([]Suite, error)
- func (s *Service) RunSuite(ctx context.Context, suiteID string) (Run, error)
- type Store
- type Suite
Constants ¶
const ( StatusRunning = "running" StatusCompleted = "completed" StatusFailed = "failed" )
Run statuses persisted on the row.
const DefaultPassThreshold int32 = 7
DefaultPassThreshold is used when AddCase is called with passThreshold <= 0.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Case ¶
type Case struct {
ID string
SuiteID string
Name string
Input string
Criteria string
PassThreshold int32
CriteriaItems []CriterionItem
CreatedAt time.Time
}
Case is one input + judge criteria + pass threshold.
CriteriaItems is the per-criterion rubric: when non-empty, the judge rates each item separately and the case passes only when every item meets its MinScore. The legacy Criteria + PassThreshold pair stays for cases authored before the rubric existed; RunSuite falls back to the single-rubric path when CriteriaItems is empty.
type CaseResult ¶
type CaseResult struct {
CaseID string `json:"case_id"`
CaseName string `json:"case_name"`
Response string `json:"response"`
Score int `json:"score"`
Rationale string `json:"rationale"`
Passed bool `json:"passed"`
Error string `json:"error,omitempty"`
CriterionResults []CriterionResult `json:"criterion_results,omitempty"`
}
CaseResult is the per-case judgment recorded on a run. Error is set (and every other field left at its zero value except CaseID/CaseName) when the agent or judge call failed — a failing case does not abort the rest of the suite.
type CriterionItem ¶
CriterionItem is one row in a case's per-criterion rubric. Label is the human-readable assertion the judge rates; MinScore is the 0-10 threshold the item must clear to count as passed. MinScore defaults to the case's PassThreshold when zero.
type CriterionResult ¶
type CriterionResult struct {
Label string `json:"label"`
Score int `json:"score"`
MinScore int `json:"min_score"`
Rationale string `json:"rationale"`
Passed bool `json:"passed"`
}
CriterionResult is the judge's verdict for one rubric item on one case run. Recorded inside CaseResult.CriterionResults; empty when the case used the legacy single-rubric path.
type Run ¶
type Run struct {
ID string
SuiteID string
Status string
StartedAt time.Time
FinishedAt *time.Time
Results []CaseResult
Summary RunSummary
}
Run is one execution of a suite — header fields plus the per-case results.
type RunSummary ¶
type RunSummary struct {
Total int `json:"total"`
Passed int `json:"passed"`
Failed int `json:"failed"`
}
RunSummary rolls up the per-case results.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service is the eval-harness entry point. CRUD methods are thin pass-throughs to the Store; RunSuite is the real logic: dispatch each case through runner, have judge rate the response, persist the run.
func NewService ¶
func NewService(store Store, runner agentloop.Loop, judge llm.Client, redactor *redact.Redactor) *Service
NewService wires the harness. store and runner are required and panic if nil — a misconfigured harness should fail at construction, not on the first RunSuite call. judge may be nil if the harness is only used for Suite/Case CRUD (e.g. an admin UI) and RunSuite is never called.
redactor, if non-nil, is applied to the agent's response before it reaches the judge prompt or CaseResult.Response — an eval case exercises the same capabilities production traffic does, so a case whose input happens to trigger secret(...) or a credential-echoing fetch() response would otherwise send that value on to a third-party judge LLM and persist it in the stored run. Pass nil for no redaction.
func (*Service) AddCase ¶
func (s *Service) AddCase(ctx context.Context, suiteID, name, input, criteria string, passThreshold int32, items []CriterionItem) (Case, error)
AddCase defaults passThreshold to DefaultPassThreshold when <= 0, and defaults each item's MinScore to that same threshold when unset — a caller can leave most items at the case default and override only the strict or lenient ones.
func (*Service) CreateSuite ¶
func (*Service) RunSuite ¶
RunSuite dispatches every case in the suite through runner and has judge rate each response. Synchronous — returns the finished run. For long suites the caller should set a generous ctx timeout. Errors during a single case are surfaced inside that case's result (Error field) rather than aborting the suite, so the rest of the cases still run and RunSuite still returns a completed Run.
type Store ¶
type Store interface {
CreateSuite(ctx context.Context, name, judgeModel string) (Suite, error)
ListSuites(ctx context.Context) ([]Suite, error)
GetSuite(ctx context.Context, id string) (Suite, error)
DeleteSuite(ctx context.Context, id string) error
AddCase(ctx context.Context, suiteID, name, input, criteria string, passThreshold int32, items []CriterionItem) (Case, error)
ListCases(ctx context.Context, suiteID string) ([]Case, error)
DeleteCase(ctx context.Context, id string) error
CreateRun(ctx context.Context, suiteID string) (Run, error)
FinishRun(ctx context.Context, id, status string, results []CaseResult, summary RunSummary) error
ListRuns(ctx context.Context, suiteID string) ([]Run, error)
GetRun(ctx context.Context, id string) (Run, error)
}
Store is the persistence contract an application implements to back the harness with whatever it already uses — Postgres, SQLite, evalmem.InMemoryStore for local dev/CI. All methods return domain types, never a driver-specific row.
type Suite ¶
type Suite struct {
ID string
Name string
JudgeModel string // passed as llm.CompletionRequest.Model for every judge call; empty uses the judge client's default
CreatedAt time.Time
UpdatedAt time.Time
}
Suite is one collection of Cases run against the Service's configured agentloop.Loop.
Source Files