Documentation
¶
Overview ¶
Package evaltype holds the shared value types and small interfaces of the eval harness. It is a dependency-light leaf package so that env, driver, usersim, judge and report can all reference these types without forming an import cycle (notably: env collects ToolCallRecords and driver consumes *env.Env, so the records must live below both).
Index ¶
Constants ¶
const ( StatusOK = "ok" StatusError = "error" )
Status discriminates whether a scenario ran to completion or errored.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Answer ¶
Answer is the simulated user's reply to one QuestionItem. Value is used for select / free_text; Values for multi_select.
type Answers ¶
type Answers struct {
Items []Answer
}
Answers bundles the replies to all items of a Question.
type Artifact ¶
Artifact is the workflow-type-specific produced result. Render returns a human-readable snapshot the judge evaluates each check against.
type CaseArtifact ¶
type CaseArtifact struct {
Case *model.Case
Transcript []TurnRecord
ToolCalls []ToolCallRecord
}
CaseArtifact is the thread-mode produced result: the final case state plus the turn transcript and the tool-call trajectory.
func (*CaseArtifact) Render ¶
func (a *CaseArtifact) Render() string
Render implements Artifact: a structured, judge-readable snapshot. Field values and tool calls are surfaced explicitly so objectively-decidable checks (state grounding) do not require re-parsing prose.
type CheckVerdict ¶
CheckVerdict is the judge's assessment of one check.
type Completer ¶
type Completer interface {
Complete(ctx context.Context, systemPrompt, userPrompt string, schema *gollem.Parameter) (string, error)
}
Completer is the minimal LLM surface the eval components (judge, usersim, toolsim) need: produce a completion for a system+user prompt, optionally constrained to a JSON response schema. Keeping this behind an interface lets those components be unit-tested with a canned completer instead of driving the full gollem agent loop. The gollem-backed implementation lives in the llmrun package.
type JobArtifact ¶
type JobArtifact struct {
JobID string
Outcome JobOutcome
Case *model.Case
Actions []*model.Action
ToolCalls []ToolCallRecord
}
JobArtifact is the job workflow's produced result: the job run outcome, the case state after the run, the actions present, and the tool-call trajectory (from the job run event timeline).
func (*JobArtifact) Render ¶
func (a *JobArtifact) Render() string
Render implements Artifact: a judge-readable snapshot of what the job did.
type JobOutcome ¶
type JobOutcome struct {
Stage string // "SUCCESS" | "FAILED" | "RUNNING"
Error string
Summary string // final LLM response text
}
JobOutcome is the terminal result of a Job run.
type Question ¶
type Question struct {
Reason string
Items []QuestionItem
}
Question is a clarification request surfaced by the agent during a turn.
type QuestionItem ¶
type QuestionItem struct {
ID string
Text string
Type QuestionType
Options []string
}
QuestionItem is one item the agent asks the user.
type QuestionType ¶
type QuestionType string
QuestionType mirrors the planner's question item control type.
const ( QuestionSelect QuestionType = "select" QuestionMultiSelect QuestionType = "multi_select" QuestionFreeText QuestionType = "free_text" )
type ScenarioResult ¶
type ScenarioResult struct {
ScenarioID string
EvalID string
Workflow string
Status string // StatusOK | StatusError
Err string
Score Score
Checks []CheckVerdict
Artifact Artifact
DumpDir string
}
ScenarioResult is the per-scenario outcome the reporter renders. The final OK/NG decision is left to a human reviewer; there is deliberately no overall pass/fail field.
type Simulator ¶
Simulator produces answers to an agent question. It is implemented by the usersim package; the persona it answers with is bound at construction time so this interface stays free of scenario types.
type ToolCallRecord ¶
type ToolCallRecord struct {
Seq int // 1-based call order
Tool string // tool name (e.g. slack_search)
Args any // arguments (query etc.)
Mode string // "sim" | "live"
Result any // returned content
}
ToolCallRecord captures one tool invocation for verification and diagnosis. Result holds the (simulated or real) returned content; it is rendered in full in diagnostic dumps and summarized for the judge.
Source Files