Documentation
¶
Overview ¶
Package scenario defines the TOML schema for eval scenarios and loads / validates them. A scenario file carries both the system-under-test workspace configuration (authored with the same top-level layout as a normal workspace config file) and the eval-specific tables (meta / input / cases / tools / persona / expect). The workspace part is extracted by handing the same file to config.LoadWorkspaceConfigs, which ignores the eval-only keys; the eval-only tables are decoded into Scenario. Keeping both in one file lets a scenario be fully self-contained.
Index ¶
Constants ¶
const WorkflowJob = "job"
WorkflowJob is the workflow kind that runs a Job against a case.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CaseSeed ¶
type CaseSeed struct {
Title string `toml:"title"`
Description string `toml:"description"`
BoardStatus string `toml:"board_status"`
Fields map[string]any `toml:"fields"`
}
CaseSeed is a pre-existing case injected into the memory repository before the run (for future search/context use; optional).
type Check ¶
Check is one natural-language yes/no question the judge answers against the produced artifact (case state + transcript + tool calls).
type Expect ¶
type Expect struct {
Checks []Check `toml:"checks"`
}
Expect holds the checklist the judge evaluates the produced artifact against.
type GitHubRepoRef ¶
GitHubRepoRef is one repository in a github source.
type GitHubSource ¶
type GitHubSource struct {
Repositories []GitHubRepoRef `toml:"repositories"`
}
GitHubSource configures a github source.
type Input ¶
type Input struct {
Text string `toml:"text"`
// Reporter optionally pins the posting user id. Empty means the harness
// synthesizes one (the reporter identity does not affect the evaluation).
Reporter string `toml:"reporter"`
}
Input is the triggering input. For thread_mode_initial it is the first top-level post in the monitored channel.
type JobSpec ¶
JobSpec selects which workspace Job to run and against which seeded case. Used by the job workflow. The Job must be declared in the workspace config ([[job]]); the target case must be one of the seeded [[cases]] (matched by title, or the first when TargetCase is empty).
type Meta ¶
type Meta struct {
ID string `toml:"id"`
Description string `toml:"description"`
Workflow string `toml:"workflow"`
// Language is the i18n language the system-under-test agent responds in
// (the simulated end-user's locale). Distinct from the eval output language.
Language string `toml:"language"`
}
Meta identifies the scenario and selects the workflow driver.
type NotionDBSource ¶
type NotionDBSource struct {
DatabaseID string `toml:"database_id"`
DatabaseTitle string `toml:"database_title"`
DatabaseURL string `toml:"database_url"`
}
NotionDBSource configures a notion_db source.
type NotionPageSource ¶
type NotionPageSource struct {
PageID string `toml:"page_id"`
PageTitle string `toml:"page_title"`
PageURL string `toml:"page_url"`
Recursive bool `toml:"recursive"`
MaxDepth int `toml:"max_depth"`
}
NotionPageSource configures a notion_page source.
type Persona ¶
type Persona struct {
Description string `toml:"description"`
Knowledge string `toml:"knowledge"`
// MaxAnswerTurns bounds the question/answer loop. Zero means no follow-up
// answers are produced.
MaxAnswerTurns int `toml:"max_answer_turns"`
}
Persona is the simulated end-user that answers the agent's questions.
type Scenario ¶
type Scenario struct {
Meta Meta `toml:"meta"`
Input Input `toml:"input"`
Cases []CaseSeed `toml:"cases"`
Sources []Source `toml:"sources"`
Tools map[string]Tool `toml:"tools"`
Persona Persona `toml:"persona"`
// Job (table key `run_job`) selects which workspace Job to run for the
// job workflow. The key is `run_job`, not `job`, to avoid
// colliding with the workspace config's `[[job]]` array in the same file.
Job *JobSpec `toml:"run_job"`
Expect Expect `toml:"expect"`
// Workspace is loaded from the same file via config.LoadWorkspaceConfigs.
// Not a TOML field of this struct.
Workspace *config.WorkspaceConfig `toml:"-"`
// Path is the source file path, retained for diagnostics.
Path string `toml:"-"`
}
Scenario is one eval case. The TOML-decoded eval tables plus the workspace configuration extracted from the same file.
func Load ¶
Load reads and parses a scenario file: it decodes the eval-specific tables and extracts the workspace configuration from the same file (reusing the existing config loader, which validates the workspace and ignores eval-only keys).
func (*Scenario) Validate ¶
func (sc *Scenario) Validate(opts ValidateOptions) error
Validate checks the eval-specific parts of the scenario. The workspace part was already validated by Load via config.LoadWorkspaceConfigs.
type SlackChannelRef ¶
SlackChannelRef is one Slack channel in a slack source.
type SlackSource ¶
type SlackSource struct {
Channels []SlackChannelRef `toml:"channels"`
}
SlackSource configures a slack source.
type Source ¶
type Source struct {
Name string `toml:"name"`
Type string `toml:"type"` // notion_db | notion_page | slack | github
Description string `toml:"description"`
Enabled *bool `toml:"enabled"` // default true
NotionDB *NotionDBSource `toml:"notion_db"`
NotionPage *NotionPageSource `toml:"notion_page"`
Slack *SlackSource `toml:"slack"`
GitHub *GitHubSource `toml:"github"`
}
Source is a workspace data source seeded into the memory repository before the run. Source-aware tools / workflows (e.g. the workspace-metadata tool, Job runs) read these from the same repo. Exactly one type-specific config block must be present, matching Type.
type Tool ¶
Tool describes how one agent tool behaves during the run. By default a tool is simulated: when called, the ToolSimulator LLM produces a response from Background. Setting Live runs the real client instead (Background ignored).
type ValidateOptions ¶
type ValidateOptions struct {
// KnownWorkflows is the set of registered workflow driver kinds.
KnownWorkflows []string
// KnownTools is the catalog of valid tool names. When empty, tool names are
// not checked (e.g. during partial validation).
KnownTools []string
}
ValidateOptions injects the sets the validator checks scenario references against. They are passed in (rather than imported) to avoid an import cycle with the driver / tool packages.