scenario

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 28, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package scenario defines the TOML schema for eval scenarios and loads / validates them. A scenario file carries both the system-under-test workspace configuration (authored with the same top-level layout as a normal workspace config file) and the eval-specific tables (meta / input / cases / tools / persona / expect). The workspace part is extracted by handing the same file to config.LoadWorkspaceConfigs, which ignores the eval-only keys; the eval-only tables are decoded into Scenario. Keeping both in one file lets a scenario be fully self-contained.

Index

Constants

View Source
const WorkflowJob = "job"

WorkflowJob is the workflow kind that runs a Job against a case.

Variables

This section is empty.

Functions

This section is empty.

Types

type CaseSeed

type CaseSeed struct {
	Title       string         `toml:"title"`
	Description string         `toml:"description"`
	BoardStatus string         `toml:"board_status"`
	Fields      map[string]any `toml:"fields"`
}

CaseSeed is a pre-existing case injected into the memory repository before the run (for future search/context use; optional).

type Check

type Check struct {
	ID       string `toml:"id"`
	Question string `toml:"question"`
}

Check is one natural-language yes/no question the judge answers against the produced artifact (case state + transcript + tool calls).

type Expect

type Expect struct {
	Checks []Check `toml:"checks"`
}

Expect holds the checklist the judge evaluates the produced artifact against.

type GitHubRepoRef

type GitHubRepoRef struct {
	Owner string `toml:"owner"`
	Repo  string `toml:"repo"`
}

GitHubRepoRef is one repository in a github source.

type GitHubSource

type GitHubSource struct {
	Repositories []GitHubRepoRef `toml:"repositories"`
}

GitHubSource configures a github source.

type Input

type Input struct {
	Text string `toml:"text"`
	// Reporter optionally pins the posting user id. Empty means the harness
	// synthesizes one (the reporter identity does not affect the evaluation).
	Reporter string `toml:"reporter"`
}

Input is the triggering input. For thread_mode_initial it is the first top-level post in the monitored channel.

type JobSpec

type JobSpec struct {
	ID         string `toml:"id"`
	TargetCase string `toml:"target_case"`
}

JobSpec selects which workspace Job to run and against which seeded case. Used by the job workflow. The Job must be declared in the workspace config ([[job]]); the target case must be one of the seeded [[cases]] (matched by title, or the first when TargetCase is empty).

type Meta

type Meta struct {
	ID          string `toml:"id"`
	Description string `toml:"description"`
	Workflow    string `toml:"workflow"`
	// Language is the i18n language the system-under-test agent responds in
	// (the simulated end-user's locale). Distinct from the eval output language.
	Language string `toml:"language"`
}

Meta identifies the scenario and selects the workflow driver.

type NotionDBSource

type NotionDBSource struct {
	DatabaseID    string `toml:"database_id"`
	DatabaseTitle string `toml:"database_title"`
	DatabaseURL   string `toml:"database_url"`
}

NotionDBSource configures a notion_db source.

type NotionPageSource

type NotionPageSource struct {
	PageID    string `toml:"page_id"`
	PageTitle string `toml:"page_title"`
	PageURL   string `toml:"page_url"`
	Recursive bool   `toml:"recursive"`
	MaxDepth  int    `toml:"max_depth"`
}

NotionPageSource configures a notion_page source.

type Persona

type Persona struct {
	Description string `toml:"description"`
	Knowledge   string `toml:"knowledge"`
	// MaxAnswerTurns bounds the question/answer loop. Zero means no follow-up
	// answers are produced.
	MaxAnswerTurns int `toml:"max_answer_turns"`
}

Persona is the simulated end-user that answers the agent's questions.

type Scenario

type Scenario struct {
	Meta    Meta            `toml:"meta"`
	Input   Input           `toml:"input"`
	Cases   []CaseSeed      `toml:"cases"`
	Sources []Source        `toml:"sources"`
	Tools   map[string]Tool `toml:"tools"`
	Persona Persona         `toml:"persona"`
	// Job (table key `run_job`) selects which workspace Job to run for the
	// job workflow. The key is `run_job`, not `job`, to avoid
	// colliding with the workspace config's `[[job]]` array in the same file.
	Job    *JobSpec `toml:"run_job"`
	Expect Expect   `toml:"expect"`

	// Workspace is loaded from the same file via config.LoadWorkspaceConfigs.
	// Not a TOML field of this struct.
	Workspace *config.WorkspaceConfig `toml:"-"`
	// Path is the source file path, retained for diagnostics.
	Path string `toml:"-"`
}

Scenario is one eval case. The TOML-decoded eval tables plus the workspace configuration extracted from the same file.

func Load

func Load(path string) (*Scenario, error)

Load reads and parses a scenario file: it decodes the eval-specific tables and extracts the workspace configuration from the same file (reusing the existing config loader, which validates the workspace and ignores eval-only keys).

func (*Scenario) Validate

func (sc *Scenario) Validate(opts ValidateOptions) error

Validate checks the eval-specific parts of the scenario. The workspace part was already validated by Load via config.LoadWorkspaceConfigs.

type SlackChannelRef

type SlackChannelRef struct {
	ID   string `toml:"id"`
	Name string `toml:"name"`
}

SlackChannelRef is one Slack channel in a slack source.

type SlackSource

type SlackSource struct {
	Channels []SlackChannelRef `toml:"channels"`
}

SlackSource configures a slack source.

type Source

type Source struct {
	Name        string            `toml:"name"`
	Type        string            `toml:"type"` // notion_db | notion_page | slack | github
	Description string            `toml:"description"`
	Enabled     *bool             `toml:"enabled"` // default true
	NotionDB    *NotionDBSource   `toml:"notion_db"`
	NotionPage  *NotionPageSource `toml:"notion_page"`
	Slack       *SlackSource      `toml:"slack"`
	GitHub      *GitHubSource     `toml:"github"`
}

Source is a workspace data source seeded into the memory repository before the run. Source-aware tools / workflows (e.g. the workspace-metadata tool, Job runs) read these from the same repo. Exactly one type-specific config block must be present, matching Type.

func (Source) IsEnabled

func (s Source) IsEnabled() bool

IsEnabled reports the effective enabled flag (default true when unset).

type Tool

type Tool struct {
	Background string `toml:"background"`
	Live       bool   `toml:"live"`
}

Tool describes how one agent tool behaves during the run. By default a tool is simulated: when called, the ToolSimulator LLM produces a response from Background. Setting Live runs the real client instead (Background ignored).

type ValidateOptions

type ValidateOptions struct {
	// KnownWorkflows is the set of registered workflow driver kinds.
	KnownWorkflows []string
	// KnownTools is the catalog of valid tool names. When empty, tool names are
	// not checked (e.g. during partial validation).
	KnownTools []string
}

ValidateOptions injects the sets the validator checks scenario references against. They are passed in (rather than imported) to avoid an import cycle with the driver / tool packages.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL