Documentation
¶
Overview ¶
Package generate provides pluggable session source adapters for scenario generation. It enables turning production session data (with failing assertions) into reproducible test scenarios for PromptArena.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ConvertSessionToScenario ¶
func ConvertSessionToScenario(session *SessionDetail, opts ConvertOptions) (*config.ScenarioConfig, error)
ConvertSessionToScenario converts a SessionDetail into a ScenarioConfig YAML document.
func Fingerprint ¶
func Fingerprint(session *SessionDetail) string
Fingerprint produces a stable hash for a session based on its failure pattern and user message content. Sessions with the same fingerprint are considered duplicates for deduplication purposes.
Types ¶
type ConvertOptions ¶
type ConvertOptions struct {
// Pack is the path to a .pack.json file. When set, the scenario is generated
// as a workflow scenario (with Steps) instead of a regular conversation scenario.
Pack string
}
ConvertOptions controls how a session is converted into a scenario.
type ListOptions ¶
type ListOptions struct {
// FilterPassed filters by pass/fail status: nil=all, *true=passed only, *false=failed only.
FilterPassed *bool
// FilterEvalType filters by assertion type (e.g., "content_matches").
FilterEvalType string
// Limit caps the number of results. 0 means unlimited.
Limit int
}
ListOptions controls which sessions are returned by List.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry manages named SessionSourceAdapter instances.
func (*Registry) Get ¶
func (r *Registry) Get(name string) (SessionSourceAdapter, error)
Get returns the adapter with the given name, or an error if not found.
func (*Registry) Register ¶
func (r *Registry) Register(adapter SessionSourceAdapter)
Register adds an adapter to the registry, keyed by its Name().
type SessionDetail ¶
type SessionDetail struct {
SessionSummary
// Messages is the complete conversation history.
Messages []types.Message
// EvalResults contains conversation-level assertion results (nil for recordings).
EvalResults []assertions.ConversationValidationResult
// TurnEvalResults contains per-turn assertion results keyed by turn index.
TurnEvalResults map[int][]TurnEvalResult
}
SessionDetail contains the full session data needed for scenario generation.
func DeduplicateSessions ¶
func DeduplicateSessions(sessions []*SessionDetail) []*SessionDetail
DeduplicateSessions keeps the first session per unique fingerprint.
type SessionSourceAdapter ¶
type SessionSourceAdapter interface {
// Name returns the adapter's unique identifier (e.g., "recordings", "omnia").
Name() string
// List returns session summaries matching the given options.
List(ctx context.Context, opts ListOptions) ([]SessionSummary, error)
// Get returns the full session detail including messages and eval results.
Get(ctx context.Context, sessionID string) (*SessionDetail, error)
}
SessionSourceAdapter provides session data for scenario generation. Implementations query external systems (e.g., session APIs, local recordings) and return structured session data that can be converted into Arena scenarios.