generate

package
v1.3.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 4, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package generate provides pluggable session source adapters for scenario generation. It enables turning production session data (with failing assertions) into reproducible test scenarios for PromptArena.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ConvertSessionToScenario

func ConvertSessionToScenario(session *SessionDetail, opts ConvertOptions) (*config.ScenarioConfig, error)

ConvertSessionToScenario converts a SessionDetail into a ScenarioConfig YAML document.

func Fingerprint

func Fingerprint(session *SessionDetail) string

Fingerprint produces a stable hash for a session based on its failure pattern and user message content. Sessions with the same fingerprint are considered duplicates for deduplication purposes.

Types

type ConvertOptions

type ConvertOptions struct {
	// Pack is the path to a .pack.json file. When set, the scenario is generated
	// as a workflow scenario (with Steps) instead of a regular conversation scenario.
	Pack string
}

ConvertOptions controls how a session is converted into a scenario.

type ListOptions

type ListOptions struct {
	// FilterPassed filters by pass/fail status: nil=all, *true=passed only, *false=failed only.
	FilterPassed *bool
	// FilterEvalType filters by assertion type (e.g., "content_matches").
	FilterEvalType string
	// Limit caps the number of results. 0 means unlimited.
	Limit int
}

ListOptions controls which sessions are returned by List.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages named SessionSourceAdapter instances.

func NewRegistry

func NewRegistry() *Registry

NewRegistry creates a new empty adapter registry.

func (*Registry) Get

func (r *Registry) Get(name string) (SessionSourceAdapter, error)

Get returns the adapter with the given name, or an error if not found.

func (*Registry) Names

func (r *Registry) Names() []string

Names returns the sorted list of registered adapter names.

func (*Registry) Register

func (r *Registry) Register(adapter SessionSourceAdapter)

Register adds an adapter to the registry, keyed by its Name().

type SessionDetail

type SessionDetail struct {
	SessionSummary
	// Messages is the complete conversation history.
	Messages []types.Message
	// EvalResults contains conversation-level assertion results (nil for recordings).
	EvalResults []assertions.ConversationValidationResult
	// TurnEvalResults contains per-turn assertion results keyed by turn index.
	TurnEvalResults map[int][]TurnEvalResult
}

SessionDetail contains the full session data needed for scenario generation.

func DeduplicateSessions

func DeduplicateSessions(sessions []*SessionDetail) []*SessionDetail

DeduplicateSessions keeps the first session per unique fingerprint.

type SessionSourceAdapter

type SessionSourceAdapter interface {
	// Name returns the adapter's unique identifier (e.g., "recordings", "omnia").
	Name() string

	// List returns session summaries matching the given options.
	List(ctx context.Context, opts ListOptions) ([]SessionSummary, error)

	// Get returns the full session detail including messages and eval results.
	Get(ctx context.Context, sessionID string) (*SessionDetail, error)
}

SessionSourceAdapter provides session data for scenario generation. Implementations query external systems (e.g., session APIs, local recordings) and return structured session data that can be converted into Arena scenarios.

type SessionSummary

type SessionSummary struct {
	ID          string
	Source      string
	ScenarioID  string
	ProviderID  string
	Timestamp   time.Time
	TurnCount   int
	HasFailures bool
	Tags        []string
	Metadata    map[string]interface{}
}

SessionSummary is a lightweight representation of a session for listing.

type TurnEvalResult

type TurnEvalResult struct {
	Type    string
	Passed  bool
	Message string
	Params  map[string]interface{}
}

TurnEvalResult represents the result of a single turn-level assertion.

Directories

Path Synopsis
Package sources provides built-in SessionSourceAdapter implementations.
Package sources provides built-in SessionSourceAdapter implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL