evaluation

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 10, 2026 License: MIT Imports: 6 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ExportPath

func ExportPath(sessionID string) string

ExportPath returns the filesystem path for an exported evaluation JSONL file. Exports are stored under ~/.aimux/exports/.

func WriteExport

func WriteExport(path string, turns []ExportTurn, sessionMeta *ExportSessionMeta) error

WriteExport writes all enriched turns as JSONL to the given path. The parent directory is created if it does not exist. If sessionMeta is non-nil, it is written as the first line (type: "session_meta"). Writes are performed atomically via a temp file and rename.

Types

type Annotation

type Annotation struct {
	Turn      int       `json:"turn"`
	Label     string    `json:"label"`
	Note      string    `json:"note,omitempty"`
	Timestamp time.Time `json:"timestamp"`
}

Annotation represents a human evaluation label applied to a specific conversation turn. Labels classify turns as "good", "bad", "wasteful", or "error" with an optional free-text note.

type ExportAction

type ExportAction struct {
	Tool    string `json:"tool"`
	Input   string `json:"input"`
	Success bool   `json:"success"`
	Error   string `json:"error,omitempty"`
}

ExportAction describes a single tool invocation within a conversation turn.

type ExportSessionMeta added in v0.6.0

type ExportSessionMeta struct {
	Type         string   `json:"type"` // always "session_meta"
	SessionID    string   `json:"session_id"`
	Annotation   string   `json:"annotation,omitempty"`    // achieved/partial/failed/abandoned
	FailureModes []string `json:"failure_modes,omitempty"` // failure-mode tags
	Note         string   `json:"note,omitempty"`          // free-text rationale
	Title        string   `json:"title,omitempty"`         // LLM-generated title
}

ExportSessionMeta holds session-level evaluation metadata written as the first line of a JSONL export file (type: "session_meta").

type ExportTurn

type ExportTurn struct {
	Turn       int            `json:"turn"`
	Timestamp  string         `json:"timestamp"`
	Input      string         `json:"input"`
	Output     string         `json:"output"`
	Actions    []ExportAction `json:"actions"`
	TokensIn   int64          `json:"tokens_in"`
	TokensOut  int64          `json:"tokens_out"`
	CostUSD    float64        `json:"cost_usd"`
	DurationMs int64          `json:"duration_ms"`
	Label      string         `json:"label,omitempty"`
	Note       string         `json:"note,omitempty"`
}

ExportTurn is the full enriched representation of a conversation turn, combining raw trace data with evaluation annotations. Used for serializing evaluated sessions to JSONL for downstream analysis.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store manages annotation persistence for a single agent session. Annotations are stored as JSONL files under ~/.aimux/evaluations/.

func NewStore

func NewStore(sessionID string) *Store

NewStore creates a Store for the given session ID. The backing directory (~/.aimux/evaluations/) is created lazily on the first write.

func (*Store) GetForTurn

func (s *Store) GetForTurn(turn int) *Annotation

GetForTurn returns the latest annotation for the given turn number, or nil if no annotation exists for that turn.

func (*Store) Load

func (s *Store) Load() ([]Annotation, error)

Load reads all annotations for this session from the JSONL file. Returns an empty slice (not an error) if the file does not exist.

func (*Store) Remove

func (s *Store) Remove(turn int) error

Remove deletes all annotations for the given turn by rewriting the JSONL file without that turn's entries. The rewrite is performed atomically: data is written to a temporary file first, then renamed over the original.

func (*Store) Save

func (s *Store) Save(a Annotation) error

Save appends an annotation as a single JSONL line. The directory is created on the first call. Writes are performed atomically by appending to the file with O_APPEND to avoid partial writes on concurrent access.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL