Documentation
¶
Overview ¶
Package replay compares the behaviour of two recorded agent sessions.
It answers the triage question "did the agent do something different this time, and where did it first diverge?" — the thing you actually want to know when a task that worked yesterday does not work today.
Behaviour, not prose ¶
Comparison is over the sequence of tool calls, not over the assistant's text. Model output is nondeterministic: two runs of the same task almost always word things differently while doing exactly the same work, so diffing prose reports a difference on essentially every comparison and is useless as a signal. The tool calls are what changed the world, so they are what is compared.
Assistant text is still carried on each Turn so a reporter can show what was said around a divergence; it just does not decide whether a divergence happened.
Delegated work ¶
A turn taken by a sub-agent is still a turn the run took, so sub-sessions are walked in place: their turns appear in sequence where the delegation happened. Skipping them would report "identical behaviour" for two runs whose sub-agents did entirely different things — the precise wrong answer to the question this package exists for.
First divergence only ¶
Once two runs differ, everything after that point is downstream of the difference and comparing it produces noise, not information. So the comparison stops at the first divergence and reports where it was.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Divergence ¶
type Divergence struct {
Kind Kind `json:"kind"`
// TurnIndex is the 0-based turn at which the runs first differ.
TurnIndex int `json:"turn_index"`
// A and B are the diverging turns. Either may be nil when one run had no
// turn at this index.
A *Turn `json:"a,omitempty"`
B *Turn `json:"b,omitempty"`
}
Divergence is the first behavioural difference found.
type Kind ¶
type Kind string
Kind classifies how two runs differ.
const ( // KindToolCalls means both runs made a turn at this index but called // different tools, or the same tools with different arguments. KindToolCalls Kind = "tool_calls" // KindExtraTurn means the second run kept going after the first stopped. KindExtraTurn Kind = "extra_turn" // KindMissingTurn means the second run stopped before the first did. KindMissingTurn Kind = "missing_turn" )
type Result ¶
type Result struct {
TurnsA int `json:"turns_a"`
TurnsB int `json:"turns_b"`
// TurnsMatched is how many leading turns behaved identically.
TurnsMatched int `json:"turns_matched"`
// Divergence is nil when both runs behaved identically throughout.
Divergence *Divergence `json:"divergence,omitempty"`
}
Result is the outcome of a comparison.
func CompareSessions ¶
CompareSessions is TurnsOf on both sides followed by Compare.
type Turn ¶
type Turn struct {
// Index is the turn's 0-based position in the session.
Index int `json:"index"`
// Agent is the agent that produced the turn, so a divergence in a
// multi-agent run can be attributed.
Agent string `json:"agent,omitempty"`
// Content is the assistant text. Carried for reporting; never compared.
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}
Turn is one assistant turn: what it said, and what it called.
Source Files