trajectory

package module
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package trajectory records and evaluates deterministic Agent execution facts. It keeps Agent vocabulary outside the subject-agnostic eval kernel and owns no experiment persistence, replay scheduler, or product workflow.

Index

Constants

View Source 
const (
	MetricTrajectory      eval.MetricName = "trajectory"
	MetricTaskSuccess     eval.MetricName = "task_success"
	MetricToolCalls       eval.MetricName = "tool_calls"
	MetricConsistency     eval.MetricName = "consistency"
	MetricCommittedSteps  eval.MetricName = "committed_steps"
	MetricPreparedEffects eval.MetricName = "prepared_effects"
	MetricAcceptedSignals eval.MetricName = "accepted_signals"
	MetricDroppedDeltas   eval.MetricName = "dropped_deltas"
	MetricTotalTokens     eval.MetricName = "total_tokens"
	MetricDuration        eval.MetricName = "duration"
)

These metric names are constants because a report is aggregated by exact metric identity. A renamed or restated metric silently splits a series that a reader would compare as one.

Variables

View Source 
var (
	ErrInvalidTrajectory = errors.New("eval/trajectory: invalid trajectory")
	ErrInvalidSample     = errors.New("eval/trajectory: invalid sample")
)
View Source 
var ErrIncompleteRecording = errors.New("eval/trajectory: incomplete recording")

ErrIncompleteRecording reports observations that cannot form a trustworthy Trajectory.

Functions

This section is empty.

Types

type Config 

type Config struct {
	RootProcessID agent.ProcessID
	Termination   agent.Termination
	Output        *agent.Output
	Usage         agent.Usage
	Duration      time.Duration
	Events        []agent.Event
	ModelCalls    []ModelCall
	ToolCalls     []ToolCall
}

Config supplies the complete facts owned by one Trajectory.

type Evaluator 

type Evaluator struct{}

Evaluator deterministically checks terminal success, exact Tool behavior, replay consistency, and configured resource regressions for one Sample. Its zero value is ready to use because all case-specific policy belongs to the typed Sample rather than mutable evaluator configuration.

func (Evaluator) Evaluate 

func (Evaluator) Evaluate(ctx context.Context, sample Sample) (eval.Report, error)

type Expectation 

type Expectation struct {
	Status   agent.Status  `json:"status"`
	Output   *agent.Output `json:"output,omitempty"`
	Tools    *ToolSequence `json:"tools,omitempty"`
	Baseline *Trajectory   `json:"baseline,omitempty"`
	Limits   Limits        `json:"limits,omitzero"`
}

Expectation describes case-specific success without contaminating Metric identity. Baseline is optional and enables deterministic replay comparison.

func (Expectation) Validate 

func (e Expectation) Validate() error

type Limits 

type Limits struct {
	CommittedSteps  *uint64        `json:"committed_steps,omitempty"`
	PreparedEffects *uint64        `json:"prepared_effects,omitempty"`
	AcceptedSignals *uint64        `json:"accepted_signals,omitempty"`
	DroppedDeltas   *uint64        `json:"dropped_deltas,omitempty"`
	TotalTokens     *int64         `json:"total_tokens,omitempty"`
	Duration        *time.Duration `json:"duration,omitempty"`
}

Limits defines optional upper bounds. Pointers distinguish an asserted zero from a dimension the case does not evaluate.

func (Limits) Validate 

func (l Limits) Validate() error

type ModelCall 

type ModelCall struct {
	ProcessID    agent.ProcessID `json:"process_id"`
	StepSequence uint64          `json:"step_sequence"`
	CallSequence uint32          `json:"call_sequence"`
	Response     *chat.Response  `json:"response"`
}

ModelCall is one settled model boundary attributed to an Agent Process Step.

func (ModelCall) Clone 

func (m ModelCall) Clone() ModelCall

func (ModelCall) Validate 

func (m ModelCall) Validate() error

type Recorder 

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder joins the Agent mechanics and Interaction semantic observation boundaries into owned Trajectories. Take consumes one completed root tree so a long-lived Engine does not turn evaluation capture into an unbounded log.

func (*Recorder) OnEvent 

func (r *Recorder) OnEvent(_ context.Context, event agent.Event)

func (*Recorder) OnModelResponse 

func (r *Recorder) OnModelResponse(
	_ context.Context,
	invocation interaction.ModelInvocation,
	response *chat.Response,
)

func (*Recorder) OnToolSettled 

func (r *Recorder) OnToolSettled(
	_ context.Context,
	invocation interaction.ToolInvocation,
	settlement interaction.ToolSettlement,
)

func (*Recorder) OnToolStarted 

func (r *Recorder) OnToolStarted(_ context.Context, invocation interaction.ToolInvocation)

func (*Recorder) Take 

func (r *Recorder) Take(result agent.Result) (Trajectory, error)

Take returns and releases the complete recording for one terminal root Result. The result must belong to the root Process, not one child.

type Sample 

type Sample struct {
	Actual   Trajectory  `json:"actual"`
	Expected Expectation `json:"expected"`
}

Sample is the typed subject consumed by Evaluator.

func (Sample) Validate 

func (s Sample) Validate() error

type ToolArguments 

type ToolArguments string

ToolArguments is one exact semantic JSON argument assertion. Its empty value matches a Tool call that supplied no argument text.

func (ToolArguments) Validate 

func (t ToolArguments) Validate() error

type ToolCall 

type ToolCall struct {
	ProcessID    agent.ProcessID  `json:"process_id"`
	StepSequence uint64           `json:"step_sequence"`
	ModelCall    uint32           `json:"model_call"`
	Index        uint32           `json:"index"`
	Call         chat.ToolCall    `json:"call"`
	Outcome      ToolOutcome      `json:"outcome"`
	Result       *chat.ToolResult `json:"result,omitempty"`
	Failure      string           `json:"failure,omitempty"`
}

ToolCall is one settled Tool boundary attributed to an Agent Process Step.

func (ToolCall) Clone 

func (t ToolCall) Clone() ToolCall

func (ToolCall) Validate 

func (t ToolCall) Validate() error

type ToolExpectation 

type ToolExpectation struct {
	Name      string         `json:"name"`
	Arguments *ToolArguments `json:"arguments,omitempty"`
	Outcome   ToolOutcome    `json:"outcome,omitempty"`
}

ToolExpectation asserts that a tool was called, and optionally how. Arguments and Outcome are omissible so a sample can pin the part of the behavior it cares about without freezing the rest; an expectation that had to state every field would break on unrelated prompt or model changes and stop being run.

func (ToolExpectation) Validate 

func (t ToolExpectation) Validate() error

type ToolOutcome 

type ToolOutcome string

ToolOutcome is the complete host-boundary outcome of one started Tool call. 

const (
	ToolOutcomeInvalid       ToolOutcome = ""
	ToolOutcomeSucceeded     ToolOutcome = "succeeded"
	ToolOutcomeError         ToolOutcome = "error"
	ToolOutcomeInputRequired ToolOutcome = "input_required"
	ToolOutcomeFailed        ToolOutcome = "failed"
	ToolOutcomeUnknown       ToolOutcome = "unknown"
)

Tool outcomes are a closed vocabulary because a trajectory is compared against a recorded baseline. An open set would let two runs describe the same result with different words and register as a behavior change.

func (ToolOutcome) Valid 

func (t ToolOutcome) Valid() bool

type ToolSequence 

type ToolSequence struct {
	Calls []ToolExpectation `json:"calls"`
}

ToolSequence makes an exact ordered Tool-call assertion explicit. A nil *ToolSequence skips the assertion; a non-nil empty sequence asserts no calls.

type Trajectory 

type Trajectory struct {
	// contains filtered or unexported fields
}

Trajectory is an owned, portable record of one completed root Process tree. Absolute timing and provider responses remain available in the record, but BehaviorDigest deliberately excludes them from replay comparison.

func New 

func New(config Config) (Trajectory, error)

New clones every input and then sorts by process path rather than by arrival time. Two runs of the same agent interleave concurrent siblings differently, so wall-clock order would make identical behavior compare as a regression; ordering by structural position is what makes replay comparison meaningful.

func (Trajectory) BehaviorDigest 

func (t Trajectory) BehaviorDigest() (string, error)

BehaviorDigest identifies deterministic, semantic behavior while excluding wall-clock time, attempt duration, provider responses, and token usage.

func (Trajectory) Clone 

func (t Trajectory) Clone() (Trajectory, error)

func (Trajectory) Duration 

func (t Trajectory) Duration() time.Duration

func (Trajectory) Events 

func (t Trajectory) Events() []agent.Event

func (Trajectory) MarshalJSON 

func (t Trajectory) MarshalJSON() ([]byte, error)

func (Trajectory) ModelCalls 

func (t Trajectory) ModelCalls() []ModelCall

func (Trajectory) Output 

func (t Trajectory) Output() *agent.Output

func (Trajectory) RootProcessID 

func (t Trajectory) RootProcessID() agent.ProcessID

func (Trajectory) Termination 

func (t Trajectory) Termination() agent.Termination

func (Trajectory) ToolCalls 

func (t Trajectory) ToolCalls() []ToolCall

func (Trajectory) TotalTokens 

func (t Trajectory) TotalTokens() (int64, error)

func (*Trajectory) UnmarshalJSON 

func (t *Trajectory) UnmarshalJSON(data []byte) error

func (Trajectory) Usage 

func (t Trajectory) Usage() agent.Usage

func (Trajectory) Validate 

func (t Trajectory) Validate() error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.