trajectory

package
v3.1.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package trajectory reads what an agent DID, in order, out of a recorded run — and reports on the shape of it.

Why this exists

Everything else we measure asks whether the agent ended up in the right state. The GKE drill's six boxes grade an answer; #652's eval runner grades effects on the world. Neither asks how the agent got there, and that is a separate question with separate failures (#994):

  • The #711 run reached a correct-looking end state in 44 turns and 1.4M input tokens for $1.33, against $0.26 for the comparable run. An outcome grader scores that a pass.
  • The list_agents runaway behind the #623–#627 guardrail train was a shape failure: the same call, forever, with a fine end state.
  • On 2026-09-11 a drill run scored six boxes out of six while its `cluster` subagent was dead of a Vertex 429. The parent re-did the reads itself and still produced a grounded answer. Nothing in the rubric can see any of that: the run's evidence.md never mentions the delegation, the 429, or the subagent at all, so a run where half the plan did not execute is indistinguishable on the sheet from one where it did.

What the first backfill showed

The whole archive, 15 runs, read as step tables (2026-09-11). Three things were visible that no evidence sheet says, and none of them needed a rubric — they are counts and row order:

  • The dead delegation is a NUMBER. Scenario A runs make 11-12 steps with 9-10 of them in the `cluster` subagent. The 2026-09-11 run that scored 6/6 makes 6 steps with 1 in `cluster`. The column says what the scorecard had to be told by hand.
  • #1014 is a ROW ORDER. In run 20260909T232522Z-c the child reads a resource and the parent, right after the handoff, reads it again. "A parent cannot cite its subagent's reads, so it re-does them" stops being an anecdote and becomes a shape you can point at.
  • `return_result` is present in every healthy delegation and absent from the broken one. A single column tells you whether the handoff completed, which is cheaper than any prose scan.

What it did NOT show is a loop or a redundant-call problem WITHIN one agent: no run in the archive has one. That matters for what gets built next — such a measure would report zero across the entire corpus and would therefore be untested. Delegation shape fires today.

What the first measure showed

Delegation, run over the same 15 runs, puts numbers on all three bullets above:

  • 10 repeated reads across 8 of the 15 runs. That is #1014's headline finding as a rate rather than an anecdote: in more than half the archive the parent re-issues a read its child already made.
  • 1 failed delegation, in 20260911T110503Z-a, and the parent DISCLOSED it — unprompted, in the second line of its answer. See the correction below.
  • 0 undisclosed failures. On this corpus the agent is not the thing hiding the failure.

Three design decisions in Delegation came from the corpus refusing to agree with what had been written about it, and each is worth more than the code it changed:

  • A "result dropped" measure was planned around #641 (`wait:true` returns a summary, not findings). #641 is CLOSED, and the archive confirms the fix: spawn_agent's `output` equals the child's return_result `result` byte-for-byte in all 14 healthy runs. The measure would have reported a defect that no longer exists.
  • The repeated-read signature originally matched arguments exactly and found 5 repeats in 5 runs — and MISSED the case that motivated it, where the child read a resource as YAML and the parent re-read it unformatted. Ignoring presentational arguments moved it to 10 across 8. See presentationalArgs.
  • The 2026-09-11 run's parent was described here, and in a signed evidence sheet, as having absorbed the 429 silently. It did not: it said so in its own answer. What was actually silent was evidence.md, and the rubric's blindness got read as the agent's. The bullet above is the corrected version.

What it is not

It is not a grader and it does not return a verdict. Measures emit Observations — "here is a thing that happened, at this step, with this evidence" — and a human reads them. Two reasons, and both are load-bearing:

The rubric in dev/uat/gke-drill/SCORECARD.md is still moving, and automating a moving rubric encodes whichever version happened to be current. #652 says the same thing about the eval corpus.

And the failure mode this package is most likely to have is not being wrong, it is being ignored. #790 + #844 spent 6,146 lines on a drift detector that detected correctly and changed nothing, because attention was the bottleneck rather than detection. dev/tools/focus carries that lesson in its own header. So: no gate, no CI wiring, no threshold that goes red. If this ever needs a fix, deleting it is a correct response.

The substrate

A trajectory is the ordered sequence of (agent, tool, arguments, outcome) across a parent and its subagents. The drill already records exactly that, and the recording turns out to be free to read: the `event` object in a run's transcript.jsonl is a marshalled google.golang.org/adk/v2/session.Event — the same type pkg/eventlog persists. All fifteen runs archived as of 2026-09-11 decode into it with DisallowUnknownFields set and no failures, so this package gets typed FunctionCall.Args, FunctionResponse.Response, UsageMetadata, ErrorCode and Branch for the cost of a json.Unmarshal, where score.py hand-walks the same dicts.

That also settles, in the narrow sense, the open question #994 parks: #881 (a general transcript export) is frozen while the drill, the eval runner and this package each reach for their own rendering of the same rows. They do not disagree about the data MODEL — all three already speak session.Event. They differ only in transport, SSE capture against a SQLite eventlog. So the seam here is the loader: LoadRun turns one recorded run into [Frame]s, and everything above it works on Frames. A pkg/eventlog loader is additive, needing no change to Step or to anything reading Steps.

Relationship to score.py

None, deliberately. score.py decides the drill's six rubric boxes and renders evidence.md. This answers "what did it do". They are not merged and this does not feed that, at least until the rubric stops changing.

The one place they must NOT drift is what counts as a failed tool call — see Status and responseStatus, which mirror score.py's three-way rule on purpose, down to reproducing Python's JSON encoding so that the rule's 200-character window cuts in the same place (see pyDumps). That mirroring is checked, not asserted: TestAgreesWithScorePyOnTheArchive compares this package's five counts against the tally score.py wrote into each run's own evidence.md. Fifteen runs, seventy-five numbers, zero disagreements as of 2026-09-11.

Index

Constants

View Source
const (
	// KindDelegationFailed is a spawn_agent whose child did not finish.
	KindDelegationFailed = "delegation-failed"
	// KindDelegationDisclosed is a failed delegation the parent told the
	// operator about, in its own answer, unprompted.
	KindDelegationDisclosed = "delegation-disclosed"
	// KindDelegationUndisclosed is a failed delegation the parent's
	// answer does not mention.
	KindDelegationUndisclosed = "delegation-undisclosed"
	// KindRepeatedRead is the parent re-issuing, after the handoff, a
	// call its child already made.
	KindRepeatedRead = "delegation-repeated-read"
)

Observation kinds emitted by Delegation.

View Source
const Parent = "parent"

Parent is the agent name given to frames from the parent session, so that a trajectory's Agent column is total and nothing has to test for the empty string. Subagents carry their own registered name.

Variables

View Source
var Measures = []Measure{
	Delegation{},
}

Measures is the set applied by default.

It is short on purpose. Every measure here fires on the archive as it stood on 2026-09-11 — a measure that reports nothing on every run available has not been shown to work, and shipping one is how you get an instrument nobody trusts. The candidate this rules out today is redundancy-within-an-agent, the "loop rate" #967 names: no run in the archive contains a loop, so it would be zero everywhere. It belongs with a corpus (#966) that has one.

Functions

func LoadRuns

func LoadRuns(root string) ([]*Trajectory, []SkippedRun, error)

LoadRuns reads every immediate subdirectory of root that looks like a run (it has a transcript.jsonl), oldest first, and returns the runs it could read alongside the ones it could not.

A run we cannot read is still a finding about the recorder, not a run to drop quietly — but it is not a reason to refuse the other sixty either. The archive is append-only history: one run truncated by a rig defect in September permanently blocks every analysis of every run before and after it if an unreadable directory is fatal. That happened — a jq failure left a 0-byte meta.json (since fixed in drill.sh by writing through a temp file), and it took the whole corpus down with it. So the second return value carries the casualties and the caller decides.

func WriteObservations

func WriteObservations(w io.Writer, obs []Observation) error

WriteObservations renders what the measures found, under the step table they refer to.

Every line carries the step it anchors to, because an observation you cannot walk back to a row is a claim. Nothing here is coloured, ranked or counted as a problem: see Observation.

When nothing was found it names the measures that found nothing. "observations: none" on its own invites the reading "this run was clean", and the true statement is narrower — these measures, and only these, looked and did not see anything. The gap between the two is the whole limitation of the instrument, so it is printed every time rather than documented once.

func WriteSteps

func WriteSteps(w io.Writer, t *Trajectory) error

WriteSteps renders one trajectory's step table.

The table is the product for now. Before any measure exists, the question this package has to answer is whether the ordered steps show anything the drill's evidence sheets do not — and that is a question for a human reading rows, not for code deciding.

Types

type Delegation

type Delegation struct{}

Delegation reports on the shape of parent-to-subagent handoffs.

It is the first measure in this package, and it is first because of what the archive showed rather than what was easiest to write. The obvious candidate was redundancy inside one agent — #967's "loop rate" — and no run in the archive contains a loop, so it would have reported zero fifteen times and proven nothing. Every kind below fires on the archive as it stood on 2026-09-11.

What it does NOT do is decide whether any of it was wrong. A parent that re-reads a resource its child already read may be double-checking something load-bearing; a failed delegation the parent recovered from cleanly is a good outcome, not a bad one. Those are judgements about intent, and the package doc explains why they are not made here.

func (Delegation) Name

func (Delegation) Name() string

Name implements Measure.

func (Delegation) Observe

func (d Delegation) Observe(t *Trajectory) []Observation

Observe implements Measure.

type Frame

type Frame struct {
	Agent string         `json:"agent"`
	Seq   int            `json:"seq"`
	Event *session.Event `json:"event"`
	// contains filtered or unexported fields
}

Frame is one recorded event, tagged with which agent produced it.

Seq is a total order across parent AND subagents: they write through one event log, so interleaving by Seq is what makes "which turn was the parent in when the child read that" an answerable question.

func (Frame) Partial

func (f Frame) Partial() bool

Partial reports whether this is a streaming chunk rather than a settled frame. Measures should skip partials: the same text arrives again on the final frame, and counting both double-counts everything.

func (Frame) Role

func (f Frame) Role() string

Role is the content role ("model", "user", ...), or "" if absent.

func (Frame) Text

func (f Frame) Text() string

Text returns the frame's concatenated text parts. Empty for a frame that only carried tool traffic.

type Measure

type Measure interface {
	// Name is the measure's stable identifier. It is what a run with no
	// observations reports as having looked — see [WriteObservations],
	// where naming the measures is the difference between "nothing
	// happened" and "these measures saw nothing".
	Name() string
	// Observe returns what the measure found, in trajectory order. Nil
	// is the normal result on a clean run.
	Observe(t *Trajectory) []Observation
}

Measure derives observations from one trajectory.

A measure reads [Step]s and [Frame]s and nothing else — never the run directory, never score.py's output, never the network. That is what makes a measure re-runnable over the archive years later.

type Meta

type Meta struct {
	RunID        string `json:"run_id"`
	StartedAt    string `json:"started_at"`
	ScenarioID   string `json:"scenario_id"`
	ScenarioName string `json:"scenario_name"`
	Cluster      string `json:"cluster"`
	ModelFlavor  string `json:"model_flavor"`
	DaemonImage  string `json:"daemon_image"`
	SessionID    string `json:"session_id"`
	Followup     string `json:"followup"`
}

Meta is the subset of a drill run's meta.json this package reads. Unknown fields are ignored on purpose — meta.json is the drill's file and grows on its schedule, not ours.

type Observation

type Observation struct {
	// Kind is a stable slug. Grep for it; do not parse Summary.
	Kind string `json:"kind"`
	// Agent is whose behaviour this is about, or "" when it is about the
	// run as a whole.
	Agent string `json:"agent,omitempty"`
	// Step is the [Step.Index] this anchors to, or -1 when it anchors to
	// nothing in particular.
	Step int `json:"step"`
	// Summary is one line, written to be read next to a step table.
	Summary string `json:"summary"`
	// Evidence is the specifics: quoted payload fields, step numbers,
	// counts. An observation whose evidence you cannot check by hand
	// against the table is a claim, not an observation.
	Evidence []string `json:"evidence,omitempty"`
}

Observation is one thing a measure noticed, with the evidence for it.

There is deliberately no severity, no score and no pass/fail. A measure says what happened and where; deciding whether it was bad is the reader's job, and encoding that decision here would make this a grader against a rubric that is still moving (see the package doc).

Observations are not all negative. A measure that only ever speaks up when something is wrong is indistinguishable, on a clean corpus, from a measure that is broken — so where the distinction a measure draws is the point, it reports both sides. KindDelegationDisclosed is there for exactly that reason.

func Observe

func Observe(t *Trajectory, measures ...Measure) []Observation

Observe runs measures over a trajectory and returns everything they found, grouped by measure in the order given.

type SkippedRun

type SkippedRun struct {
	Dir string
	Err error
}

SkippedRun is a run directory that looked like a run but could not be read. It is returned rather than logged so the caller decides what a broken run means: the CLI prints it, a test asserting on the archive can fail on it, and neither can miss it by forgetting to look.

func (SkippedRun) String

func (s SkippedRun) String() string

type Status

type Status string

Status is what a tool call's result carried.

The three values, and the asymmetry between them, are score.py's (dev/uat/gke-drill/score.py, response_status) and must stay that way. Two instruments reading the same transcript and disagreeing about which calls failed is the #694 failure — two surfaces disagreeing about one rule — and it would be worse here, because the disagreement would show up as a trajectory finding contradicting a scored sheet.

const (
	// StatusOK is a result with no structural error signal.
	StatusOK Status = "ok"
	// StatusError is a structural failure: isError, an error key, or a
	// status field reading error/failure/failed.
	StatusError Status = "error"
	// StatusSuspect is prose that opens like an error. Only ever a
	// suspicion: a read that SUCCEEDED and returned text about a denial
	// is the normal case in scenario C, not a failure. A false suspect
	// costs one glance at the payload; a false ok would let a turn that
	// read nothing pass as grounded.
	StatusSuspect Status = "suspect"
	// StatusNoResponse is a call that never got a result frame at all.
	StatusNoResponse Status = "no-response"
)

type Step

type Step struct {
	// Index is position within the trajectory, 0-based, across all
	// agents.
	Index int `json:"index"`
	// Agent is [Parent] or a subagent's registered name.
	Agent string `json:"agent"`
	// CallSeq and RespSeq are the frame sequence numbers. RespSeq is -1
	// when nothing answered.
	CallSeq int `json:"call_seq"`
	RespSeq int `json:"resp_seq"`
	// Invocation is the calling frame's InvocationID — the turn this
	// step belongs to. Steps sharing one Invocation happened inside one
	// model turn.
	Invocation string `json:"invocation"`
	// CallID is the provider's function-call id, the key the response
	// is matched on.
	CallID string `json:"call_id"`

	// At is the calling frame's timestamp and Done the responding
	// frame's; Done is zero when nothing answered. Kept because "where
	// did the wall clock go" is a trajectory question and the six boxes
	// cannot ask it.
	At   time.Time `json:"at"`
	Done time.Time `json:"done"`

	Tool     string         `json:"tool"`
	Args     map[string]any `json:"args"`
	Status   Status         `json:"status"`
	Response map[string]any `json:"response"`
}

Step is one tool call with its result joined on, in trajectory order.

This is the unit every measure works in. If a measure needs a field that is not here, add it here rather than reaching back into Frames — a measure that parses raw events is a measure that will disagree with the next one.

func (Step) Arg

func (s Step) Arg(name string) string

Arg returns a string-valued argument, or "".

func (Step) Elapsed

func (s Step) Elapsed() time.Duration

Elapsed is how long the call took, or 0 if it never returned.

Read it as a bound, not a measurement. A model turn routinely issues several calls in ONE frame and the runtime answers them in one frame too, so every call in such a batch reports the whole batch's span: steps 3-6 of run 20260910T170709Z-a all say 496ms because they share frames 156 and 161. What the number is good for is spotting the step that ate the wall clock, which in practice is a spawn_agent.

func (Step) Failed

func (s Step) Failed() bool

Failed reports whether the step's result was a structural failure. Suspect does not count: see StatusSuspect.

type Trajectory

type Trajectory struct {
	Meta   Meta    `json:"meta"`
	Frames []Frame `json:"frames"`
	Steps  []Step  `json:"steps"`
	Turns  []Turn  `json:"turns"`
	Usage  Usage   `json:"usage"`
}

Trajectory is one run: its identity, its ordered steps, and the raw frames the steps were derived from.

Frames is kept because a measure about what the agent SAID needs the text, and reconstructing it from Steps is impossible — Steps are tool traffic only.

func LoadRun

func LoadRun(dir string) (*Trajectory, error)

LoadRun reads one drill run directory — the layout dev/uat/gke-drill writes under ~/.gke-drill/runs/<id>/.

It merges transcript.jsonl (the parent) with subagents.json (one entry per child) and sorts by (Seq, Agent), which is the same merge score.py does. Missing subagents.json is not an error: a run with no delegation is a legitimate run.

type Turn

type Turn struct {
	PromptID  string `json:"prompt_id"`
	Model     string `json:"model"`
	TokensIn  int    `json:"tokens_in"`
	TokensOut int    `json:"tokens_out"`
	LatencyMS int    `json:"latency_ms"`
}

Turn is one completed model turn, from the drill's typed `turn-complete` frames. Carried separately from Frames because it is attach-protocol telemetry rather than an agent event.

type Usage

type Usage struct {
	TokensIn  int     `json:"tokens_in_total"`
	TokensOut int     `json:"tokens_out_total"`
	CostUSD   float64 `json:"cost_usd_total"`
	Turns     int     `json:"turns_total"`
}

Usage is the run's final cumulative spend, from the last `usage-update` frame.

Directories

Path Synopsis
cmd
trajectory command
Command trajectory prints what an agent did, in order, out of a recorded GKE-drill run.
Command trajectory prints what an agent did, in order, out of a recorded GKE-drill run.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL