replaycorpus

package
v0.71.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 17, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

codeexec.go — the response-side saving of code execution.

Every other savings figure in this harness is about the MENU: how many tokens of tool definitions an agent carries. Code execution's menu is already measured and it is the smallest of the three surfaces, but the menu is not where code execution actually pays.

It pays on the RESPONSE side. A script issues several upstream calls inside the sandbox and only the value it returns enters the model's context; the intermediate responses — which an agent driving those tools itself would have paid for in full, request and response both — never appear. That difference is what this file computes, and nothing else in bench measures it.

Package replaycorpus loads exported mcpproxy activity JSONL (`mcpproxy activity export --format json`) into grouped units of work for the token benchmark's deterministic replay (Spec 103, US1). It groups records by work session, joins code-execution sub-calls to the call that issued them, and classifies every record's usability ONCE, at load.

Two invariants define this package and must survive every future change:

  1. IT IMPORTS NOTHING FROM `bench`. The sibling loader `bench/corpusio` imports `bench`, and `bench/replay.go` — the consumer of this package — lives in `bench`. If this package mirrored corpusio, that import would be a cycle and replay could not use it at all. So the replay domain types (ReplayCall, ReplaySession, Cost, Flags) are DEFINED HERE rather than borrowed, and the tokenizer is the tiktoken library used directly rather than bench.Tokenizer. Adding a `bench` import here does not produce a compile error in this package — it breaks `bench/replay.go`, one package away, which is why the rule is stated rather than merely enforced by the build.

  2. NO CONTENT CROSSES THIS BOUNDARY — ONLY COUNTS. Replay inputs are raw recorded user traffic, and the activity EXPORT PATH DOES NOT MASK: maskActivityPayloads is wired into the list and detail handlers only, so a bodies-on export is unmasked by design. Therefore no exported type in this package has a field that holds a request body, a response body, arguments, or metadata; bodies exist only as local variables inside the load call that reads them. That is also why tokenization happens HERE (tokenize.go) rather than in the scoring layer: handing bodies out to be counted elsewhere would defeat the invariant, and the two cannot both hold. Bodies-off is the zero value, so an operator who never thinks about the flag gets the safe behaviour (privacy.go).

A third rule is a property of the data rather than of the package, and flags.go is where it lives: A TRUNCATED RECORD MUST NEVER CONTRIBUTE SILENTLY. Understating cost is the one direction of error the whole benchmark exists to prevent, so a truncated record either carries an explicitly annotated estimate derived from its pre-truncation byte length, or is excluded and counted. Every exclusion is counted and reportable (FR-002, FR-003, SC-008): silence is never success. Note that the first branch is a real one: codeexec.go's BaselineDirect COUNTS a truncated sub-call, at its full pre-truncation byte length, and stays inside this rule only because CodeExecSaving.TruncatedSubCalls and TruncatedBaseline publish how many such components contributed and how much of the baseline they are. Without that annotation direct mode would be outside the invariant, which is why those two fields are load-bearing rather than decorative.

The binding contract for the input shape is specs/103-token-bench/contracts/replay-input.md.

Index

Constants

View Source
const CodeExecToolName = "code_execution"

CodeExecToolName is the built-in whose activity records parent a sandbox's sub-calls. Sub-calls join to it by parent_id (see group.go).

View Source
const DefaultEncoding = "cl100k_base"

DefaultEncoding is the tiktoken encoding used to count recorded bodies. It matches the rest of the benchmark (bench.DefaultEncoding) so a replay figure and a corpus figure are commensurable — but it is DUPLICATED here rather than imported, because this package must not import `bench` (package doc, invariant 1). The duplication is one string; the cycle would be structural.

Variables

View Source
var ErrCSVInput = errors.New("csv activity export is not a valid replay input")

ErrCSVInput is returned for a CSV export, by name or by content.

View Source
var ErrInsideRepository = errors.New("replay input is inside the repository working tree")

ErrInsideRepository is returned for an input path inside the repository working tree.

Functions

This section is empty.

Types

type BaselineMode added in v0.65.0

type BaselineMode string

BaselineMode selects WHICH counterfactual the word "baseline" names. The two are not refinements of one another: they answer different questions and legitimately produce different numbers from the same log. Before this type existed the accounting implemented BaselineProxy and never said so, which is the defect it fixes — a saving whose denominator is unstated cannot be quoted.

It is a string type for the same reason CostBasis and ExclusionReason are: a report prints it verbatim and rows must sort deterministically (SC-002).

const (
	// BaselineDirect is the agent wired STRAIGHT to the upstream MCP servers,
	// with mcpproxy nowhere in the path. Nothing truncates, so a sub-call's
	// full pre-truncation response_bytes is exactly what that agent reads —
	// the recorded truncation is an artefact of the RECORD
	// (subCallActivityResponseLimit cuts the stored string at 8KB while
	// subCallByteSizes measures the whole result), not of the workload.
	//
	// NAME COLLISION, read this before using the word: internal/config's
	// RoutingModeDirect is ALSO "direct" and means the opposite kind of thing
	// — every upstream tool exposed THROUGH the proxy, where responses ARE cut
	// at ToolResponseLimit. Never print this value bare; always gloss it as
	// "no proxy in the path".
	//
	// It is the DEFAULT, and that is a deliberate change of published meaning.
	// tool_response_limit is an MCPPROXY feature: charging the counterfactual
	// agent for it answers "what did code execution save over call_tool_*"
	// rather than "what did mcpproxy save me". It also brings this figure into
	// line with the harness's own denominator — bench/modematrix.go's
	// baselineCell, "the same agent doing the same tasks with every upstream
	// tool inline", which every other published percentage here is measured
	// against. And the withhold it replaces is not statistically neutral: it
	// fires precisely on the LARGEST responses, the ones with most to save, so
	// the surviving population was biased small in a way no exclusion counter
	// showed as a magnitude.
	BaselineDirect BaselineMode = "direct"

	// BaselineProxy is the agent making the same calls through THIS proxy's
	// call_tool_* path, where each content block is cut at
	// config.ToolResponseLimit (20000 by default) before the model reads it.
	// Counting the full pre-truncation size then overstates the baseline, and
	// an overstated baseline inflates the saving — the one direction of error a
	// savings figure must never make — so a truncated sub-call withholds the
	// whole call. This is the behaviour that existed before the selector, bit
	// for bit.
	BaselineProxy BaselineMode = "proxy"
)

func ParseBaselineMode added in v0.65.0

func ParseBaselineMode(s string) (BaselineMode, error)

ParseBaselineMode turns operator input into the vocabulary, or refuses. Exact match only: no case folding, no trimming, no prefix. A typo'd -baseline that quietly produced the OTHER figure is the exact failure this change exists to remove, so "" is rejected too.

type BodyPolicy

type BodyPolicy int

BodyPolicy decides whether a load reads recorded request and response bodies.

The zero value is BodiesOff, and that is the whole design: an operator who constructs Options without thinking about this field gets the safe behaviour, and bodies-on is reachable only by naming it. Defaulting the other way would mean a forgotten flag silently loads raw, unmasked user traffic.

const (
	// BodiesOff loads sizes, statuses and identities only. This is the default
	// and the documented posture: it yields menu costs and the cross-mode delta
	// between the two direct cells, and NO absolute workload cost — which is
	// reported as unavailable, never as zero.
	BodiesOff BodyPolicy = iota

	// BodiesOnUnmasked loads recorded bodies so response cost can be measured
	// rather than estimated. The name says "unmasked" because that is the fact
	// an operator must weigh: the activity EXPORT PATH DOES NOT MASK —
	// maskActivityPayloads is wired into the list and detail handlers only — so
	// a bodies-on export is raw traffic, by design, as the compliance surface.
	// Selecting it prints a warning; the bodies still never leave this package.
	BodiesOnUnmasked
)

func (BodyPolicy) String

func (p BodyPolicy) String() string

String renders the policy for a report or a warning line.

type CodeExecBucket

type CodeExecBucket struct {
	Calls    int `json:"calls"`
	SubCalls int `json:"sub_calls"`
	Baseline int `json:"baseline"`
	Proxy    int `json:"proxy"`
	Saving   int `json:"saving"`

	// How much of Baseline above rests on components BaselineProxy would have
	// thrown away. Always 0 under BaselineProxy. A total that tolerates
	// truncation has to carry the magnitude beside it, not just a count — see
	// CodeExecSaving.TruncatedBaseline.
	TruncatedSubCalls int `json:"truncated_sub_calls,omitempty"`
	TruncatedBaseline int `json:"truncated_baseline,omitempty"`

	// TruncatedCalls and TruncatedCallSaving are the WHOLE-CALL figures, and
	// they are the ones to quote when asking "what does the mode change cost
	// me in confidence". TruncatedBaseline above is only the truncated
	// COMPONENT's contribution, but BaselineProxy does not drop a component —
	// it drops the entire call, its untruncated sub-calls and its proxy term
	// with it. Reporting only the component magnitude therefore understates
	// how much of the published total exists solely because direct mode
	// tolerated truncation.
	TruncatedCalls      int `json:"truncated_calls,omitempty"`
	TruncatedCallSaving int `json:"truncated_call_saving,omitempty"`
}

CodeExecBucket is the total for one accounting basis.

type CodeExecReport

type CodeExecReport struct {
	// Mode is the counterfactual every figure below is measured against. It is
	// on the report as well as on each saving so a report read alone still
	// states its own denominator.
	Mode BaselineMode `json:"baseline_mode"`

	// Buckets is keyed by basis: "measured" totals are tokens, "estimated"
	// totals are bytes.
	Buckets map[CostBasis]*CodeExecBucket `json:"buckets"`

	// Withheld counts the calls that produced no figure, by reason.
	Withheld map[ExclusionReason]int `json:"withheld,omitempty"`

	// Savings is the per-call detail, in input order.
	Savings []CodeExecSaving `json:"savings,omitempty"`
}

CodeExecReport aggregates response-side savings, bucketed BY BASIS so a measured total and an estimated one are never added together.

func CodeExecSavingsFor

func CodeExecSavingsFor(sessions []*ReplaySession, mode BaselineMode) *CodeExecReport

CodeExecSavingsFor walks sessions and reports the response-side saving of every code_execution call in them, against the counterfactual mode names (the zero value is BaselineDirect).

type CodeExecSaving

type CodeExecSaving struct {
	// ParentID is the code_execution call's own request id.
	ParentID string `json:"parent_id"`

	SubCalls int `json:"sub_calls"`

	// Basis names both the provenance AND the unit: measured figures are
	// TOKENS, estimated figures are BYTES. They are never mixed — see
	// ReasonMixedCostBasis.
	Basis CostBasis `json:"basis"`

	// Reason explains a CostUnavailable saving. Empty otherwise.
	Reason ExclusionReason `json:"reason,omitempty"`

	// Baseline, Proxy and Saving are populated only when Basis is not
	// CostUnavailable. An unavailable saving carries no numbers at all, because
	// a zero here would read as "code execution saved nothing" — a measurement
	// — when the truth is that no measurement was possible.
	Baseline int `json:"baseline,omitempty"`
	Proxy    int `json:"proxy,omitempty"`
	Saving   int `json:"saving,omitempty"`

	// Split by BILLING DIRECTION, because Saving above is not proportional to
	// cost. A tool call's ARGUMENTS are tokens the model generated (output,
	// billed around 5x input) and its RESPONSE is tokens it reads (input, billed
	// 1x, or 0.1x when the prefix is cached). Code execution trades the cheap
	// direction for the expensive one: it replaces N argument-writings with ONE
	// script, whose cost is fixed regardless of N.
	//
	// Measured consequence, on real traffic at three sub-calls: +262 tokens
	// "saved" and a NET LOSS in money. Anything that quotes the token figure as
	// a saving has to carry this split beside it.
	BaselineOutput int `json:"baseline_output,omitempty"`
	BaselineInput  int `json:"baseline_input,omitempty"`
	ProxyOutput    int `json:"proxy_output,omitempty"`
	ProxyInput     int `json:"proxy_input,omitempty"`

	// Mode is the counterfactual that produced this figure, stamped on EVERY
	// return path including a withheld one — a withheld line still has to say
	// which baseline refused it, since proxy refuses calls direct accepts. No
	// omitempty: a figure whose denominator is optional is the ambiguity being
	// removed.
	Mode BaselineMode `json:"baseline_mode"`

	// TruncatedSubCalls counts the baseline components that contributed ONLY
	// because BaselineDirect tolerates truncation; BaselineProxy withholds the
	// whole call instead, so it is always 0 there. doc.go's third invariant
	// lets a truncated record contribute but never SILENTLY: this counter and
	// TruncatedBaseline are that annotation, and direct mode is outside the
	// invariant without them.
	TruncatedSubCalls int `json:"truncated_sub_calls,omitempty"`

	// TruncatedBaseline is how much of Baseline those components contributed,
	// in the unit Basis implies. The count alone cannot answer the only
	// question worth asking of a tolerated figure — how much of this rests on
	// data the conservative mode would have thrown away.
	TruncatedBaseline int `json:"truncated_baseline,omitempty"`
}

CodeExecSaving is the response-side saving of ONE code_execution call.

Baseline is what an agent would have paid making the sub-calls itself; Proxy is what it actually paid — the script it sent plus the single result it got back. The difference can be NEGATIVE, and that is a real outcome rather than an error: a script that dispatches nothing, or whose sub-calls return less than the script's own source costs, genuinely loses tokens.

func CodeExecSavingFor

func CodeExecSavingFor(parent *ReplayCall, mode BaselineMode) CodeExecSaving

CodeExecSavingFor computes the response-side saving of one code_execution call. parent must be the code_execution record itself; its SubCalls are the sandbox dispatches joined by parent_id.

The accounting refuses three things rather than producing a plausible number:

  • a component with no honest figure (CostUnavailable) makes the whole saving unavailable, because a missing sub-call understates the baseline and so quietly flatters code execution;
  • components on DIFFERENT bases are never summed, since measured figures are tokens and estimated ones are bytes, and adding them produces a number in no unit at all;
  • a parent that is not a code_execution call, which is a caller error rather than a measurement.

mode selects the counterfactual the baseline names; the zero value is BaselineDirect. The parameter is deliberately required rather than carried on Options or inferred from CostBasis: a caller that recompiled clean and silently reported the OTHER number is the whole risk this selector exists to remove, and reporting one loaded corpus both ways — without a second, bodies-on, unmasked read — is the affordance that matters.

func (CodeExecSaving) PricedSavingUSD

func (s CodeExecSaving) PricedSavingUSD(inPerMTok, outPerMTok, cacheMult float64) (float64, bool)

PricedSavingUSD converts the saving to money, which is the only form in which the input/output asymmetry is visible.

inPerMTok and outPerMTok are USD per million tokens; cacheMult scales the INPUT side only (0.1 for a cached prefix, 1.0 for uncached). It returns ok=false rather than a number whenever pricing would be dishonest:

  • a withheld saving has no figures at all, and 0.0 would read as "code execution was free" instead of "nothing was measurable";

  • an ESTIMATED saving is BYTE lengths, not tokens, so pricing it per-token would be wrong by roughly the bytes-per-token ratio and would look entirely plausible.

  • a saving that TOLERATED a truncated sub-call is not priced, whichever mode produced it. Today that is belt-and-braces: responseCost refuses to tokenize cut text, so such a saving is CostEstimated and the basis guard above already rejects it. That is a property of the current loader, not of this function, and a future path that tokenized a partially-cut response would silently start pricing tolerated data. The rule money obeys is stated here and ENFORCED here rather than inherited from a coincidence two files away.

type Corpus

type Corpus struct {
	Sessions   []*ReplaySession `json:"sessions"`
	Exclusions ExclusionReport  `json:"exclusions"`

	// Bodies and Encoding record the posture this corpus was produced under, so
	// a report can state it rather than assume it. Encoding is empty with
	// bodies off, where nothing was tokenized.
	Bodies   BodyPolicy `json:"bodies"`
	Encoding string     `json:"encoding,omitempty"`

	// RecordsRead is every line decoded, including those later dropped. It is
	// the denominator an exclusion report is read against.
	RecordsRead int `json:"records_read"`

	// FleetChecked reports whether a fleet resolver was supplied. False means
	// the `unreplayable` flag was never evaluated — not that nothing was
	// unreplayable.
	FleetChecked bool `json:"fleet_checked"`

	// Warnings mirrors everything sent to Options.Warnf.
	Warnings []string `json:"warnings,omitempty"`
}

Corpus is the loader's whole output: grouped sessions, the accounting for everything that did not contribute, and the posture the load ran under. It carries counts and identities — never content.

func Load

func Load(r io.Reader, opts Options) (*Corpus, error)

Load decodes activity JSONL from r.

It is the boundary the package doc's second invariant is stated about: bodies read here exist only as locals inside this call and the helpers it hands them to, and what it returns holds counts alone.

func LoadFile

func LoadFile(path string, opts Options) (*Corpus, error)

LoadFile loads a replay input from disk.

The path is refused BEFORE it is opened if it lies inside the repository working tree, and refused if it names a CSV export. Both checks precede I/O on purpose: the first is a privacy rule that must not depend on the file existing, and the second saves an operator from a confusing parse error when the real problem is the export format.

func (*Corpus) TotalCalls

func (c *Corpus) TotalCalls() int

TotalCalls counts every call in every session, sub-calls included, each once.

type Cost

type Cost struct {
	Basis CostBasis `json:"basis"`

	// Tokens is populated only when Basis is CostMeasured. An estimate carries
	// no token count on purpose: converting bytes to tokens here would hide the
	// conversion's assumptions inside a number that looks measured.
	Tokens int `json:"tokens,omitempty"`

	// Bytes is the pre-truncation byte LENGTH, populated for measured and
	// estimated figures alike so a report can show the basis it worked from.
	Bytes int `json:"bytes,omitempty"`

	// Truncated marks an estimate taken from a record whose stored content was
	// cut. This is the annotation that keeps FR-002 honest: a truncated record
	// may contribute, but never silently.
	Truncated bool `json:"truncated,omitempty"`

	// Reason explains a CostUnavailable figure. Empty otherwise.
	Reason ExclusionReason `json:"reason,omitempty"`
}

Cost is one accounted cost component of a call: its request, or its response.

type CostBasis

type CostBasis string

CostBasis is the provenance of one cost figure. The three values must never be presented interchangeably: FR-013 requires measured and estimated figures to coexist inside one block, which only works if each carries its own basis.

const (
	// CostMeasured means the text was tokenized inside this package. Only
	// reachable with bodies on.
	CostMeasured CostBasis = "measured"

	// CostEstimated means the figure is derived from a pre-truncation BYTE
	// LENGTH. Byte lengths are not token counts; this is an estimate and must
	// be labelled as one wherever it appears.
	CostEstimated CostBasis = "estimated"

	// CostUnavailable means no honest figure exists. It is reported as
	// unavailable with a Reason — never as zero, which would read as a free
	// call and understate the workload.
	CostUnavailable CostBasis = "unavailable"
)

type ExclusionReason

type ExclusionReason string

ExclusionReason names one reason a record, or one cost component of a record, could not contribute to a replay figure. It is a CLOSED vocabulary and a string type so a report can print it verbatim and so rows sort deterministically — a replay report has to be byte-reproducible (SC-002), and a map's iteration order is not.

const (
	// ReasonNotACall marks a record that is not a tool call at all — a
	// quarantine change, a policy decision, a server change. The activity log
	// is wider than tool calls, so these arrive in every export and are
	// dropped rather than counted as zero-cost calls.
	ReasonNotACall ExclusionReason = "not_a_call"

	// ReasonMissingTool marks a call record with no tool name. It cannot be
	// resolved against any mode's tool surface, so it can carry no menu cost.
	ReasonMissingTool ExclusionReason = "missing_tool_name"

	// ReasonUnattributed marks a record with no work_session_id and no parent
	// to inherit one from. The work session is the unit of work for US1, so an
	// unattributed record belongs to no unit and is reported as such rather
	// than being folded into an arbitrary one.
	ReasonUnattributed ExclusionReason = "unattributed"

	// ReasonOrphanedSubCall marks a sub-call whose parent fell outside the
	// exported window AND which carries no work_session_id of its own, so
	// there is no unit of work left to attribute it to.
	//
	// It is separated from ReasonUnattributed on purpose. Both are records
	// with no session, but they tell the operator different things: an
	// unattributed record never had a session, whereas this one had a parent
	// that the EXPORT WINDOW cut off. The second is fixable — re-export with a
	// wider window and the record comes back — and folding it into the first
	// would hide the one exclusion reason the operator can actually act on.
	ReasonOrphanedSubCall ExclusionReason = "orphaned_sub_call"

	// ReasonTruncated marks a record whose stored response was cut at capture.
	// It is a FLAG, not a drop: the record still contributes its call shape and
	// an annotated byte-length estimate. It is counted so that no truncated
	// record can contribute without the report saying so.
	ReasonTruncated ExclusionReason = "truncated"

	// ReasonBodiesMissing marks a record loaded with bodies off — the default.
	// The content is absent, so no response cost is measurable, only estimated.
	ReasonBodiesMissing ExclusionReason = "bodies_missing"

	// ReasonSensitive marks a record carrying the sensitive-data flag. This is
	// a BEST-EFFORT reducer, never a guarantee: there is no persisted
	// sensitivity field, the flag is derived at export from detection metadata
	// added asynchronously AFTER the record is first persisted, so a freshly
	// exported record may be sensitive and not yet flagged. Anything relying on
	// it must say so (Principle IV, contracts/replay-input.md §4).
	ReasonSensitive ExclusionReason = "sensitive"

	// ReasonUnreplayable marks a call whose tool no longer resolves against the
	// supplied fleet. Only ever set when a fleet resolver was supplied — with
	// no fleet to resolve against, "unreplayable" would be an assertion the
	// loader has no basis for.
	ReasonUnreplayable ExclusionReason = "unreplayable"

	// ReasonNoByteCounts marks a response cost withheld because the record
	// carries no pre-truncation byte length. Zero bytes means UNKNOWN, not
	// free; treating it as free would silently understate the workload.
	ReasonNoByteCounts ExclusionReason = "no_byte_counts"

	// ReasonInternalNoByteCounts is ReasonNoByteCounts for the specific, known
	// gap: the internal tool-call emission never populates the byte fields, so
	// EVERY built-in call (retrieve_tools above all) is unaccountable with
	// bodies off. It has its own name because it is a systematic gap an
	// operator should see as one line, not as noise inside a general bucket.
	ReasonInternalNoByteCounts ExclusionReason = "internal_no_byte_counts"

	// ReasonSubCallZeroBytes is the second known gap: a code-execution sub-call
	// records BOTH byte counts as zero (internal/server/mcp_code_execution.go),
	// so a sandbox's sub-calls cannot use the byte-length estimate either.
	ReasonSubCallZeroBytes ExclusionReason = "sub_call_zero_bytes"

	// ReasonTruncatedRetrieveOverstates is the asymmetry that makes truncated
	// built-in records worse than useless. For an ordinary upstream call,
	// truncation cuts the STORED text while the agent consumed the whole thing,
	// so the pre-truncation byte length is an honest estimate. For
	// retrieve_tools it is the other way round: the log stores the FULL
	// response while the agent consumed the cut text, so both the stored text
	// and the byte length describe something LARGER than what was paid for.
	// Counting either would overstate mcpproxy's cost — flattering, and still
	// wrong — so the cost is withheld and counted. The alternative permitted by
	// the contract is to re-apply the recorded truncation limit before
	// counting; that needs a limit the export does not carry, so exclusion is
	// what is implementable here.
	ReasonTruncatedRetrieveOverstates ExclusionReason = "truncated_retrieve_tools_overstates"

	// ReasonMixedCostBasis marks a figure withheld because its components did
	// not share an accounting basis. A measured cost is a TOKEN count and an
	// estimated one is a BYTE length; adding them yields a number in no unit at
	// all, and it would look entirely plausible. This is the never-sum rule
	// applied at the point of addition rather than at publication.
	ReasonMixedCostBasis ExclusionReason = "mixed_cost_basis"

	// ReasonTruncatedSubCallOverstates marks a code-execution saving withheld
	// because one of the sandbox's sub-calls was truncated. response_bytes is
	// the FULL pre-truncation size, but the baseline it feeds means "what an
	// agent would have paid making this call itself" — and that agent receives
	// a response cut to ToolResponseLimit. Charging the baseline the full size
	// overstates it, and an overstated baseline INFLATES the saving, which is
	// the one direction of error a savings figure must never make.
	//
	// Withholding, rather than substituting the cut length, follows the policy
	// ReasonTruncatedRetrieveOverstates already sets. The alternative worth
	// considering later is to count the truncated component at its STORED
	// length and publish the result as a lower bound — more informative, and it
	// needs a saving type that can express "at least".
	//
	// SCOPE, since this is now conditional: everything above describes
	// BaselineProxy, the counterfactual where the agent's own call goes through
	// call_tool_* and IS cut at ToolResponseLimit. Under BaselineDirect the
	// agent talks to the upstream servers with no proxy in the path, nothing
	// truncates, and the full pre-truncation size is exactly what it reads — so
	// codeexec.go counts the sub-call and annotates it (TruncatedSubCalls /
	// TruncatedBaseline) instead of firing this reason. The PARENT half is
	// unconditional: the parent's response is the proxy term, mcpproxy really
	// did cut it, and this reason still fires for it in both modes.
	//
	// Corollary for anyone comparing two runs: a call withheld here under proxy
	// is re-tested under direct and may withhold under a DIFFERENT reason, so
	// per-reason tallies are not comparable across baselines.
	ReasonTruncatedSubCallOverstates ExclusionReason = "truncated_sub_call_overstates_saving"
)

type ExclusionReport

type ExclusionReport struct {
	Dropped  map[ExclusionReason]int `json:"dropped,omitempty"`
	Withheld map[ExclusionReason]int `json:"withheld,omitempty"`
	Flagged  map[ExclusionReason]int `json:"flagged,omitempty"`

	// OrphanedSubCalls counts sub-calls whose parent_id resolved to no
	// request_id in the export — typically because the parent code_execution
	// fell outside the exported window.
	//
	// Two outcomes, and this counter spans BOTH so the attribution loss is
	// visible either way:
	//   - the sub-call carries its own work_session_id: it is KEPT at top
	//     level, because dropping it would understate the workload;
	//   - it does not: there is no unit of work left to attribute it to, so it
	//     is dropped under ReasonOrphanedSubCall — which is a DIFFERENT and
	//     more actionable reason than ReasonUnattributed, since a wider export
	//     window would recover it.
	// Sandbox sub-calls commonly fall in the second case: they inherit their
	// session from the parent, so losing the parent loses the session too.
	OrphanedSubCalls int `json:"orphaned_sub_calls,omitempty"`
}

ExclusionReport accounts for everything that did NOT contribute, in three distinct senses that a single counter would blur:

  • Dropped — records that entered no session at all.
  • Withheld — RESPONSE costs suppressed on records that WERE admitted. Only the response side is counted: a request cost withheld on the same record has the same cause, is visible on the call's RequestCost, and counting it too would make "12 withheld" mean neither 12 records nor 12 responses.
  • Flagged — records admitted and counted, but carrying a usability flag a report must surface beside the figure they contributed to.

The distinction is load-bearing for FR-003/SC-008: "12 exclusions" means nothing if a dropped record, a suppressed response cost and a truncation annotation are all folded into one number.

func (*ExclusionReport) DroppedRows

func (r *ExclusionReport) DroppedRows() []ExclusionRow

DroppedRows returns the dropped-record counts sorted by reason.

func (*ExclusionReport) FlaggedRows

func (r *ExclusionReport) FlaggedRows() []ExclusionRow

FlaggedRows returns the admitted-but-flagged counts sorted by reason.

func (*ExclusionReport) TotalDropped

func (r *ExclusionReport) TotalDropped() int

TotalDropped is how many records entered no session.

func (*ExclusionReport) TotalFlagged

func (r *ExclusionReport) TotalFlagged() int

TotalFlagged is how many usability flags were raised across admitted records.

func (*ExclusionReport) TotalWithheld

func (r *ExclusionReport) TotalWithheld() int

TotalWithheld is how many cost components were suppressed.

func (*ExclusionReport) WithheldRows

func (r *ExclusionReport) WithheldRows() []ExclusionRow

WithheldRows returns the withheld cost-component counts sorted by reason.

type ExclusionRow

type ExclusionRow struct {
	Reason ExclusionReason `json:"reason"`
	Count  int             `json:"count"`
}

ExclusionRow is one line of an exclusion report: a reason and how many times it fired. Rows are what a report renders; the maps behind them are what the loader accumulates.

type Flags

type Flags struct {
	Truncated     bool `json:"truncated,omitempty"`
	BodiesMissing bool `json:"bodies_missing,omitempty"`
	Sensitive     bool `json:"sensitive,omitempty"`
	Unreplayable  bool `json:"unreplayable,omitempty"`
}

Flags is the usability classification of a call or a session, computed once at load and never re-derived downstream. A session's flags are the union of its calls': one truncated call makes the whole unit of work suspect, because the unit's total is what a report quotes.

func (Flags) Reasons

func (f Flags) Reasons() []ExclusionReason

Reasons lists the set flags as exclusion reasons, in a fixed order, so a report can print why a session is not usable without re-deriving anything.

func (Flags) Usable

func (f Flags) Usable() bool

Usable reports whether nothing is wrong with this call or session. It is deliberately strict — a caller that wants to use a partly-flagged session must look at WHICH flag is set and say so in its report, rather than consulting one boolean.

type Options

type Options struct {
	// Bodies selects whether recorded request and response bodies are read.
	// The zero value is BodiesOff — see BodyPolicy for why that direction.
	Bodies BodyPolicy

	// Encoding is the tiktoken encoding used to count bodies on a bodies-on
	// load. Empty means DefaultEncoding. Ignored with bodies off, where there
	// is nothing to count.
	Encoding string

	// Warnf receives operator-facing warnings. Nil means stderr. Every warning
	// is also appended to Corpus.Warnings, so a caller that renders a report
	// does not have to intercept the printing to know a warning fired.
	Warnf func(format string, args ...any)

	// FleetResolver answers whether a recorded (server, tool) still exists in
	// the fleet a replay will score against. Nil means NO FLEET WAS SUPPLIED,
	// and the loader then refuses to assert either replayability or its
	// absence — Corpus.FleetChecked says which happened. A replay run needs a
	// fleet input regardless (a recording-only invocation computes nothing);
	// this field is how that fleet reaches the usability classification.
	FleetResolver func(serverName, toolName string) bool
	// contains filtered or unexported fields
}

Options configures a load. Its ZERO VALUE is the safe, documented default: bodies off, no fleet resolution claimed, warnings to stderr, the repository root discovered from the working directory.

type ReplayCall

type ReplayCall struct {
	ID         string    `json:"id"`
	RequestID  string    `json:"request_id,omitempty"`
	ParentID   string    `json:"parent_id,omitempty"`
	ServerName string    `json:"server_name,omitempty"`
	ToolName   string    `json:"tool_name,omitempty"`
	Status     string    `json:"status,omitempty"`
	Timestamp  time.Time `json:"timestamp"`

	// Internal marks a built-in proxy tool call (retrieve_tools, call_tool_*,
	// code_execution) as opposed to an upstream one. It decides which
	// byte-coverage gap applies and whether the truncation asymmetry bites.
	Internal bool `json:"internal,omitempty"`

	// RequestBytes and ResponseBytes are byte LENGTHS measured pre-truncation,
	// NOT token counts. Zero means unknown, not free.
	RequestBytes  int `json:"request_bytes,omitempty"`
	ResponseBytes int `json:"response_bytes,omitempty"`

	Flags        Flags `json:"flags,omitempty"`
	RequestCost  Cost  `json:"request_cost"`
	ResponseCost Cost  `json:"response_cost"`

	// SubCalls are the calls a code_execution sandbox issued, joined by
	// parent_id. They live here and NOT in their session's top-level Calls, so
	// that a naive sum over Calls cannot double-count them.
	SubCalls []*ReplayCall `json:"sub_calls,omitempty"`
}

ReplayCall is one recorded tool call. It carries identity, sizes and derived measurements — and, by construction, NO content: see the package doc's second invariant.

type ReplaySession

type ReplaySession struct {
	WorkSessionID string `json:"work_session_id"`

	// Calls holds the TOP-LEVEL calls only. Code-execution sub-calls hang off
	// their parent's SubCalls, so summing over Calls cannot double-count them
	// and AllCalls is the single way to see every record exactly once.
	Calls []*ReplayCall `json:"calls"`

	// Usability is the union of the calls' flags, computed once at load. It is
	// a union because a report quotes the SESSION total: one truncated call
	// makes that total suspect, so the session has to say so.
	Usability Flags `json:"usability,omitempty"`

	CallCount    int `json:"call_count"`
	SubCallCount int `json:"sub_call_count,omitempty"`

	FirstSeen time.Time `json:"first_seen"`
	LastSeen  time.Time `json:"last_seen"`
}

ReplaySession is one unit of real recorded work: everything that shares a work_session_id (spec 082 — one client, one project, across reconnects). That, not the transport session, is the unit US1 scores, because a transport session is regenerated on every reconnect and would split one piece of work into several.

func (*ReplaySession) AllCalls

func (s *ReplaySession) AllCalls() []*ReplayCall

AllCalls returns every call in the session — parents followed by their sub-calls — each exactly once. It is the accessor anything that sums or counts should use: walking Calls alone misses sandbox sub-calls, and walking a flattened list built by hand is how they come to be counted twice.

func (*ReplaySession) Span

func (s *ReplaySession) Span() time.Duration

Span is the wall-clock extent of the session, first record to last.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL