Documentation
¶
Overview ¶
Package toonenc implements the adaptive TOON encoder for tool-call result text blocks (spec 084). It decides, per block, whether to replace a JSON rendering with a smaller TOON encoding (marker + decode hint + body) or to pass the block through byte-identically.
Layering rule: this package depends ONLY on the standard library and github.com/toon-format/toon-go. It is imported by internal/server (production seam) and designed to be imported by the spec-083 bench adaptive-results arm (FR-012; lands after PR #851 merges and this branch rebases), so it must never import internal/server, internal/config, or any other mcpproxy package — that is what keeps "the profiler exercises the exact production code path" literally true without dragging the server into the bench build.
All exported functions are pure and deterministic (FR-011): identical input produces an identical decision and identical bytes. Observability (logging, metrics) for encoder failures is the caller's responsibility (FR-006).
Index ¶
Constants ¶
const Marker = "" /* 128-byte string literal not displayed */
Marker is the deterministic one-line header that precedes every TOON-encoded block (FR-005). It is part of the response contract with agents and is asserted byte-for-byte against specs/084-toon-output/contracts/marker-format.md.
Strict ASCII by contract: the separator is " - " (space-hyphen-space), NOT an em dash, so the marker's byte cost is stable across tokenizers and fully counted in the size comparison (FR-003c, FR-004).
const MinToonRowBytes = 16
MinToonRowBytes is a small fixed estimate of one TOON data row, used by the too-small-budget guard (FR-008/FR-009): when the truncator would retain fewer bytes than marker + newline + one data row, encoding is pointless — the block passes through and truncation behaves exactly as today. Fixed constant → determinism preserved (FR-011).
Variables ¶
This section is empty.
Functions ¶
func AssembleEmission ¶
AssembleEmission builds the complete encoded emission for a TOON body: exactly Marker + "\n" + body. Passthrough blocks never go through this function — they carry no marker (FR-005).
Types ¶
type Classification ¶
type Classification struct {
Tabular bool
Envelope bool // true if unwrapped from a single-key object envelope
Rows int // element count of the classified array
Cols int // size of the >=90%-present union key set
Reason NotTabularReason // set only when Tabular == false
}
Classification is the deterministic result of the tabular-uniform predicate (FR-003b, data-model.md §2).
func Classify ¶
func Classify(v interface{}) Classification
Classify is the deterministic tabular-uniform predicate (FR-003b, v1 rules). It expects a value parsed from JSON with json.Number decoding (as EncodeBlock produces) and never mutates it.
v1 rules:
- a JSON array of at least 4 objects;
- every row value scalar or null (no nested objects/arrays);
- every key appearing in any row must be present in at least 90% of rows (the union key set = all keys, each >=90%-present; a key below the tolerance makes the array too ragged);
- an object with exactly one key whose value is such an array ("envelope") also qualifies;
- empty arrays and arrays of non-objects do not qualify;
- key order is irrelevant.
The classifier is a pure predicate: it never encodes. The FR-004 size comparison in EncodeBlock is an independent backstop for "uniform enough" borderline cases.
type Decision ¶
type Decision struct {
// BlockIndex is the text-block index within a multi-block response.
// EncodeBlock leaves it zero; the caller (server seam / bench arm) sets it.
BlockIndex int
Mode Mode
// Classification is always computed for JSON-parseable input, even in
// always mode where it does not gate encoding (informational).
Classification Classification
// PassthroughEmissionBytes is len(original block text) — the exact
// passthrough emission the agent would receive with the feature off
// (FR-003c).
PassthroughEmissionBytes int
// EncodedEmissionBytes is len(Marker + "\n" + toonBody) — the complete
// encoded emission. Zero on every passthrough outcome.
EncodedEmissionBytes int
// ThresholdPct is the toon_min_savings_pct in effect for this decision.
ThresholdPct int
Outcome Outcome
}
Decision is the per-text-block outcome record — feeds the tool_call activity metadata (FR-010) and the spec-083 profiler (FR-012).
func EncodeBlock ¶
EncodeBlock is the single deterministic entry point called by internal/server (production seam) and designed to be imported by the spec-083 bench adaptive-results arm (FR-012; lands after PR #851 merges and this branch rebases). It decides, for one tool-result text block, whether to emit TOON or pass the block through unchanged, and returns a full Decision record.
Inputs (contracts/encoder-decision.md):
- text: the exact text-block content the agent would receive with the feature off (the passthrough emission, FR-003c), already sanitised;
- mode: ModeAdaptive or ModeAlways — the caller never invokes EncodeBlock when the resolved mode is ModeOff;
- minSavingsPct: toon_min_savings_pct in effect (validated 1-90); ignored in ModeAlways;
- retainedBudget: the truncator's actual retained-prefix budget in bytes (Truncator.SimpleTruncateBudget()); 0 means unlimited.
EncodeBlock is a pure function: no logging, no metrics, no I/O. FR-006 observability for OutcomePassthroughError is the caller's responsibility.
Guarantees:
- G1 byte-identity: every non-encoded outcome returns out == text.
- G2 never-larger (adaptive): encoded ⇒ len(out) <= len(text).
- G3 determinism: identical input ⇒ identical out and Decision.
- G4 no data loss: any failure ⇒ passthrough, never an error return.
type Mode ¶
type Mode string
Mode is the resolved TOON output mode for a tool call (global config value or per-server override, resolved by config.ResolveToonOutput and parsed here at the server/bench boundary).
const ( // ModeOff disables the feature: the encoder is never invoked and // responses are byte-identical to pre-feature behavior (FR-002). Default. ModeOff Mode = "off" // ModeAdaptive encodes a block iff it is tabular-uniform (FR-003b) AND // the complete encoded emission beats the exact passthrough emission by // at least the configured threshold (FR-003c). ModeAdaptive Mode = "adaptive" // ModeAlways encodes every JSON-parseable text block regardless of // tabular classification or size comparison (benchmark/debug only, // FR-009). Non-JSON text still passes through unmarked. ModeAlways Mode = "always" )
func ParseMode ¶
ParseMode parses a config-level toon_output string into a Mode. The empty string maps to ModeOff (unset = default). Unknown values return ("", false); callers must treat that as off (config validation rejects such values before they reach a live call, so this is a defensive backstop).
type NotTabularReason ¶
type NotTabularReason string
NotTabularReason explains why a block did not classify as tabular-uniform (data-model.md §2). Set only when Classification.Tabular is false.
const ( // ReasonNotJSON — the block text did not parse as a single JSON value // (plain text, base64, binary, malformed JSON, trailing garbage). ReasonNotJSON NotTabularReason = "not-json" // ReasonNotArray — the parsed value is not an array (and not a // single-key envelope object wrapping an array). ReasonNotArray NotTabularReason = "not-array" // ReasonTooFewRows — the array has fewer than 4 elements (empty // included). ReasonTooFewRows NotTabularReason = "too-few-rows" // ReasonNonObjectElements — at least one array element is not a JSON // object. ReasonNonObjectElements NotTabularReason = "non-object-elements" // ReasonNestedValues — at least one row field holds a nested object or // array (v1 is flat-scalar-only, FR-003b). ReasonNestedValues NotTabularReason = "nested-values" // ReasonTooRagged — the rows disagree beyond the 90% key-presence // tolerance, or the union key set collapses to empty. ReasonTooRagged NotTabularReason = "too-ragged" // ReasonNonRoundtrippableNumber — the value contains a JSON number whose // literal does not survive the float64 round-trip toon-go applies to // json.Number (e.g. integers beyond 2^53, large uint64). Encoding would // silently corrupt the number, so the block passes through (FR-004/FR-006 // no data loss). Not an encoder fault — it belongs to the // passthrough-not-tabular family, never logged or counted. ReasonNonRoundtrippableNumber NotTabularReason = "non-roundtrippable-number" )
type Outcome ¶
type Outcome string
Outcome is the per-block encoding decision outcome (data-model.md §3).
const ( // OutcomeEncoded — the block was replaced with Marker + "\n" + TOON body. OutcomeEncoded Outcome = "encoded" // OutcomePassthroughNotTabular — non-JSON input, or (adaptive mode) a // JSON value that did not classify tabular-uniform. Ordinary traffic, // never logged. OutcomePassthroughNotTabular Outcome = "passthrough-not-tabular" // OutcomePassthroughBelowThreshold — the encoding did not beat the // passthrough emission by the configured margin, or the truncation // budget was too small to hold marker + one data row. OutcomePassthroughBelowThreshold Outcome = "passthrough-below-threshold" // OutcomePassthroughError — a genuine encoder failure on input that // already parsed as JSON. The only outcome the caller logs and counts // (FR-006). OutcomePassthroughError Outcome = "passthrough-error" )