Documentation
¶
Overview ¶
Package tokenopt reduces the token footprint of session raw.jsonl streams before they are handed to a summarization LLM. Deterministic, streaming, single-pass, no LLM calls.
The one public entry point is Compress. It reads a raw.jsonl stream from r, applies a fixed set of transforms, and writes the compressed stream to w. Memory stays bounded (a small dedup set of content hashes) regardless of session size.
This package is intentionally self-contained — it depends only on the standard library and a small LRU — so it can be used by the CLI, the sessionsummary package, and a future server-side distiller with no coupling. No persistence, no sidecar manifest: the raw.jsonl on disk is the audit trail.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Mode ¶
type Mode int
Mode controls how aggressively Compress reduces the stream.
const ( // ModeConversationOnly (default) emits header + user + assistant entries // verbatim and replaces every tool/system entry with a compact marker // (name + brief gist of the input). Optimal for downstream summarization: // the summarizer needs the conversation arc, not tool I/O. ModeConversationOnly Mode = 0 // ModeLossless keeps every entry but applies content-level transforms // (ANSI strip, progress collapse, image elision, large-Read truncation, // system-reminder + tool_result dedup). Use when a downstream consumer // still needs tool details (replay, debugging, contract-testing). ModeLossless Mode = 1 )
type Options ¶
type Options struct {
// Mode selects the compression strategy. Defaults to ModeConversationOnly.
Mode Mode
// LargeReadMaxLines is the line threshold above which a Read tool_result
// body is truncated to head+tail. ModeLossless only. Defaults to 120.
LargeReadMaxLines int
// LargeReadKeepLines is how many lines to keep from the start and end when
// a large Read body is truncated. ModeLossless only. Defaults to 40.
LargeReadKeepLines int
// ToolResultLRUSize caps the content-hash LRU for tool_result dedup.
// ModeLossless only. Defaults to 1024 unique payloads.
ToolResultLRUSize int
// ToolResultMinBytes is the minimum content size worth deduping.
// ModeLossless only. Defaults to 512.
ToolResultMinBytes int
}
Options configures a Compress run. Zero value is a reasonable default (ModeConversationOnly).
type Stats ¶
type Stats struct {
EntriesIn int
EntriesOut int // differs from EntriesIn in ModeConversationOnly where tool entries collapse to markers
BytesIn int64
BytesOut int64
ToolsMarked int // ModeConversationOnly: count of tool entries replaced with compact markers
SystemDropped int // ModeConversationOnly: count of system entries dropped
ANSIStripped int // ModeLossless: entries with ANSI sequences removed
ProgressCollapsed int // ModeLossless: entries with \r progress frames collapsed
ImagesElided int // ModeLossless: base64 image payloads replaced
LargeReadsElided int // ModeLossless: large Read tool_result bodies truncated
RemindersDeduped int // ModeLossless: <system-reminder> blocks replaced with a ref
ToolResultsRefd int // ModeLossless: tool_result bodies replaced with a tool_ref
}
Stats reports what Compress did. Zero values are meaningful (no matches).
func Compress ¶
Compress reads raw.jsonl entries from r, applies streaming transforms, and writes compressed jsonl to w. Returns Stats describing what was done.
Guarantees (both modes):
- Single pass over r. Surviving entries emit in original order.
- Constant memory relative to session size.
- User turns and header entries are preserved verbatim, always.
Mode-specific behavior:
- ModeConversationOnly (default): assistant turns verbatim; tool entries collapse to compact tool_mark entries with a brief gist; system entries are dropped. Typical reduction 50–80% on real sessions.
- ModeLossless: every entry preserved; content-level transforms only (ANSI strip, image elision, dedup, etc.).
func CompressWith ¶
CompressWith is Compress with tunable options.
func (*Stats) Add ¶
Add accumulates other into s. Useful for aggregating across many sessions (e.g., a daemon summarizing a nightly batch).
Source Files