Documentation
¶
Overview ¶
Package compaction keeps an agent turn inside the model's context window by replacing older conversation with a structured summary.
The cut point is chosen by token budget alone — never by "keep the current turn intact". A ReAct loop is a single turn that can run twenty rounds and produce a hundred thousand tokens of tool traffic; a compactor that refuses to touch the current turn has nothing left to compact and burns an LLM call per round achieving nothing.
Index ¶
- Constants
- Variables
- func Apply(messages []chat.Message, p *Preparation, summary string) []chat.Message
- func IsOverflowError(err error) bool
- func ResponseHitContextLimit(resp *types.ChatResponse, contextWindow, completionBudget int) bool
- func SummaryMessage(summary string) chat.Message
- type Checkpoint
- type Compactor
- type CutPoint
- type Preparation
- type Reason
- type Result
- type Settings
Constants ¶
const DefaultKeepRecentTokens = 20000
DefaultKeepRecentTokens is how much recent conversation survives a compaction untouched. It is what makes compaction terminate: whatever the context grew to, it comes out of a compaction at roughly summary + this, which is far below any sane threshold, so the next round cannot immediately re-trigger.
const DefaultReserveTokens = 16384
DefaultReserveTokens is the floor on room kept free for the next response.
const DefaultSummarizationStallTimeout = 120 * time.Second
DefaultSummarizationStallTimeout matches the engine's default stall budget for its own rounds.
Variables ¶
var ErrNothingToCompact = errors.New("compaction: nothing outside the keep-recent budget")
ErrNothingToCompact means the context cannot be made smaller: everything outside the keep-recent budget is already gone. Callers must treat this as a stop signal rather than retrying, or they reproduce the every-round summarization loop this package exists to prevent.
Functions ¶
func Apply ¶
Apply rebuilds the message list as system prompt, summary, and the verbatim tail. Nothing between the system prompt and the cut point survives.
func IsOverflowError ¶
IsOverflowError reports whether a failed LLM call failed because the request did not fit in the context window.
func ResponseHitContextLimit ¶
func ResponseHitContextLimit( resp *types.ChatResponse, contextWindow, completionBudget int, ) bool
ResponseHitContextLimit reports whether a *successful* response was shaped by a full context window rather than by the completion budget we asked for. Three shapes:
- The provider accepted an oversized request silently and reported a prompt larger than the window (z.ai).
- The provider truncated the input to fill the window exactly, leaving no room to generate, and stopped on length with no output (Xiaomi MiMo).
- The provider stopped on length having emitted less than we allowed, so our own cap cannot be what stopped it (the general recoverable case).
completionBudget is the max_tokens we requested, before any context-based clamping; passing the clamped value would hide exactly the case this detects.
func SummaryMessage ¶
SummaryMessage wraps summary text in the envelope the model sees, tagged so the next compaction recognizes it as its own output.
Types ¶
type Checkpoint ¶
type Checkpoint struct {
// TurnID is the stored turn the summary covers through, inclusive.
TurnID string
Summary string
// Degraded reports that Summary holds a raw archive because the
// summarizer failed. It is still a checkpoint: the next compaction
// folds it in as the previous summary.
Degraded bool
}
Checkpoint is a summary that ends exactly on a stored turn. Persisted onto that turn, it lets the next turn load the summary in place of everything up to and including the turn, instead of summarizing the same history again.
type Compactor ¶
type Compactor struct {
// contains filtered or unexported fields
}
Compactor summarizes conversation history into a checkpoint.
func New ¶
New builds a compactor. It returns nil when compaction cannot run, so the caller can treat a nil compactor as "feature off" without a second flag.
type CutPoint ¶
type CutPoint struct {
// FirstKeptIdx is the index of the first message kept verbatim.
FirstKeptIdx int
// TurnStartIdx is the user message that opened the turn the cut lands in,
// or -1 when the cut falls on a turn boundary.
TurnStartIdx int
// IsSplitTurn reports that the cut divides a single turn, so the part
// before it needs its own summary to explain the part after it.
IsSplitTurn bool
}
CutPoint is where the conversation is divided into "summarize this" and "keep this verbatim".
func FindCutPoint ¶
func FindCutPoint( messages []chat.Message, start, keepRecentTokens int, estimator *agenttoken.Estimator, ) CutPoint
FindCutPoint walks backwards from the newest message accumulating estimated size, and keeps the largest suffix that still fits keepRecentTokens.
The budget is a ceiling, not a floor. Stopping at the first message that reaches the budget and keeping that message too is the obvious reading and the wrong one: a single knowledge_search result is routinely ten thousand tokens on its own, so "keep the recent 16k" would hand back 25k and leave barely a round of headroom before the next compaction.
Nothing here consults the current turn. That is deliberate: the budget is the only rule, and a turn large enough to exceed it on its own gets split rather than exempted.
type Preparation ¶
type Preparation struct {
// FirstKeptIdx is where the verbatim tail begins.
FirstKeptIdx int
// MessagesToSummarize are the complete turns being replaced by prose.
MessagesToSummarize []chat.Message
// TurnPrefixMessages is the discarded head of a split turn, summarized
// separately so the retained tail still has its originating request.
TurnPrefixMessages []chat.Message
IsSplitTurn bool
// PreviousSummary is the prior compaction's text, updated in place rather
// than summarized again.
PreviousSummary string
TokensBefore int
// OmittedHistory and OmittedTurnPrefix count the oldest messages of each
// range left out because one summarization request could not carry them
// (see fitSummarizerInput). Their file paths are still recorded.
OmittedHistory int
OmittedTurnPrefix int
// contains filtered or unexported fields
}
Preparation is everything the summarization step needs, computed before any LLM call so the caller can decide not to make one.
func Prepare ¶
func Prepare(messages []chat.Message, s Settings, estimator *agenttoken.Estimator) *Preparation
Prepare picks the cut point and collects the message ranges around it.
It returns nil when compaction cannot help — no messages fall outside the keep-recent budget, or the last compaction already left nothing to remove. That nil is the guard against the failure mode where every round spends a summarization call on a context it cannot shrink.
type Reason ¶
type Reason string
Reason records what asked for a compaction, for logging and for the UI.
type Result ¶
type Result struct {
Messages []chat.Message
Summary string
Reason Reason
TokensBefore int
TokensAfter int
MessagesBefore int
MessagesAfter int
// Degraded reports that at least one summary came from the raw archive
// because the summarizer failed. The context still shrank; its memory is
// just coarser.
Degraded bool
// SplitTurn reports that the cut divided a single turn, so a turn-prefix
// summary was generated alongside the history summary.
SplitTurn bool
// Checkpoint is the part of this compaction a later turn can start from,
// or nil when there is none (see Preparation.checkpoint).
Checkpoint *Checkpoint
// Omitted counts the oldest messages left out of the summarization
// requests because they did not fit (see fitSummarizerInput).
Omitted int
}
Result describes what one compaction did.
type Settings ¶
type Settings struct {
// Enabled turns automatic compaction on. Disabled still allows an
// explicit overflow-recovery compaction, which is a repair, not a policy.
Enabled bool
// MaxContextTokens is the model's context window. Zero disables
// compaction entirely: without a window there is no budget to enforce.
MaxContextTokens int
// ReserveTokens is the part of the window history may not occupy because
// the next response has to fit there.
ReserveTokens int
// KeepRecentTokens is the token budget of recent messages kept verbatim.
KeepRecentTokens int
// MaxSummaryTokens caps the summarization completion. Zero falls back to
// the reserve-derived budget alone.
MaxSummaryTokens int
// StallTimeout is how long a summarization stream may produce no output
// before it is cancelled. Zero means DefaultSummarizationStallTimeout.
StallTimeout time.Duration
}
Settings configures when compaction runs and how much it keeps.
func (Settings) Normalize ¶
Normalize fills in defaults and reconciles budgets that cannot all be honored at once.
func (Settings) ShouldCompact ¶
ShouldCompact reports whether the current context has crossed the threshold.