compaction

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: Apache-2.0, MIT Imports: 12 Imported by: 0

Documentation

Overview

Package compaction keeps an agent turn inside the model's context window by replacing older conversation with a structured summary.

The cut point is chosen by token budget alone — never by "keep the current turn intact". A ReAct loop is a single turn that can run twenty rounds and produce a hundred thousand tokens of tool traffic; a compactor that refuses to touch the current turn has nothing left to compact and burns an LLM call per round achieving nothing.

Index

Constants

View Source
const DefaultKeepRecentTokens = 20000

DefaultKeepRecentTokens is how much recent conversation survives a compaction untouched. It is what makes compaction terminate: whatever the context grew to, it comes out of a compaction at roughly summary + this, which is far below any sane threshold, so the next round cannot immediately re-trigger.

View Source
const DefaultReserveTokens = 16384

DefaultReserveTokens is the floor on room kept free for the next response.

View Source
const DefaultSummarizationStallTimeout = 120 * time.Second

DefaultSummarizationStallTimeout matches the engine's default stall budget for its own rounds.

Variables

View Source
var ErrNothingToCompact = errors.New("compaction: nothing outside the keep-recent budget")

ErrNothingToCompact means the context cannot be made smaller: everything outside the keep-recent budget is already gone. Callers must treat this as a stop signal rather than retrying, or they reproduce the every-round summarization loop this package exists to prevent.

Functions

func Apply

func Apply(messages []chat.Message, p *Preparation, summary string) []chat.Message

Apply rebuilds the message list as system prompt, summary, and the verbatim tail. Nothing between the system prompt and the cut point survives.

func IsOverflowError

func IsOverflowError(err error) bool

IsOverflowError reports whether a failed LLM call failed because the request did not fit in the context window.

func ResponseHitContextLimit

func ResponseHitContextLimit(
	resp *types.ChatResponse, contextWindow, completionBudget int,
) bool

ResponseHitContextLimit reports whether a *successful* response was shaped by a full context window rather than by the completion budget we asked for. Three shapes:

  • The provider accepted an oversized request silently and reported a prompt larger than the window (z.ai).
  • The provider truncated the input to fill the window exactly, leaving no room to generate, and stopped on length with no output (Xiaomi MiMo).
  • The provider stopped on length having emitted less than we allowed, so our own cap cannot be what stopped it (the general recoverable case).

completionBudget is the max_tokens we requested, before any context-based clamping; passing the clamped value would hide exactly the case this detects.

func SummaryMessage

func SummaryMessage(summary string) chat.Message

SummaryMessage wraps summary text in the envelope the model sees, tagged so the next compaction recognizes it as its own output.

Types

type Checkpoint

type Checkpoint struct {
	// TurnID is the stored turn the summary covers through, inclusive.
	TurnID  string
	Summary string
	// Degraded reports that Summary holds a raw archive because the
	// summarizer failed. It is still a checkpoint: the next compaction
	// folds it in as the previous summary.
	Degraded bool
}

Checkpoint is a summary that ends exactly on a stored turn. Persisted onto that turn, it lets the next turn load the summary in place of everything up to and including the turn, instead of summarizing the same history again.

type Compactor

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor summarizes conversation history into a checkpoint.

func New

func New(chatModel chat.Chat, estimator *agenttoken.Estimator, settings Settings) *Compactor

New builds a compactor. It returns nil when compaction cannot run, so the caller can treat a nil compactor as "feature off" without a second flag.

func (*Compactor) Compact

func (c *Compactor) Compact(
	ctx context.Context, messages []chat.Message, reason Reason,
) (*Result, error)

Compact replaces history older than the keep-recent budget with a summary. It returns ErrNothingToCompact when no such history exists.

func (*Compactor) Settings

func (c *Compactor) Settings() Settings

Settings returns the normalized settings, including the derived threshold.

type CutPoint

type CutPoint struct {
	// FirstKeptIdx is the index of the first message kept verbatim.
	FirstKeptIdx int
	// TurnStartIdx is the user message that opened the turn the cut lands in,
	// or -1 when the cut falls on a turn boundary.
	TurnStartIdx int
	// IsSplitTurn reports that the cut divides a single turn, so the part
	// before it needs its own summary to explain the part after it.
	IsSplitTurn bool
}

CutPoint is where the conversation is divided into "summarize this" and "keep this verbatim".

func FindCutPoint

func FindCutPoint(
	messages []chat.Message,
	start, keepRecentTokens int,
	estimator *agenttoken.Estimator,
) CutPoint

FindCutPoint walks backwards from the newest message accumulating estimated size, and keeps the largest suffix that still fits keepRecentTokens.

The budget is a ceiling, not a floor. Stopping at the first message that reaches the budget and keeping that message too is the obvious reading and the wrong one: a single knowledge_search result is routinely ten thousand tokens on its own, so "keep the recent 16k" would hand back 25k and leave barely a round of headroom before the next compaction.

Nothing here consults the current turn. That is deliberate: the budget is the only rule, and a turn large enough to exceed it on its own gets split rather than exempted.

type Preparation

type Preparation struct {
	// FirstKeptIdx is where the verbatim tail begins.
	FirstKeptIdx int
	// MessagesToSummarize are the complete turns being replaced by prose.
	MessagesToSummarize []chat.Message
	// TurnPrefixMessages is the discarded head of a split turn, summarized
	// separately so the retained tail still has its originating request.
	TurnPrefixMessages []chat.Message
	IsSplitTurn        bool
	// PreviousSummary is the prior compaction's text, updated in place rather
	// than summarized again.
	PreviousSummary string
	TokensBefore    int
	// OmittedHistory and OmittedTurnPrefix count the oldest messages of each
	// range left out because one summarization request could not carry them
	// (see fitSummarizerInput). Their file paths are still recorded.
	OmittedHistory    int
	OmittedTurnPrefix int
	// contains filtered or unexported fields
}

Preparation is everything the summarization step needs, computed before any LLM call so the caller can decide not to make one.

func Prepare

func Prepare(messages []chat.Message, s Settings, estimator *agenttoken.Estimator) *Preparation

Prepare picks the cut point and collects the message ranges around it.

It returns nil when compaction cannot help — no messages fall outside the keep-recent budget, or the last compaction already left nothing to remove. That nil is the guard against the failure mode where every round spends a summarization call on a context it cannot shrink.

type Reason

type Reason string

Reason records what asked for a compaction, for logging and for the UI.

const (
	// ReasonThreshold is the routine case: context crossed the budget.
	ReasonThreshold Reason = "threshold"
	// ReasonOverflow is repair after the provider rejected or truncated a
	// request because the window was full.
	ReasonOverflow Reason = "overflow"
)

type Result

type Result struct {
	Messages       []chat.Message
	Summary        string
	Reason         Reason
	TokensBefore   int
	TokensAfter    int
	MessagesBefore int
	MessagesAfter  int
	// Degraded reports that at least one summary came from the raw archive
	// because the summarizer failed. The context still shrank; its memory is
	// just coarser.
	Degraded bool
	// SplitTurn reports that the cut divided a single turn, so a turn-prefix
	// summary was generated alongside the history summary.
	SplitTurn bool
	// Checkpoint is the part of this compaction a later turn can start from,
	// or nil when there is none (see Preparation.checkpoint).
	Checkpoint *Checkpoint
	// Omitted counts the oldest messages left out of the summarization
	// requests because they did not fit (see fitSummarizerInput).
	Omitted int
}

Result describes what one compaction did.

func (*Result) Freed

func (r *Result) Freed() int

Freed is how much room the compaction actually recovered. A non-positive value means the summary cost as much as the history it replaced.

type Settings

type Settings struct {
	// Enabled turns automatic compaction on. Disabled still allows an
	// explicit overflow-recovery compaction, which is a repair, not a policy.
	Enabled bool
	// MaxContextTokens is the model's context window. Zero disables
	// compaction entirely: without a window there is no budget to enforce.
	MaxContextTokens int
	// ReserveTokens is the part of the window history may not occupy because
	// the next response has to fit there.
	ReserveTokens int
	// KeepRecentTokens is the token budget of recent messages kept verbatim.
	KeepRecentTokens int
	// MaxSummaryTokens caps the summarization completion. Zero falls back to
	// the reserve-derived budget alone.
	MaxSummaryTokens int
	// StallTimeout is how long a summarization stream may produce no output
	// before it is cancelled. Zero means DefaultSummarizationStallTimeout.
	StallTimeout time.Duration
}

Settings configures when compaction runs and how much it keeps.

func (Settings) Normalize

func (s Settings) Normalize() Settings

Normalize fills in defaults and reconciles budgets that cannot all be honored at once.

func (Settings) ShouldCompact

func (s Settings) ShouldCompact(contextTokens int) bool

ShouldCompact reports whether the current context has crossed the threshold.

func (Settings) Threshold

func (s Settings) Threshold() int

Threshold is the context size above which history must be compacted. It is an absolute reserve rather than a fraction of the window, because what has to fit is the response, whose size does not scale with the window.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL