Documentation
¶
Overview ¶
Package loopdetector answers one question about a model's output: is it going round in a loop, and how many times has it gone round.
llm.RepetitionGuard is on by default and catches one shape, the same bytes back to back. Loops seen in practice also come with random data between the copies, or go round ten or more variations before repeating, and those have no period for it to find. This package finds them, for a host that installs it.
THE INPUT IS THE SESSION PLUS THE STREAM. Begin takes the messages the request was sent with, and reads the turn since a person last spoke; Watch takes the reply's deltas as they arrive, thinking included. There is no cache to size or to go stale, and nothing lives past the reply: what the detector compares against is the conversation itself, which any party that can see the request already holds, the corrallm proxy included. The history has no thinking in it (clients do not send it back), so a loop in thinking is found within the reply being built.
THE OUTPUT IS A COUNT. Loops says how many times the output has gone round; Config.Tolerance says at what count Watch asks for the stream to be cut. A host that wants to watch before it trusts this sets a negative Tolerance and reads the Counters.
Only output being built can be interrupted. A session whose history already loops is counted, but a reply is cut only once it continues the loop: a model told to stop must be given the reply in which to stop.
Four detectors feed the count:
exact the same block back to back: llm.RepetitionGuard.
cycle the same sentences recurring, in any order, around other output.
phrase the same, reworded a little. Words are stemmed and sentences matched
by BM25: word order and word forms do not matter, synonyms do. BM25
rather than a count of shared words, because it discounts what every
sentence has: rows of a table share almost all their words and
differ in a few, and it is those few that decide the score.
tool the same call returning the same result. A result that changed
is progress, so the count for that call starts again.
Index ¶
Constants ¶
const ( KindExact = "exact" KindCycle = "cycle" KindPhrase = "phrase" KindTool = llm.KindTool )
Kinds of loop, as reported in llm.RepetitionInfo.Kind and the Counters.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Off disables detection: a Session counts nothing and interrupts nothing.
Off bool
// Tolerance is the loop count at which Watch returns a finding, which cuts
// the stream. 0 → 3; negative → never, the count is only reported.
Tolerance int
// ToolTolerance is Tolerance for the tool detector, for a host that refuses
// repeated calls itself and wants text cut but tools only counted. 0 →
// Tolerance; negative → never.
ToolTolerance int
// Exact configures the exact detector.
Exact llm.RepetitionGuard
// Window is how many recent sentences the cycle and phrase detectors look at
// (default 48), and MinShare the share of them that must be repeats (default
// 0.7). What is not a repeat is the room left for other output between the
// copies. A list worked through in thinking is a false hit, and such lists
// are shorter than 48 sentences: over 13,454 blocks of real thinking, 24 and
// 0.6 gave 14 cuts at Tolerance 3, these gave none, and a real reply of 40
// sentences said about 21 times was still found.
Window int
// MinSpan is the bytes the Window must cover (default 1024), so that a run
// of short lines does not count as a loop.
MinSpan int
// MinTokens is the fewest words that make a sentence (default 4). Shorter
// ones, closing braces and "Let me check.", are not evidence of anything.
MinTokens int
// MaxSegment splits text with no sentence end in it (default 512 bytes).
MaxSegment int
// Similarity is the BM25 score, as a share of a sentence's score against
// itself, at which two sentences are the same (default 0.85). Words are
// stemmed first. Most false hits came from here: lower, a list whose lines
// differ in a few words is taken for one line said many times. Negative
// turns the phrase detector off.
Similarity float64
// SampleBytes caps the First, Last and Sample of a finding (default 200).
SampleBytes int
// ToolWindow is how many recent tool calls the tool detector looks at
// (default 12). MinShare applies to them too. No tool is exempt: polling for
// a result that has not changed is the waste this finds. A host waits on an
// event instead.
ToolWindow int
}
Config is the detector's settings. The zero value is usable.
type Counters ¶
type Counters struct {
// Sessions is how many replies were watched. Bytes and Sentences are how much
// of them was; Repeats is how many sentences matched an earlier one.
Sessions, Bytes, Sentences, Repeats int64
// Detected counts sessions, by kind, in which a loop was found, whatever the
// Tolerance. Interrupted counts those where Watch asked for a cut.
Detected, Interrupted map[string]int64
}
Counters are totals since New, for a host to export as metrics.
type LoopDetector ¶
type LoopDetector struct {
// contains filtered or unexported fields
}
LoopDetector holds the settings and the counters. It is safe for concurrent use; each reply gets its own Session from Begin.
func (*LoopDetector) Begin ¶
func (d *LoopDetector) Begin(history []llm.Message) *Session
Begin starts watching a reply to history.
func (*LoopDetector) Counters ¶
func (d *LoopDetector) Counters() Counters
Counters returns the totals so far.
func (*LoopDetector) Install ¶
func (d *LoopDetector) Install(c *llm.Client)
Install makes the detector watch every stream of c, in place of c.Repetition.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session watches one reply against the conversation it continues. It belongs to one stream and is not safe for concurrent use.
func (*Session) Finding ¶ added in v0.16.1
func (s *Session) Finding() *llm.RepetitionInfo
Finding returns the highest finding so far, cut or not, nil when the session is not looping. It is for a caller who wants to pass it on.
func (*Session) Loops ¶
Loops reports the highest loop count seen so far and the kind of loop it was in, 0 and "" when the session is not looping.
func (*Session) Watch ¶
func (s *Session) Watch(channel, delta string) *llm.RepetitionInfo
Watch feeds one delta of the reply: llm.StreamWatcher. A non-nil return means the loop count reached Tolerance and the stream should be cut. It says so once.
func (*Session) WatchToolCall ¶
func (s *Session) WatchToolCall(name, args string) *llm.RepetitionInfo
WatchToolCall feeds a finished tool call of the reply: llm.ToolCallWatcher.
Source Files