watchdog

package
v3.0.0-dev.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 21, 2026 License: Apache-2.0 Imports: 5 Imported by: 0

Documentation

Overview

The windowed-density loop detector (#702).

The two loop signals that shipped before this one divide the space cleanly on paper: RepeatedToolCallSignal owns a *run* of one call, AlternatingCycleSignal owns a repeating *sequence* and explicitly skips blocks made of a single repeated call because the repeat detector already covers them. The shape that falls between them is mostly one repeated call with occasional other calls wedged in. The repeat detector resets its run length on any non-matching call, so an interleave restarts the count from zero; the cycle detector sees a nearly-uniform block and hands it back to the repeat detector. Neither converges quickly.

This is not a miss — it is a convergence gap, and the difference matters for how the detector is tuned. In GKE UAT session 019ffb77-3616-77a6-b06d-a5815b0929fe the backstop did fire: 22 byte-identical gke_list_clusters calls over two minutes and twenty seconds, halted by repeated-tool-call at call 22. The loop was visibly degenerate from about the fourth call, and the interleaves were what stretched a threshold of 5 into 22 calls and a large share of a $0.77 session.

So the job here is to reach the same verdict sooner on the same shape, without loosening either existing detector's contract and without raising a second alert for a pattern one of them already owns. Density over a sliding window does that: it does not care where the interleaves fall, only that one call dominates recent activity.

Index

Constants

View Source
const (
	DefaultCycleMaxPeriod = 4
	DefaultCycleRepeats   = 3
)

Default tuning for AlternatingCycleSignal. Exported so an operator building a custom signal list can see what they are deviating from.

View Source
const (
	DefaultDominantWindow      = 12
	DefaultDominantThreshold   = 8
	DefaultDominantDeferRun    = DefaultRepeatThreshold
	DefaultDominantDeferPeriod = DefaultCycleMaxPeriod
)

Default tuning for DominantToolCallSignal. Exported so an operator building a custom signal list can see what they are deviating from.

8-of-12 is chosen against the failure it was drawn from: the UAT loop reaches it around the tenth call rather than the twenty-second, while still requiring that two thirds of a full window be one call. DefaultDominantDeferRun tracks DefaultRepeatThreshold — see the DeferRun field for why the two are tied.

View Source
const DefaultFailureStreak = 3

DefaultFailureStreak is the number of consecutive failed tool calls that trips ToolFailureStreakSignal. Three is chosen to sit above ordinary exploration — a 404, a missing file, one RBAC denial — and below the point where a model has usually stopped gathering evidence and started composing an answer.

View Source
const DefaultNoOpStreak = 3

DefaultNoOpStreak is the number of consecutive self-reported no-op tool results that trips NoOpStreakSignal.

Three, and low on purpose. The other detectors' thresholds price in the chance they are wrong about a legitimate workload; this one cannot be wrong in that direction, because a tool asserting it did nothing is not a heuristic about the model's intent. Two would be defensible and is not chosen only because a single retry after a transient is an ordinary shape; by the third the agent has spent three turns' worth of calls to change nothing.

View Source
const DefaultRepeatThreshold = 5

DefaultRepeatThreshold is the consecutive-run length that trips RepeatedToolCallSignal in the default signal set. Exported because DominantToolCallSignal defers to that detector and has to know where its territory starts — see DominantToolCallSignal.DeferRun.

View Source
const DefaultStallMemory = 64

DefaultStallMemory bounds how many distinct result digests the signal remembers. A stall is a local phenomenon — the repeats that matter are within a few calls of each other — and an unbounded set on a long-running daemon is a slow leak charged to a guardrail. 64 is far more than DefaultStallRun needs and small enough to be free.

View Source
const DefaultStallRun = 6

DefaultStallRun is the number of consecutive results carrying only already-seen payloads that trips NoNewStateSignal. See the file comment for why it sits at 6.

View Source
const DefaultToolNameRun = 15

DefaultToolNameRun is the consecutive same-name run length that trips RepeatedToolNameSignal in the default signal set.

The number is a false-positive budget, not a loop-detection preference. Dropping args from the key means this signal cannot tell a stuck agent from a productive one working through a list, so the threshold has to sit above the longest legitimate single-tool run we expect to see.

What the budget costs depends entirely on the severity, and that is why this signal is Warn (see RepeatedToolNameSignal). At Critical the threshold had to clear every plausible sweep, because a false positive under --watchdog=enforce halts a working agent; twenty was the smallest number that did, and twenty is high enough that the signal has no demonstrated catch — a guardrail priced so as never to fire. At Warn a false positive costs one line in the operator log and, under --watchdog=feedback, one paragraph of next-turn context aimed at a model that is free to disregard it. That is cheap enough to buy a threshold low enough to be useful.

Fifteen is where those meet. It clears the one legitimate sweep this package actually certifies — TestDominantToolCallSignal_DoesNotTripOnLegitimateWork asserts that twelve read_file calls over twelve distinct paths is work, not a loop — with margin, while sitting low enough to catch grinding before it has run for twenty calls. Do not raise it back toward twenty without also raising the severity; the two numbers are one decision.

It is still NOT tuned to catch the nine-call mark_task_done loop above. Nine is under fifteen, and no threshold low enough to catch it survives the paragraphs above. That specific shape is fixed at the tool layer instead (mark_task_done now reports the repeat rather than acknowledging it again), which is the better place for it anyway: a tool that cannot do anything the second time in a turn should say so itself, not wait for a behavioral detector to infer it. This signal is the backstop for the general case — any tool ground on long enough with varying arguments — and it is deliberately the last line of defence, not the first.

View Source
const DefaultToolsWithoutText = 12

DefaultToolsWithoutText is the number of consecutive tool calls with no intervening assistant text that trips ToolsWithoutTextSignal.

Twelve, and unlike most thresholds in this package that number was measured rather than argued. The design doc guessed 15 in a table of "initial guesses"; sixty-four archived GKE-drill sessions — real runs against a live cluster, all of them judged good at the time — put the longest textless run at 10, the 95th percentile at 5 and the median at 2. Twelve clears the observed ceiling by a fifth and sits under #905's recorded loop, which ran fifteen calls without a word.

The corpus is committed (testdata/corpus) and the margin is asserted, not remembered: TestDefaultToolsWithoutTextClearsTheRecordedCorpus fails if a future extraction contains a good session that would trip this. A threshold whose justification is a sentence in a comment rots; one whose justification is a test does not.

What the corpus cannot say is how a coding agent behaves — every run in it is Kubernetes triage, which is the workload this project aims at but not the only one it runs. A twelve-call silent sweep through a codebase is plausible in a way a twelve-call silent sweep through a cluster is not, and an operator who sees this fire on one should raise the threshold rather than treat it as a finding. That it costs only a log line is the reason the tighter number is affordable; see the Severity paragraph on the type.

View Source
const FeedbackHeader = "[watchdog]"

FeedbackHeader opens the block FormatFeedback renders. Exported so hosts, tests and transcript tooling can find the boundary without re-typing the literal.

Variables

This section is empty.

Functions

func FormatFeedback

func FormatFeedback(alerts []Alert) string

FormatFeedback renders alerts as the model-facing block --watchdog=feedback prepends to the next turn's prompt. Returns "" for an empty slice so callers can skip the prepend cheaply.

Two things the wording has to do. It must be unmistakably *about the model's own last turn* rather than a message from the user — a model that reads "you called grep 5 times" as a user complaint will apologize instead of changing behavior. And it must carry an instruction, not a description: the whole point of routing this to the model is that a description of a loop is what it already had.

Not a trust boundary. A user prompt can contain the literal "watchdog" string, exactly as it can contain "[Inbox]" or "[Background reports]"; this block is a steering signal, and nothing downstream grants authority based on it.

Types

type Alert

type Alert struct {
	Signal   string
	Severity Severity
	Reason   string
	Guidance string

	// Scope bounds what a Critical alert is allowed to stop. Zero value
	// (ScopeSession) is the original contract; see AlertScope. Read only
	// under --watchdog=enforce, and only for Critical alerts — a Warn
	// stops nothing either way.
	Scope AlertScope

	// ClearsOnText marks an alert whose condition ends when the model
	// produces text, as tools-without-text's does. A host that holds the
	// model-facing half for a later turn drops it once the model has
	// spoken, because by then it describes something no longer true: in
	// a turn that runs for hours, "the next turn" can be hours after the
	// silence it names (#1166).
	ClearsOnText bool

	// Cleared is set by DefaultWatchdog on a buffered ClearsOnText alert
	// when the model spoke before the alert was drained. Check still
	// returns it, so the operator log and the metric count it; only the
	// model-facing route should skip it.
	//
	// DefaultWatchdog does this for every Signal it runs. A custom
	// Watchdog that buffers ClearsOnText alerts must set Cleared itself:
	// the agent drops an alert it has already queued when the model
	// speaks, but it cannot see inside another implementation's buffer.
	Cleared bool
}

Alert is what a triggered signal returns. Signal is the stable string ID the rest of the system can dispatch on (future "auto" mode picks behavior per signal).

Reason and Guidance are the same observation addressed to two different readers, and the split is load-bearing. Reason is operator-facing — the agent's wiring logs it verbatim, and it may name operator affordances ("/interrupt", "--max-turn-cost-usd") that only a human at a terminal can act on. Guidance is model- facing: what the *agent* should do differently on its next turn, written as an instruction with no reference to operator controls, because under --watchdog=feedback (#159) it is injected into the model's next-turn context. An unattended daemon has no operator to read Reason, so for it Guidance is the only half that does work.

Guidance is optional. A third-party Signal that leaves it empty still produces feedback — FormatFeedback falls back to Reason — but the fallback carries operator advice the model can't take, so built-in signals should always set it.

func (Alert) String

func (a Alert) String() string

String implements fmt.Stringer for Alert so log lines stay uniform. Format: "[severity] signal: reason".

type AlertScope

type AlertScope int

AlertScope says how much a Critical alert entitles enforce mode to stop: the whole session, or only the turn the evidence came from (#1090).

Severity says how sure the signal is that something is wrong; scope says what stopping it costs. They were the same question while every Critical meant the same thing, and the approval-gate drill showed they are not: a no-op streak is a loop inside one turn, and cutting the turn ends it exactly as dead as halting the session does, while the halt additionally takes the daemon off the air until a human arrives. A signal whose evidence spans turns has no such option.

const (
	// ScopeSession is the zero value, deliberately: a signal written
	// before this existed — or by a third party — keeps enforce's
	// original contract, where a Critical refuses every subsequent turn
	// until the operator resets. Opting DOWN to ScopeTurn is a claim a
	// signal has to earn.
	ScopeSession AlertScope = iota

	// ScopeTurn cuts the turn the alert arrived in and nothing further.
	// Only sound when ending the turn ends the behavior — which, for a
	// within-turn loop, it does, because the model cannot call a tool
	// inside a turn that is over. What it gives up is protection against
	// the model restarting the loop next turn; the agent covers that with
	// an escalation after consecutive turns end this way (see
	// maxConsecutiveWatchdogTurnCuts), not by widening the scope.
	ScopeTurn
)

func (AlertScope) String

func (s AlertScope) String() string

String renders the scope for a log line or a trip reason.

type AlternatingCycleSignal

type AlternatingCycleSignal struct {
	// MaxPeriod is the longest cycle considered. Periods start at 2 —
	// period 1 is a consecutive repeat, which RepeatedToolCallSignal
	// owns.
	MaxPeriod int
	// Cycles is how many consecutive repetitions of the block are
	// required before the signal trips.
	Cycles int
	// contains filtered or unexported fields
}

AlternatingCycleSignal trips when the agent repeats the same short *sequence* of tool calls over and over — a → b → a → b → a → b (#649).

This is the evasion the consecutive-repeat detector named in its own docstring and could not catch: "consecutive" means a run of one call, so any loop with a second call wedged in it reads as varied activity. The shape is not hypothetical. The live UAT that motivated the enforce-mode backstop (#623) was an agent cycling list_agents → check_agent, and it survived both an operator "stop" and /interrupt.

Detection is a period scan over the recent call history: for each period p in 2..MaxPeriod, the last p*Cycles observations trip the signal when they consist of Cycles byte-identical blocks of length p. Keys are canonicalized (canonicalArgs), so a cycle that alternates "./main.go" with "main.go" still reads as one call — but unlike the repeat detector this cannot use the pairwise path-suffix relation, which is not transitive and therefore cannot key a history buffer.

Two deliberate limits on false positives, because a Critical alert halts the agent under --watchdog=enforce:

  • A block whose entries are all the same call is skipped. It is a plain repeat, already the other signal's job, and alerting twice for one pattern doubles the noise and (under feedback mode) the tokens.
  • Cycles defaults to 3, i.e. six calls for the a→b→a→b→a→b shape. Two passes through a sequence is normal work — read a file, grep, read another, grep again. Three passes with byte-identical arguments each time is not: nothing in the inputs changed, so nothing in the results can have.

A polling loop written as alternating tool calls is the known false positive. That is what wait_and_verify (#648) exists for, and an operator who wants the pattern anyway can drop the signal by constructing DefaultWatchdog with their own signal list.

func NewAlternatingCycleSignal

func NewAlternatingCycleSignal(maxPeriod, cycles int) *AlternatingCycleSignal

NewAlternatingCycleSignal constructs the cycle detector. maxPeriod below 2 is clamped to 2 (period 1 is the repeat detector's job) and cycles below 2 is clamped to 2 (a single pass through a sequence is not a cycle).

func (*AlternatingCycleSignal) Name

func (s *AlternatingCycleSignal) Name() string

Name implements Signal.

func (*AlternatingCycleSignal) ObserveToolCall

func (s *AlternatingCycleSignal) ObserveToolCall(tc ToolCall) *Alert

ObserveToolCall implements Signal. Appends the call to a bounded history and reports the shortest cycle covering the tail.

func (*AlternatingCycleSignal) Reset

func (s *AlternatingCycleSignal) Reset()

Reset implements Signal.

type AssistantTextObserver

type AssistantTextObserver interface {
	ObserveAssistantText(string)
}

AssistantTextObserver is the optional half of Watchdog for the model's own output: an implementation that wants to see what the agent SAID, as opposed to what it called, implements it, and the agent feeds text through a type assertion.

Optional for the reason ToolResultObserver and TurnObserver are. Watchdog is documented as a plug-in point, so widening it would break every third-party watchdog at a minor version to add one signal, and a custom watchdog that only counts calls stays perfectly valid.

type DefaultWatchdog

type DefaultWatchdog struct {
	// contains filtered or unexported fields
}

DefaultWatchdog is the package-default implementation. Fans observations across the configured signals; Check collects alerts from each.

func NewDefaultWatchdog

func NewDefaultWatchdog() *DefaultWatchdog

NewDefaultWatchdog returns a DefaultWatchdog wired with the default signal set:

  • RepeatedToolCall (threshold 5): 5 consecutive calls to the same tool with path-canonicalized-identical args.
  • AlternatingCycle (period ≤ 4, 3 laps): the same short sequence of calls repeated three times — the a → b → a → b shape the repeat detector structurally cannot see (#649).
  • DominantToolCall (8 of the last 12): one call dominating recent activity even though interleaves keep breaking its run — the shape that falls between the two detectors above and that neither converges on quickly (#702).
  • RepeatedToolName (15 in a row): the same tool ground on with varying arguments, which every detector above misses because they all key on (name, args) and a model-authored free-text argument hashes differently each iteration.
  • ToolFailureStreak (3 in a row): every call erroring with none succeeding in between, i.e. an agent with no verified evidence about anything (#639).
  • NoOpStreak (3 in a row): three tool RESULTS each declaring the call changed nothing. The only signal that needs no inference — the tools said it themselves — and the one that reaches the free-text loop the four above structurally cannot (#907).
  • NoNewState (6 in a row): six results whose payloads this turn has already seen. Keys on the ANSWER rather than the call, which is how it catches "these two calls ask the same question differently" without having to understand either one (#655).
  • ToolsWithoutText (12 in a row): twelve calls with no assistant text between them. The only signal that compares nothing, so the only one with no evasion other than talking (#655).

Four of the eight are Critical, so they halt under --watchdog=enforce. The three args-keyed loop detectors qualify because each can prove the calls are identical, so the agent is provably learning nothing; NoOpStreak qualifies for the opposite reason — it proves nothing and infers nothing, because a no-op is a claim the tool made about its own result. RepeatedToolName is Warn despite being a loop signal, because keying on the name alone cannot distinguish a loop from a sweep and halting a working agent is worse than missing a slow one — see RepeatedToolNameSignal. ToolFailureStreak is Warn for a related reason: a run of denials may be a legitimate RBAC probe, and see NoOpStreakSignal for why that argument does not carry across to it. NoNewState is Warn on the RepeatedToolName argument transposed from the call to the result: a repeated payload cannot distinguish a loop from a poll, and polling a cluster until it converges is most of what this project's agents legitimately do. ToolsWithoutText is Warn for the same reason at its limit: it has made no comparison at all, so a sweep and a runaway are the same observation to it. A Warn never halts; it reaches the operator log plus — under --watchdog=feedback — the model's own next turn. Operators wanting different thresholds, or a subset, construct DefaultWatchdog directly with a custom signal list.

func (*DefaultWatchdog) Check

func (w *DefaultWatchdog) Check() []Alert

Check returns any alerts that accumulated since the last Check and resets the buffer. Returns nil (not an empty slice) when no alerts are pending — lets the caller skip work cheaply.

func (*DefaultWatchdog) ObserveAssistantText

func (w *DefaultWatchdog) ObserveAssistantText(text string)

ObserveAssistantText fans one piece of model output across every wired signal that implements SignalTextObserver. Implements AssistantTextObserver.

func (*DefaultWatchdog) ObserveToolCall

func (w *DefaultWatchdog) ObserveToolCall(tc ToolCall)

ObserveToolCall fans the observation across every wired signal.

func (*DefaultWatchdog) ObserveToolResult

func (w *DefaultWatchdog) ObserveToolResult(tr ToolResult)

ObserveToolResult fans a tool outcome across every wired signal that implements SignalResultObserver. Implements ToolResultObserver.

func (*DefaultWatchdog) ObserveTurnStart

func (w *DefaultWatchdog) ObserveTurnStart()

ObserveTurnStart fans a turn boundary across every wired signal that implements SignalTurnObserver. Implements TurnObserver.

Deliberately does not touch w.alerts: Check drains them and the post-turn hook is what calls it, so clearing here would discard the alerts raised by the turn that just ended.

func (*DefaultWatchdog) Reset

func (w *DefaultWatchdog) Reset()

Reset clears alerts + every signal's state. Called on logical session boundaries (e.g. operator-initiated /clear).

type DominantToolCallSignal

type DominantToolCallSignal struct {
	// Window is how many recent tool calls are considered.
	Window int
	// Threshold is how many of those must be the same call.
	Threshold int
	// DeferRun is the consecutive-run length that hands the window
	// back to RepeatedToolCallSignal: when the dominant call has ever
	// reached a run that long inside the current window, this signal
	// stays quiet, because that run is what the repeat detector
	// exists to catch and one behavior should not raise two alerts —
	// under --watchdog=feedback a duplicate alert is duplicated
	// prompt text, not just a duplicated log line.
	//
	// This is a deliberate, explicit coupling to the repeat
	// detector's threshold rather than a guess: AlternatingCycleSignal
	// makes the same deference structurally, by skipping uniform
	// blocks. Zero or negative disables it, which is what an operator
	// wiring this signal *without* the repeat detector wants.
	DeferRun int

	// DeferCyclePeriod is the same deference toward
	// AlternatingCycleSignal: when the whole window is a clean
	// repetition of a block of period 2..DeferCyclePeriod, that
	// detector owns it. Without this, a → a → b → a → a → b reads as
	// a dominant call (8 of 12 are `a`) *and* as a 3-call cycle, and
	// would raise two Critical alerts for one loop. Zero or negative
	// disables it.
	//
	// This is not a hole: the cycle detector reaches such a window
	// first — three laps of a 3-call block is nine calls, where the
	// density threshold needs twelve — so deferring costs nothing but
	// the duplicate. A period longer than the cycle detector scans
	// (a → a → a → a → b, period 5) does not match here and is
	// exactly the case this signal is for.
	DeferCyclePeriod int
	// contains filtered or unexported fields
}

DominantToolCallSignal trips when a single tool call accounts for Threshold of the last Window observations — the interleaved loop neither the consecutive-repeat nor the alternating-cycle detector converges on quickly (#702).

Args are compared canonicalized (canonicalArgs, see canonical.go), the same treatment AlternatingCycleSignal uses and for the same reason: counting occurrences means keying a map, and the repeat detector's pairwise path-suffix relation is not transitive, so it cannot key one.

Severity is Critical, matching the other two loop detectors. The justification is theirs as well: the dominant call's arguments are equivalent every time, so its results are too, and an agent spending two thirds of a window re-asking one answered question is not making progress.

The known false positive is a hand-rolled polling loop — the same one the cycle detector documents, and the same answer: wait_and_verify (#648) is the supported way to wait, and an operator who wants the pattern anyway constructs DefaultWatchdog with their own signal list. Note that a poll tight enough to put 8 identical calls in a 12-call window already trips the repeat detector at 5 consecutive unless something interleaves, so this signal does not make a legitimate poll materially more halt-prone than it already was.

func NewDominantToolCallSignal

func NewDominantToolCallSignal(window, threshold, deferRun, deferCyclePeriod int) *DominantToolCallSignal

NewDominantToolCallSignal constructs the density detector. window below 2 is clamped to 2, threshold below 2 is clamped to 2 (one call is not a density), and a threshold above window is clamped to window — a signal that can never trip is a silent misconfiguration, and clamping makes it a strict one instead.

func (*DominantToolCallSignal) Name

func (s *DominantToolCallSignal) Name() string

Name implements Signal.

func (*DominantToolCallSignal) ObserveToolCall

func (s *DominantToolCallSignal) ObserveToolCall(tc ToolCall) *Alert

ObserveToolCall implements Signal. Appends to the sliding window and reports the dominant call when it crosses the density threshold.

func (*DominantToolCallSignal) Reset

func (s *DominantToolCallSignal) Reset()

Reset implements Signal.

type NoNewStateSignal

type NoNewStateSignal struct {
	Threshold int
	Memory    int
	// contains filtered or unexported fields
}

NoNewStateSignal trips when Threshold tool results in a row all carry a payload this turn has already seen.

"In a row" counts successful, non-no-op, non-exempt results. A failure is ToolFailureStreakSignal's territory and a self-declared no-op is NoOpStreakSignal's; passing them through here would let one runaway raise three alerts saying the same thing, and the operator reading three lines learns less than the one who reads one.

func NewNoNewStateSignal

func NewNoNewStateSignal(threshold, memory int) *NoNewStateSignal

NewNoNewStateSignal constructs the signal. A threshold below 3 is clamped to 3: two identical answers is an ordinary retry, and at 2 this would fire on every agent that read a file twice.

func (*NoNewStateSignal) Name

func (s *NoNewStateSignal) Name() string

Name implements Signal.

func (*NoNewStateSignal) ObserveToolCall

func (s *NoNewStateSignal) ObserveToolCall(ToolCall) *Alert

ObserveToolCall implements Signal. The evidence this signal reads is the result, not the call; the method exists because DefaultWatchdog fans every call across every signal.

func (*NoNewStateSignal) ObserveToolResult

func (s *NoNewStateSignal) ObserveToolResult(tr ToolResult) *Alert

ObserveToolResult implements SignalResultObserver.

func (*NoNewStateSignal) ObserveTurnStart

func (s *NoNewStateSignal) ObserveTurnStart()

ObserveTurnStart implements SignalTurnObserver. The seen set is scoped to one turn, and this is the boundary that enforces it.

Without it the signal would be wrong on the workload this project cares most about. A watchdog is cleared only by an operator Reset, so its memory otherwise spans the daemon's whole life — and a daemon woken every ten minutes to read six resources off a cluster that has not changed would, from the second wake onward, produce six consecutive already-seen results and trip. That agent is monitoring correctly; the stall this signal is looking for is one turn spinning in place, not two turns that legitimately saw the same world.

A loop that survives a turn boundary is therefore not this detector's to catch, and that is the right split: it is re-driven work, which the auto-continue and cost-ceiling machinery already bound.

func (*NoNewStateSignal) Reset

func (s *NoNewStateSignal) Reset()

Reset implements Signal. It clears the seen set as well as the run, because "already seen" is scoped to the window the watchdog is currently observing — after an operator reset, the agent starting over with the same reads is starting over, not stalling.

type NoOpStreakSignal

type NoOpStreakSignal struct {
	Threshold int
	// contains filtered or unexported fields
}

NoOpStreakSignal trips when Threshold tool results in a row all report themselves as no-ops. Any result that did something — or failed — resets the run.

Resetting on a single productive call is the conservative choice and is affordable here in a way it is not for RepeatedToolNameSignal: at a run of 15, one unrelated call one-twelfth of the way in destroys the detection, which is how #905 escaped. At 3 the same interruption costs at most two observations, and the loop that is actually stuck re-accumulates immediately. #905's own trace splits 13 into 7 + 6 and trips on either half.

Severity is Critical, unlike ToolFailureStreakSignal's Warn. A failure streak is an evidence problem — the agent might still be doing something useful and halting it three RBAC denials into a legitimate probe would make the backstop the outage. A no-op streak is the runaway itself: the runtime has already established that the calls accomplished nothing, so under --watchdog=enforce there is nothing being interrupted that was going anywhere.

That is a real availability trade, it was taken deliberately, and #1090 repriced one half of it. enforce is the unattended default (#642) and mark_task_done is registered by default, so on a headless daemon three redundant checkpoints used to cost an operator reset — and #907 accepted that because the failure it replaces is unbounded: #905's loop ended when a human hit interrupt, and Warn would not have ended it any sooner. What that reasoning did not have was a measurement. The approval-gate drill supplied one: the reset is not a rare price, it is the modal ending of an ordinary denial, and it lands on a turn where the model had already stopped doing the thing anybody wanted stopped (it was recording the denial, four times, in four wordings).

The argument was also comparing against the wrong alternative. Warn is not the only other option: the alert is Scope: ScopeTurn, so enforce cuts the turn instead of the session, and the agent halts the session only when consecutive turns keep ending that way. Nothing #907 bought is given up — the model cannot keep calling a tool inside a turn that has ended, and #905's sixteen calls were one turn — while the agent that says "there is nothing further to record" three times loses a turn rather than its working life. Deployments that would rather absorb the loop still have the two levers that predate this signal — --watchdog=feedback, which reaches the only party that can stop calling, and --checkpoint=operator, which withholds the tool.

func NewNoOpStreakSignal

func NewNoOpStreakSignal(threshold int) *NoOpStreakSignal

NewNoOpStreakSignal constructs the signal. Threshold below 2 is clamped to 2: a tool reporting one no-op is an ordinary event — a retry, an idempotent write, an operator asking twice — and threshold 1 would alert on every one of them.

func (*NoOpStreakSignal) Name

func (s *NoOpStreakSignal) Name() string

Name implements Signal.

func (*NoOpStreakSignal) ObserveToolCall

func (s *NoOpStreakSignal) ObserveToolCall(ToolCall) *Alert

ObserveToolCall implements Signal. A call carries no outcome, and the whole point of this signal is that the outcome is the evidence; the method exists because DefaultWatchdog fans every call across every signal.

func (*NoOpStreakSignal) ObserveToolResult

func (s *NoOpStreakSignal) ObserveToolResult(tr ToolResult) *Alert

ObserveToolResult implements SignalResultObserver.

func (*NoOpStreakSignal) Reset

func (s *NoOpStreakSignal) Reset()

Reset implements Signal.

type RepeatedToolCallSignal

type RepeatedToolCallSignal struct {
	Threshold int
	// contains filtered or unexported fields
}

RepeatedToolCallSignal trips when the same (name, args) tool call appears Threshold times consecutively. Catches the read_file loop pattern from issue #144 and similar runaway-tool-call shapes.

"Consecutive" is the key word: a → b → a → b → a doesn't trip (no run of identical calls), but a → a → a → a → a does. This matches operator intuition ("the agent is stuck on the same thing") without flagging legitimate patterns like alternating-tool exploration loops.

Args comparison is path-canonicalized (#649, see canonical.go), not literal-string as in v1: "main.go", "./main.go" and "/workspace/main.go" are one call, because an agent re-reading one file under three spellings is as stuck as one re-reading it under the same spelling. Everything else still compares exactly — a detector that generalizes too eagerly halts legitimate work.

func NewRepeatedToolCallSignal

func NewRepeatedToolCallSignal(threshold int) *RepeatedToolCallSignal

NewRepeatedToolCallSignal constructs a signal with the given threshold. Threshold must be ≥ 2 (a "repeated call" requires at least two of the same in a row); values < 2 are clamped to 2 to avoid the degenerate case where every tool call trips the signal.

func (*RepeatedToolCallSignal) Name

func (s *RepeatedToolCallSignal) Name() string

Name implements Signal.

func (*RepeatedToolCallSignal) ObserveToolCall

func (s *RepeatedToolCallSignal) ObserveToolCall(tc ToolCall) *Alert

ObserveToolCall implements Signal. Tracks the running count of consecutive identical calls; emits an alert when count reaches Threshold. Returns nil on subsequent observations within the same run (already-tripped guard) so we don't re-emit on every extra call — operators want one notice per stuck pattern, not one per tool call past the threshold.

func (*RepeatedToolCallSignal) Reset

func (s *RepeatedToolCallSignal) Reset()

Reset implements Signal.

type RepeatedToolNameSignal

type RepeatedToolNameSignal struct {
	// Threshold is the consecutive same-name run length that trips.
	Threshold int

	// DeferRun is the run length that hands the window back to
	// RepeatedToolCallSignal. When the current run is not just
	// same-name but same-name-and-same-args, that detector owns it and
	// has already alerted at its own (lower) threshold; re-reporting
	// here would put two alerts, and under --watchdog=feedback two
	// blocks of prompt text, in front of one behavior. It would also be
	// the weaker report shadowing the stronger one — that detector's
	// alert is Critical and can prove the calls are identical, which
	// this one cannot. Mirrors DominantToolCallSignal.DeferRun,
	// including the convention that zero or negative disables the
	// deference for an operator wiring this signal without the repeat
	// detector.
	DeferRun int
	// contains filtered or unexported fields
}

RepeatedToolNameSignal trips when the same tool NAME is called Threshold times consecutively, whatever the arguments.

Deliberately the coarsest of the loop detectors. It exists for the shape the other three structurally cannot see: a loop whose arguments change every iteration without the call meaning anything different. Free-text arguments are the common source — a summary, a goal, a message, a commit note — because the model rephrases them naturally and each rephrasing defeats an args-keyed comparison.

Severity is Warn — the only loop detector that is not Critical, and deliberately so. The other three assert something they can prove: the calls are identical, so the agent is provably learning nothing. This one cannot. A long same-name run is a loop *or* a sweep, and the signal has no way to tell which. Under --watchdog=enforce a Critical alert halts the agent, so being wrong here would stop an agent that was working — the backstop becomes the outage. ToolFailureStreakSignal reached the same conclusion from the same place ("halting three denials into a legitimate RBAC probe"), and this signal has strictly less information than that one does.

Warn still does work. It reaches the operator log in every mode, and under --watchdog=feedback it reaches the model's own next turn as Guidance — which is the party that can actually stop making the call. An unattended daemon gets the feedback path and no halt, which is the right trade for an inference this soft.

Known false positive, now survivable: a genuine sweep longer than Threshold — an agent reading twenty files in a row, or issuing twenty reads against twenty distinct resources. It costs a log line and a paragraph of next-turn context, not a halt. This is why the alert text names the possibility instead of asserting a loop, and why Guidance tells a sweeping agent to keep going. An operator who runs workloads with long single-tool sweeps and wants silence should construct DefaultWatchdog with their own signal list, or raise Threshold.

func NewRepeatedToolNameSignal

func NewRepeatedToolNameSignal(threshold, deferRun int) *RepeatedToolNameSignal

NewRepeatedToolNameSignal constructs a signal with the given threshold and repeat-detector deference. Threshold must be ≥ 2; a smaller value is clamped, matching NewRepeatedToolCallSignal.

func (*RepeatedToolNameSignal) Name

func (s *RepeatedToolNameSignal) Name() string

Name implements Signal.

func (*RepeatedToolNameSignal) ObserveToolCall

func (s *RepeatedToolNameSignal) ObserveToolCall(tc ToolCall) *Alert

ObserveToolCall implements Signal. Tracks the consecutive run of calls sharing a tool name, alongside the trailing sub-run of calls that also share canonically-identical args so the deference to RepeatedToolCallSignal can be evaluated.

func (*RepeatedToolNameSignal) Reset

func (s *RepeatedToolNameSignal) Reset()

Reset implements Signal.

type Severity

type Severity string

Severity classifies the urgency of an alert. Warn is the operator- visible-but-not-action-blocking level; Critical marks a runaway the agent should act on — under --watchdog=enforce (#623) a Critical alert halts the agent (refuses further turns until the operator resets), while warn mode logs it like any other alert.

const (
	SeverityWarn     Severity = "warn"
	SeverityCritical Severity = "critical"
)

type Signal

type Signal interface {
	// Name returns the stable signal ID used in Alert.Signal.
	Name() string

	// ObserveToolCall updates the signal's internal state with one
	// tool invocation. Returning a non-nil Alert means the signal
	// tripped on this observation; DefaultWatchdog appends it to
	// the pending-alerts buffer.
	ObserveToolCall(ToolCall) *Alert

	// Reset clears the signal's state. Called from
	// DefaultWatchdog.Reset.
	Reset()
}

Signal is the per-detector interface inside DefaultWatchdog. Each signal owns its own state and decides when to emit an alert. Implementations must be safe to call serially from DefaultWatchdog (which holds a mutex across observations); they do NOT need to be concurrency-safe themselves.

Adding a new signal: implement Signal, append to NewDefaultWatchdog's signal list (or to a constructor variant). No changes to DefaultWatchdog itself.

type SignalResultObserver

type SignalResultObserver interface {
	ObserveToolResult(ToolResult) *Alert
}

SignalResultObserver is the same extension one level down, for signals inside DefaultWatchdog. A Signal that doesn't implement it simply never sees results.

type SignalTextObserver

type SignalTextObserver interface {
	ObserveAssistantText(string) *Alert
}

SignalTextObserver is that extension one level down, for signals inside DefaultWatchdog. A Signal that doesn't implement it never sees assistant text.

type SignalTurnObserver

type SignalTurnObserver interface {
	ObserveTurnStart()
}

SignalTurnObserver is that extension one level down, for signals inside DefaultWatchdog. A Signal that doesn't implement it keeps its state across turns, which is right for every streak detector here: a streak is already self-limiting, because the next call that breaks it clears it whether or not a turn ended in between.

type ToolCall

type ToolCall struct {
	Name string
	Args string
}

ToolCall is the per-tool-call observation the watchdog needs. Name is the canonical tool name (e.g. "read_file", "mcp.gke.list_clusters"). Args is the JSON-serialized argument blob, passed through as the caller produced it: the detectors canonicalize path-shaped values themselves (#649, canonical.go) rather than requiring every caller to agree on a normal form, and a third-party Watchdog implementation still sees the raw args.

type ToolFailureStreakSignal

type ToolFailureStreakSignal struct {
	Threshold int
	// contains filtered or unexported fields
}

ToolFailureStreakSignal trips when Threshold tool calls in a row all return errors. Any successful call resets the run — the point is "this agent currently has no verified evidence," and one success is evidence.

Severity is Warn, not Critical, and that is a decision rather than an oversight. Under --watchdog=enforce a Critical alert halts the agent, and since #642 enforce is the default for unattended runs; halting a daemon three denials into a legitimate RBAC probe would make the backstop the outage. A failure streak is an evidence problem, so it goes where evidence problems belong — the operator log, and (under --watchdog=feedback) the model's own next turn. Runaway *behavior* is already Critical via the loop detectors.

func NewToolFailureStreakSignal

func NewToolFailureStreakSignal(threshold int) *ToolFailureStreakSignal

NewToolFailureStreakSignal constructs the signal. Threshold below 2 is clamped to 2: a single failed call is an ordinary event, not a signal, and threshold 1 would alert on every one of them.

func (*ToolFailureStreakSignal) Name

func (s *ToolFailureStreakSignal) Name() string

Name implements Signal.

func (*ToolFailureStreakSignal) ObserveToolCall

func (s *ToolFailureStreakSignal) ObserveToolCall(ToolCall) *Alert

ObserveToolCall implements Signal. Calls carry no outcome, so this signal ignores them; the interface requires the method because DefaultWatchdog fans every call across every signal.

func (*ToolFailureStreakSignal) ObserveToolResult

func (s *ToolFailureStreakSignal) ObserveToolResult(tr ToolResult) *Alert

ObserveToolResult implements SignalResultObserver.

func (*ToolFailureStreakSignal) Reset

func (s *ToolFailureStreakSignal) Reset()

Reset implements Signal.

type ToolResult

type ToolResult struct {
	Name  string
	Error string

	// NoOp is the tool's own assertion that this invocation changed
	// nothing — the reserved "no_op" key in the response, opted into
	// per tool. It is deliberately not inferred: a signal that guessed
	// inertness from repetition would be the thing every other detector
	// in this package already does, and #905 is the proof that guessing
	// is what fails. See noop.go.
	//
	// A failed call is not a no-op. The two are independent claims and
	// a tool that sets both is reporting an error, which
	// ToolFailureStreakSignal already covers.
	NoOp bool

	// Digest is a stable fingerprint of the result payload — what the
	// call actually returned, as opposed to what it was asked (#655).
	// Empty means the caller could not compute one, which every
	// consumer must read as "unknown", never as "unchanged": see
	// NoNewStateSignal, which resets on it.
	//
	// The value is opaque and its algorithm is the producer's business;
	// the only contract is that equal payloads give equal digests and
	// the encoding is stable across calls within a process. The agent
	// tap computes it in pkg/agent/watchdog.go.
	Digest string
}

ToolResult is the outcome half of a tool invocation. Error is the tool's error text, empty for success — the ADK convention is a reserved "error" key inside FunctionResponse.Response, and the agent wiring flattens that to this field so the watchdog never has to know the provider's response shape.

func (ToolResult) Failed

func (r ToolResult) Failed() bool

Failed reports whether the call errored.

type ToolResultObserver

type ToolResultObserver interface {
	ObserveToolResult(ToolResult)
}

ToolResultObserver is the optional half of Watchdog: an implementation that also wants to see tool *outcomes* implements it, and the agent feeds results through a type assertion.

Deliberately not folded into Watchdog. That interface is documented as a plug-in point ("consumers can plug in their own implementation"), so widening it would break every third-party watchdog at a minor version to add one signal — and a custom watchdog that only counts calls stays perfectly valid.

type ToolsWithoutTextSignal

type ToolsWithoutTextSignal struct {
	Threshold int
	// contains filtered or unexported fields
}

ToolsWithoutTextSignal trips when Threshold tool calls go by with no assistant text between them. Any non-whitespace text from the model resets the run.

Severity is Warn, and this is the least close call in the package. Under --watchdog=enforce a Critical alert stops something, and the only thing this signal knows is that a number got large — it has not compared two calls, read a result, or been told anything by a tool. A deep sweep and a runaway look identical to it by construction. So it goes where weak evidence belongs: the operator log, and under --watchdog=feedback the model's own next turn, where the note to "say what you have found" costs a working agent one paragraph and gives a stuck one the only nudge that has ever stopped one of these without a human. The loop detectors that CAN prove redundancy are the ones that halt.

Scope is left at its zero value because scope is only read for a Critical alert (see AlertScope); there is no turn-versus-session judgement to make about an alert that never stops anything.

Not turn-scoped, for the reason every streak here is not: an ordinary turn ends with the model answering, so the run clears itself at the boundary without anyone having to clear it. A turn that ended without the model saying anything at all is the shape worth carrying into the next one — the person waiting for an answer did not get one, and the next turn starting does not change that.

Parallel tool calls count individually. Twelve calls issued at once with nothing said about any of them is the same silence as twelve issued in sequence, which is the same reading NoOpStreakSignal takes.

func NewToolsWithoutTextSignal

func NewToolsWithoutTextSignal(threshold int) *ToolsWithoutTextSignal

NewToolsWithoutTextSignal constructs the signal. Threshold below 2 is clamped to 2: at 1 every tool call an agent ever makes would alert, since a call necessarily precedes the text that reports on it.

func (*ToolsWithoutTextSignal) Name

func (s *ToolsWithoutTextSignal) Name() string

Name implements Signal.

func (*ToolsWithoutTextSignal) ObserveAssistantText

func (s *ToolsWithoutTextSignal) ObserveAssistantText(text string) *Alert

ObserveAssistantText implements SignalTextObserver. Never alerts — text is what clears this signal, not what trips it.

func (*ToolsWithoutTextSignal) ObserveToolCall

func (s *ToolsWithoutTextSignal) ObserveToolCall(tc ToolCall) *Alert

ObserveToolCall implements Signal.

func (*ToolsWithoutTextSignal) Reset

func (s *ToolsWithoutTextSignal) Reset()

Reset implements Signal.

type TurnObserver

type TurnObserver interface {
	ObserveTurnStart()
}

TurnObserver is the same optional extension for turn boundaries: a watchdog that scopes any of its evidence to a single turn implements it, and the agent calls it at the top of every turn through a type assertion. Separate from Watchdog for the reason above — a custom watchdog that never needed a turn boundary must not have to grow one.

Scoping matters because the watchdog outlives the turn. Its state is cleared only by an operator Reset, so a signal that remembers what it has already seen would, on a daemon woken every ten minutes to check a stable cluster, read the second wake's identical readings as a stall. That agent is doing its job; see NoNewStateSignal.

type Watchdog

type Watchdog interface {
	// ObserveToolCall records one tool invocation. Called by the
	// agent's event-tap as tool calls stream by; safe to call from
	// any goroutine.
	ObserveToolCall(ToolCall)

	// Check returns alerts triggered since the last Check call and
	// resets the per-call alert buffer. Returns nil when no signal
	// has tripped. Typically called from the agent's post-turn
	// hook; an alert returned here is "for the turn just ended."
	Check() []Alert

	// Reset clears all accumulated state. Called when the agent
	// resets (e.g. via a hypothetical /clear that clears history)
	// so signals don't carry across a logical session boundary.
	Reset()
}

Watchdog observes per-turn telemetry and returns any alerts that triggered during observation. Implementations must be safe for concurrent use — the agent calls Observe* methods from the streaming event handler and Check from the post-turn hook; concurrency is bounded but real.

The interface is intentionally narrow for v1: just tool-call observation + alert reporting. Richer telemetry (turn timing, per-turn cost delta, files-touched diff) can be added as additional Observe* methods as new signals need them.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL