watchdog

type Alert ¶

type Alert struct {
	Signal   string
	Severity Severity
	Reason   string
	Guidance string
}

Alert is what a triggered signal returns. Signal is the stable string ID the rest of the system can dispatch on (future "auto" mode picks behavior per signal).

Reason and Guidance are the same observation addressed to two different readers, and the split is load-bearing. Reason is operator-facing — the agent's wiring logs it verbatim, and it may name operator affordances ("/interrupt", "--max-turn-cost-usd") that only a human at a terminal can act on. Guidance is model- facing: what the *agent* should do differently on its next turn, written as an instruction with no reference to operator controls, because under --watchdog=feedback (#159) it is injected into the model's next-turn context. An unattended daemon has no operator to read Reason, so for it Guidance is the only half that does work.

Guidance is optional. A third-party Signal that leaves it empty still produces feedback — FormatFeedback falls back to Reason — but the fallback carries operator advice the model can't take, so built-in signals should always set it.

func (a Alert) String() string

type AlternatingCycleSignal ¶ added in v2.9.0

type AlternatingCycleSignal struct {
	// MaxPeriod is the longest cycle considered. Periods start at 2 —
	// period 1 is a consecutive repeat, which RepeatedToolCallSignal
	// owns.
	MaxPeriod int
	// Cycles is how many consecutive repetitions of the block are
	// required before the signal trips.
	Cycles int
	// contains filtered or unexported fields
}

AlternatingCycleSignal trips when the agent repeats the same short *sequence* of tool calls over and over — a → b → a → b → a → b (#649).

This is the evasion the consecutive-repeat detector named in its own docstring and could not catch: "consecutive" means a run of one call, so any loop with a second call wedged in it reads as varied activity. The shape is not hypothetical. The live UAT that motivated the enforce-mode backstop (#623) was an agent cycling list_agents → check_agent, and it survived both an operator "stop" and /interrupt.

Detection is a period scan over the recent call history: for each period p in 2..MaxPeriod, the last p*Cycles observations trip the signal when they consist of Cycles byte-identical blocks of length p. Keys are canonicalized (canonicalArgs), so a cycle that alternates "./main.go" with "main.go" still reads as one call — but unlike the repeat detector this cannot use the pairwise path-suffix relation, which is not transitive and therefore cannot key a history buffer.

Two deliberate limits on false positives, because a Critical alert halts the agent under --watchdog=enforce:

• A block whose entries are all the same call is skipped. It is a plain repeat, already the other signal's job, and alerting twice for one pattern doubles the noise and (under feedback mode) the tokens.
• Cycles defaults to 3, i.e. six calls for the a→b→a→b→a→b shape. Two passes through a sequence is normal work — read a file, grep, read another, grep again. Three passes with byte-identical arguments each time is not: nothing in the inputs changed, so nothing in the results can have.

A polling loop written as alternating tool calls is the known false positive. That is what wait_and_verify (#648) exists for, and an operator who wants the pattern anyway can drop the signal by constructing DefaultWatchdog with their own signal list.

func NewAlternatingCycleSignal(maxPeriod, cycles int) *AlternatingCycleSignal

func (s *AlternatingCycleSignal) Name() string

func (s *AlternatingCycleSignal) ObserveToolCall(tc ToolCall) *Alert

func (s *AlternatingCycleSignal) Reset()

type DefaultWatchdog ¶

type DefaultWatchdog struct {
	// contains filtered or unexported fields
}

DefaultWatchdog is the package-default implementation. Fans observations across the configured signals; Check collects alerts from each.

func NewDefaultWatchdog() *DefaultWatchdog

func (w *DefaultWatchdog) Check() []Alert

func (w *DefaultWatchdog) ObserveToolCall(tc ToolCall)

func (w *DefaultWatchdog) ObserveToolResult(tr ToolResult)

func (w *DefaultWatchdog) Reset()

type DominantToolCallSignal ¶ added in v2.9.0

type DominantToolCallSignal struct {
	// Window is how many recent tool calls are considered.
	Window int
	// Threshold is how many of those must be the same call.
	Threshold int
	// DeferRun is the consecutive-run length that hands the window
	// back to RepeatedToolCallSignal: when the dominant call has ever
	// reached a run that long inside the current window, this signal
	// stays quiet, because that run is what the repeat detector
	// exists to catch and one behavior should not raise two alerts —
	// under --watchdog=feedback a duplicate alert is duplicated
	// prompt text, not just a duplicated log line.
	//
	// This is a deliberate, explicit coupling to the repeat
	// detector's threshold rather than a guess: AlternatingCycleSignal
	// makes the same deference structurally, by skipping uniform
	// blocks. Zero or negative disables it, which is what an operator
	// wiring this signal *without* the repeat detector wants.
	DeferRun int

	// DeferCyclePeriod is the same deference toward
	// AlternatingCycleSignal: when the whole window is a clean
	// repetition of a block of period 2..DeferCyclePeriod, that
	// detector owns it. Without this, a → a → b → a → a → b reads as
	// a dominant call (8 of 12 are `a`) *and* as a 3-call cycle, and
	// would raise two Critical alerts for one loop. Zero or negative
	// disables it.
	//
	// This is not a hole: the cycle detector reaches such a window
	// first — three laps of a 3-call block is nine calls, where the
	// density threshold needs twelve — so deferring costs nothing but
	// the duplicate. A period longer than the cycle detector scans
	// (a → a → a → a → b, period 5) does not match here and is
	// exactly the case this signal is for.
	DeferCyclePeriod int
	// contains filtered or unexported fields
}

DominantToolCallSignal trips when a single tool call accounts for Threshold of the last Window observations — the interleaved loop neither the consecutive-repeat nor the alternating-cycle detector converges on quickly (#702).

Args are compared canonicalized (canonicalArgs, see canonical.go), the same treatment AlternatingCycleSignal uses and for the same reason: counting occurrences means keying a map, and the repeat detector's pairwise path-suffix relation is not transitive, so it cannot key one.

Severity is Critical, matching the other two loop detectors. The justification is theirs as well: the dominant call's arguments are equivalent every time, so its results are too, and an agent spending two thirds of a window re-asking one answered question is not making progress.

The known false positive is a hand-rolled polling loop — the same one the cycle detector documents, and the same answer: wait_and_verify (#648) is the supported way to wait, and an operator who wants the pattern anyway constructs DefaultWatchdog with their own signal list. Note that a poll tight enough to put 8 identical calls in a 12-call window already trips the repeat detector at 5 consecutive unless something interleaves, so this signal does not make a legitimate poll materially more halt-prone than it already was.

func NewDominantToolCallSignal(window, threshold, deferRun, deferCyclePeriod int) *DominantToolCallSignal

func (s *DominantToolCallSignal) Name() string

func (s *DominantToolCallSignal) ObserveToolCall(tc ToolCall) *Alert

func (s *DominantToolCallSignal) Reset()

type RepeatedToolCallSignal ¶

type RepeatedToolCallSignal struct {
	Threshold int
	// contains filtered or unexported fields
}

RepeatedToolCallSignal trips when the same (name, args) tool call appears Threshold times consecutively. Catches the read_file loop pattern from issue #144 and similar runaway-tool-call shapes.

"Consecutive" is the key word: a → b → a → b → a doesn't trip (no run of identical calls), but a → a → a → a → a does. This matches operator intuition ("the agent is stuck on the same thing") without flagging legitimate patterns like alternating-tool exploration loops.

Args comparison is path-canonicalized (#649, see canonical.go), not literal-string as in v1: "main.go", "./main.go" and "/workspace/main.go" are one call, because an agent re-reading one file under three spellings is as stuck as one re-reading it under the same spelling. Everything else still compares exactly — a detector that generalizes too eagerly halts legitimate work.

func NewRepeatedToolCallSignal(threshold int) *RepeatedToolCallSignal

func (s *RepeatedToolCallSignal) Name() string

func (s *RepeatedToolCallSignal) ObserveToolCall(tc ToolCall) *Alert

func (s *RepeatedToolCallSignal) Reset()

type RepeatedToolNameSignal ¶ added in v2.9.0

type RepeatedToolNameSignal struct {
	// Threshold is the consecutive same-name run length that trips.
	Threshold int

	// DeferRun is the run length that hands the window back to
	// RepeatedToolCallSignal. When the current run is not just
	// same-name but same-name-and-same-args, that detector owns it and
	// has already alerted at its own (lower) threshold; re-reporting
	// here would put two alerts, and under --watchdog=feedback two
	// blocks of prompt text, in front of one behavior. It would also be
	// the weaker report shadowing the stronger one — that detector's
	// alert is Critical and can prove the calls are identical, which
	// this one cannot. Mirrors DominantToolCallSignal.DeferRun,
	// including the convention that zero or negative disables the
	// deference for an operator wiring this signal without the repeat
	// detector.
	DeferRun int
	// contains filtered or unexported fields
}

RepeatedToolNameSignal trips when the same tool NAME is called Threshold times consecutively, whatever the arguments.

Deliberately the coarsest of the loop detectors. It exists for the shape the other three structurally cannot see: a loop whose arguments change every iteration without the call meaning anything different. Free-text arguments are the common source — a summary, a goal, a message, a commit note — because the model rephrases them naturally and each rephrasing defeats an args-keyed comparison.

Severity is Warn — the only loop detector that is not Critical, and deliberately so. The other three assert something they can prove: the calls are identical, so the agent is provably learning nothing. This one cannot. A long same-name run is a loop *or* a sweep, and the signal has no way to tell which. Under --watchdog=enforce a Critical alert halts the agent, so being wrong here would stop an agent that was working — the backstop becomes the outage. ToolFailureStreakSignal reached the same conclusion from the same place ("halting three denials into a legitimate RBAC probe"), and this signal has strictly less information than that one does.

Warn still does work. It reaches the operator log in every mode, and under --watchdog=feedback it reaches the model's own next turn as Guidance — which is the party that can actually stop making the call. An unattended daemon gets the feedback path and no halt, which is the right trade for an inference this soft.

Known false positive, now survivable: a genuine sweep longer than Threshold — an agent reading twenty files in a row, or issuing twenty reads against twenty distinct resources. It costs a log line and a paragraph of next-turn context, not a halt. This is why the alert text names the possibility instead of asserting a loop, and why Guidance tells a sweeping agent to keep going. An operator who runs workloads with long single-tool sweeps and wants silence should construct DefaultWatchdog with their own signal list, or raise Threshold.

func NewRepeatedToolNameSignal(threshold, deferRun int) *RepeatedToolNameSignal

func (s *RepeatedToolNameSignal) Name() string

func (s *RepeatedToolNameSignal) ObserveToolCall(tc ToolCall) *Alert

func (s *RepeatedToolNameSignal) Reset()

type Severity ¶

type Severity string

Severity classifies the urgency of an alert. Warn is the operator- visible-but-not-action-blocking level; Critical marks a runaway the agent should act on — under --watchdog=enforce (#623) a Critical alert halts the agent (refuses further turns until the operator resets), while warn mode logs it like any other alert.

const (
	SeverityWarn     Severity = "warn"
	SeverityCritical Severity = "critical"
)

type Signal ¶

type Signal interface {
	// Name returns the stable signal ID used in Alert.Signal.
	Name() string

	// ObserveToolCall updates the signal's internal state with one
	// tool invocation. Returning a non-nil Alert means the signal
	// tripped on this observation; DefaultWatchdog appends it to
	// the pending-alerts buffer.
	ObserveToolCall(ToolCall) *Alert

	// Reset clears the signal's state. Called from
	// DefaultWatchdog.Reset.
	Reset()
}

Signal is the per-detector interface inside DefaultWatchdog. Each signal owns its own state and decides when to emit an alert. Implementations must be safe to call serially from DefaultWatchdog (which holds a mutex across observations); they do NOT need to be concurrency-safe themselves.

Adding a new signal: implement Signal, append to NewDefaultWatchdog's signal list (or to a constructor variant). No changes to DefaultWatchdog itself.

type SignalResultObserver ¶ added in v2.9.0

type SignalResultObserver interface {
	ObserveToolResult(ToolResult) *Alert
}

SignalResultObserver is the same extension one level down, for signals inside DefaultWatchdog. A Signal that doesn't implement it simply never sees results.

type ToolCall ¶

type ToolCall struct {
	Name string
	Args string
}

ToolCall is the per-tool-call observation the watchdog needs. Name is the canonical tool name (e.g. "read_file", "mcp.gke.list_clusters"). Args is the JSON-serialized argument blob, passed through as the caller produced it: the detectors canonicalize path-shaped values themselves (#649, canonical.go) rather than requiring every caller to agree on a normal form, and a third-party Watchdog implementation still sees the raw args.

type ToolFailureStreakSignal ¶ added in v2.9.0

type ToolFailureStreakSignal struct {
	Threshold int
	// contains filtered or unexported fields
}

ToolFailureStreakSignal trips when Threshold tool calls in a row all return errors. Any successful call resets the run — the point is "this agent currently has no verified evidence," and one success is evidence.

Severity is Warn, not Critical, and that is a decision rather than an oversight. Under --watchdog=enforce a Critical alert halts the agent, and since #642 enforce is the default for unattended runs; halting a daemon three denials into a legitimate RBAC probe would make the backstop the outage. A failure streak is an evidence problem, so it goes where evidence problems belong — the operator log, and (under --watchdog=feedback) the model's own next turn. Runaway *behavior* is already Critical via the loop detectors.

func NewToolFailureStreakSignal(threshold int) *ToolFailureStreakSignal

func (s *ToolFailureStreakSignal) Name() string

func (s *ToolFailureStreakSignal) ObserveToolCall(ToolCall) *Alert

func (s *ToolFailureStreakSignal) ObserveToolResult(tr ToolResult) *Alert

func (s *ToolFailureStreakSignal) Reset()

type ToolResult ¶ added in v2.9.0

type ToolResult struct {
	Name  string
	Error string
}

ToolResult is the outcome half of a tool invocation. Error is the tool's error text, empty for success — the ADK convention is a reserved "error" key inside FunctionResponse.Response, and the agent wiring flattens that to this field so the watchdog never has to know the provider's response shape.

func (r ToolResult) Failed() bool

type ToolResultObserver ¶ added in v2.9.0

type ToolResultObserver interface {
	ObserveToolResult(ToolResult)
}

ToolResultObserver is the optional half of Watchdog: an implementation that also wants to see tool *outcomes* implements it, and the agent feeds results through a type assertion.

Deliberately not folded into Watchdog. That interface is documented as a plug-in point ("consumers can plug in their own implementation"), so widening it would break every third-party watchdog at a minor version to add one signal — and a custom watchdog that only counts calls stays perfectly valid.

type Watchdog ¶

type Watchdog interface {
	// ObserveToolCall records one tool invocation. Called by the
	// agent's event-tap as tool calls stream by; safe to call from
	// any goroutine.
	ObserveToolCall(ToolCall)

	// Check returns alerts triggered since the last Check call and
	// resets the per-call alert buffer. Returns nil when no signal
	// has tripped. Typically called from the agent's post-turn
	// hook; an alert returned here is "for the turn just ended."
	Check() []Alert

	// Reset clears all accumulated state. Called when the agent
	// resets (e.g. via a hypothetical /clear that clears history)
	// so signals don't carry across a logical session boundary.
	Reset()
}

Watchdog observes per-turn telemetry and returns any alerts that triggered during observation. Implementations must be safe for concurrent use — the agent calls Observe* methods from the streaming event handler and Check from the post-turn hook; concurrency is bounded but real.

The interface is intentionally narrow for v1: just tool-call observation + alert reporting. Richer telemetry (turn timing, per-turn cost delta, files-touched diff) can be added as additional Observe* methods as new signals need them.