runner

package
v3.0.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package runner drives the agent through a real conversation — either a single one-shot prompt (Headless) or a multi-turn stdin REPL (REPL). Both share an Agent under the hood and route partial text + tool-call summaries through the same streaming consumer.

Wake-loop health, and the one thing /healthz can honestly say about it (#978).

A wake loop cannot die — it logs every turn error and goes back to blocking, which is right for liveness and useless for recovery. From outside, a daemon whose every turn fails looks exactly like a daemon with nothing to do: a running pod, a green readiness probe, and no work getting done. This file is the state that tells those apart.

The reporting granularity is deliberate and is the one design decision here worth arguing about. A multi-session daemon hosts many loops, and the tempting shape — one HealthCheck per session — is wrong twice over: the check list would grow without bound under a kubelet re-probing it on a fixed period, and HealthCheck.Name is documented as a compile-time constant precisely because the endpoint is unauthenticated and a session id is not the caller's business. So the daemon registers ONE aggregate check and the per-session detail goes to the daemon log, where the healthz handler already writes the error text for a failing check and the operator reading it has already authenticated to the node.

The aggregate fails only when every live loop is failing. One session whose RBAC was revoked must not pull a pod serving forty-nine healthy ones out of its Service — readiness is a routing decision, not a notification channel. "Every loop is failing with nothing getting done" is the condition the issue actually describes, and it is the one a readiness gate should act on.

And it takes unhealthyAfterFailures consecutive failures, not one. The evidence from #935 is that a single provider 429 kills a turn roughly every six minutes under load; descheduling a pod for one of those would make the readiness gate a worse signal than no gate, because an operator who watches it flap learns to ignore it.

Index

Constants

View Source
const (
	ExitOK          = 0
	ExitAgentError  = 1
	ExitConfigError = 2
)

Exit codes — kept distinct so CI can disambiguate failure modes.

Variables

View Source
var ErrNotTerminal = errors.New("runner: stdin is not a terminal; turnInterrupter unavailable")

ErrNotTerminal is returned when newTurnInterrupter is called with a stdin that isn't a terminal. The REPL falls back to legacy behavior in that case (no mid-turn cancel; the existing process- level SIGINT continues to mean "exit").

Functions

func AnsiMagenta

func AnsiMagenta() string

AnsiMagenta returns the ANSI escape used for ↪-prefixed lines so downstream alert sinks can match the runner's color choice without duplicating the constant.

func FormatAlertLine

func FormatAlertLine(from, kind, text string) string

FormatAlertLine renders one background-agent Alert as a single "↪ <from> <kind>: <text>" line in the same shape as the ↪ google_search lines from v1.1.0. Reuses the magenta sigil so consumers see a consistent "server-side / out-of-band activity" signal regardless of source.

Exposed so consumers wiring their own alert sinks (a Slack adapter, a web UI, a logger) format alerts consistently with what the CLI shows. Use with paint(line, runner.AnsiMagenta(), color) when you want ANSI styling; raw string otherwise.

func Headless

func Headless(ctx context.Context, m adkmodel.LLM, prompt string, stdout, stderr io.Writer, tracker *usage.Tracker, pricing usage.Pricing, agentOpts []agent.Option, eventsOpts ...EventsOption) (int, error)

Headless executes prompt against m and streams the assistant's text to stdout as partial events arrive. Tool-call summaries are written to stderr as one line per call. Returns an exit code suitable for os.Exit.

agentOpts lets the caller pass extra agent.Options (typically WithTools, WithToolsets, WithSystemInstructionPrefix). Pass nil for no tools and the default instruction.

tracker (optional) records per-turn usage; when supplied, the caller can write a summary using its totals after Headless returns. Pass nil to skip accounting.

eventsOpts forwards through to WriteEvents (e.g. WithColor) so callers can opt into ANSI styling without reaching into the formatter directly.

A trailing newline is always added to stdout when at least one chunk was written, so shell pipelines see a clean terminator.

func IsTerminal

func IsTerminal(w io.Writer) bool

IsTerminal reports whether w is connected to a terminal (TTY). Use to gate WithColor: pass WithColor(IsTerminal(os.Stdout)) so color renders interactively but not when output is captured to a file or piped to another process.

Returns false for any writer that isn't a *os.File (buffers, pipes, network connections). On Unix and modern Windows, character devices report ModeCharDevice; that's the signal we trust.

func REPL deprecated

func REPL(ctx context.Context, m adkmodel.LLM, stdin io.Reader, stdout, stderr io.Writer, tracker *usage.Tracker, pricing usage.Pricing, agentOpts []agent.Option, eventsOpts ...EventsOption) (int, error)

REPL drives a multi-turn stdin loop against m. Each line is sent through the same Agent so the ADK runner accumulates conversation history across turns (the in-memory session service appends events per session ID, which the Agent reuses).

Prompts are written to stdout, partial assistant text streams back inline, and tool-call summaries go to stderr — same shape as Headless. Built-in commands: `/exit`, `/quit`, EOF (Ctrl-D).

agentOpts mirrors Headless. The same tracker/pricing pair is used across every turn so the final summary is meaningful. eventsOpts (e.g. WithColor) forward through to WriteEvents for every turn.

SIGINT (Ctrl+C) is owned by the REPL when stdin is a TTY: the double-press-within-1s exits gesture works both during a turn (the per-turn turnInterrupter handles raw-mode bytes) and between turns (this function's signal handler manages the SIGINT state machine). The bundled CLI's main.go does NOT include SIGINT in its signal.NotifyContext for that reason; passing a ctx whose parent cancels on SIGINT would defeat the gesture.

Deprecated: use Run with RunOptions{Model: m, AgentOptions: agentOpts, …}.

func REPLWithAgent deprecated

func REPLWithAgent(ctx context.Context, a *agent.Agent, m adkmodel.LLM, stdin io.Reader, stdout, stderr io.Writer, tracker *usage.Tracker, pricing usage.Pricing, eventsOpts ...EventsOption) (int, error)

REPLWithAgent runs the REPL against a pre-constructed Agent. Useful for tests that need a reference to the agent (to call Inject from outside the loop, for example) and for library consumers that construct the Agent themselves. Equivalent to REPL minus the agent.New() call at the top.

Deprecated: use Run with RunOptions{Agent: a, Model: m, …}.

func REPLWithAgentAndInitialPrompt deprecated

func REPLWithAgentAndInitialPrompt(ctx context.Context, a *agent.Agent, m adkmodel.LLM, initialPrompt string, stdin io.Reader, stdout, stderr io.Writer, tracker *usage.Tracker, pricing usage.Pricing, eventsOpts ...EventsOption) (int, error)

REPLWithAgentAndInitialPrompt is REPLWithAgent plus the seed-prompt behavior of REPLWithInitialPrompt: when initialPrompt is non-empty, exactly one turn fires before the stdin/wake select loop. Exists so hosts that construct (and decorate — e.g. wrap with pkg/attachadapter and register for attach-mode) the Agent themselves keep the --prompt startup behavior the construct-inside variants provide.

Deprecated: use Run with RunOptions{Agent: a, Model: m, InitialPrompt: initialPrompt, …}.

func REPLWithInitialPrompt deprecated

func REPLWithInitialPrompt(ctx context.Context, m adkmodel.LLM, initialPrompt string, stdin io.Reader, stdout, stderr io.Writer, tracker *usage.Tracker, pricing usage.Pricing, agentOpts []agent.Option, eventsOpts ...EventsOption) (int, error)

REPLWithInitialPrompt behaves like REPL but seeds the first turn with initialPrompt before entering the main select loop (issue #291). The seed runs through the same runREPLTurn used mid-loop, so ESC interrupt, usage tracking, and tool-approval prompts all work identically to a user-typed submission.

initialPrompt == "" is exactly equivalent to REPL — no seed, no behavior change.

Deprecated: use Run with RunOptions{Model: m, InitialPrompt: initialPrompt, …}.

func Run

func Run(ctx context.Context, opts RunOptions) (int, error)

Run drives the multi-turn stdin REPL described on REPL, configured by a single options struct. Returns the process exit code and any terminal error, same contract as the deprecated positional variants it replaces.

func WakeLoop

func WakeLoop(ctx context.Context, a *agent.Agent, opts WakeLoopOptions)

WakeLoop is the wake-driven inbox drain every headless agent surface runs: block until an attach client's POST /inject (or any other Inject caller) fires WakeRequested, run one empty-prompt turn so the inbox drains into a real model turn, account its usage, repeat. Returns when ctx is cancelled.

The empty prompt means "no user text this turn, just drain the inbox" — the same path the REPL uses between submissions. The turn's events flow through the eventlog to the attach broadcaster, which is what a remote operator's TUI renders.

This consolidates the previously duplicated loops in cmd/core-agent (the --no-repl inline loop and the per-session multi-session loop) behind pkg/runner, which already owns the "drive the agent through a conversation" surface. Extracted as part of the pkg/compose work (#386, docs/compose-extraction-design.md).

Repeated failure (#978)

A turn that ends in a fault increments a consecutive-failure count and moves opts.Health to LoopFailing; any clean turn resets both. From the second consecutive failure the loop holds off for a capped, doubling interval before driving another, which is what stops a persistent fault — revoked RBAC, a dead credential, an exhausted quota — from burning one turn per arriving event forever. Operator cancellations and guardrail halts do not count; see countsAsFailure.

What the loop deliberately does NOT do is give up. #978 asks it to "stop after a bound", and there is no honest way to do that here:

  • Returning closes the inbox (the defer below), so every later inject fails with ErrInboxClosed — including the operator's message telling the session what to do about the failure. The loop would destroy the only channel through which it could be fixed.
  • Staying parked while refusing to drive turns is worse in a quieter way. The inbox is bounded and drops the OLDEST message when full, so a producer pointed at a stopped loop silently loses its earliest signals — the #1040 harm, except #1040 is safe to fence precisely because a guardrail halt has a reset verb that releases it. A loop that stopped itself has none.

So the bound is expressed as a RATE, not as a stop: at most one failing turn per MaxFailureBackoff, indefinitely, with the condition visible on GET /healthz the whole time. An operator who wants it to actually stop has the verbs for that already — pause the session, or delete it.

The cap is minutes rather than hours for the same drop-oldest reason. A hold-off IS a park, just a bounded one, and the inbox keeps filling through it; the next turn drains the lot, so nothing is lost as long as the wait stays short against how fast a producer can push 256 messages. Raising MaxFailureBackoff past that trades the rate limit for the harm it was avoiding.

func WriteEvents

func WriteEvents(events iter.Seq2[*session.Event, error], out, info io.Writer, opts ...EventsOption) error

WriteEvents formats events from agent.Run(...) for human-readable streaming display. It's the library-friendly counterpart to the formatter inside Headless / REPL — library callers who want their agent loop to look like an interactive chat session can pass the returned iterator straight in.

Routing:

  • Partial assistant text (event.Partial == true) streams to out as it arrives, with no prefix — so a model's reply renders character-by-character like an interactive chat.
  • Tool calls render as `→ name(key=value, ...)` to info.
  • Tool responses render as `← name(key=value, ...)` to info.
  • Server-side built-in evidence (Gemini grounding metadata) renders as `↪ google_search: ...` to info, AFTER the model's text, since the metadata only lands on the aggregated final event.
  • Final TurnComplete events are skipped (they repeat the text already streamed via Partial events).

Tool → / ← lines render off non-partial events only. ADK's streaming aggregator yields one partial event per chunk and then one aggregated event carrying the whole model call, so a streamed tool call reaches WriteEvents twice; the aggregate carries every part and is the event the runtime executes the tool from, so rendering that one is both complete and exactly once (#926). Repeated → / ← lines are NOT deduplicated: an agent calling the same tool with the same args five times in a row is a runaway loop, and the whole value of watching the stream is seeing it happen (a content key cannot tell "the same part twice" from "the same call twice" — the distinction #915 turned on).

Identical ↪ lines ARE still deduplicated within one WriteEvents call: Vertex re-emits the same grounding metadata across events, and unlike a tool call there is no operator signal in seeing the same search evidence twice.

out and info may point at the same writer (e.g. both os.Stdout) when you want a single combined stream — useful for tmux capture or piping. They're separate parameters so the default CLI path can keep tool chatter on stderr away from the assistant's reply on stdout.

Pass WithColor(true) to enable ANSI styling — typically gated on IsTerminal(out) so piped output stays clean.

Returns the first iterator error, or nil on clean completion. A trailing newline is written to out if any text was streamed.

func WriteSummary

func WriteSummary(w io.Writer, t *usage.Tracker, modelID string)

WriteSummary emits a one-line usage tally suitable for shell pipelines / CI logs. No-op when no turns were recorded.

When the resolved pricing for the model used during the session is zero (e.g. a custom / fine-tuned variant not in the built-in rate table and without a cfg.Model.Pricing override), we surface "$—" rather than "$0.0000" — the latter implied "free", which is misleading and historically confusing for operators who actually spent money.

Types

type EventsOption

type EventsOption func(*eventsConfig)

EventsOption configures WriteEvents. Use the With* helpers below.

func WithColor

func WithColor(on bool) EventsOption

WithColor enables ANSI color codes in WriteEvents output. Tool calls and responses render in cyan; partial assistant text in green. Off by default — colored output looks like garbage when piped to a file, so callers must opt in (typically guarded by IsTerminal(out)).

type LoopHealth

type LoopHealth struct {
	// contains filtered or unexported fields
}

LoopHealth is one wake loop's health, safe for concurrent reads while the loop writes it. The zero value is usable and reports an idle loop with no session name; WakeLoop fills the session in when it starts.

func NewLoopHealth

func NewLoopHealth(session string) *LoopHealth

NewLoopHealth returns a handle for a loop on the named session.

func (*LoopHealth) Snapshot

func (h *LoopHealth) Snapshot() LoopStatus

Snapshot returns the current reading. Nil-safe.

type LoopHealthSet

type LoopHealthSet struct {
	// contains filtered or unexported fields
}

LoopHealthSet is the daemon's registry of live wake loops, and the thing the aggregate /healthz check reads. The zero value is ready to use and every method is nil-safe, so a host that wants no health reporting leaves the field unset.

func (*LoopHealthSet) Err

func (s *LoopHealthSet) Err() error

Err reports the aggregate condition: non-nil only when at least one loop is live and EVERY live loop has failed unhealthyAfterFailures turns in a row without a clean one since.

The partial case returns nil on purpose. A readiness probe decides whether to route traffic to this pod, and one broken session is not a reason to stop serving the others — the operator learns about it from the daemon log, which is where the detail can safely go. See the file comment.

Three details are easy to get wrong and all three are deliberate. The predicate is the failure COUNT, not LoopFailing: a loop that is mid-retry reads as LoopWorking, and a probe landing inside that turn must not see a broken session as recovered. A stopped loop is skipped rather than counted as healthy — it unregisters immediately in every in-tree host, but if one lingered, counting it would let a dead loop mask a genuine all-failing daemon. And a loop that has never completed a turn is skipped for the same reason in the other direction: it is not evidence of health, and counting it as such would silently disable this check on the deployment it was written for. A multi-session `--no-repl` daemon — every bundled GKE recipe — starts a bootstrap `default` session whose loop nothing ever drives, so "every live loop is failing" could never be true while it sat there reading idle.

func (*LoopHealthSet) Register

func (s *LoopHealthSet) Register(session string) (*LoopHealth, func())

Register adds a loop for the named session and returns it along with the func that removes it again. Call the remover when the loop returns: a stopped loop that stays registered would keep an evicted session's last failure in the aggregate forever, which is how a health signal becomes a thing operators learn to ignore.

func (*LoopHealthSet) Snapshot

func (s *LoopHealthSet) Snapshot() []LoopStatus

Snapshot returns a reading of every registered loop, ordered by session id so a log line is stable between probes.

type LoopState

type LoopState string

LoopState is the coarse condition of one wake loop.

const (
	// LoopIdle: parked on the wake channel with nothing to do. The
	// overwhelmingly common state, and indistinguishable from
	// LoopFailing to anything outside the process before #978.
	LoopIdle LoopState = "idle"
	// LoopWorking: a turn is running right now.
	LoopWorking LoopState = "working"
	// LoopFailing: consecutive turns have ended in a fault. The loop
	// is still listening and still accepting input — see WakeLoop for
	// why giving up is not an option it has.
	LoopFailing LoopState = "failing"
	// LoopStopped: ctx was cancelled and the loop returned. Its inbox
	// is closed; nothing more will run.
	LoopStopped LoopState = "stopped"
)

type LoopStatus

type LoopStatus struct {
	// Session is the loop's session id. Used for the daemon log and
	// never for the /healthz body.
	Session string
	State   LoopState
	// ConsecutiveFailures counts TURNS, not yielded errors: a turn
	// that yields three errors is one failure. Reset by any turn that
	// ends clean.
	ConsecutiveFailures int
	// LastErrorKind is the stable attach.ClassifyTurnError kind of the
	// most recent counted failure, not the provider's prose. The prose
	// can carry a DSN or a hostname; the kind is an enum.
	LastErrorKind string
	// TurnsObserved counts completed turns of every outcome. Zero means
	// the loop has never run one, which is the difference between "this
	// session is healthy" and "this session has never been asked to do
	// anything" — see LoopHealthSet.Err.
	TurnsObserved int
	// Since is when the loop entered State.
	Since time.Time
}

LoopStatus is a point-in-time reading of one wake loop.

type RunOptions

type RunOptions struct {
	// Model drives every turn. Required when Agent is nil (it is
	// what the constructed Agent runs on). When Agent is non-nil,
	// leave Model nil — Run derives it from Agent.Model() (#510) —
	// or, if set, it MUST be the same model the agent was built
	// with; a mismatch produces confusing turns.
	Model adkmodel.LLM

	// Agent, when non-nil, is used as-is — for hosts that construct
	// (and decorate, e.g. wrap with pkg/attachadapter and register
	// for attach-mode) the Agent themselves. When nil, Run constructs
	// one from Model + AgentOptions.
	Agent *agent.Agent

	// AgentOptions configure the Agent Run constructs. Ignored when
	// Agent is non-nil.
	AgentOptions []agent.Option

	// InitialPrompt, when non-empty, seeds the first turn before the
	// stdin/wake select loop — the --prompt startup behavior. The
	// seed runs through the same turn machinery as a typed
	// submission (ESC interrupt, usage tracking, tool prompts).
	InitialPrompt string

	// Stdin / Stdout / Stderr default to the process's os.Stdin /
	// os.Stdout / os.Stderr when nil. Mid-turn ESC interrupt and the
	// double-Ctrl+C exit gesture require Stdin to be a real terminal
	// (*os.File on a TTY); anything else degrades gracefully.
	Stdin  io.Reader
	Stdout io.Writer
	Stderr io.Writer

	// Tracker + Pricing accumulate per-turn cost across the whole
	// session so the final summary is meaningful. Optional.
	Tracker *usage.Tracker
	Pricing usage.Pricing

	// EventsOptions forward to WriteEvents on every turn (e.g.
	// WithColor).
	EventsOptions []EventsOption
}

RunOptions configures Run, the consolidated multi-turn REPL entry point (#492). It replaces four positional variants (REPL, REPLWithInitialPrompt, REPLWithAgent, REPLWithAgentAndInitialPrompt) whose widest form had grown to ten parameters — and whose agent-taking forms invited an agent/model mismatch by accepting both.

type WakeLoopOptions

type WakeLoopOptions struct {
	// Tracker receives one AppendUsage per completed turn (keyed by
	// Model, priced by Pricing) via the usage.TurnTap discipline —
	// overwrite-per-event, commit exactly once on TurnComplete.
	// Nil disables accounting.
	Tracker *usage.Tracker
	Model   string
	Pricing usage.Pricing

	// OnTurnError is invoked for each error the turn iterator
	// yields. The loop always keeps running — one bad turn must not
	// kill an attach-only daemon or a hosted session. Nil writes
	// "core-agent: session <sid> turn: <err>" to stderr, matching
	// the historical inline loops.
	OnTurnError func(error)

	// Health, when non-nil, is updated with this loop's state after
	// every turn so GET /healthz can tell "idle, waiting for events"
	// from "awake and failing repeatedly" (#978). Register it with the
	// daemon's LoopHealthSet to have it counted in the aggregate check.
	// Nil disables health reporting and changes nothing else.
	Health *LoopHealth

	// FailureBackoff is the base wait after a SECOND consecutive failed
	// turn; each further failure doubles it up to MaxFailureBackoff.
	// Zero means defaultFailureBackoff. Negative disables the backoff
	// entirely, which is only sensible in a test.
	FailureBackoff time.Duration

	// MaxFailureBackoff caps the doubling. Zero means
	// defaultMaxFailureBackoff.
	MaxFailureBackoff time.Duration

	// Debugf, when non-nil, receives trace-level lifecycle lines
	// (loop start/stop, wake fired, turn finished). Wire the host's
	// debug logger; nil is silent.
	Debugf func(format string, args ...any)
	// contains filtered or unexported fields
}

WakeLoopOptions configures WakeLoop. The zero value is usable: no usage accounting, turn errors to stderr, no debug tracing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL