persona-probe

Overview ¶

persona-probe measures whether deleting the persona line ("You are a helpful assistant. Be concise and accurate.") from the system prompt regresses Gemini — the #459 merge gate (resolved question 1 in docs/system-prompt-layering-design.md). Same posture and evidence standard as dev/parallel-probe: run a real tool workflow under both prompt variants, emit per-run behavioral metrics, and eyeball the delta before committing.

The layered prompt (agent.CoreInstruction + quirks + overlay) ships WITHOUT the persona line; if this probe shows a regression on Gemini — degraded task completion, runaway verbosity, tool-use breakdown — the line ships as a GeminiPersonaQuirk with the probe result cited in its doc comment. The layer architecture is identical in both outcomes.

Usage (from the repo root, with Vertex env sourced):

go run ./dev/persona-probe                       # layered prompt, no persona
go run ./dev/persona-probe --persona             # same, with the persona line appended
go run ./dev/persona-probe --task=multiread [--persona]
go run ./dev/persona-probe --runs=3 [...]        # repeat for stability

Compare the metric lines (turns, tool calls, output chars, final answer present) and the transcripts across the two variants. No automated verdict on purpose — the parallel-probe precedent is human-eyeballed evidence recorded in the doc comment of whatever ships.