Documentation
¶
Overview ¶
Package testutil contains shared helpers for agent-deck's test suites.
Performance regression API (this file) ======================================
All TestPerf_* tests use this small set of primitives. The design has three deliberate parts:
- Cold vs warm budget classification.
- n=11 trimmed-mean measurement.
- 1 ms budget floor.
── Cold vs warm ────────────────────────────────────────────────────────
The budget formula differs based on whether the test crosses a process or syscall boundary that introduces external variance the test code can't control:
COLD tests use ColdBudget(t, base) = max(base × 5, 1 ms) × multiplier. For: cold-start exec, real-disk fsync (Tier 2 — see docs/perf-budget-suite.md), child-process spawn, network round-trips. The 5× factor reflects that runner / loader / fsync variance can't be capped tighter without false positives.
WARM tests use WarmBudget(t, base) = max(base × 3, 1 ms) × multiplier. For: pure in-process Go work measurable under controlled GC. The tighter 3× factor is safe because TrimmedMeanWarm forces a GC cycle and disables auto-GC during each timed window — the largest noise source is eliminated. WARM tests run on tmpfs / in memory (Tier 1) so disk speed is irrelevant.
Both formulas are scaled by PERF_BUDGET_MULTIPLIER (default 1.0; CI sets 2.0 to absorb shared-runner variance). The effective CI gate is 10× local for cold, 6× local for warm.
── n=11 trimmed mean ───────────────────────────────────────────────────
TrimmedMean runs n=11 timed iterations, sorts, drops the top 2 and bottom 2 samples, and averages the middle 7 (a 36% trimmed mean).
Why n=11:
- Odd: median is well-defined as a fallback diagnostic.
- Cheap: 11 × ~10 ms typical = ~110 ms per test, well under the 60 s perf-suite timeout.
- Drop 2/2: handles one GC pause + one scheduler hiccup without losing too many samples. In practice, ~1 in 10 samples on a loaded host is an outlier; dropping 2 from each end is robust.
- Middle 7: variance of the mean scales as 1/√7 ≈ 0.38 — about 2× noise reduction vs a single sample, ~1.5× better than the median of 5 used previously.
Larger n (21, 51) was considered but rejected: marginal variance reduction (~30%) at 2–5× test cost. n=11 is the sweet spot.
── 1 ms floor ──────────────────────────────────────────────────────────
PerfBudgetFloor caps the minimum budget at 1 ms regardless of the caller's base × multiplier. Anything faster is either "just fast" (sub-millisecond timing is dominated by clock resolution + scheduler jitter) or a sign that the unit under test is too small to be a meaningful regression target. Move such tests to Benchmark* (Track A) instead of TestPerf_* (Track B).
Index ¶
- Constants
- func CleanGitEnv(base []string) []string
- func ColdBudget(t *testing.T, base time.Duration) time.Duration
- func IsolateHome() func()
- func IsolateTmuxSocket() func()
- func ShortTmuxSocket() (socket string, cleanup func())
- func SkipIfShort(t *testing.T)
- func TrimmedMean(fn func()) time.Duration
- func TrimmedMeanWarm(fn func()) time.Duration
- func TrimmedMeanWarmWithSetup(setup, op func()) time.Duration
- func TrimmedMeanWithSetup(setup, op func()) time.Duration
- func UnsetGitRepoEnv()
- func WarmBudget(t *testing.T, base time.Duration) time.Duration
- type FsyncCounter
Constants ¶
const HomeIsolationMarkerEnv = "AGENT_DECK_TEST_HOME_ISOLATED"
HomeIsolationMarkerEnv is set during HOME+XDG isolation. Runtime guards (and the pathsafety guard test) read this to confirm a test context is sandboxed.
const PerfBudgetFloor = 1 * time.Millisecond
PerfBudgetFloor is the minimum budget any TestPerf_* gate may apply. Sub-millisecond budgets are dominated by clock resolution and scheduler jitter, not by the unit under test. See package docblock for rationale.
const PerfBudgetMultiplierEnv = "PERF_BUDGET_MULTIPLIER"
PerfBudgetMultiplierEnv is the env var read by ColdBudget and WarmBudget. CI sets this to 2.0 to absorb shared-runner variance; developers on slow laptops can bump it locally. Default is 1.0.
const RaceEnabled = false
RaceEnabled reports whether the binary was built with -race. See race_on.go.
const TestIsolationMarkerEnv = "AGENT_DECK_TEST_ISOLATED"
Name of the marker env var set during test isolation. Runtime guards in internal/tmux use this to detect a test context so they can panic loudly on an isolation leak instead of silently attacking the user's real tmux server.
Variables ¶
This section is empty.
Functions ¶
func CleanGitEnv ¶
CleanGitEnv returns a copy of base with git repository-routing env vars removed.
func ColdBudget ¶ added in v1.9.20
ColdBudget returns the budget for a COLD test: one that crosses a process or syscall boundary (cold-start exec, real-disk fsync, child-process spawn, network). Formula:
max(base * 5, PerfBudgetFloor) * PERF_BUDGET_MULTIPLIER
Pair with TrimmedMean(fn) for measurement.
func IsolateHome ¶ added in v1.9.48
func IsolateHome() func()
IsolateHome makes it safe for tests to resolve and write agent-deck runtime paths (~/.agent-deck/config.json, profiles/<p>/state.db, worker-scratch, logs, hooks) without ever touching the developer's real home directory.
WHY THIS EXISTS (2026-06-04 data-loss incident — third of its class):
`go test` resolves runtime paths via the HOME env var (os.UserHomeDir reads $HOME on Unix), NOT via the test's working directory. So running the suite from inside a git worktree still wrote to the real ~/.agent-deck. Test isolation up to that point relied solely on AGENTDECK_PROFILE=_test, which only scopes the *profile subdirectory* — GetAgentDeckDir(), config.json, worker-scratch/, and logs/ all still resolved under the real HOME. The concrete trigger: internal/ui's TestMain set AGENTDECK_PROFILE=_test but did NOT override HOME/XDG, so an un-sandboxed `go test ./internal/ui/...` wiped the live profile index + config.
IsolateHome closes that gap by pointing HOME at a fresh per-call temp dir, CLEARING every XDG_* base dir so they track HOME, and setting AGENTDECK_PROFILE=_test as a second belt. It mirrors IsolateTmuxSocket (internal/testutil/tmuxenv.go).
WHY XDG IS CLEARED, NOT PINNED (2026-06-07 ~96-test isolation regression):
IsolateHome runs once per package (from TestMain), so its temp dir is SHARED by every test in the package. After #1294's XDG refactor, path resolution prefers an XDG location *if the file already exists* on disk. Pinning XDG_CONFIG_HOME to <pkg-tempdir>/.config meant one test writing config.toml there left a stale file that LATER tests then read instead of their own fresh slate — config bled across tests (catalog "empty", profile resolving to `default`, ~96 cross-package failures). Worse, the many tests that swap only HOME via t.TempDir() left XDG_CONFIG_HOME still pointing at the shared package dir, so their per-test config redirection silently did nothing.
By clearing XDG_* instead, every base dir falls back to $HOME/.config, $HOME/.local/share, etc. (see agentpaths.xdgDir). Now any test that swaps HOME — which is the overwhelmingly common pattern — automatically gets a fully isolated, clean config/data/cache slate, with zero per-test wiring. Tests that genuinely need a specific XDG dir still set it explicitly via t.Setenv (auto-restored), which takes precedence.
It sets:
- HOME -> <tempdir> (os.UserHomeDir source on Unix)
- XDG_CONFIG_HOME -> "" (cleared; resolves under $HOME/.config)
- XDG_DATA_HOME -> "" (cleared; resolves under $HOME/.local/share)
- XDG_CACHE_HOME -> "" (cleared; resolves under $HOME/.cache)
- XDG_STATE_HOME -> "" (cleared; resolves under $HOME/.local/state)
- AGENTDECK_PROFILE -> _test
- AGENT_DECK_TEST_HOME_ISOLATED -> 1 (marker for guard/runtime checks)
Call it from every package-level TestMain that transitively resolves an agent-deck path:
func TestMain(m *testing.M) {
cleanupHome := testutil.IsolateHome()
defer cleanupHome()
cleanupTmux := testutil.IsolateTmuxSocket()
defer cleanupTmux()
os.Exit(m.Run())
}
Returns a cleanup function that removes the temp dir and restores the original env so the parent process is not permanently altered.
func IsolateTmuxSocket ¶ added in v1.7.3
func IsolateTmuxSocket() func()
IsolateTmuxSocket makes it safe to spawn real tmux servers from tests even when `go test` is invoked from inside a live tmux session (the default on every developer host that uses agent-deck).
The helper does THREE things:
Unsets TMUX and TMUX_PANE. Tmux's client discovery order is: `$TMUX → -S path → -L name → $TMUX_TMPDIR`. If TMUX is set, every later step is ignored — so setting TMUX_TMPDIR alone provides zero isolation when the test process inherits TMUX from a parent tmux pane. This was the 2026-04-17 three-cascade bug: v1.7.3 set TMUX_TMPDIR but left TMUX set, so every test-spawned tmux session joined the user's real server and eventually destabilised it.
Sets TMUX_TMPDIR to a fresh per-call temp dir. Tests that use `-L <name>` or `$TMUX_TMPDIR`-derived sockets will land here, never at /tmp/tmux-<uid>/default.
Sets AGENT_DECK_TEST_ISOLATED=1. Production code paths in internal/tmux read this marker at tmux-spawn time and panic with a clear message if TMUX is still set and points to a non-isolated socket — the "make the failure loud, not silent" belt to the TMUX-unset suspender.
Call this from every package-level TestMain that transitively spawns tmux:
func TestMain(m *testing.M) {
cleanup := testutil.IsolateTmuxSocket()
defer cleanup()
os.Exit(m.Run())
}
Returns a cleanup function that removes the temp dir and restores the original TMUX / TMUX_PANE / AGENT_DECK_TEST_ISOLATED values so the parent process's env is not permanently altered.
func ShortTmuxSocket ¶ added in v1.9.53
func ShortTmuxSocket() (socket string, cleanup func())
ShortTmuxSocket returns a tmux -S socket path under a short base dir (/tmp when writable, via shortTmuxTmpBase) that fits the darwin sun_path 104-byte limit regardless of $TMPDIR length or test name, plus a cleanup func that removes the dir. Use for any test that passes its own `tmux -S <path>`; t.TempDir() on darwin resolves under /var/folders/<hash>/T/<TestName>... and overshoots the limit for long test names ("File name too long").
socket, cleanup := testutil.ShortTmuxSocket() t.Cleanup(cleanup)
func SkipIfShort ¶ added in v1.9.20
SkipIfShort skips the test when `go test -short` is in effect. TestPerf_* tests are expensive enough that contributors running quick unit-test loops shouldn't pay for them. CI always runs in long mode.
func TrimmedMean ¶ added in v1.9.20
TrimmedMean runs fn n=11 times (plus 1 warm-up), sorts the samples, drops the top 2 and bottom 2, and returns the average of the middle 7.
Use for COLD tests where GC manipulation in the parent process doesn't help (the timed work happens in a child process or kernel).
func TrimmedMeanWarm ¶ added in v1.9.20
TrimmedMeanWarm is TrimmedMean with controlled GC: forces a runtime.GC() before each timed iteration and disables auto-GC during the timed window via debug.SetGCPercent(-1), restoring the original setting on return.
Use for WARM tests measuring pure-Go in-process work, where GC pauses are the dominant noise source.
func TrimmedMeanWarmWithSetup ¶ added in v1.9.46
TrimmedMeanWarmWithSetup is TrimmedMeanWithSetup with controlled GC: it runs setup() before each timed op() (excluded from the timing window) and, like TrimmedMeanWarm, forces a runtime.GC() before each timed iteration and disables auto-GC during the timed window via debug.SetGCPercent(-1), restoring the original setting on return.
This is the correct partner for WarmBudget when the timed primitive mutates state and therefore needs a fresh fixture per iteration (e.g. insert/delete against a DB, or a prune over a pre-filled table). TrimmedMeanWithSetup is the COLD variant and does NOT control GC, so pairing it with WarmBudget's tighter base×3 gate would measure GC noise the budget assumes away.
func TrimmedMeanWithSetup ¶ added in v1.9.20
TrimmedMeanWithSetup runs setup() before each timed op() but excludes setup from the timing window. Use when the timed primitive needs a fresh fixture per iteration (e.g. a populated tree before timing DeleteAll). Setup and op share state via closure capture in the caller.
COLD variant — no GC manipulation. For WARM work that also needs a per-iteration fixture, use TrimmedMeanWarmWithSetup instead: pairing WarmBudget's tighter base×3 gate with this COLD measurement is unsound, because the GC noise WarmBudget assumes away is not actually controlled.
func UnsetGitRepoEnv ¶
func UnsetGitRepoEnv()
UnsetGitRepoEnv removes git repository-routing env vars from the current process. This prevents subprocess git commands from accidentally targeting the caller's repo.
func WarmBudget ¶ added in v1.9.20
WarmBudget returns the budget for a WARM test: pure in-process Go work that can be measured under controlled GC. Formula:
max(base * 3, PerfBudgetFloor) * PERF_BUDGET_MULTIPLIER
Pair with TrimmedMeanWarm(fn) — that variant forces a GC cycle and disables auto-GC around each timed iteration, eliminating the largest noise source.
No callers in this PR; exported to capture the cold/warm convention in code so future contributors classify their tests correctly.
Types ¶
type FsyncCounter ¶ added in v1.9.47
type FsyncCounter struct {
// contains filtered or unexported fields
}
FsyncCounter is the reusable Tier 2 instrumentation the perf-budget doc anticipates (docs/perf-budget-suite.md, "real-disk count assertions"):
"Disk walltime varies 100× across SSDs / HDDs / cloud volumes — un-normalizable. Instead instrument the storage layer (*sql.DB and *os.File wrappers exposing fsync count, transaction count, rows written) and assert on counts: 'create one group = exactly 1 transaction + 1 fsync'."
It wraps a `func(*os.File) error` fsync seam (e.g. the one a package routes (*os.File).Sync through) and tallies the calls, bucketed by whether the descriptor is a regular file or a directory. The two carry different regression signals: a durable atomic write fsyncs the DATA file (content durability) and then best-effort fsyncs the PARENT directory (rename durability), so a test that asserts on both pins the exact I/O shape.
Today only the durable per-parent outbox drain wires it (one caller), but it is package-level testutil rather than a one-off in that test because the doc frames the wrapper as shared infrastructure every future persistence-touching Tier 2 gate should reuse — a single counting convention beats a bespoke counter per save path.
Construct with new(FsyncCounter); install via the seam's test hook:
fc := new(testutil.FsyncCounter)
restore := session.SetFsyncHookForTest(fc.Wrap(nil)) // nil → real Sync
defer restore()
... exercise the code under test ...
fc.Reset() // discard setup syncs
... run the timed/counted op ...
if fc.FileSyncs() != 2 { ... }
All methods are safe for concurrent use; the seam may fire from background goroutines in the code under test.
func (*FsyncCounter) DirSyncs ¶ added in v1.9.47
func (c *FsyncCounter) DirSyncs() int
DirSyncs returns the number of fsyncs issued against directories.
func (*FsyncCounter) FileSyncs ¶ added in v1.9.47
func (c *FsyncCounter) FileSyncs() int
FileSyncs returns the number of fsyncs issued against regular files.
func (*FsyncCounter) Reset ¶ added in v1.9.47
func (c *FsyncCounter) Reset()
Reset zeroes both counters. Call it after fixture setup (whose own durable writes are not part of the operation under test) and before the counted op.
func (*FsyncCounter) Wrap ¶ added in v1.9.47
Wrap returns a fsync-seam function that records each call then delegates to real. Pass real=nil to delegate to (*os.File).Sync (the production default), which is the common case — the count is the assertion, the underlying fsync is left intact (and is cheap on the tmpfs scratch dir a Tier 2 test uses).
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package crossfixture supplies the shared scaffolding for tests that must verify TUI ↔ web ↔ CLI parity (TEST-PLAN.md §6.4 / TUI-TEST-PLAN.md §6.4 crossProcessFixture).
|
Package crossfixture supplies the shared scaffolding for tests that must verify TUI ↔ web ↔ CLI parity (TEST-PLAN.md §6.4 / TUI-TEST-PLAN.md §6.4 crossProcessFixture). |
|
Package fakeclock provides an injectable Clock for tests that exercise time-sensitive logic — hook freshness windows, heartbeat staleness, log-rotation maintenance — without sleeping or depending on wall-clock progress.
|
Package fakeclock provides an injectable Clock for tests that exercise time-sensitive logic — hook freshness windows, heartbeat staleness, log-rotation maintenance — without sleeping or depending on wall-clock progress. |
|
Package fakeinotify provides a controllable filesystem event source for tests that exercise hook-status-watcher overflow / fallback paths (TEST-PLAN.md J2 regression, TUI-TEST-PLAN.md §6.2 fakeInotify).
|
Package fakeinotify provides a controllable filesystem event source for tests that exercise hook-status-watcher overflow / fallback paths (TEST-PLAN.md J2 regression, TUI-TEST-PLAN.md §6.2 fakeInotify). |
|
Package logassert captures slog records during a test and offers assertion helpers so tests can verify "this code path emitted hook_overflow with dropped>0" without grepping stderr.
|
Package logassert captures slog records during a test and offers assertion helpers so tests can verify "this code path emitted hook_overflow with dropped>0" without grepping stderr. |
|
Package multiclienttmux boots an isolated tmux server with aggressive-resize=on and lets a test attach N pty clients at chosen sizes — the harness from TEST-PLAN.md §6.1 / TUI-TEST-PLAN.md §6.8 for the "two web clients hijacking pane size" regression (J4 / F2).
|
Package multiclienttmux boots an isolated tmux server with aggressive-resize=on and lets a test attach N pty clients at chosen sizes — the harness from TEST-PLAN.md §6.1 / TUI-TEST-PLAN.md §6.8 for the "two web clients hijacking pane size" regression (J4 / F2). |
|
Package profilefixture builds the controlled environment that profile-resolution parity tests require: a known AGENTDECK_PROFILE, CLAUDE_CONFIG_DIR, agent-deck config-dir override, and isolated tempdir.
|
Package profilefixture builds the controlled environment that profile-resolution parity tests require: a known AGENTDECK_PROFILE, CLAUDE_CONFIG_DIR, agent-deck config-dir override, and isolated tempdir. |
|
Package teatesthelper wraps charmbracelet/x/exp/teatest with the conventions our TUI tests need (TUI-TEST-PLAN.md §6.1):
|
Package teatesthelper wraps charmbracelet/x/exp/teatest with the conventions our TUI tests need (TUI-TEST-PLAN.md §6.1): |