testharness

package
v1.2.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 12, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package testharness wires the real MCP dispatch path (auth → enforcement pipeline → tool handler → Clockify client) against a fake Clockify upstream so tests can assert policy, auth, and idempotency properties end-to-end without bypassing the layers that enforce them.

Existing service-layer tests in internal/tools call `svc.Foo(ctx, args)` directly — that path is fine for happy-path coverage but skips the enforcement pipeline, which is precisely where policy and rate-limit regressions live. Use testharness.Invoke for tests whose premise is "this call SHOULD/SHOULD NOT reach Clockify" — the UpstreamHit field on the result is the canonical way to assert the call was rejected before an HTTP request was emitted.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BenchHarness

type BenchHarness struct {
	// contains filtered or unexported fields
}

BenchHarness is an amortised variant of Invoke for benchmarks. It builds the full MCP stack (tools.Service, Registry, enforcement Pipeline, mcp.Server, initialized state) ONCE, then the Call method dispatches a single tools/call through the already-wired server.

Why this helper exists: memory profiling of the tier-1 write bench (internal/tools/writes_bench_test.go) showed ~82% of per-iteration allocations came from Service.Registry() — the bench was measuring "cold server boot + one call" instead of "one call against a warm server." Amortising the setup makes the measurement reflect real per-dispatch cost the way production clients actually pay it.

Production use of the full pipeline is NOT affected: the MCP server already calls Registry exactly once at startup. This helper only exists to stop benchmarks from paying that cost every iteration.

Tests that assert policy / enforcement / auth properties must continue to use Invoke — each test case needs an isolated pipeline so that state (rate-limit counters, initialized flag) from a prior assertion does not leak into the next one.

The default benchmark pipeline intentionally contains only policy enforcement. Rate-limit, dry-run, and truncation benchmarks must construct those collaborators explicitly so the measured path is obvious at the call site.

func NewBenchHarness

func NewBenchHarness(tb testing.TB, opts InvokeOpts) *BenchHarness

NewBenchHarness wires the full MCP stack once and returns a handle whose Call method reuses it across iterations. See the BenchHarness doc for the rationale.

opts.Tool is ignored at construction time (BenchHarness.Call takes the tool name per dispatch); everything else mirrors Invoke's defaults exactly so the measured path is identical.

func (*BenchHarness) Call

func (h *BenchHarness) Call(ctx context.Context, tool string, args map[string]any) InvokeResult

Call dispatches one tools/call through the already-initialized MCP server. The returned InvokeResult is shape-identical to the one from Invoke. Call is safe to invoke from the benchmark's b.Loop() body.

type FakeClockify

type FakeClockify struct {
	// contains filtered or unexported fields
}

FakeClockify is a counted wrapper around httptest.Server. The harness reads the request count before and after each Invoke to decide whether the call reached Clockify (UpstreamHit) — this is how tests assert "policy rejected the call before any HTTP request was made."

func NewFakeClockify

func NewFakeClockify(t testing.TB, handler http.Handler) *FakeClockify

NewFakeClockify constructs a counted httptest.Server fronting the supplied handler. The server is automatically closed via t.Cleanup.

Accepts testing.TB so benchmarks (writes_bench_test.go) can reuse the same fake-upstream wiring without duplicating the httptest plumbing. *testing.T continues to satisfy testing.TB transparently for existing callers.

func (*FakeClockify) RequestCount

func (f *FakeClockify) RequestCount() int64

RequestCount returns the total number of HTTP requests received since construction. Tests usually don't call this directly — Invoke compares it against a pre-call snapshot to populate InvokeResult.UpstreamHit.

func (*FakeClockify) Reset

func (f *FakeClockify) Reset()

Reset zeroes the request counter. Useful when a test invokes the harness multiple times against the same fake and wants independent UpstreamHit assertions without tearing down the upstream between calls.

func (*FakeClockify) URL

func (f *FakeClockify) URL() string

URL returns the base URL of the fake upstream, suitable for passing to clockify.NewClient.

type InvokeOpts

type InvokeOpts struct {
	// Tool is the MCP tool name (e.g. "clockify_delete_entry"). Required.
	Tool string
	// Args is the tools/call arguments map. Nil means an empty args object.
	Args map[string]any
	// PolicyMode defaults to policy.Standard. Set to policy.ReadOnly or
	// policy.SafeCore to assert that write tools are rejected before the
	// handler runs.
	PolicyMode policy.Mode
	// DeniedTools is an optional per-call override for the policy deny list.
	// Lets a single test assert that an explicitly-denied tool is rejected
	// under an otherwise-permissive mode.
	DeniedTools []string
	// Principal is attached to the call context via authn.WithPrincipal.
	// Nil means no principal (rate-limit falls back to the global scope);
	// set this to simulate per-subject rate limiting or auth failures that
	// still flow through the dispatcher.
	Principal *authn.Principal
	// Upstream is the fake Clockify server the tool's HTTP client talks to.
	// Required — the harness panics if nil, because there's nothing sensible
	// to default here.
	Upstream *FakeClockify
	// ClockifyAPIKey is the bearer key the clockify client sends upstream.
	// Defaults to "test-api-key". The fake upstream decides whether to
	// reject it — tests that want to assert upstream auth errors provide a
	// handler that returns 401 for mismatched keys.
	ClockifyAPIKey string
	// WorkspaceID is written into tools.Service.WorkspaceID. Defaults to
	// "test-workspace".
	WorkspaceID string
	// RequestID is the JSON-RPC request id. Defaults to 1; set this when a
	// test needs to correlate multiple invocations in the same log stream.
	RequestID int
	// Client lets callers supply a pre-built clockify.Client whose HTTP
	// transport is reused across many Invoke calls. Default behaviour is
	// "construct a fresh client per call" (correct for tests where each
	// case must be independent). Benchmarks pass a shared client so they
	// don't burn an ephemeral port per iteration — without this the
	// loopback exhausts its port range after a few thousand calls.
	Client *clockify.Client
	// ActivateTier2Groups is the list of Tier-2 group names to register
	// on the server via Server.ActivateGroup after initialize. Only
	// honoured by NewBenchHarness; Invoke ignores this field because its
	// fresh-per-call server doesn't need lazy activation for unit tests.
	// Use this when a benchmark dispatches a Tier-2 tool (expenses,
	// invoices, approvals, custom_fields, scheduling, ...). A group name
	// that doesn't exist in Tier2Groups causes the harness to Fatal.
	ActivateTier2Groups []string
}

InvokeOpts configures a single tool-call dispatch through the full MCP pipeline. Only Tool and Upstream are required; everything else has a sane default that mirrors the Standard production profile.

type InvokeResult

type InvokeResult struct {
	// Result is the decoded tools/call result envelope. For a successful
	// call this is `{"content":[{"type":"text","text":"<json>"}]}`; tests
	// that need the tool-specific shape should decode the "text" field.
	// Nil when the call failed at the JSON-RPC protocol layer (-32xxx).
	Result map[string]any
	// ResultText is the JSON text returned inside Result.content[0].text
	// when the call succeeded. Tests that want to assert on the concrete
	// tool response shape typically json.Unmarshal this. Empty string when
	// the call errored.
	ResultText string
	// IsError is the MCP spec's tool-error flag on the result envelope.
	// True when the tool handler (or the enforcement pipeline) returned an
	// error without triggering a JSON-RPC -32xxx protocol error. Policy
	// denials, handler errors, and upstream 4xx/5xx all surface here.
	IsError bool
	// ErrorMessage is the human-readable error string extracted from either
	// the JSON-RPC error envelope or the isError:true content block. Empty
	// on success.
	ErrorMessage string
	// RPCError is the JSON-RPC protocol error (schema validation -32602,
	// uninitialized server -32002, etc.) or nil when the call went through.
	// Tool errors come back as IsError:true on the result envelope, NOT as
	// a JSON-RPC error.
	RPCError *mcp.RPCError
	// Outcome classifies the call result for assertion. See the Outcome*
	// constants below.
	Outcome Outcome
	// UpstreamHit reports whether any HTTP request reached the fake
	// Clockify server during this call. False when the enforcement pipeline
	// rejected the call (policy, rate limit, schema validation) before the
	// handler had a chance to run.
	UpstreamHit bool
	// Raw is the full JSON-RPC response bytes in case a test needs more
	// detail than the structured fields expose.
	Raw []byte
}

InvokeResult captures everything a test wants to assert about one dispatch. The canonical assertion for "policy blocked the call before any HTTP request was made" is `!result.UpstreamHit && result.Outcome == OutcomePolicyDenied`.

func Invoke

func Invoke(t testing.TB, opts InvokeOpts) InvokeResult

Invoke dispatches a single tools/call through a freshly constructed server with real enforcement, a real tools.Service, and a Clockify client pointed at opts.Upstream. The server is initialized (initialize → initialized flag) before the call so tools/call passes the spec-compliance guard.

Each Invoke gets a fresh server so independent calls can't leak state through the dispatcher. Tests that want shared state across calls should share the *FakeClockify upstream and assert on its RequestCount directly.

type Outcome

type Outcome string

Outcome classifies how a dispatch call ended. Kept as a string type (not an int enum) so test failure messages are self-explanatory.

const (
	// OutcomeSuccess means the tool handler ran and returned without error.
	OutcomeSuccess Outcome = "success"
	// OutcomePolicyDenied means the enforcement pipeline rejected the call
	// because the policy mode forbids this tool. UpstreamHit is false.
	OutcomePolicyDenied Outcome = "policy_denied"
	// OutcomeInvalidParams means the request failed JSON schema validation
	// (JSON-RPC -32602). UpstreamHit is false.
	OutcomeInvalidParams Outcome = "invalid_params"
	// OutcomeToolError means the tool handler returned an error (upstream
	// 4xx/5xx, business rule violation, etc.). UpstreamHit is usually true
	// — the error originated from a completed HTTP exchange.
	OutcomeToolError Outcome = "tool_error"
	// OutcomeProtocolError means the JSON-RPC layer rejected the request
	// (unknown method, uninitialized server, etc.) before the dispatcher ran.
	OutcomeProtocolError Outcome = "protocol_error"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL