Documentation
¶
Overview ¶
Package e2ecalls declares the record the end-to-end suite writes down while it runs, and the coverage audit reads back afterwards: what a test asked the server to do, what the server dispatched, and on which runtime, surface and mode.
Why the record exists ¶
Coverage used to be credited by mentions in the source: the audit that ran before this one globbed the suite's test files and credited any catalog action whose ID appeared on a code line. That counts an action nothing runs, and it counted eighteen of them that neither Docker target can reach. A call counts here only when the server dispatched it, in a test that passed, on a named runtime, surface and mode, which is what these lines carry.
Why it is a package of its own ¶
Two programs read and write it: the e2e harness, which is built with the e2e tag, and cmd/audit_e2e_coverage, which is an ordinary command. A shared Go type is what keeps the two halves from drifting, and it can only be shared from a package both may import. internal/testutil cannot be that package: it embeds the pinned GraphQL schema and net/http/httptest, which a command has no use for and would carry anyway. So this package is untagged and holds the record and nothing else.
The shard mechanism under it is github.com/jmrplens/gitlab-mcp-server/v3/internal/testutil/shardio, which is the only thing this package imports beyond the standard library and is bound by the same two rules: untagged, and never importing testing. What lives here is the content, which lines exist, what each field means and which of them a reader joins on; what lives there is how a line reaches a file and comes back. The two were one package until a second record needed the same mechanism and copied it.
Being untagged means it is compiled by an ordinary build, enters make test, the coverage job and Sonar, and is held to the same bar as production code. It must still never reach the server binary: cmd/server's TestDependencies_TestSupport_NeverReachesTheServerBinary names it, because that test matches exact import paths and internal/testutil alone does not cover a subpackage of it.
What a shard is ¶
One test process writes one shard, named ShardPattern, into the directory DirEnv names. Nothing is written when that variable is unset. Each line is one JSON Record: a schema version, a line type, and exactly one payload of Run, Session, Call, Dispatch or Skip. Read merges a directory of them, subdirectories included, so one run per Docker target can be compared against another without moving files around.
Index ¶
- Constants
- func CredentialKind(kind string) string
- func IsShard(name string) bool
- func RefusedOutcome(reason string) string
- func Release()
- func RunIDDate(runID string) (time.Time, bool)
- type Call
- type Dispatch
- type Enveloper
- type FixtureProfile
- type Record
- type Reporter
- type Run
- type Session
- type Shard
- type Skip
- type Writer
Constants ¶
const ( // DirEnv names the directory the end-to-end harness records its calls // into. Recording is off unless it is set, so an ordinary suite run pays // nothing for it. // // The path must be absolute. A test binary runs with its own package // directory as the working directory, so a relative path would drop one // shard under each of the suite's package directories and the merge would // find none of them. A relative value is refused rather than resolved, // because resolving it would produce exactly that scattering silently. The // sibling GITLAB_MCP_TEST_INVENTORY_DIR is refused on the same terms. // // It carries the project's prefix and is not one of the settings // internal/config resolves: it configures the test harness, cmd/server // never reads it, and there is no legacy spelling of it to warn anybody // about. That is the standing the developer-only MODELEVAL_* variables // have. DirEnv = "GITLAB_MCP_TEST_E2E_CALLS_DIR" // SchemaVersion is the version every line carries. A reader refuses a line // written under another one rather than guessing which fields it holds: // these shards are read by a command in the same tree, so a mismatch means // a stale artifact and not an old peer to be tolerated. // // Version 2 (issue 920) changed what a session line's DispatchObserved // means, a server span of any method now counting where only one carrying // the dispatch facts did, and added Idle, whose absence carries meaning // too. A version 1 line cannot be told apart from a non-idle one, and its // false DispatchObserved said only that no fact-carrying span arrived, so // folding it under the new reading would mark sessions unobserved that // were only never asked a tool. Such shards are refused, to be re-recorded // rather than re-folded. [ReadShardsForCalls] and its merged form // [ReadForCalls] are the exception, for a reader of the lines version 2 // left readable: the coverage audit's -baseline and R-PATH's -e2e-calls. SchemaVersion = 2 // OldestCallsSchemaVersion is the oldest schema whose run, call, dispatch // and skip lines read the way [SchemaVersion]'s do. Version 2 changed the // meaning of the session line, and it also stopped writing a dispatch line // whose span named neither a tool nor an action, so a version 1 shard can // carry such lines; every reader skips a dispatch line with no action, so // its action cells read the same. A reader that judges nothing by a // session line can therefore still read a version 1 shard, through // [ReadShardsForCalls]. A version that changes one of those four lines in a // way such a reader sees moves this to itself. // // It exists for the old suite's baseline: its shards were written under // version 1 by a suite that has since been deleted, so they can never be // recorded again, and what they are compared on is what their calls // credited. OldestCallsSchemaVersion = 1 // ShardPattern is the [os.CreateTemp] pattern a shard file is named with. // One process writes one shard, so package binaries running side by side // never write to the same file and no cross-process locking is needed. ShardPattern = shardPrefix + "*" + shardExt )
const ( // TypeRun names the line a test package writes once: which runtime it // found, what it required, and whether it went on to run. TypeRun = "run" // TypeSession names the line one server configuration writes when it // starts, holding what that session serves. TypeSession = "session" // TypeCall names the line one MCP call writes: what a test asked for and // what came back. TypeCall = "call" // TypeDispatch names the line the server's own span writes: what it // decided to run for a call the harness made. TypeDispatch = "dispatch" // TypeSkip names the line a skipped test writes, with its reason. TypeSkip = "skip" )
The five line types. Every Record carries exactly one of them, and the reader refuses a record whose payload is not the one its type names.
const ( // PurposeTest is a call the test body made and asserted on. PurposeTest = "test" // PurposeCleanup is a call made from a cleanup, after the test body. PurposeCleanup = "cleanup" // PurposeSweep is a call made by a sweep over what a session serves. PurposeSweep = "sweep" // PurposeRaw is a call made through the harness's raw escape hatch, where // the protocol rather than the action is what the test is about. PurposeRaw = "raw" // PurposeModel is a call a language model chose and the model evaluation // harness sent on its behalf. // // It is never credited as coverage, for a reason stronger than the one // that exempts PurposeRaw: a raw call at least names a tool a test wrote // down, while what a model calls is decided at run time by a provider. A // shard of model calls pointed at the coverage audit by mistake would // otherwise raise this suite's coverage by whatever the model happened to // try, and lower it again when the next run tried something else. PurposeModel = "model" )
What a call was made for. A call made to build or tear down fixture state is worth less as coverage than one a test asserted on, so the purpose is recorded rather than inferred from the test name.
const ( // ExpectationOK is a call the test expected to succeed. ExpectationOK = "ok" // ExpectationAny is a call whose outcome the test did not constrain. ExpectationAny = "any" )
What the caller expected. Anything other than these two names a harness failure class, which is why the field is a string here: the classes belong to the harness, and this package is the record rather than the vocabulary.
const ( // OutcomeOK is a call the server answered without an error. OutcomeOK = "ok" // OutcomeToolError is a call answered with a tool error result. OutcomeToolError = "tool_error" // OutcomeProtocolError is a call answered with a JSON-RPC error. OutcomeProtocolError = "protocol_error" // OutcomeTransportError is a call that never got an answer. OutcomeTransportError = "transport_error" // OutcomePreview is a safe-mode preview: the server answered the call with // the preview of a mutation it did not make. OutcomePreview = "preview" // OutcomeRefusedPrefix starts the outcome of a call the server refused. // The refusal reason follows it. OutcomeRefusedPrefix = "refused:" )
What came back. A refusal carries its reason, which is why it is a prefix and not a value: RefusedOutcome spells one.
const ( // CredentialClassic is a token whose scopes decide what it may do, which // is every token the suite ran with before fine-grained ones. A line that // carries no kind was written by such a session, and reads as this. CredentialClassic = "classic" // CredentialFineGrained is a fine-grained personal access token, judged // per action from its grant. CredentialFineGrained = "fine-grained" )
The kinds of credential a session runs with. The kind is a coordinate of what a session serves, beside its surface and mode: a fine-grained personal access token is listed only what its grant reaches and refused the rest with a reason no classic token is ever given (issue 952), so a call credited without it would fold that refusal into the classic session's cells.
const ( // RunStarted is a package whose runtime met its requirement. RunStarted = "started" // RunRefused is a package that found a runtime it cannot run on. RunRefused = "refused" )
Whether a package ran at all. A refused run is recorded rather than left silent, because a missing run line and a run that refused to start are different answers to "was this runtime exercised".
const ( // StatusPassed is a test that ended without a failure. StatusPassed = "passed" // StatusFailed is a test that failed. StatusFailed = "failed" // StatusSkipped is a test that skipped. StatusSkipped = "skipped" )
How the test a call belongs to ended. Records are buffered per test and written from its first-registered cleanup, so every call knows this by the time it is written.
const RunIDStampLayout = "20060102t150405z"
RunIDStampLayout is the UTC timestamp a run identifier opens with, spelled the way the harness formats it and the way a reader parses it back. It is lowercased so the whole identifier stays a legal GitLab path segment.
It lives here rather than in either side because it is the same contract as the field names above: the harness under the e2e build tag mints the run ID, and cmd/audit_e2e_coverage, which is not under that tag, reads the day of the run off it to date the committed coverage record. While the layout was spelled in both, nothing tied the two spellings together -- the committed shard fixtures carry invented identifiers, so a layout change on one side would leave every test green and be found by the first real Docker run producing a record with no date at all.
Variables ¶
This section is empty.
Functions ¶
func CredentialKind ¶ added in v3.2.0
CredentialKind reads the kind a session or call line carries, the empty value of a line written before the field existed reading as classic.
func IsShard ¶
IsShard reports whether a file name is one of this package's shards.
It is exported because the coverage audit asks the same question of a directory before it walks it, and spelling the two halves of the name a second time there is how the answer would come to disagree with the one the reader gives.
func RefusedOutcome ¶
RefusedOutcome spells the outcome of a call the server refused for the given reason, as OutcomeRefusedPrefix followed by that reason.
It exists so the writer and the reader agree on one spelling: the coverage audit classifies a refusal by this prefix, and a second hand-written concatenation is how the two would come to disagree.
func Release ¶
func Release()
Release closes every open shard and forgets it, so a later Open of the same directory starts a new one. Windows is why it exists: a directory holding an open file cannot be removed.
func RunIDDate ¶
RunIDDate reads the day a run started off its identifier, and reports false for an identifier carrying no stamp this can read.
A run ID is the stamp, a hash and the package name joined by dashes, so the stamp is everything before the first one; an identifier with no dash carries no stamp this recognizes even if it parses, since the caller overrode it and whatever it says is not a time this package wrote. It sits beside the layout so the format and the parse cannot drift apart.
Types ¶
type Call ¶
type Call struct {
// Test names the top-level test the call is attributed to.
Test string `json:"test"`
// Purpose is one of the Purpose* constants.
Purpose string `json:"purpose"`
// Expectation is [ExpectationOK], [ExpectationAny] or a harness failure
// class the test declared.
Expectation string `json:"expectation"`
// Session is the label of the session the call went to.
Session string `json:"session"`
// Surface repeats the session's surface, as Mode and Capabilities repeat
// the rest of its shape, so one call line can be classified on its own
// rather than only after a join back to the session that served it.
Surface string `json:"surface"`
// Mode is the protective mode the session runs in.
Mode string `json:"mode"`
// Capabilities is the resource and prompt surface of the session.
Capabilities string `json:"capabilities"`
// Credential repeats the session's credential kind, written only for a
// fine-grained one, as [Session.Credential] is.
Credential string `json:"credential,omitempty"`
// Requirement is the runtime requirement of the package that made the
// call.
Requirement string `json:"requirement"`
// Method is the MCP method, so the non-tool verbs are recorded on the same
// terms as tools/call.
Method string `json:"method"`
// Tool is the tool name the call named, empty for a method that names
// none.
Tool string `json:"tool,omitempty"`
// Action is the canonical catalog ID the test asked for.
Action string `json:"action,omitempty"`
// Dispatched is the canonical ID the server reported running, joined from
// the span on [Call.TraceID]. It is empty when no span arrived.
Dispatched string `json:"dispatched,omitempty"`
// Target names what a non-tool call addressed: a resource URI, a prompt or
// a completion reference.
Target string `json:"target,omitempty"`
// Arguments are the argument names the call carried, sorted. The values
// are left out: a value is a fixture, a name is part of the call.
Arguments []string `json:"arguments,omitempty"`
// Outcome is one of the Outcome* constants, or [RefusedOutcome] of the
// reason the server gave.
Outcome string `json:"outcome"`
// DurationMS is how long the call took, in milliseconds.
DurationMS float64 `json:"duration_ms,omitempty"`
// TraceID is the W3C trace the harness stamped into the call, which is
// what a [Dispatch] line joins on.
TraceID string `json:"trace_id,omitempty"`
// TestStatus is how the test ended: one of the Status* constants. A call
// made by a test that failed is not coverage.
TestStatus string `json:"test_status"`
}
Call is one MCP call a test made.
It holds both halves of the question coverage asks: what the test requested, which the client knows, and what the server dispatched, which only the server's own span can say. The two differ wherever an alias is rewritten, and crediting the requested one would credit an action that never ran.
type Dispatch ¶
type Dispatch struct {
// TraceID is the trace the harness stamped into the call this describes.
TraceID string `json:"trace_id"`
// Tool is the span's gen_ai.tool.name.
Tool string `json:"tool,omitempty"`
// Action is the span's gitlab_mcp.action, which names the route that ran
// rather than the one the call asked for.
Action string `json:"action,omitempty"`
// Domain is the span's gitlab_mcp.domain.
Domain string `json:"domain,omitempty"`
// RefusalReason is the span's gitlab_mcp.refusal_reason, set when the
// server declined to run the action.
RefusalReason string `json:"refusal_reason,omitempty"`
// ErrorType is the span's error.type.
ErrorType string `json:"error_type,omitempty"`
// Status is the span status.
Status string `json:"status,omitempty"`
// Requests is how many GitLab requests the handler made while running this
// action, counted from the client spans of the same trace.
//
// It is the one fact here that comes from a span other than the server's
// own, and it is what lets a reader ask per action what the committed
// request inventory can only answer per package: nothing on the wire names
// an action, but a trace does, and every GitLab call the handler made is a
// child of the span that names it.
//
// Read it as a floor rather than as an exact count. The spans travel
// through a batching processor with a bounded queue, so a run that
// overflows it drops client spans silently and this reads low; and a retry
// is a separate round trip, so two attempts at one endpoint count two. Zero
// means no client span of this trace arrived, which is usually an action
// that reached no GitLab (a refusal, a safe-mode preview) and is
// occasionally that drop.
Requests int `json:"requests,omitempty"`
// Routes are the routes those requests were sent to, each a method and a
// path template as GitLab's own record spells the route ("GET
// /projects/:id/issues"), sorted and deduplicated. A request whose span
// named no route, one to an endpoint the server's table does not hold,
// is counted in Requests and named here by its method alone, so a reader
// can tell a route nobody derived from a span that carried none.
//
// It is what lets a reader compare what a handler was seen calling with
// what the derivation says it calls route by route rather than by count,
// which is the one per-action oracle for a request the derivation left
// out. The same floor applies to it as to Requests, since both are read
// from the same spans.
Routes []string `json:"routes,omitempty"`
}
Dispatch is what the server said it ran, read from its own OTLP span.
It is a line of its own rather than only a field of Call because the span arrives over the network after the answer does. A call flushed before its span lands carries no dispatched action, and this line is what the audit joins to it afterwards.
type Enveloper ¶ added in v3.2.0
type Enveloper interface {
// contains filtered or unexported methods
}
Enveloper is one thing a shard can hold: a line that knows the envelope it is written in. The set is closed on purpose: Run, Session, Call, Dispatch and Skip are the only implementations, because the reader dispatches on Record.Type and a sixth shape it has never heard of would be dropped or guessed at.
type FixtureProfile ¶
type FixtureProfile struct {
// Runner is whether a CI runner was registered.
Runner bool `json:"runner"`
// FixtureService is whether the compose fixture service was up.
FixtureService bool `json:"fixture_service"`
// Bitbucket is whether the Bitbucket import source was up.
Bitbucket bool `json:"bitbucket"`
// GHToken is whether a GitHub token was configured for import scenarios.
GHToken bool `json:"gh_token"`
// Seeds names the per-consumer seeds the setup script created.
Seeds []string `json:"seeds,omitempty"`
}
FixtureProfile is what a runtime had available to the test package that ran on it.
It travels with the run line because a scenario that needs a CI runner is absent rather than failing when there is none, and a coverage figure that cannot tell those apart is not worth reading.
type Record ¶
type Record struct {
// Schema is [SchemaVersion] as of the run that wrote the line.
Schema int `json:"schema"`
// Type names which payload is set: one of the Type* constants.
Type string `json:"type"`
// Run is set when Type is [TypeRun].
Run *Run `json:"run,omitempty"`
// Session is set when Type is [TypeSession].
Session *Session `json:"session,omitempty"`
// Call is set when Type is [TypeCall].
Call *Call `json:"call,omitempty"`
// Dispatch is set when Type is [TypeDispatch].
Dispatch *Dispatch `json:"dispatch,omitempty"`
// Skip is set when Type is [TypeSkip].
Skip *Skip `json:"skip,omitempty"`
}
Record is one line of a shard: the schema it was written under, which line it is, and exactly one payload.
It is an envelope rather than one flat struct because the five lines share almost no fields, and a flat struct would answer "which of these fields apply" with a convention instead of with the type.
func Read ¶
Read merges every shard under dir, subdirectories included, in the order the directory tree walks. It is ReadShards with the file boundaries dropped, for a reader that wants the lines and not their provenance.
func ReadForCalls ¶
ReadForCalls is ReadShardsForCalls with the file boundaries dropped, as Read is ReadShards: for a reader of the run, call, dispatch and skip lines that wants the lines and not their provenance, such as R-PATH's per-action observation, which folds dispatch lines alone.
type Reporter ¶
Reporter is the part of testing.TB the writer reports a broken shard through. See shardio.Reporter for why it is an interface and why it is taken per call.
type Run ¶
type Run struct {
// Package is the test package that wrote the line.
Package string `json:"package"`
// Requirement is the runtime the package asked for.
Requirement string `json:"requirement"`
// Edition is what the instance reported: a community or enterprise build.
Edition string `json:"edition"`
// Tier is the licensing tier the catalog was built at.
Tier string `json:"tier"`
// TierConfirmed is whether the tier came from the instance license rather
// than from a setting. An unconfirmed tier means the catalog may hold
// actions the instance will refuse.
TierConfirmed bool `json:"tier_confirmed"`
// GitLabVersion is the version the instance reported.
GitLabVersion string `json:"gitlab_version"`
// RunID is the identifier every name this run created is scoped to, which
// is also what the orphan sweep deletes by.
RunID string `json:"run_id"`
// Commit is the revision under test, from E2E_COMMIT.
Commit string `json:"commit,omitempty"`
// Filter is the -run expression the binary was given, empty when it ran
// everything. A partial run is not a coverage claim about the rest.
Filter string `json:"filter,omitempty"`
// Fixtures is what the runtime had available.
Fixtures FixtureProfile `json:"fixtures"`
// Status is [RunStarted] or [RunRefused].
Status string `json:"status"`
// Reason says why a refused run refused. It is empty on a started run.
Reason string `json:"reason,omitempty"`
}
Run is what one test package found when it started, written once per package.
It is the denominator of everything else in the shard: a call means nothing without the runtime it was made against, since the served catalog, the pruned schemas and the actions that exist at all follow from the edition and the tier.
type Session ¶
type Session struct {
// Label names the session in the call lines that reference it.
Label string `json:"label"`
// Surface is dynamic, meta or individual.
Surface string `json:"surface"`
// Mode is default, read-only or safe.
Mode string `json:"mode"`
// Capabilities is the resource and prompt surface: full or minimal.
Capabilities string `json:"capabilities"`
// Transport is how the harness reached the binary.
Transport string `json:"transport"`
// Credential is the kind of credential the session ran with: one of the
// Credential* constants, written only for a fine-grained one, so a line
// with none reads as classic ([CredentialKind]) and every line written
// before the field reads as what it was.
Credential string `json:"credential,omitempty"`
// Tools are the tool names the session listed.
Tools []string `json:"tools,omitempty"`
// Resources are the static resource URIs the session listed.
Resources []string `json:"resources,omitempty"`
// ResourceTemplates are the URI templates the session listed.
ResourceTemplates []string `json:"resource_templates,omitempty"`
// Prompts are the prompt names the session listed.
Prompts []string `json:"prompts,omitempty"`
// Completions are the completion references the session offers.
Completions []string `json:"completions,omitempty"`
// SubscribableKinds are the resource kinds the session accepts a
// subscription for.
SubscribableKinds []string `json:"subscribable_kinds,omitempty"`
// DispatchObserved is whether the server's own span arrived for at least
// one call this session made, of whatever method: a resource read, a
// prompt or a completion counts as much as a tool call, since each says
// the session's telemetry reaches the harness.
//
// It is false when that telemetry never arrived, and then every tool call
// of the session is a claim about what was asked for and not about what
// ran. It is also false for a session that made no traced call at all,
// which Idle says.
//
// It decides no credit. A resource read, a prompt and a completion are
// judged on their answer, and a tool call is judged on the action its own
// span named, joined on the trace id; this is the statement about the
// session a reader checks those against.
DispatchObserved bool `json:"dispatch_observed"`
// Idle is whether the session issued no trace: it was started, listed what
// it serves, and made no call that carried one (a subscribe on protocol
// 2026-07-28 carries none). Its DispatchObserved is then false for want of
// a traced call rather than for want of telemetry, and a reader folding
// sessions sets it apart instead of letting it read as a session whose
// spans never came.
//
// It is written only when true. A line written before the field existed
// cannot be told apart from a non-idle one, and its false DispatchObserved
// said only that no fact-carrying span arrived, which is why
// [SchemaVersion] moved to 2 with it: such a line is refused, or dropped
// by [ReadShardsForCalls], rather than read under the new meaning.
Idle bool `json:"idle,omitempty"`
}
Session is one server configuration, written when that session starts.
What a session serves is the denominator the coverage audit divides by: an action no session served is absent rather than untested, and the two are different findings.
type Shard ¶
Shard is one shard file read back: where it was, and every line it held.
It exists because a shard is the one unit that says which process wrote a line. No call, skip or session line names its package, only the run line does, and one process writes one shard, so a reader that wants to place a call in its package reads the shards apart and joins each to the run line beside it. Merged, that attribution is gone.
func ReadShards ¶
ReadShards reads every shard under dir, subdirectories included, one entry per file in the order the directory tree walks.
Subdirectories are read because one run per Docker target writes into a directory of its own, and the coverage audit compares the runtimes against each other. A directory holding no shard is an error rather than an empty result, since reporting nothing recorded as zero coverage would be a claim about the server made from a claim about the harness.
func ReadShardsForCalls ¶
ReadShardsForCalls is ReadShards for a reader of the run, call, dispatch and skip lines alone, such as a comparison of what two runs credited.
It accepts a line written under any schema from OldestCallsSchemaVersion to SchemaVersion, since those four lines read the same under each to a reader that skips a dispatch line naming no action (version 1 wrote some that version 2 no longer writes, and every reader skips them), and drops the session lines of an older schema, which are the lines a later version changed the meaning of: a reader handed one would fold it under a reading it was not written for, which is what ReadShards refuses a whole shard for. A session line of the current schema is kept. Every other rule ReadShards holds a line to still applies.
type Skip ¶
type Skip struct {
// Test names the test that skipped.
Test string `json:"test"`
// Reason is what it said when it skipped.
Reason string `json:"reason"`
}
Skip is a test that did not run, with the reason it gave.
A skip is recorded because it is the honest third answer beside covered and absent: an action whose only scenario skips for want of a runner is not covered, and saying so is more useful than leaving the cell empty.
type Writer ¶
Writer appends lines to this process's shard of the call record.
The zero value is not usable: Open and OpenDir make one. A nil *Writer is usable and does nothing, which is what recording being off looks like to a caller, so the harness has no branch of its own to get wrong.