keycheck

Overview ¶

Command keycheck verifies that the string keys a service handler writes into its map[string]<T> wire responses, AND the json struct tags on any locally-declared *Output-suffixed struct it constructs, actually exist in the pinned AWS SDK's own response deserializer -- not the Go field name, the deserializer's literal switch-case string. It exists for gopherstack-zquj (the map-key half) and gopherstack-v4a4 (the struct-tag half, added after glue's querySchemaVersionMetadataOutput was found tagged json:"MetadataInfo" where the SDK expects "MetadataInfoMap", commit c3aa73e59): a wrong key or tag is checked by no compiler and no existing scan, so it is silently dropped by any real client and invisible to a raw-body test (which asserts the same key the author typed).

For each op it builds the real wire key set from the pinned SDK's <prefix>deserializeOpDocument<Op>Output case-switch, recursing through nested <prefix>deserializeDocument<Type> calls, and diffs that set against every string key the handler's reachable call graph writes into a map[string]<T>.

PROTOCOL COVERAGE. Validated on awsjson1.1 (shield, ssoadmin, gopherstack-zquj first pass, 115 ops) and on restjson1 document(body)-bound members (scheduler's pre-fix awsvpcConfiguration/capacityProvider bug, commit 8469dcdd9). Both protocols' codegen emits the same <prefix>deserializeOpDocument<Op>Output / <prefix>deserializeDocument<Type> functions with a map[string]interface{} type-switch body, which is the only shape this scanner parses -- pass the matching -prefix (awsAwsjson11_, awsAwsjson10_, awsRestjson1_) and it works.

It does NOT understand restjson1 members bound to an HTTP header or the status line: those never appear in a deserializeOpDocument function, so a handler that legitimately writes such a key will false-positive as NotInTree. Hand-check any restjson1 MISMATCH against the op's http.header trait before trusting it.

It does NOT understand query, ec2query, or restxml/xml protocols at all -- their deserializers are xml.Decoder based with no map[string]interface{} type assertion, so -sdk parsing resolves zero ops and zero types against them. See FAIL-LOUD: that state is reported as an explicit error, never a silent zero.

FAIL-LOUD. Any state meaning "this service was not actually checked" is an explicit stderr row and a non-zero exit, never a bare zero:

• -sdk yields zero deserializeOpDocument/deserializeDocument matches for -prefix: wrong prefix, or a protocol this tool can't read.
• the -svc package's dispatcher resolves zero op-to-handler bindings.
• a dispatched op has no way to resolve its allowed key set: neither a deserializeOpDocument<Op>Output function NOR a wrapper type (<prefix>deserializeOp<Op>) confirming a genuinely empty output. (The SDK omits the document deserializer entirely when an Output struct has no members beyond ResultMetadata; that is confirmed by the wrapper's HandleDeserialize body calling no deserializeOpDocument* function, not merely by the function's absence -- six ssoadmin ops are this case and are correctly resolved as empty, not unresolved.)
• an op is bound to more than one distinct handler name (KNOWN BLIND SPOT #6, sqs's dual JSON/Query handlers) -- neither is silently preferred; the op is reported ambiguous instead of checked.

The service writing zero map[string]<T> literal keys anywhere is reported as N/A, not as "0 mismatches, clean": it means the service builds responses from tagged structs rather than hand-written maps, which is a different construction this tool has nothing to check.

A found MISMATCH exits non-zero too (a different code from an unresolved service), so this can gate CI once trusted.

KNOWN BLIND SPOT #1, disclosed rather than fixed: this checks whether a written key exists ANYWHERE in the op's transitively reachable shape, not whether it sits at the correct nesting level. A key real at one depth but wrongly placed at another will not be caught. Hand-check the highest-surface op in any service audited with this tool.

KNOWN BLIND SPOT #2, found live during the gopherstack-zquj sweep: the written-key BFS walks the op handler's full same-package call graph (writtenKeys, capped at 200 funcs) with no way to tell "this map literal becomes the wire response" from "this map literal is written somewhere else entirely" -- internal persisted state (event-history records), request-body transformation (rewriting a state-machine definition before handing it to an internal executor), or any other side effect reachable from the handler. A handler that calls into unrelated backend code (openCountsLocked -> the timeout-sweep janitor -> terminateExecutionLocked in swf, confirmed live) can pull in keys that were never going on this op's wire response at all, and will misreport as MISMATCH. The signal to distrust a hit: a large FuncsWalked count (or hitting the 200-func cap outright, as dynamodb does) relative to what the op plausibly needs to build its own response. Re-run with -op and KEYCHECK_DEBUG_WALK=<Op> (see below) to print the exact call chain and hand-verify which function actually wrote the flagged key, and whether that write reaches the HTTP response, before trusting any MISMATCH this tool reports.

TWO RECURRING SHAPES of blind spot #2, both found live re-sweeping for gopherstack-zquj and worth naming explicitly rather than re-discovering per service: (a) a shared helper writes a key inside an "if cond" conditional (e.g. comprehend's matchResult only sets "Type" when its kind argument is non-empty); every caller of the helper is credited with the key regardless of whether that call site's arguments ever satisfy the condition, so a caller that always passes the empty case (DetectKeyPhrases calling matchResult with kind="") gets a false MISMATCH for a key it can never actually emit. (b) the walk reaches an op's own error-path construction (a *_test.go-free exception/failure type built for the same op, e.g. timestreamwrite's RejectedRecords[].ExistingVersion), which is real and correct on the error response but doesn't appear in the success deserializer this tool diffs against.

KNOWN BLIND SPOT #4, found live during the gopherstack-0kk8 dispatch-table sweep: analyzeFunc/extractCases only understands a deserializer function shaped as a map[string]interface{} type-assert followed by a switch on fixed, known key strings (an AWS "structure" shape). A genuine AWS *map* member (map[string]T with caller-controlled dynamic keys, e.g. glue's QuerySchemaVersionMetadata MetadataInfoMap or personalize's GetSolutionMetrics Metrics) deserializes with the same map[string] interface{} type-assert but a for-range-and-single-call loop instead of a switch -- extractCases finds no *ast.SwitchStmt, so that type resolves with an empty case list. Every key the handler legitimately writes into the corresponding hand-written map then reports as a false NotInTree MISMATCH. Confirmed live sweeping glue (MetadataInfoMap), ssm (aggregation-result Count fields) and personalize (Metrics) after fixing their dispatch-table resolution: all of their post-fix MISMATCH rows trace to this gap or to blind spot #2, not to a real dropped key -- see gopherstack-0kk8. Hand-check any MISMATCH whose written keys look like data (language codes, entity-type names, metric names) rather than members of a fixed schema before trusting it.

KNOWN BLIND SPOT #5, the struct-tag half added for gopherstack-v4a4: only a composite literal of a locally-declared struct type whose NAME ends in "Output" is recognised (this repo's overwhelming convention, 542 non-test occurrences) -- an anonymous struct literal, a differently-named type, or a struct built field-by-field via `var out X; out.Field = ...` rather than one literal contributes nothing and is invisible to this scan. A field with no json tag at all is assumed to marshal under its Go field name (encoding/ json's real default), which is occasionally wrong on its own and is not itself flagged. Neither gap is fixed to keep the corresponding NoWrittenKeys (N/A) path honest rather than silently under-reporting as clean. FIXED, in the same fallback: an UNEXPORTED field with no json tag used to fall back to its lowercase Go name too, which encoding/json never does -- it never marshals an unexported field regardless of a tag's presence. Found live triaging batch's newly-resolved (KNOWN BLIND SPOT #7) ops: ComputeEnvironment's deliberately-unexported `region` field (kept off the wire on purpose, see its own doc comment in services/batch/models.go) fabricated a false "region" MISMATCH on every op reachable from DescribeComputeEnvironmentsOutput -- five real ops, zero real bugs. structTagFields now skips any field whose Names[0] is unexported entirely (no key, no recursion into its type), pinned by TestRunCheck_StructTagIgnoresUnexportedField.

KNOWN BLIND SPOT #6, found live sweeping sqs for gopherstack-v4a4, FIXED for gopherstack-kiwf: a package can declare TWO handler functions for the same op name -- sqs hosts both a "handle<Op>" JSON handler (the one the pinned aws-sdk-go-v2 client actually talks to, confirmed against deserializers.go: sqs is awsAwsjson10_, case-sensitive) and a legacy "query<Op>" XML/Query-protocol handler for the same op string, left over from before SQS's protocol switch. Every op-to-handler write now goes through pkgScan.bindOp, which detects a second, DIFFERENT handler name claiming an already-bound op and refuses to pick either: the op is pulled out of normal checking and reported as its own ERROR row (AmbiguousOps / AmbiguousHandlers in checkResult) naming every conflicting handler, rather than one silently winning by file-processing order. Before the fix, sqs's ~13 dually-bound ops (handler.go's sqsDispatchTable vs query.go's queryActionTable, e.g. DeleteMessageBatch) resolved to the XML handler only because query.go sorts after handler.go, producing 85 MISMATCH rows that were all comparing the wrong handler's fields against the JSON SDK's key set -- confirmed by hand not real; sqs's real JSON handlers (handleDeleteMessageBatch etc.) already write correctly. TestRunCheck_AmbiguousHandlerBinding reproduces this exact shape as a fixture and fails against the unfixed tool.

KNOWN BLIND SPOT #3: a written key absent from the real reachable shape is reported identically whether it REPLACES a real required key (the real value is silently dropped on every client -- the gopherstack-6flj/zquj class this tool exists to catch) or sits ALONGSIDE all the real keys as a harmless extra the real client's typed struct has no field to receive. Telling these apart requires reading whether the handler also writes (or omits) the correspondingly-named real key -- confirmed both ways live in wafv2 (CheckCapacity's "ConsumedCapacity" replaced the real "Capacity" and dropped the value entirely; GetWebACLForResource's "LockToken" sits beside a correct response and is just ignored noise).

KNOWN BLIND SPOT #7, found live during the gopherstack-zquj re-sweep, FIXED for op-naming, VERIFIED by a full sweep of all 70 restjson1 services (not the 69 last estimated -- see the sweep note below): op resolution used to match the handler's dispatch-table KEY against the SDK's PascalCase operation name verbatim. Several restjson1 services key their dispatch table by REST path (or method+path) instead of the operation name -- account ("/acceptPrimaryEmailUpdate"), batch ("/v1/canceljob"), mgn ("DELETE tags"), xray ("/CancelTraceRetrieval") -- so every op reported UnresolvedOps even when HandlerOpsResolved showed the dispatch table itself resolved fully (mgn: 95/95, resiliencehub: 63/63, xray: 38/38). recoverOpName/resolveOpNames now recover the real op name from the handler's OWN name (this repo's "handle<Op>"/"json<Op>" convention, already used elsewhere in this file to find the handler in the first place) whenever the raw dispatch key itself isn't found in the SDK's op index, and refuse to guess (report AmbiguousOps instead) if that recovery would make two differently-bound keys collide on the same real op name.

SWEPT 2026-08-22 against all 70 restjson1 services (protocol verified per-service by reading each pinned deserializers.go directly, not assumed from services/_PROTOCOLS.md, though that doc's independent classification agreed): of the 13 services gopherstack-zquj's prior triage named as affected, 10 were CONFIRMED and newly resolve at least one op -- account, apigatewayv2, appmesh, batch, bedrock, mgn, opensearch, pinpoint, resiliencehub, xray -- and 5 of those 10 (account, bedrock, mgn, pinpoint, resiliencehub) now resolve EVERY op their dispatch table binds, exiting clean. The other 3 named services (amplify, appsync, outposts) get ZERO benefit from this fix: their dispatch KEY is a bare resource path ("branches", "webhooks", "capacity") whose single bound HANDLER itself internally multiplexes several distinct real ops by HTTP method (handleBranches serves both ListBranches and CreateBranch) -- there is no single real op name to recover, so recoverOpName correctly finds nothing and (where two such multiplexing handlers collide on one dispatch key, amplify's actual shape) resolveOpNames's own ambiguity guard fires instead. That is a genuinely different, deeper gap (a dispatch KEY resolving to more than one real op, not a naming mismatch) and is not fixed here. opensearch separately mixes in an unrelated-SDK gap: its already-PascalCase dispatch keys (CreateCollection, CreateAccessPolicy, ...) belong to OpenSearch Serverless, a different SDK package than classic opensearch's pinned deserializers.go, so no op-name recovery changes them; only 1 of its 96 SDK ops (a REST-path-keyed classic-domain op) was actually fixed by this change.

ENUM/TYPE-STRING DISPATCH TABLE MISREAD AS OP DISPATCH, found sweeping the 12-service PARTIAL tier for gopherstack-85e3, FIXED: a per-item classification switch or map keyed by an enum string that happens to look like an op name (apigateway's IntegrationType, glacier's job-type Action, lightsail's ResourceType, swf's DecisionType) got recorded as a real op-to-handler binding, then reported as a false "no deserializeOpDocument" ERROR once it failed SDK resolution -- 90%+ of the noise across those 12 services. filterEnumGroups groups every candidate op by the single switch statement or map/slice literal that bound it (ps.opGroup) and reclassifies the WHOLE group as FILTERED, not unresolved, only when it has 2+ candidates and EVERY one failed resolution: a real dispatch table drawn from the same SDK/prefix almost always resolves at least one member, so batting 0-for-N is the corroborating signal, not a name pattern. A lone failing candidate (N=1) has no sibling to corroborate it and is never filtered -- that stays ordinary KNOWN BLIND SPOT #7 territory. Filtered ops are still printed in full (FILTERED: ...), never silently dropped.

LAMBDA-TRIGGER-ENVELOPE POLLUTION, a further refinement of blind spot #2 found sweeping cognitoidp for gopherstack-ck9f, FIXED: cognitoidp's auth ops (SignUp, InitiateAuth, RespondToAuthChallenge, ...) each reach a shared Lambda-trigger-invocation helper whose own envelope map (version, triggerSource, userName, callerContext, request, response, ...) and each caller's own request/response maps got attributed wholesale to the op being checked -- ~85% of cognitoidp's 304 pre-triage mismatches, plus a coincidental CASE-MISMATCH on lowercase envelope keys (userName, challengeName, session) that collide with the op's own correctly-cased struct fields written by an unrelated code path. isBoundaryCall recognizes two structural shapes, neither a name pattern: (a) a call that crosses an injected-dependency boundary -- a method invoked on a struct field whose declared type is a package-local interface (cognitoidp's b.lambdaInvoker.LambdaTriggerInvoker), marked transitively up the call graph (computeCrossesBoundary); (b) a call to a same-package function whose signature converts a map into a slice of some OTHER named type (computeMapConversionFuncs, cognitoidp's sortedAttributeList(map[string]string) []attributeType -- map KEYS become list-item Name VALUES, never JSON keys). A composite literal or variable passed to either is excluded from writtenKeys UNLESS it is independently part of what the enclosing function itself returns (returnedRoots), so a value that legitimately crosses the boundary AND is handed back as real output is never suppressed. computeBoundaryProducerFuncs extends the same idea one hop further for a helper (cognitoidp's userAttrsWithSub) that exists solely to build a map every one of its callers feeds into a conversion func: its OWN return-bound writes are excluded too, at their construction site.

DETERMINISTIC-VS-GENUINE AMBIGUITY, a refinement of blind spot #6 found sweeping cognitoidp for gopherstack-ck9f, FIXED: cognitoidp keeps both a legacy handler and a hardened "Full"/"Accurate" variant for many ops, bound in separate OpsA/OpsB/OpsC family maps that dispatchTable() merges via SEQUENTIAL maps.Copy calls -- Go's maps.Copy overwrites on collision, so whichever family is copied LAST deterministically wins, unlike sqs's real ambiguity (two tables queried independently, never merged). Before this fix all 27 such ops were pulled into AmbiguousOps/ERROR and masked from checking entirely. resolveDeterministicOverrides finds, for a conflicting op, whether every candidate handler's own enclosing function (handlerSourceFunc) appears in the SAME maps.Copy chain to the SAME destination (findCopyChains) -- and only then resolves to the textually-last one, printing a DETERMINISTIC OVERRIDE line naming both sides so the choice stays independently verifiable against the assembler's own call order. An op whose conflicting handlers are never merged into a shared destination (sqs's shape) is left exactly as ambiguous as before.

SHARED-ERROR-HELPER POLLUTION, a named instance of blind spot #2 shape (b), found live re-sweeping medialive and quicksight for gopherstack-v4a4 (disclosed, not fixed -- same rationale as blind spot #1): both services route nearly every handler's error branch through one package-level helper (medialive's respondErr, quicksight's writeError) whose own map[string]any{"Message": ...}/{"Code": ..., "Message": ..., "Status": ...} literal gets attributed to every calling op's writtenKeys, producing a near-op-total MISMATCH set (medialive: 122/123 ops) that is entirely the error envelope, not the success response the SDK deserializer this tool diffs against ever covers. The tell: the SAME 1-3 keys recur, verbatim, across dozens of otherwise-unrelated ops. KEYCHECK_DEBUG_WALK confirms the call chain terminates in the shared helper every time.

OUTPUT-SUFFIX NAME COLLISION WITH AN INTERNAL BACKEND CONTRACT, found live sweeping kinesis for gopherstack-v4a4 (disclosed, not fixed): blind spot #5's "*Output"-suffix heuristic assumes that name means "this literal is what gets marshaled to the wire" (this repo's overwhelming convention). kinesis instead names its StorageBackend-interface return values <Op>Output (models.go's DescribeStreamOutput, ListShardsOutput, ...) as a domain-modeling convention unrelated to marshaling -- the actual wire response is built separately, per op, from correctly-tagged jsonXxx structs (handler_shards.go's jsonShardDescription, handler_consumers.go's jsonConsumer/jsonConsumerDescription, both confirmed against the pinned SDK's deserializers.go by their own doc comments) that this scan's naming heuristic never looks at because they don't end in "Output". Every one of kinesis's CASE-MISMATCH/MISMATCH rows this session traced back to this gap, confirmed by hand against handler_shards.go/handler_consumers.go: the real wire structs were already correctly tagged throughout.

FOUR MORE DISPATCH CONVENTIONS, added for gopherstack-zquj's 17-service wholly-unresolved tier (HandlerOpsResolved was 0 for all 17; 13 now substantially or fully resolve):

• DECLARED-OPS NAME-MATCHED HANDLER RECOVERY (resolveDeclaredOpsFallback). apigatewaymanagementapi, appconfigdata, mediastoredata, bedrockagent, elasticsearch, lambda and mwaa route via a maze of nested method/path if-else and switch trees with no dispatch TABLE at all -- nothing else in this file can bind an op it never sees as a table entry. This trusts two things already independently authoritative in this repo, never a guess: GetSupportedOperations()'s own literal []string return (this repo's hand-maintained, tested contract for what a service implements) and the handle<Op>/json<Op> naming convention. For every op literalStringReturns finds with no existing binding, if "handle"+op or "json"+op is declared in the package, it's bound, each under its OWN groupID so a genuinely-missing op recovered this way can never be swept into filterEnumGroups's enum-table reclassification. Deliberately conservative about extraction: only a direct `return []string{...}` composite (string literals or resolvable consts) is read -- iotwireless's GetSupportedOperations flattens several supportedXOps() helper calls in a for-range append loop and forecast's builds off a live h.ops map plus 8 appended literals, and BOTH yield nothing here rather than guessing. An op GetSupportedOperations() advertises with no discoverable handle<Op> function (mwaa's InvokeRestApi; forecast's ~63 ops that route through one generic data-driven execute(), never a per-op function at all) stays invisible -- never bound, never forced into UnresolvedOps -- the same as any other never-bound op has always been in this file; that silence is a known, disclosed limit of this convention, not new dishonesty.

• NAMED-STRUCT ROUTE-TABLE DISPATCH, an extension of recordSliceBindingDispatch. networkmanager's real shape is a package-level `type route struct{ fn dispatchFunc; op string; method string; pattern []string }` NAMED type (not glue's anonymous struct), scattered across many small per-resource-family []route{...} literals (globalNetworkCoreRoutes, siteRoutes, ...) rather than one big table. Gated on structHasFuncField: the named type must itself declare a func-typed field, the same "this literal is genuinely a table of handler functions" signal mapValueIsFuncType already requires for map dispatch, so an unrelated named-struct slice (a validation-rule list) that merely has a string field can never be misattributed. The handler half tries strict findHandlerSelector first, falling back to structFieldHandler (networkmanager's handlers are named dispatch<Op>, not handle<Op>).

• PAIRED STRING+HANDLER RETURN DISPATCH (recordPairedReturnDispatch). grafana and s3tables resolve routes through a tree of small `func (h *Handler) routeX(...) (string, dispatchFunc)` helpers whose terminal case co-locates the op name and its handler in one statement: `return "CreateWorkspace", h.handleCreateWorkspace`. There is no dispatch table anywhere -- no switch, no map, no slice literal -- so none of this file's other conventions could ever see it; the binding only ever exists at the return site. Gated on the enclosing func's OWN declared return signature being exactly (string, func-shaped) (funcReturnsStringAndHandler), checked structurally, not by scanning for the shape opportunistically, so an unrelated two-value return elsewhere in the package can never be misread even if it happens to return a string literal next to a handle<Op>-shaped selector. A nested closure's own return statements are excluded (ast.Inspect stops at any *ast.FuncLit) so they are never attributed to the enclosing func's signature.

• LOOSE SWITCH-CASE DISPATCH, an extension of findHandlerCall. polly and iotwireless switch on a real op-name string (`switch op { case opCreateWirelessDevice: ... }`, resolved via ps.constVals same as everywhere else) but call a bare-lowercase-named method (h.synthesizeSpeech, h.createWirelessDevice -- no handle/json prefix), the same "bare lowercase method value" shape already trusted for map dispatch (findHandlerSelectorLoose) but never extended to switch-case bodies before now. Gated on the case body being EXACTLY one return statement -- every real instance of this shape in this repo is a pure one-line delegation; a case that does real work first (validation, a helper call) before finally delegating -- a shape this repo's real op-dispatch switches never use -- stays strict-only, so the loose match can never grab an earlier, unrelated call and misattribute it as the handler. This is also why dynamodbstreams stays correctly unresolved despite its own single-statement-return switch cases (`case "DescribeStream": return dispatchDescribeStream(ctx, body, h.Streams.DescribeStream)`): the call target is a bare package-level function identifier (dispatchDescribeStream), not a `h.foo`-shaped selector, which matchHandlerCall has never matched in either strict or loose mode -- inherited from the original strict matcher's shape, not a new carve-out, but load-bearing here: extending to bare identifiers would bind DescribeStream to dispatchDescribeStream, whose own body crosses into the sibling dynamodb package (ddbbackend.ToWireGetRecordsOutput) that this same-package-only walk cannot follow, producing "0 written keys, N/A" -- a false clean, strictly worse than staying an honest ERROR. See gopherstack-zquj.

Sweeping what these four newly resolve found the same false-positive shapes recurring, not new bugs: SHARED-ERROR-HELPER POLLUTION again (apigatewaymanagementapi's writeModeledError embeds connectionId in every error envelope, credited to all 3 ops; appconfigdata's structured validation-error Details maps the same way), the OUTPUT-SUFFIX COLLISION class again (mediastoredata's internal ListItemsOutput/Item backend return types, never marshaled directly, collide with the *Output heuristic), KNOWN BLIND SPOT #4's dynamic-map gap again (elasticsearch's LimitsByRole map[string]Limits), an httpPayload/raw-blob passthrough gap adjacent to the existing header/status-line disclosure (iotwireless's GetPositionEstimate GeoJSON blob has no document deserializer at all to check against), and a JSON-encoded-as-a-STRING variant of blind spot #2 (lambda's GetLayerVersionPolicy builds an IAM policy map that gets marshaled into the wire response's "Policy" STRING field, not written as top-level document keys). One real bug found this way: gopherstack-wla0 (s3tables GetTable/ListTables write "tableBucketARN", which is not a real member of either shape at all -- the real member is "tableBucketId", a system-assigned identifier genuinely distinct from the ARN and never tracked anywhere in this service's internal models, so every real client's TableBucketId decodes empty on both ops; filed structural, not tag-fixed, since a real fix needs a new ID synthesized and threaded through table-bucket creation).

Usage:

go run ./cmd/keycheck -sdk <path to deserializers.go> -prefix awsAwsjson11_ -svc <service dir> [-op OpName]

Exit codes: 0 clean or N/A, 1 NOTHING in the service was verified (see FAIL-LOUD -- zero ops checked), 2 every dispatched op resolved and a real key mismatch was found, 3 SOME ops were resolved and checked (real MISMATCH data or a real clean result) but at least one other op remains unresolved or ambiguous -- a substantially-checked service, never to be conflated with exit 1's "nothing checked" (see VERDICT in the report, added because exit 1 alone let 13 substantially-checked services, cognitoidp alone 102 ops/304 mismatches, hide behind the same code as a service with zero dispatch resolved).