reqfielddiff

Overview ¶

Command reqfielddiff finds SDK request-input fields the emulator never declared at all -- gopherstack-4glf's class, invisible to cmd/reqfieldscan by construction, since that scan enumerates fields the emulator's own decode structs DECLARE and checks each is read: a field with no struct field to enumerate is invisible to it. Confirmed concretely before this tool existed: apigateway's GetResources drops the SDK's documented Embed parameter, and "Embed" appears nowhere in services/apigateway; reqfieldscan reports zero findings for that service.

GROUND TRUTH, for one operation, is two independently-resolved field sets: the pinned aws-sdk-go-v2 <Op>Input struct's own top-level fields (sdkfields.go, adapted from cmd/structfielddiff's identical parse, which already "dumps SDK shapes for manual comparison" per gopherstack-4glf -- this tool automates the other half, the diff against the emulator, that issue says nothing joins), and the fields the union of every struct type the emulator's own handler for that op actually decodes into declares (structs.go/resolve.go). A field present in the first set with no normalized-name match in the second is reported.

RESOLVING THE EMULATOR'S DECODE TARGET is the hard half, and it is a STRICTLY HARDER problem than cmd/reqfieldscan's, not the same one reused: reqfieldscan's whole dispatch-table machinery is built around service.JSONOpFunc / service.WrapOp, and this tool's own confirmed ground truth sits OUTSIDE that world entirely. omics -- the service carrying the three-undeclared-defaulted-parameter finding this tool was built to catch -- dispatches through `map[string]func(*Handler,*echo.Context,string) error` closures that call a plain `h.handleStartRun(c)`, decoding into an ANONYMOUS INLINE struct with no WrapOp anywhere. apigateway -- the service carrying the other two confirmed instances (GetResources' Embed, GetBasePathMapping's DomainNameId) -- dispatches through `map[string]actionFn` (a locally named func type, not service.JSONOpFunc) into functions decoding a NAMED local struct via a bare json.Unmarshal call. cloudfront's third confirmed instance (ListDistributionsByRealtimeLogConfig's RealtimeLogConfigName) resolves only through a helper function called FROM the dispatched closure, whose return type -- not any decode call inside it -- IS the request struct.

So resolution here generalizes cmd/reqfieldscan's two building blocks rather than reusing them outright (dispatch.go, resolve.go):

• Dispatch-table recognition (isDispatchMapType) accepts ANY map[string]<func-shaped-value> composite literal -- a literal func type, a locally-declared named func type (apigateway's actionFn), or service.JSONOpFunc specifically -- not only the latter. The slice-of-struct binder shape generalizes the same way (any bind field of function type, not only one returning JSONOpFunc).
• A resolved dispatch value is unwrapped through func-literal closures (their first return statement, recursively) to either a service.WrapOp call (resolved exactly as cmd/reqfieldscan does, via the handler's own *In parameter type) OR a plain function/method call or reference, whose OWN BODY is then scanned directly (resolve.go's scanBody) for a decode signal: a json/xml.Unmarshal or echo Bind call binding a locally-known struct type, a call to a bare package function OR an `h.<Method>(...)` call whose declared return type IS a known struct (cloudfront's decodeXBody(c) shape), or a literal QueryParam/Param/FormValue("name") call, harvested directly as a declared wire name with no struct behind it at all (apigateway's resourceActions shape, and the many services that read echo params with no struct in between). The returns-a-struct signal is deliberately gated to that same bare-func-or-`h.<Method>` shape (matchReturnsStructCall's isBareOrHandlerCall) rather than any selector call: lookupFuncDecl resolves a method by NAME ALONE, ignoring the receiver's real type, so an ungated call to some other receiver (a backend/business-logic call like `lambdaBk.UpdateFunctionURLConfig(...)`) could resolve to a same-named method on a completely different type and merge in ITS return struct's fields as falsely "declared" -- exactly how UpdateFunctionUrlConfig's genuinely undeclared InvokeMode field went unreported end to end (gopherstack-id70): the backend method of the same name returns *FunctionURLConfig, the response struct, which also happens to declare InvokeMode. Exactly ONE hop of recursion into a `h.<Method>(...)` or bare package-func call the handler makes is followed (maxHop in resolve.go) -- never into `h.Backend.X`, so a backend's own internal field names can never leak in as false "declared" matches. This is the same single-hop discipline cmd/reqfieldscan discloses for its own field-coverage pass.
• When a dispatch-table entry doesn't exist AT ALL for an op, or resolves to nothing usable, a name-convention search (findHandlerByName) looks for "handle"+Op (then the suffixed variants cmd/reqfieldscan's package doc names -- Full/Accurate/ WithOpts -- then case-insensitively), and this repo's other observed convention, lowerCamel(Op)+"Action" / Op+"Action" (apigateway's own shape). resolveOp in resolve.go runs BOTH the dispatch-table and name-convention searches and UNIONS whatever each finds, rather than picking one and stopping the moment either "succeeds" -- deliberately over-inclusive, so an unresolved dispatch value can never suppress a handler sitting right there under its conventional name.

THIS TOOL'S OWN INHERITED BLIND SPOTS, checked against cmd/reqfieldscan's seven:

1. Slice-of-struct dispatch table (glue): generalized in binderFields, same as reqfieldscan's fix.

2. Local generic wrapper (cognitoidp's wrapAccuracy[I,O](fn)): collectLocalWrapOpWrappers is reqfieldscan's identical logic, type-parameter-agnostic since it only inspects the function body's first return statement.

3. Handler name suffixes (handle<Op>Full/Accurate/WithOpts): findHandlerByName tries all three explicitly.

4. Go type alias in the struct collector: resolveStructAliases, reqfieldscan's identical logic.

5. Anonymous inline struct decoding (opsworks, and THIS TOOL'S OWN omics ground truth): collectAnonReqStructs, reqfieldscan's identical logic, keyed by file:line.

6. Method receiver not bound during local-binding collection: bindFieldList binds fd.Recv exactly as cmd/reqfieldscan's coverage.go does.

7. A second in-package dispatch table behind suffixed/colliding names: the collectDispatchEntries half (unioning every map/binder literal package-wide, with no de-duplication by which "logical" table an op belongs to) is UNCHANGED FROM REQFIELDSCAN, still unpatched there, and this tool still inherits that exposure -- no concrete failing instance has surfaced for THAT half specifically, so it stays undesigned rather than guessed at.

   The OTHER half of this blind spot -- findHandlerByName's case-insensitive fold fallback, resolve.go -- has moved from theoretical to OBSERVED (gopherstack-fr30). It used to pick whichever match Go's randomized map iteration produced first, which is a determinism bug independent of whether a real collision exists; fixing that determinism bug required actually enumerating every collision, and the census (every op in every services/<dir>, current repo state) found 177 operations across 26 services where the fold matches 2+ names case-insensitively for the same op: amplify, apigatewayv2, appsync, cleanrooms, cloudfront, cognitoidentity, ec2, ecr, elbv2, glue, grafana, identitystore, lambda, lightsail, mwaa, opsworks, quicksight, rds, rdsdata, route53resolver, s3, sagemaker, servicediscovery, sesv2, sqs, transfer. Every single instance is the SAME shape: an exported PascalCase method on a Backend/ InMemoryBackend (business logic, e.g. appsync's `(b *InMemoryBackend) CreateAPI`, s3's `(b *InMemoryBackend) GetBucketACL`) colliding with the real unexported dispatch handler spelled identically but for case (appsync's `(h *Handler) createAPI`, s3's `(h *S3Handler) getBucketACL`) -- never two genuinely different DECODE sites. The tie-break rule (findHandlerByNameFold's doc comment) resolves every one of these correctly by preferring an unexported name over an exported one, so this is now a resolved, deterministic collision class rather than an open blind spot -- but it confirms the blind spot's premise (a same-named second table sitting behind the real one) is real in this repo, not hypothetical.

TRIAGE (triage.go) ranks each undeclared field by, in order: a documented default (defaultLanguageRe) -- a field the campaign's own history says produced 19 of its confirmed bugs, and the entirety of this tool's omics ground truth (RetentionMode/ScratchStorageMode/ StorageCapacity/StorageType on StartRun, each with a stated default, none declared); a filter/range/page-size field on a List/Describe/Search op; a sibling operation in the same service that DOES declare the same normalized field name; and SDK-required. A field whose doc comment starts "Deprecated:" is excluded from findings entirely, counted separately. Everything else ranks lowest, explicitly labeled "no strong signal" rather than omitted -- this tool reports a raw, ranked queue, it does not decide what's a bug.

WHAT THIS TOOL CANNOT TELL YOU, stated plainly rather than left implicit:

• It cannot distinguish a missing field from a deliberate, already- recorded structural gap (a capability this backend does not model at all -- a cross-account view, a VPC association). Roughly thirty such gaps are on record across this campaign, reasoned individually ("its enum has exactly one legal value and every record carries it", "the listing returns an empty slice unconditionally") -- this tool has no access to that reasoning and will re-flag every one of them. Every finding is a LEAD for a human or a sweep, never a verdict.
• It only compares an operation's TOP-LEVEL Input fields, never fields nested inside a sub-struct (a Filter type's own members, a nested config object). A field missing one level down is invisible to this scan by construction, the same way an undeclared field was invisible to cmd/reqfieldscan. This was a deliberate scope cut, not an oversight: every ground-truth instance this tool was validated against (omics' four StartRun parameters, apigateway's Embed and DomainNameId, cloudfront's RealtimeLogConfigName) is a top-level Input field, so the cut cost nothing against known ground truth -- but it means a nested filter struct missing members entirely would not be caught here.
• Name matching (normalizeWireName) is a case-and-separator-insensitive fold, nothing more. A wire name that diverges semantically from a simple case-fold of the Go field name -- an abbreviation expanded or contracted, a genuine rename -- will not match, and reads as a false "undeclared" finding.
• It says nothing about whether a DECLARED field is read correctly, or at all -- that's cmd/reqfieldscan's axis for whether it's read, and gopherstack-uox6's axis entirely for whether it's read CORRECTLY. A field this tool calls "declared" might still be silently ignored or misapplied; those are different bugs on different axes.
• The coverage guard (report.go) catches an implausible RESOLUTION number, never an implausible TRIAGE. A field ranked "no strong signal" that is in fact a real bug will not be elevated by anything here -- the triage signals are a ranking heuristic over a raw diff, not a classifier, and the tool says so in its own output rather than implying otherwise.
• QUERY-PROTOCOL FORM-READ DETECTION (formreads.go, gopherstack-99nj). An AWS query-protocol service (ec2, rds, s3, iam, autoscaling, elb, ses, cloudwatch, ...) reads its fields off a raw url.Values, not any struct decode call this scan otherwise recognises -- a field read this way is invisible to every other signal in this file, so a correctly-handled field still reads as undeclared. A blanket `.Get("literal")` signal was deliberately rejected as too risky: that name is used for every unrelated map/cache Get() call in this repo, and matching it by name alone would trade a real resolution gain for a worse one -- false "declared" matches that silently suppress genuine findings. What's actually implemented is narrower, because this scan already resolves each operation's own handler AND already has that operation's own SDK Input field names in hand (sdkfields.go) before it ever scans a body: the candidate key set a form-read call is allowed to match is restricted to THIS operation's own field names, normalized, plus a singular variant for the query-protocol convention where a plural field (KeyNames) is read from singular indexed member keys (KeyName.1, KeyName.2, ...). Two shapes are recognised, both gated on the receiver/argument being a url.Values- typed PARAMETER of the function being scanned (never a reassigned local, never a package-level cache): `vals.Get("Name")` directly, and a call to a package-level helper whose own first parameter is url.Values (ec2's parseMemberList, rds's extractMemberList/ extractIndexedList, iam's parseIndexedValues, autoscaling/elb's parseMembers, ses's parseSESMemberList, cloudwatch's parseMemberList/ parseDimensionsFromForm, ... -- recognised structurally by that signature, not by name) carrying a PascalCase string-literal argument. A nested-prefix literal ("AssociationTarget.InstanceId") is matched by its first dot-segment against the top-level field this scan is scoped to. A url.Values held in a reassigned local (`q := c.Request().URL.Query()`) and a fully chained accessor with no intermediate variable at all (`c.Request().URL.Query().Get(...)`) are both now covered too (isURLQueryCall, matchFormGetCall's second branch -- lambda's durable-execution family and apigatewayv2's ImportApi/validateFailOnWarnings are the confirmed instances; the latter drops addFormReadLiteral's uppercase-first-letter gate, since non-query-protocol services spell these camelCase). NOT covered, deliberately left as findings rather than guessed at: a helper that is a method rather than a bare package function; a nested-prefix literal more than one dot-segment deep; irregular English plurals singularVariant's simple suffix-strip doesn't cover; and an indexed read directly off url.Values itself (`q["Statuses"]`, distinct from mapFieldRead's map[string]any indexing below). Validated against ground truth: ec2's 26 hand-verified identifier-list fields and the six MaxResults/NextToken fields fixed in 427bd2b15 are no longer reported (both confirmed present before this change and absent after); ecs and omics -- neither query-protocol, neither using url.Values at all -- produce byte- identical findings before and after.
• dynamodbstreams decodes directly into the real aws-sdk-go-v2 input type itself (`var input dynamodbstreams.GetRecordsInput`, and a generic `dispatchStreamsOp[In any, Out any]` helper inferring In from the backend method's own signature) -- a foreign, imported qualified type this scan's struct collector (locally-declared types only) cannot see. Hand-confirmed: this makes dynamodbstreams's true coverage 100% by construction, not the "0/4, zero declared fields" the coverage guard reports for it -- a case where the guard's own loud failure is the CORRECT caution (a human must still read the flagged service to learn this), not a false alarm to silence.

SECOND ROUND OF FIXES (wrappers.go, mapfields.go, gopherstack-99nj's xhu2t slices 1-4 and gopherstack-7fve), closing five more blind spots gopherstack-xhu2t found roughly halve tier-1 on the services they hit -- all "already correctly read, invisible to this scan by construction", the same class as the query-form fix above, not real gaps:

1. A NAMED decode struct one indirection away from any call this scan recognised as a decode: iot's readBody(c, dst any) error decodes directly into its own `any`-typed parameter with no address-of at all, since the caller already passed one (`&input`) -- collectLocalDecodeDstWrappers finds every such wrapper structurally (a decode-verb call inside the function passing one of the function's OWN `any` params directly), matchDecodeDstWrapperCall then treats a call to it exactly like a direct decode call.
2. A LOCAL GENERIC dispatch-wrapper function (not service.WrapOp itself, and not a package-level function forwarding verbatim to it either -- collectLocalWrapOpWrappers' existing, narrower check) whose callback parameter's OWN signature carries the request struct: ssm's jsonOp[I,O](fn func(ctx,*I)(O,error)), apigatewayv2's handleCreate/handleCreateMulti/handleUpdate[I,O](..., backendFn func(I)(*O,error)) (request passed BY VALUE, not pointer), and dynamodb's handleOp[WireIn,...](..., toSDK func(*WireIn)*SDKIn, ...). collectGenericDecodeWrapperFuncs recognises the shape structurally (a generic func with a func-typed parameter whose own last parameter targets one of the wrapper's type parameters, body unexamined); resolveGenericCallbackReqType/matchGenericCallbackCall then resolve the CALL SITE's own callback argument (a dispatch-table value or a plain statement inside an already-resolved handler) to a concrete struct -- which only ever succeeds when that argument's own signature names a struct this scan already knows about, so matching an unrelated same-shaped generic helper (a Map/Filter utility, say) can never manufacture a false "declared" field on its own. One of these callback arguments is itself a subpackage-qualified selector (dynamodb's `models.ToSDKCreateTableInput`) -- see point 5.
3. QUERY, HEADER, and PATH-VALUE reads this scan's existing per-op literal matching couldn't reach at all: a local helper forwarding one of its own string parameters straight into a queryParamSelectors call (cleanrooms' qp(c,"key"), iot's parseInt32QueryParam(c,"name") -- collectQueryAccessorWrappers/matchQueryAccessorWrapperCall, recognised structurally the same way collectLocalDecodeDstWrappers is); an HTTP header read (`<expr>.Header.Get("X-Amz-Acl")`, gopherstack-7fve's own confirmed s3 instance) matched after stripping a known header prefix (matchHeaderReadCall, stripHeaderPrefix, formreads.go); and a REST-path value that never appears in any call at all because an upstream dispatcher already extracted it and threaded it straight through as a plain scalar parameter (matchOwnParamNames, apigatewayv2's handleUpdateStage(c, apiID, stageName string)) or a local bound from a `func([]string, T) string`-shaped path-segment accessor (matchPathSegmentLocalNames, quicksight's `namespace := seg(segs, segResID)`) -- both gated to hop 0 only and to formKeys, this op's own SDK field names, exactly like the query-form fix above.
4. A hand-decoded `body map[string]any` request (quicksight's own documented shape, and personalize's independent instance of the identical pattern): a field read via a strField/mapField/ boolField-style accessor call (isMapFieldAccessorSig, structurally any func(map[string]any, string) ...), or via direct `body["Key"]` indexing -- mapfields.go, gated to a scan-recognised map[string]any local exactly the way form-reads are gated to a recognised url.Values local, including a local established one call removed via a multi-return wrapper (`body, err := readBody(c)`).
5. A dispatch table built from a composite literal of a package-level `type X = map[string]F` alias (or `type X map[string]F` defined type) rather than a literal map type -- appstream's `opTable{...}`, spelled as a bare *ast.Ident at the composite-literal site, entirely invisible to a check gated on *ast.MapType alone (collectNamedDispatchMapTypes). And, orthogonally, a per-family REST path/method switch this scan cannot statically map to an operation name at all (lambda's dispatchSpecialRoutes -> handleESMRoute), reached instead through ONE new fully-deterministic exact-name candidate -- "handle"+verb+ACRONYM, built from the op name's own PascalCase words (abbreviatedHandlerName, pascalWords) -- for the one confirmed instance of this repo abbreviating a handler name to initials (handleCreateESM for CreateEventSourceMapping). Finally, dynamodb's handleOp callback argument (`models.ToSDKCreateTableInput`, point 2 above) is itself a bare function in an imported IN-REPO SUBPACKAGE (services/dynamodb/ models), one level of indirection past what a single-package struct/func collector can ever see on its own -- collectInRepoSubPackages finds every import textually rooted at this exact service's own "/services/<dir>/" path (never an arbitrary unrelated package by coincidence) and buildSubPackageIndexes parses it into the same structural index handlerResolveCtx keeps for the service package itself; resolveHandlerReqType and resolveTypeExprDef both consult it wherever a qualified selector appears, whether as a callback argument or as a LOCAL wrapper's own qualified parameter type (dynamodb's `toSDKPutItemInputChecked(input *models.PutItemInput)`). Validated against ground truth (before/after -dir counts, hand confirmed against source for every drop not already in this set): ssm 52->1, iot 36->6, quicksight 37->14, lambda 25->4, cleanrooms 22->1, apigatewayv2 22->1, appstream 19->0, dynamodb 17->3 -- zero services regressed (no tier-1 count increased) across the full repo scan.

Usage:

go run ./cmd/reqfielddiff                       # scan every services/<dir>
go run ./cmd/reqfielddiff -dir omics,apigateway  # scan only these
go run ./cmd/reqfielddiff -json out.json         # also write the full report as JSON

Exit codes: 0 no findings and no coverage warning in any scanned service, 1 a run error, 2 at least one non-deprecated undeclared field found, or at least one service tripped a coverage warning.