scanner

scanner — maintainer notes

This document is the long-form companion to the scanner package code. The source files keep godoc concise; complex invariants, design trade-offs, and known quirks live here.

The scanner package owns package loading and entity discovery. It turns a set of Go package patterns into a ScanCtx that exposes the classified per-decl inventory (meta, routes, operations, models, parameters, responses) consumed by the builder layer.

Table of contents

• §options — Options.DescWithRef shape and rationale
• §descwithref — the description-only-decoration $ref shape and why it has a flag
• §diagnostics — OnDiagnostic contract and experimental-API caveat
• §prune — PruneUnusedModels reachability and why it runs before name reduction
• §subtypes — discriminator subtype discovery — the reverse swagger:allOf index
• §model-lookup — GetModel vs FindModel — pure read vs implicit registration
• §classifier — detectNodes bitmask semantics and struct-annotation exclusivity
• §after-decl — AfterDeclComments — reading annotations inside / below a declaration
• §enum-values — where swagger:enum member values come from, and what the degraded reading can still see
• §clean-godoc — CleanGoDoc — filtering godoc syntax out of carried-over title / description prose
• §quirks-open — deferred follow-ups

§options — Options overview

Options is the externally-visible configuration struct. It is re-exported from the package root as codescan.Options. The default zero value is a valid configuration: every flag defaults to false and every slice/map defaults to nil.

Most fields are simple toggles (scope inclusion, debug, vendor extension suppression). Two fields carry non-trivial semantics that warrant the inline godoc and the deeper notes below:

• DescWithRef — controls the $ref shape used when a struct field resolves to a named type and its only decoration is a description. See §descwithref.
• OnDiagnostic — diagnostic callback hook. See §diagnostics.
• PruneUnusedModels — drop discovered definitions unreachable from any root, on top of ScanModels. See §prune.

§descwithref — description-only-decoration $ref shape

When a struct field's Go type resolves to a named type (so the spec emits a $ref to its definition) and its only field-level decoration is a description (no validations, no user-authored vendor extensions), the spec has two possible shapes:

1. Bare $ref — {$ref: ...}. The field's description is dropped. This is the conservative default when DescWithRef is false.
2. Single-arm allOf — {description: "...", allOf: [{$ref}]}. The description is preserved by wrapping the $ref in a single-arm allOf compound. This is JSON-Schema-draft-4 correct for sibling description.

DescWithRef=true opts into the second shape. The default is false because the bare-$ref shape interoperates more broadly with Swagger 2.0 tooling that does not implement the allOf compound.

When the field also carries validation overrides (pattern, enum, example, etc.) or user-authored vendor extensions, the allOf compound is mandatory regardless of DescWithRef — the override would be lost otherwise.

§diagnostics — OnDiagnostic callback

Options.OnDiagnostic, when non-nil, is invoked for every grammar.Diagnostic the builder layer records: lexer/parser warnings, semantic-validation failures from the validations package, and any future diagnostic class wired into the builder pipeline.

Contract:

• The callback fires once per diagnostic, in source order.
• Diagnostics never block the build. An invalid construct is silently dropped from the output spec; the explanation flows through this channel instead.
• The callback may be called from any per-decl builder; it is the caller's responsibility to make it goroutine-safe if the consumer ever drives codescan.Run concurrently (today it is single- goroutine, but the callback contract makes no such guarantee).

The diagnostic surface is experimental. Once the LSP integration matures the shape is expected to grow: typed severity classes, structural deduplication, per-position provenance. Callers that adopt OnDiagnostic today should treat the signature as subject to breaking change in a future minor release.

ScanCtx.OnDiagnostic returns the user-supplied callback verbatim; builders pipe diagnostics through it via common.Builder.RecordDiagnostic.

§prune — PruneUnusedModels reachability

Options.PruneUnusedModels is a modifier on ScanModels (-m). The three emission modes:

1. no ScanModels — only models transitively reachable from routes/responses/parameters are emitted (discovery-driven).
2. ScanModels — every swagger:model type is emitted, reachable or not.
3. ScanModels + PruneUnusedModels — discovery runs as in (2), then unreachable definitions are pruned again. The middle ground a shared-library scan wants: keep only the $ref'd subset (go-swagger/go-swagger#2639).

Without ScanModels the flag is a no-op (the set is already reachable-only) and raises one positionless scan.pruned-unused Hint.

Shared objects pruned first (C4). Before the definition walk, the shared parameters (#/parameters/*) and responses (#/responses/*) that no operation and no path-item references are themselves pruned (spec/prune.go, pruneUnusedSharedObjects; the read-only "is referenced" mirror is collectSharedRefs). InputSpec-supplied shared objects are pinned (never pruned), mirroring the definitions rule. Each drop raises a located scan.pruned-unused Hint. Because this precedes the definition walk, a definition kept alive only by a now-pruned shared object becomes prunable in turn. A pruned shared response's buffered provenance anchors are dropped (DropDeferredOrigins) so none dangle — shared-response anchors are buffered (BeginDeferredOrigins) and flushed verbatim after the prune only when PruneUnusedModels is set, so the non-prune anchor stream is unchanged.

Reachability. Roots are the paths (operation body parameters + response schemas), the surviving shared responses and parameters, and every definition supplied via InputSpec. Overlay definitions are pinned: never pruned and seeded as roots so their $ref targets survive. The walk (spec/prune.go, collectDefRefs) is the read-only mirror of the ref-rewriter (reduce.go, rewriteSchemaRefs) and must cover the same container set; a visited set handles recursive / cyclic models. A model referenced only by another unreferenced model is itself pruned.

Ordering — before name reduction. The prune runs before reduceDefinitionNames, in the fully-qualified #/definitions/<pkgpath>/ <name> key namespace. This is the point of the feature, not an implementation detail: name reduction deconflicts cross-package leaf collisions (a.Thing / b.Thing → AThing / BThing). Pruning an unused twin first means the collision never materialises, so the surviving model keeps its bare leaf name — no spurious concat churn. Each prune raises a located scan.pruned-unused Hint; the buffered provenance for a pruned node is dropped so no anchor dangles. The collision renames the reduce stage does perform are surfaced as scan.renamed-definition Hints (located at the Go type).

Discriminated families are kept whole. A subtype $refs its base, never the reverse, so the walk above cannot see the subtypes of a discriminated base and would prune a polymorphic family down to its base alone. A reachable definition carrying discriminator therefore also marks its subtypes reachable, via the reverse swagger:allOf index (spec/subtypes.go, subtypeKeysOf). Note the rule keeps a reached base's family — it does not make bases roots: a discriminated base nothing references is still pruned, together with its subtypes. See §subtypes.

§subtypes — discriminator subtype discovery

Interface-based polymorphism emits a base definition carrying discriminator plus one allOf: [{$ref base}, {own props}] definition per subtype — a struct embedding the base under swagger:allOf.

The reference direction is the problem: a subtype $refs its base; nothing $refs a subtype. So a route that references only the base reaches the base and stops, and the emitted family is the base alone — useless to a consumer that has to unmarshal the polymorphic payload (go-swagger/go-swagger#1913). Before v0.37 the only way to get the subtypes was ScanModels, which emits every annotated model whether it belongs to the family or not.

Reverse index. spec/subtypes.go builds, once per scan, a base Go type identity → subtype declarations map from the model index (TypeIndex.Models, which classification populates whether or not ScanModels is set — that independence is what makes the pull possible). A subtype relation is an embedded member carrying swagger:allOf — a struct's anonymous field or an interface's anonymous interface, which is how a mid-level type in a multi-level hierarchy is written:

• the pointer is unwrapped (*Base composes like Base), and an alias embed is indexed under both its own and the aliased type's identity, so the relation is found whichever definition the allOf member ends up $refing under RefAliases / TransparentAliases;
• swagger:ignore on the embed drops it, exactly as in the schema builder — the index never claims a relation the document lacks;
• a plain (unannotated) embed is not a subtype: it inlines the base's properties, no allOf member. DefaultAllOfForEmbeds is deliberately not honoured here — it is a rendering knob, and letting it decide which definitions exist would be a surprising coupling.

The index is keyed by Go type identity (<pkgpath>.<TypeName>), not by swagger name: it is the one fact both ends can compute without knowing the other's swagger:model override. Entries are ordered by definition key so the pull order — and hence the Hint order — does not follow map iteration.

Two hooks, one per hole.

1. Discovery (spec.go, buildDiscoveredSchema → discriminatedSubtypesOf): when a definition has just been built and carries a discriminator, its subtypes are appended to s.discovered, so the existing fixpoint loop builds them, discovers their dependencies, and cascades if a subtype is itself a discriminated base. Each genuinely new pull raises a located scan.discovered-subtype Hint. A no-op under ScanModels, where every model is built up front anyway.
2. Prune reachability (prune.go): see §prune — required because under ScanModels the family is built and then lost, not never-built.

The gate is the built definition's own discriminator (isDiscriminated), not a source-level re-derivation. It reads identically for an interface base with a discriminator: true member and a struct base with a discriminator: true property, and it is the same fact the emitted document exposes. A base with no discriminator pulls nothing: its allOf users are ordinary compositions, not a polymorphic family.

Multi-level hierarchies. A mid-level type — a subtype that is itself a base — renders as allOf: [{$ref parent}, {own props, discriminator}]: its properties, and therefore its discriminator, sit in its own compound member, not at the top level. So the gate looks for an inline discriminator anywhere in the definition's own schema. The $ref member is deliberately not followed: a leaf must not inherit its base's discriminator, or every subtype would pull in its own siblings. Because hook A feeds the discovery fixpoint, the levels cascade — the root pulls the mid-level, and the mid-level (only just pulled in itself) pulls the leaves on the next round.

Fixtures: fixtures/enhancements/discriminated-subtypes (edges/ holds the embed-shape corner cases, in a family no route references — which also locks the other half of the gate: an unreached base pulls nothing) and fixtures/enhancements/discriminated-subtypes-nested (two-level hierarchy: Shape → Polygon → Square/Triangle).

§model-lookup — GetModel vs FindModel

ScanCtx exposes two lookup helpers with similar signatures but different side-effect contracts. The choice between them is load-bearing for the shape of the emitted spec.

GetModel(pkgPath, name) — pure read

Looks up a model decl across three sources, in order:

1. Models — decls annotated with swagger:model. Always emitted as top-level definitions regardless of lookup.
2. ExtraModels — decls discovered as dependencies of other emitted shapes. Already enqueued for top-level emission.
3. FindDecl — fall through to a syntactic search over the loaded packages.

No side effect. A FindDecl hit through GetModel does not register the decl in ExtraModels. Callers that want the lookup to also surface the decl as a top-level definition must follow up with AddDiscoveredModel explicitly.

FindModel(pkgPath, name) — implicit registration

The older sibling of GetModel. It does the same three-source lookup, but a FindDecl hit also writes the decl into ExtraModels as a side effect.

FindModel is deprecated. The implicit registration surprises readers and pulls stdlib types (notably time.Time, json.RawMessage) into the spec's top-level definitions when they should be inlined where referenced. Builders that need the registration should use the explicit GetModel + AddDiscoveredModel pair.

AddDiscoveredModel — explicit registration

Registers a decl in ExtraModels. No-op for decls already in Models (annotated decls are emitted unconditionally — registering them as discovered would create a Models↔ExtraModels bouncing loop in the spec orchestrator's joinExtraModels pass). Nil and Ident-less decls are silently ignored, which is defensive against the scanner emitting partial decls during error recovery.

§classifier — detectNodes bitmask

TypeIndex.detectNodes scans every comment group in a file and returns a bitmask of detected annotation kinds. Each kind drives downstream processing:

route, operation, and meta accumulate freely across comment groups in a file. The three struct-level annotations (model, parameters, response) are mutually exclusive within a single comment group — a struct cannot simultaneously be a model and a parameters bag, for instance. checkStructConflict enforces the rule per comment group and returns an error if the constraint is violated.

The annotation vocabulary recognised by the classifier is a closed set. Unknown annotations beginning with swagger: raise a classifier error. A handful of annotation tokens (strfmt, name, enum, default, alias, type, title, description, …) are recognised but produce no bit — they are field/decl-level decorations that downstream builders parse out of the comment block directly. (title / description are the godoc title/description overrides; see the schema builder's §user-overrides.)

§after-decl — AfterDeclComments

Options.AfterDeclComments (opt-in, default false) lets swagger annotations live inside a declaration or inlined as a trailing comment, so the godoc above the declaration stays clean and human-facing. It is solely a scanner concern — the located comments are folded into the comment source the builders already consume (EntityDecl.Comments and ast.Field.Doc), so the grammar and builders are untouched. Same annotation grammar, no new syntax.

What the scanner folds, by shape (index.go):

The clean godoc above still provides the title/description: the merged group is docAbove ++ located, and because positions stay ascending (doc above < the inside/trailing comment below), the grammar reconstructs a blank-line gap and parses it without change. Discovery works because detectNodes already scans every file.Comments group (the file bitmask flips), and the merged EntityDecl.Comments makes the per-decl HasModelAnnotation gate pass.

Idempotency. Decl-level folding is pure construction — ts.Doc is never mutated, so re-processing is safe with no guard. Field-level folding is the only place the shared AST is mutated (Field.Doc is repointed to the merged group), guarded by TypeIndex.enrichedFields so a field is rewritten at most once.

Routes / operations are already position-agnostic (collectRoute/OperationPathAnnotations scan all file.Comments), so a swagger:route inside a func body is discovered with or without this option.

Out of scope. A standalone const X = … // swagger:enum: swagger:enum is type-based (it resolves a type and collects that type's consts via FindEnumValues), so a lone const is not an enum carrier and has no builder semantics today. Supporting it would mean new builder behaviour, which this scanner-only feature deliberately avoids. Nested/anonymous inline structs are likewise not enriched (only named struct type decls are walked).

§enum-values — reading swagger:enum members

FindEnumValues walks the const declarations of a package and emits one row per constant whose type is the annotated enum type. Two decisions shape it.

Membership is decided per name, from the type-checker

The spec's syntactic type (vs.Type) is not usable as the membership test, because inside an iota block only the first spec carries a type at all:

const (
    Sunday Weekday = iota   // Type=Weekday  Values=[iota]
    Monday                  // Type=<nil>    Values=<nil>
    Tuesday                 // Type=<nil>    Values=<nil>
)

Monday and Tuesday inherit both implicitly, so a syntactic reader sees two specs that declare nothing. Membership therefore comes from TypesInfo.Defs[name].(*types.Const).Type() — the type the checker assigned — which also covers a constant declared without a written type (const Extra = StatusOn).

The test is type identity, not name: the named type must also come from the package being walked. A constant declared in the annotated package can perfectly well have an imported type (const ForeignDay foreign.Weekday = 13 next to a local Weekday enum), and it is a member of neither. The syntactic reading ruled that out structurally — a qualified type is a selector expression, not the bare ident it required — so the package check is what keeps the type-checked reading from being wider than the one it replaced.

An enum cannot be hosted on an alias to a basic type (type Unsigned = uint64): the checker erases the alias, so const Zero Unsigned = 0 is indistinguishable from any other uint64 constant and there is nothing left to match on. The annotation is a no-op there — as it was before this change, since the classifier never reaches an alias decl either. An alias to a named enum type (type Weekday2 = Weekday) is fine: the underlying named type survives.

Values come from the type-checker, not from the literal

A const's right-hand side is only incidentally a literal. It can be iota, an expression (1 << 3), a reference to an earlier member (Prev * 2), a rune literal ('a', whose constant is the integer 97), or true / false — which are predeclared identifiers, since Go has no boolean literal token. Reading the value out of the syntax means reimplementing Go's constant evaluator: iota counting, implicit repetition, and constant folding.

go/types has already done that, exactly and with arbitrary precision, so enumConstantValue converts the resulting constant.Value by kind (Int → int64, or uint64 past MaxInt64; Float → float64; String; Bool). A constant with no JSON representation (complex) or one the checker could not evaluate is dropped rather than emitted as a null member.

The degraded reading. When the package only partially type-checked (see ErrDegradedLoad), a constant may have no value in Defs. Rather than let an annotated enum vanish, enumValue falls back to the literal syntax — a lone literal, optionally signed, with rune literals and raw/escaped strings handled. It is a strict subset: iota, expressions and references are invisible to it by construction, and its values keep the kind their literal implies rather than the kind of their declared type. The builder's validations.CoerceConstant closes that last gap — see §enum-const-values.

§clean-godoc — CleanGoDoc

Options.CleanGoDoc (opt-in, default false) rewrites godoc-specific syntax that reads as bracket noise when a title / description is carried from godoc into the spec, and recomposes resolvable doc-links to the name the referenced schema is exposed under. Off ⇒ output is byte-identical.

The scanner side is thin: it holds the flag (CleanGoDoc()) and a shared mangling.NameMangler (Mangler(), used for humanization). The transform, the consumption-seam wiring, the go/types resolver, and the post-reduce marker substitution all live in the builders — see internal/builders/godoclink/README.md for the two-phase marker contract and the full mechanics.

Like swagger:title / swagger:description (overrides) and AfterDeclComments, this is part of the clean-godoc cluster: keep the Go-facing doc clean while the API spec carries curated text. Crucially it touches only godoc-derived prose — author-written overrides (harvested separately) are never filtered.

§quirks-open — deferred follow-ups

Where open quirks live. This section documents caveats of this package. The project-wide register of what is actually open — verified, with the stale historical registers called out — is .claude/plans/quirks-open.md.

• FindModel deprecation. The deprecated alias is still on the ScanCtx surface for in-tree callers. Once every builder has been audited and migrated to the GetModel + AddDiscoveredModel pair, the deprecated method can be removed in a future major release.
• Recognised-but-unused annotation tokens. detectNodes recognises a list of field-level tokens (strfmt, name, discriminated, file, enum, default, alias, type, allOf, ignore, title, description) only to avoid raising the "unknown annotation" error. Promoting them to per-file bits would let downstream builders skip whole files that carry no decorations — an optimisation, not a correctness change.
• shouldAcceptTag precedence. When both includeTags and excludeTags are populated, includeTags wins (a tag in includeTags admits the operation even if it also appears in excludeTags). This is deliberate but easy to mis-read; an explicit "the include list takes precedence" doc on Options would help callers, but the field-level prose is already dense.