Documentation
¶
Overview ¶
Package packages loads and type-checks the Go packages codescan scans.
A Loader can resolve a package graph two ways, and WithStrategy picks between them:
- StrategyGoPackages delegates to golang.org/x/tools/go/packages — the go command's own answer, and the default.
- StrategyToolchainFree does the same job in pure Go, for the environments the first cannot reach.
Owning the choice here rather than at the call site keeps the two honest: options such as WithGoEnv mean the same thing under both, and a caller swaps strategies without rewriting how it asks for anything.
The motivation for the second strategy is that packages.Load shells out to `go list`, so it cannot run where there is no toolchain and no exec — a WASI guest being the case that forced the issue, and "whichever Go version happens to be installed" being the case that makes it worth having anyway.
The entry point mirrors the original on purpose:
upstream: packages.Load(cfg, patterns...) ([]*packages.Package, error) here: packages.NewLoader().Load(cfg, patterns...) ([]*packages.Package, error)
Config, Package, Error and LoadMode are type aliases of the upstream types, so a caller can switch back to packages.Load by changing the call and nothing else.
Whichever strategy runs, the result has the same shape: named packages with their files, their transitive imports, and full syntax plus type information.
Layout ¶
The go command's own complications live one level down, in the list subpackage: pattern matching, module boundaries, workspaces, vendoring, the module cache.
They are separated because the two halves are inherited from different places — this package is a simplified go/packages, that one is cmd/go — and because quirks are easier to check against upstream when they are quarantined rather than diffused. vfs holds the filesystem seam both halves read through.
The loading itself stays here: choosing a strategy, parsing, type-checking, and the two ways of standing in for a dependency that cannot be read from source — export data, and synthesis.
What the toolchain-free strategy does not implement ¶
The upstream Config fields Context, Logf, Fset, ParseFile, Tests and Overlay are accepted and ignored, as is Mode — there is no cheaper mode on offer. Overlay in particular is not supported by design: it is documented as slow, it requires every source file to be held in memory, and it works against demand-driven parsing.
Use WithFS instead, which virtualizes the filesystem one level lower.
Dir, BuildFlags and Env are honoured: they select which files a package is built from.
Package fields that codescan does not consume are left unset rather than hydrated.
StrategyGoPackages passes Config through to upstream untouched, apart from forcing Tests off and filling in Mode when the caller left it zero.
Build constraints ¶
This section describes StrategyToolchainFree; under StrategyGoPackages file selection is the go command's own.
File selection goes through go/build, which resolves //go:build expressions and GOOS/GOARCH filename suffixes in pure Go.
The loader always sets GOOS and GOARCH explicitly from the scan target and never inherits build.Default: a loader running inside a WASI guest would otherwise inherit GOOS=wasip1 and silently drop every _linux.go file, producing a different spec than the same scan run natively.
Cgo follows CGO_ENABLED from the environment, as the go command does, and files importing "C" are parsed alongside the ordinary ones rather than skipped. go/build sorts them into CgoFiles because compiling them needs the cgo tool first, but codescan only reads declarations and comments, and annotated types do live in them (go-swagger#1096).
The "C" pseudo-package itself is unloadable and resolves through synthesis, like any other import whose source cannot be found.
Index ¶
- Constants
- Variables
- type Config
- type Error
- type ExportOnly
- type GoEnv
- type LoadMode
- type Loader
- type Option
- func WithCompiledDependencies() Option
- func WithExportData(fsys fs.FS) Option
- func WithFS(fsys fs.FS) Option
- func WithGoEnv(env GoEnv) Option
- func WithOnExportOnly(fn func(ExportOnly)) Option
- func WithOnSynthesized(fn func(Synthesized)) Option
- func WithStrategy(s Strategy) Option
- func WithStubbedStdlib() Option
- type Package
- type Strategy
- type Synthesized
Constants ¶
const ( ListError = packages.ListError ParseError = packages.ParseError TypeError = packages.TypeError )
Error kinds, re-exported for callers that classify Errors.
Variables ¶
var ( // ErrImportCycle reports that the package graph loops back on itself. // // A well-formed Go program has no import cycle, so this means either the tree under scan does not compile, or the // resolver mapped two directories to the same import path. ErrImportCycle = errors.New("import cycle") // WebAssembly, which has no process model to run `go list` with. ErrStrategyUnavailable = errors.New("loading strategy unavailable in this build") )
Sentinels for the conditions a caller might reasonably branch on.
The detail (which package, which pattern) goes in the wrapping message; errors.Is matches the sentinel.
Functions ¶
This section is empty.
Types ¶
type ExportOnly ¶
type ExportOnly struct {
// Path is the import path.
Path string
// Reason says which half failed: the source was not on the filesystem, or it would not parse.
Reason string
}
ExportOnly reports a dependency whose types were read from export data but whose source was not available.
What it says about those types — its annotations — could not be read.
type GoEnv ¶
type GoEnv struct {
// GOOS and GOARCH select the platform the scanned code is built for: //go:build lines and _linux.go / _amd64.go
// filename suffixes resolve against them.
GOOS string
GOARCH string
// GOFLAGS supplies default command-line flags, in the go command's own format ("-tags=integration -mod=vendor").
//
// Flags given explicitly in Config.BuildFlags win, as they do for the go command.
//
// Only the flags that change what is built are interpreted by the toolchain-free strategy; the go/packages strategy
// passes the whole string through to `go list`, which understands all of them.
GOFLAGS string
// GOWORK selects the workspace.
//
// "off" disables workspace mode; a path names a go.work file explicitly; empty means search upwards from Config.Dir,
// as the go command does.
//
// It matters most for imports: inside a workspace, a module listed in `use` resolves to its directory rather than to
// the module cache, and missing that means reading a stale copy — or synthesizing an empty one.
GOWORK string
// GOEXPERIMENT enables toolchain experiments, in the go command's format ("jsonv2,noaliastypeparams").
//
// Each enabled experiment contributes a goexperiment.<name> build tag.
//
// The baseline is the set the codescan binary was itself built with, since go/build computes ToolTags at init from its
// own build configuration and there is no way to ask it about another toolchain.
// This is exact when both were built by the same Go release, which is the ordinary case, and approximate otherwise.
GOEXPERIMENT string
}
GoEnv is the part of the go environment that decides WHAT gets built, as opposed to how fast or where the output lands.
Every field here changes the set of files a package is made of, or the set of packages a pattern matches, so it changes the emitted spec.
That is why they are parameters rather than ambient state: a scan that silently inherits them from whatever shell it happened to start in is not reproducible, and — as GOOS/GOARCH proved — an inherited value reaches one loading strategy and not the other.
An empty field means "whatever Config.Env says, and failing that the process environment", which is what packages.Load does. Set a field to pin it.
type LoadMode ¶
LoadMode mirrors packages.LoadMode.
It is accepted for signature compatibility; this loader always produces syntax and type information, since that is the only mode codescan asks for.
type Loader ¶
type Loader struct {
// contains filtered or unexported fields
}
Loader loads and type-checks Go packages.
The Loader supports two loading strategies: using the standard go tool chain (golang.org/x/tools/go/packages, bends on "go list" and requires a go compiler to be installed) or as pure go, without toolchain.
The latter strategy remains experimental for now: we use it to support a WASI run. The go toolchain has evolved over time and we'll always support the standard toolchain strategy.
However, our pure go loader comes with some other benefits (significant less memory is used, may usee fs.FS) and we may transition to pure go being the default strategy in future releases.
A Loader is single-use per Load call in the sense that it caches nothing across calls. It is safe to keep one around and call Load repeatedly.
func NewLoader ¶
NewLoader returns a Loader reading through the real filesystem unless WithFS says otherwise.
func (*Loader) Load ¶
Load resolves patterns into type-checked packages, using whichever strategy the Loader was configured with.
See WithStrategy.
The signature mirrors golang.org/x/tools/go/packages.Load so the two are interchangeable at the call site.
Both strategies deliver the same shape: named packages with files, transitive imports, syntax and type information, so a caller reads the result the same way whichever one ran.
func (*Loader) ReadBackSource ¶
ReadBackSource parses a package's source onto it and reports whether it now carries syntax.
The scanner calls this when a declaration is wanted from a dependency the load took types-only.
This gives callers of the Loader the ability to call for parsed source on-demand: they are not compelled to resort to an eager full compilation of the entire dependency graph.
Both Loader strategies pass over such a dependency on the same reasoning — export data holds types and not comments, so a package that says nothing about its own types has nothing to say.
Both are asking the wrong question the moment some scanned code names one of its types as a model. A package says things in its comments and declares them in its source, and a definition renders from that declaration or not at all.
Asking here rather than reading every dependency up front keeps it affordable: the cost is one parse per declaration wanted, against one per dependency loaded, typically single digits against several hundred.
This is a method on the Loader because reading is: WithFS means a scan's whole world can be a virtual tree, and a read-back going to the real filesystem would answer from outside it.
There is deliberately no marker check because the caller has already established that the source is wanted, by a better question than the marker asks.
It is idempotent: asking twice only costs one parse.
func (*Loader) Strategy ¶
Strategy reports which strategy this Loader will actually use, resolving the WithFS override.
Exported because the caller's preference is not the answer: a virtual filesystem forces the toolchain-free strategy whatever was asked for, and options that only one strategy can honour (WithCompiledDependencies) are quietly dropped on the other. Anyone announcing what a load is about to do has to ask here rather than re-deriving it from the options, which is how such an announcement came to contradict the load.
type Option ¶
type Option func(*options)
Option configures a Loader.
func WithCompiledDependencies ¶
func WithCompiledDependencies() Option
WithCompiledDependencies takes dependency types from the compiler's export data instead of reading their source.
It applies to StrategyGoPackages only (go toolchain's loader). The toolchain-free strategy has WithExportData, which is the same idea supplied by hand; here the go command produces the data itself, from its build cache.
The speed is not marginal — parsing and type-checking dependencies is most of what a load does, and on a warm cache this removes nearly all of it. On a cold one it is markedly slower, because the dependencies have to be compiled before their export data exists.
It costs dependency SOURCE. Export data carries the exported type surface — fields, method sets and interface identity are all real — but no syntax and no comments, so anything a dependency says ABOUT its types is out of reach until something reads it.
For codescan that is load-bearing rather than incidental: for example go-openapi's strfmt annotates its own types, and those annotations give a strfmt.DateTime field its date-time format.
So the source comes back twice over: [attachAnnotatedDependencies] for the packages carrying the marker, and Loader.ReadBackSource for a declaration the spec turns out to want. This option skips the parsing and type-checking of everything neither of those reaches.
It stays an option here while being the default above, because the loader is the mechanism and the policy is the scanner's. The scanner adds a retry on top: `go list -export` BUILDS what it is asked about, so a scanned package that does not compile fails the load outright, and it is reloaded without this option rather than allowed to abort. See Options.CompiledDependencies.
func WithExportData ¶
WithExportData serves dependencies from pre-computed export data instead of reading their source.
fsys holds one file per package, named by import path with a ".export" suffix — the layout hack/genexportdata produces. Whole compiled archives are accepted as well as bare export sections.
It applies to dependencies only. The module under scan is always read from source: its comments are the annotations, and export data carries none.
This is the fast path with none of the fidelity loss of WithStubbedStdlib: the types are the ones the compiler computed, so fields, method sets and interface identity are all real. A full scan otherwise spends nearly all of its time parsing and type-checking its dependencies, and a WebAssembly guest pays a five- to six-fold compute tax on top of that.
hack/genexportdata produces the tree, as a directory or as a zip (archive/zip's reader is an fs.FS, so an embedded build carries one file and still reads per package).
It is consulted per dependency, and the decision is whole. A package whose source carries swagger annotations is read from source in the ordinary way: export data holds types and not comments, and the two cannot be combined after the fact, since go/types records what a type expression denotes behind an unexported field.
Every other package is taken from here and never parsed at all.
Nothing is lost by that, and little is given up: the saving was never in the handful of packages a scan reads, but in the closure behind them, which this still serves. Where a dependency's source cannot be found at all, WithOnExportOnly says so.
The data is only valid for the toolchain that produced it, since the export format is tied to the Go release. A package the tree does not cover falls back to source, and then to synthesis.
func WithFS ¶
WithFS makes the loader read source through fsys instead of the real filesystem.
Every path the loader is given — Config.Dir, the patterns, and the paths it derives from them — is then interpreted relative to the root of fsys, following io/fs conventions: slash-separated, no leading slash, no "..". A leading slash or an OS-specific separator is normalised away rather than rejected, so a caller can pass the same patterns it would use natively.
This is the seam that makes a virtualized source tree possible: an in-memory tree in a WASI guest, a testing/fstest.MapFS in a unit test, an archive reader, or an overlay composed from several roots.
The default (no WithFS) reads through the os package.
Notice that this also forces StrategyToolchainFree, overriding WithStrategy: `go list` runs against the real filesystem, so it could not honour fsys even if asked.
func WithGoEnv ¶
WithGoEnv pins the parts of the go environment that decide what gets built.
It replaces the older WithTarget: GOOS and GOARCH were never the only environment variables that change the answer, and having one of them explicit while the rest stayed ambient let the two strategies build for different platforms without anyone noticing.
func WithOnExportOnly ¶
func WithOnExportOnly(fn func(ExportOnly)) Option
WithOnExportOnly registers a callback fired once per dependency whose types came from export data without its source.
Export data plus source is the intended shape — the compiler's answer for the types, the file's own words for the annotations. This fires when only the first half was available, which is a real loss and an invisible one: the spec comes out valid and quieter.
func WithOnSynthesized ¶
func WithOnSynthesized(fn func(Synthesized)) Option
WithOnSynthesized registers a callback fired once per import path that had to be synthesized.
Without it, the loss is invisible: a package that only mentions a synthesized type in a field position type-checks cleanly and simply produces a thinner spec. Otherwise only the downstream wreckage surfaces: a value-position use of a fabricated type reads as an error in the scanned code rather than as a missing dependency.
func WithStrategy ¶
WithStrategy selects how the package graph is resolved.
The default is StrategyGoPackages. WithFS overrides this to StrategyToolchainFree: `go list` runs against the real filesystem, so it could not honour a virtual one even if asked, which means there is no coherent configuration being overridden.
func WithStubbedStdlib ¶
func WithStubbedStdlib() Option
WithStubbedStdlib keeps the standard library out of the package graph.
Standard-library imports are then synthesized from the names selected through them — opaque types carrying the right package path and name — instead of being parsed and type-checked out of GOROOT.
The trade is fidelity for reach. Everything keyed on a type's identity survives: codescan recognizes time.Time, json.RawMessage and friends by (package, name), never by shape. Everything structural is lost: a synthesized type has no fields to drill into and no method set, so a spec that renders json.RawMessage as a byte array, time.Duration as an integer, or that depends on a type implementing encoding.TextMarshaler, comes out different.
The reach bought is a small footprint and no Go installation: GOROOT no longer has to exist and no module cache has to be populated, which for a WASI guest or a browser is the difference between scanning and not.
It is not failsafe, and the failure mode is quiet — the spec comes out subtly thinner rather than erroring. Across codescan's own fixture corpus 133 of 138 scans are byte-identical; the rest lose a byte-array rendering, an integer format, or a TextMarshaler-derived string, and stdlib interfaces such as io.Reader have no identity recognizer to fall back on at all.
Loader.ReadBackSource does not recover any of this. It gives a dependency back the source the load declined to read, and a synthesized package never had source to decline: it was fabricated from names, not read from files.
Note that synthesis is not exclusive to this option: an import that cannot be resolved is synthesized whether or not the standard library was withheld. The option only makes it deliberate for the one dependency every Go program has.
Prefer a full graph wherever GOROOT is available; reach for this where it is not.
type Strategy ¶
type Strategy int
Strategy names a way of resolving a package graph.
Both strategies answer the same question and, across codescan's fixture corpus, give the same answer. They differ in what they need to run, which is the only reason a caller picks one.
const ( // StrategyGoPackages delegates to [golang.org/x/tools/go/packages], which resolves the graph by running `go list`. // // Authoritative — it is the go command's own answer — at the price of needing an installed toolchain and the // ability to start a process. // // The zero value, so a Loader that is asked for nothing behaves as codescan always has. StrategyGoPackages Strategy = iota // StrategyToolchainFree resolves the graph in pure Go: patterns to directories, build-constraint selection through // go/build, parse, type-check. // // Needs no toolchain, no GOROOT beyond the source it reads, and no exec — which is what makes it the only strategy // available inside a WebAssembly guest, or against a virtual filesystem. StrategyToolchainFree )
type Synthesized ¶
type Synthesized struct {
// Path is the import path.
Path string
// Pos is the import that triggered the synthesis — the first one seen for this path.
Pos token.Position
// Deliberate distinguishes the standard library withheld by [WithStubbedStdlib] from an import that simply could not
// be found.
//
// The first is the caller's own choice; the second is usually a mounting or module-cache problem.
Deliberate bool
// Cgo marks the "C" pseudo-package, which is neither of the above: it has no source anywhere to find, and is
// fabricated because this loader does not run the cgo tool.
//
// Worth telling apart, since "could not be resolved" invites a reader to go looking for something
// that was never there.
Cgo bool
}
Synthesized reports an import whose types were fabricated rather than loaded.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package list answers the question `go list` answers: given a pattern or an import path, which directory holds that package, and what is it called.
|
Package list answers the question `go list` answers: given a pattern or an import path, which directory holds that package, and what is it called. |
|
Package vfs is the loader's single point of filesystem contact.
|
Package vfs is the loader's single point of filesystem contact. |