Documentation
¶
Overview ¶
Package python implements dependency extraction for Python codebases.
This file holds the naming rules. Python package names are compared after normalization (PEP 503): Flask, flask and flask are all one project, and an index, a lockfile and a wheel may each spell the name differently. Every name read from project metadata goes through NormalizeName before it is used as an identity anywhere.
Index ¶
- func InstalledNodeList(dists []*InstalledDistribution, includeFiles bool) (*sbom.NodeList, error)
- func NormalizeName(name string) string
- type Artifact
- type Decomposer
- type Dependency
- type DirectURL
- type Environment
- type InstalledDistribution
- type InstalledFile
- type Lockfile
- type Marker
- type Options
- type Package
- type PoetryDependency
- type PoetryFile
- type PoetryLockfile
- type PoetryMetadata
- type PoetryPackage
- type PoetrySource
- type PyPIClient
- type PyProject
- type Source
- type Version
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func InstalledNodeList ¶
func InstalledNodeList(dists []*InstalledDistribution, includeFiles bool) (*sbom.NodeList, error)
InstalledNodeList builds the graph an installed environment holds. When includeFiles is set, every file the RECORDs own becomes a node related to its package.
func NormalizeName ¶
NormalizeName returns the canonical form of a Python package name: every run of dashes, underscores and dots becomes one dash, and the result is lowercased. This is the form purls and lockfile lookups use.
Types ¶
type Decomposer ¶
type Decomposer struct{}
Decomposer reads dependency data from Python codebases managed by uv. The whole graph comes from uv.lock: versions, edges, markers and artifact hashes are all resolved in the lock, so extraction needs no network and no Python interpreter.
func New ¶
func New() *Decomposer
func (*Decomposer) DefaultOptions ¶
func (d *Decomposer) DefaultOptions() any
DefaultOptions returns the default options for the Python decomposer.
func (*Decomposer) Extract ¶
func (d *Decomposer) Extract(opts *api.DecomposerOptions) (*sbom.NodeList, error)
Extract builds the dependency graph the target environment sees, from whichever lockfile the codebase has. A project migrating between tools may carry both; uv.lock wins, being the one uv keeps current.
func (*Decomposer) FindCodeBases ¶
func (d *Decomposer) FindCodeBases(index *code.PathIndex) ([]string, error)
FindCodeBases locates Python codebases by their lockfiles.
func (*Decomposer) Requirements ¶
func (d *Decomposer) Requirements(_ *api.DecomposerOptions) []api.Requirement
Requirements returns nothing: extraction is pure Go, offline.
type Dependency ¶
type Dependency struct {
Name string `toml:"name"`
// Version and Source disambiguate the target when the resolution
// forked: a lock holding numpy at three versions states which one this
// edge points to. Both empty mean the name alone identifies it.
Version string `toml:"version"`
Source Source `toml:"source"`
// Extra lists the target's extras this dependency enables, as in
// requests[socks].
Extra []string `toml:"extra"`
// Marker is the environment marker the edge is conditional on, empty
// when it holds everywhere.
Marker string `toml:"marker"`
}
Dependency is one edge of the resolved graph.
type DirectURL ¶
type DirectURL struct {
URL string `json:"url"`
VCSInfo struct {
VCS string `json:"vcs"`
CommitID string `json:"commit_id"`
RequestedRevision string `json:"requested_revision"`
} `json:"vcs_info"`
DirInfo struct {
Editable bool `json:"editable"`
} `json:"dir_info"`
}
DirectURL is a parsed direct_url.json.
type Environment ¶
type Environment struct {
// The PEP 508 environment variables, named as the markers name them.
OSName string // os_name: "posix", "nt"
SysPlatform string // sys_platform: "linux", "darwin", "win32"
PlatformSystem string // platform_system: "Linux", "Darwin", "Windows"
PlatformMachine string // platform_machine: "x86_64", "aarch64", "arm64", "AMD64"
PythonVersion string // python_version: "3.12"
PythonFullVersion string // python_full_version: "3.12.0"
ImplementationName string // implementation_name: "cpython"
PlatformPythonImplementation string // platform_python_implementation: "CPython"
// Extras are the enabled extras, normalized. The "extra" marker
// variable evaluates against this set.
Extras []string
}
Environment is the concrete target environment markers evaluate against: one operating system, one architecture, one Python version, one set of enabled extras. A lockfile resolves for every environment at once; an extraction is for one of these.
func NewEnvironment ¶
func NewEnvironment(goos, goarch, pythonVersion string) (*Environment, error)
NewEnvironment builds the environment for an operating system and architecture in Go's vocabulary (GOOS and GOARCH names), and a Python version stated as "3.12" or "3.12.4". Empty os or arch mean the platform unpack itself runs on.
The translation is the reason this constructor exists: Python spells the same platform several ways, and the values must agree with what real interpreters report — linux/arm64 is aarch64, but a Mac's arm64 stays arm64 and Windows spells it ARM64.
type InstalledDistribution ¶
type InstalledDistribution struct {
// Name is normalized; the dist-info directory spells it however the
// wheel did (PySocks-1.7.1.dist-info).
Name string
Version string
Summary string
RequiresPython string
HomePage string
ProjectURLs map[string]string
// The three licence tiers, as METADATA states them; LicensesFromMetadata
// of the node builder triages them.
LicenseExpression string
License string
Classifiers []string
// RequiresDist are the declared dependencies, raw PEP 508.
RequiresDist []string
// Installer names the tool that installed the package (the INSTALLER
// file), and Requested says the install was asked for rather than
// pulled in (the REQUESTED marker, PEP 376).
Installer string
Requested bool
// DirectURL is the provenance of a package installed from a URL or a
// repository rather than an index (direct_url.json, PEP 610).
DirectURL *DirectURL
// Files is the RECORD: every file the installation owns.
Files []InstalledFile
// Path is the dist-info directory, relative to the FS the environment
// was read from.
Path string
}
InstalledDistribution is one installed package, read from its dist-info.
func FindDistributions ¶
func FindDistributions(fsys fs.FS) ([]*InstalledDistribution, error)
FindDistributions walks a filesystem and reads every installed distribution on it, wherever its site-packages lives: system interpreters, virtualenvs and --target directories all qualify. A distribution whose metadata cannot be read is skipped: one broken package should not hide an environment.
func ReadDistribution ¶
func ReadDistribution(fsys fs.FS, dir string) (*InstalledDistribution, error)
ReadDistribution reads one dist-info directory.
type InstalledFile ¶
type InstalledFile struct {
// Path is relative to the site-packages directory (entries may climb
// out of it: ../../../bin/pip).
Path string
// Algorithm and Digest are the recorded hash, the digest in the
// urlsafe base64 RECORD states it in. Empty for the files an
// installer rewrites, RECORD itself among them.
Algorithm string
Digest string
Size int64
}
InstalledFile is one RECORD entry.
type Lockfile ¶
type Lockfile struct {
// Version is the lock schema version, and Revision the revision within
// it. Additions bump the revision; only breaking changes bump the
// version, which is the one compatibility is judged by.
Version int `toml:"version"`
Revision int `toml:"revision"`
// RequiresPython is the project's Python requirement, as a PEP 440
// specifier set such as ">=3.10".
RequiresPython string `toml:"requires-python"`
// ResolutionMarkers split the environments the resolution forked over;
// SupportedMarkers restrict the environments the project supports at
// all (tool.uv.environments). Both absent mean one universal
// resolution.
ResolutionMarkers []string `toml:"resolution-markers"`
SupportedMarkers []string `toml:"supported-markers"`
Packages []Package `toml:"package"`
}
Lockfile is a parsed uv.lock.
func ParseLockfile ¶
ParseLockfile reads a uv.lock document.
func ReadLockfile ¶
ReadLockfile reads a uv.lock from a file.
type Marker ¶
type Marker struct {
// contains filtered or unexported fields
}
Marker is a parsed environment marker, ready to evaluate.
func ParseMarker ¶
ParseMarker parses an environment marker expression such as "python_full_version < '3.11' and sys_platform == 'linux'".
type Options ¶
type Options struct {
// Platform targets an operating system and architecture, in GOOS/GOARCH
// vocabulary ("linux/arm64"), an os alone ("windows"), or empty for the
// platform unpack runs on. A lockfile resolves for every platform at
// once; the extraction reads it for this one.
Platform string
// PythonVersion targets an interpreter version, "3.12" or fuller.
// Empty picks the newest version the lockfile's own resolution forks
// mention, so the default describes a current install with no Python
// on the machine unpack runs on.
PythonVersion string
// Concurrency controls the parallel requests to PyPI when enriching
// (default: 10).
Concurrency int
}
Options configures the Python dependency extraction.
type Package ¶
type Package struct {
Name string `toml:"name"`
Version string `toml:"version"`
Source Source `toml:"source"`
// Dependencies are the package's resolved runtime dependencies.
// OptionalDependencies holds those of each extra, keyed by extra name,
// and DevDependencies those of each dependency group (PEP 735), keyed
// by group name.
Dependencies []Dependency `toml:"dependencies"`
OptionalDependencies map[string][]Dependency `toml:"optional-dependencies"`
DevDependencies map[string][]Dependency `toml:"dev-dependencies"`
// ResolutionMarkers claim this entry for part of the environment space
// when the resolution forked: the numpy for one Python version range
// carries the markers telling it apart from the numpy for another.
ResolutionMarkers []string `toml:"resolution-markers"`
// Sdist and Wheels are the package's distribution artifacts. Hashes
// live here: PyPI has no package-level hash, only per-artifact ones.
Sdist *Artifact `toml:"sdist"`
Wheels []Artifact `toml:"wheels"`
}
Package is one locked package. A package appears once per resolved version: the same name shows up several times when the resolution forked over environments, each entry claimed by its resolution markers.
type PoetryDependency ¶
PoetryDependency is one edge of the resolved graph, normalized.
type PoetryFile ¶
PoetryFile is one distribution artifact.
func (*PoetryFile) HashValue ¶
func (f *PoetryFile) HashValue() (algorithm, value string)
HashValue splits the file's hash into its algorithm and hex value, written as "sha256:abc...".
type PoetryLockfile ¶
type PoetryLockfile struct {
Packages []PoetryPackage `toml:"package"`
Metadata PoetryMetadata `toml:"metadata"`
}
PoetryLockfile is a parsed poetry.lock.
func ParsePoetryLockfile ¶
func ParsePoetryLockfile(data []byte) (*PoetryLockfile, error)
ParsePoetryLockfile reads a poetry.lock document.
func ReadPoetryLockfile ¶
func ReadPoetryLockfile(path string) (*PoetryLockfile, error)
ReadPoetryLockfile reads a poetry.lock from a file.
type PoetryMetadata ¶
type PoetryMetadata struct {
LockVersion string `toml:"lock-version"`
PythonVersions string `toml:"python-versions"`
ContentHash string `toml:"content-hash"`
}
PoetryMetadata is the lock's trailer.
type PoetryPackage ¶
type PoetryPackage struct {
Name string `toml:"name"`
Version string `toml:"version"`
Description string `toml:"description"`
Optional bool `toml:"optional"`
PythonVersions string `toml:"python-versions"`
// Groups lists the dependency groups the package serves (lock 2.1).
// Empty in a 2.0 lock, where membership is derived from the manifest.
Groups []string `toml:"groups"`
// markers says where the package applies at all. A package serving
// several groups may apply to each under a different condition, so the
// lock writes either one marker or a group-to-marker table; the parser
// normalizes both into the table, the plain form under every group.
RawMarkers any `toml:"markers"`
Markers map[string]string `toml:"-"`
// RawDependencies is the resolved dependency table: target name to a
// constraint, which may be a bare version string, an object carrying a
// marker and extras, or a list of such objects when the constraint
// differs by environment. Dependencies is its normalized form.
RawDependencies map[string]any `toml:"dependencies"`
Dependencies []PoetryDependency `toml:"-"`
// Extras names the package's extras and their requirements. The values
// are requirement strings kept only for completeness: the packages an
// enabled extra pulls in are resolved through Dependencies.
Extras map[string][]string `toml:"extras"`
// Files are the distribution artifacts with their hashes, named by
// filename rather than URL.
Files []PoetryFile `toml:"files"`
// Source is set when the package does not come from an index.
Source *PoetrySource `toml:"source"`
}
PoetryPackage is one locked package.
type PoetrySource ¶
type PoetrySource struct {
Type string `toml:"type"` // "git", "url", "directory", "legacy"
URL string `toml:"url"`
// Reference is what was asked for (a branch, tag or rev) and
// ResolvedReference the commit it resolved to.
Reference string `toml:"reference"`
ResolvedReference string `toml:"resolved_reference"`
}
PoetrySource says where a non-index package comes from.
type PyPIClient ¶
PyPIClient fetches package metadata from the PyPI JSON API.
func NewPyPIClient ¶
func NewPyPIClient(concurrency int) *PyPIClient
NewPyPIClient creates a client for the PyPI JSON API.
func (*PyPIClient) FetchAll ¶
func (c *PyPIClient) FetchAll(packages []packageKey) map[packageKey]*pypiInfo
FetchAll fetches metadata for the packages in parallel. Failures are skipped: enrichment adds what it can and never breaks an extraction.
type PyProject ¶
type PyProject struct {
Project struct {
Name string `toml:"name"`
Version string `toml:"version"`
RequiresPython string `toml:"requires-python"`
Dependencies []string `toml:"dependencies"`
OptionalDependencies map[string][]string `toml:"optional-dependencies"`
} `toml:"project"`
// DependencyGroups is the standard groups table (PEP 735). Entries are
// requirement strings, or include tables this reader skips.
DependencyGroups map[string][]any `toml:"dependency-groups"`
Tool struct {
Poetry struct {
Name string `toml:"name"`
Version string `toml:"version"`
Dependencies map[string]any `toml:"dependencies"`
Extras map[string][]string `toml:"extras"`
Group map[string]struct {
Dependencies map[string]any `toml:"dependencies"`
} `toml:"group"`
// DevDependencies is the oldest spelling of the dev group.
DevDependencies map[string]any `toml:"dev-dependencies"`
} `toml:"poetry"`
} `toml:"tool"`
}
PyProject is a parsed pyproject.toml, reduced to what the graph needs.
func ReadPyProject ¶
ReadPyProject reads a pyproject.toml file.
func (*PyProject) ExtraDependencies ¶
ExtraDependencies returns the direct dependencies of every extra.
func (*PyProject) GroupDependencies ¶
GroupDependencies returns the direct dependencies of every dependency group, whichever of the three spellings declares them.
func (*PyProject) MainDependencies ¶
MainDependencies returns the project's direct runtime dependencies.
In the legacy table, python is a pseudo-dependency and an entry marked optional belongs to the extras rather than to the runtime set.
func (*PyProject) RootName ¶
RootName returns the project's name, normalized, from whichever table states it.
func (*PyProject) RootVersion ¶
RootVersion returns the project's version from whichever table states it.
type Source ¶
type Source struct {
// Registry is the index URL the package was resolved from.
Registry string `toml:"registry"`
// Virtual and Editable mark the project's own packages: Virtual a
// package that is not installed (the usual project root), Editable one
// installed in place (workspace members). Both hold the directory,
// relative to the lock.
Virtual string `toml:"virtual"`
Editable string `toml:"editable"`
// Git, Path, Directory and URL are direct dependencies on a repository,
// a local archive, a local directory, and a remote archive.
Git string `toml:"git"`
Path string `toml:"path"`
Directory string `toml:"directory"`
URL string `toml:"url"`
}
Source says where a package comes from. Exactly one field is set.
type Version ¶
type Version struct {
Epoch int
Release []int
// Pre is the pre-release phase, in order: "a" < "b" < "rc". The long
// spellings (alpha, beta, c, pre, preview) normalize into those three.
Pre int
PreN int
HasPre bool
Post int
HasPost bool
Dev int
HasDev bool
Local string
}
Version is a parsed PEP 440 version.
func ParseVersion ¶
ParseVersion parses a PEP 440 version string.
func (*Version) Compare ¶
Compare orders two versions per PEP 440: -1, 0 or 1 as v sorts before, with or after o. Trailing zeros in the release are insignificant (1.0 == 1.0.0), a dev release sorts before pre-releases, pre-releases before the final, the final before post-releases, and a local version after the same version without one.