resolver

package
v1.18.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 28, 2026 License: Apache-2.0 Imports: 48 Imported by: 0

Documentation

Overview

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com/) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

Package resolver implements Terraform module resolution for tfmodules.RemoteResolver.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

  • Unless explicitly stated otherwise all files in this repository are licensed under the Apache-2.0 License. *

  • This product includes software developed at Datadog (https://www.datadoghq.com) Copyright 2024 Datadog, Inc.

Index

Constants

View Source
const (
	DefaultMaxCacheBytes = 2 * 1024 * 1024 * 1024

	CacheSubdirModules  = "modules"
	CacheSubdirGitBare  = "git-bare"
	CacheSubdirGitLocal = "git-local"
)
View Source
const (
	FailureTimeout         = "timeout"
	FailureBudgetExceeded  = "budget_exceeded"
	FailureAllowlistDenied = "allowlist_denied"
	FailureNotFound        = "registry_not_found"
	FailureHTTPClient      = "http_4xx"
	FailureHTTPServer      = "http_5xx"
	FailureNetwork         = "network"
	FailureUnpack          = "unpack_failed"
	FailureUnknown         = "unknown"
)

Failure codes for module resolution telemetry. The set is intentionally bounded: consumers turn these into metric tags, so raw error strings must never be used. Keep in sync with the taxonomy in K9CODESEC-5303.

View Source
const (
	DefaultFetchTimeout   = 90 * time.Second
	DefaultMaxModuleBytes = 0
	DefaultMaxTotalBytes  = 1024 * 1024 * 1024
)
View Source
const (
	ManifestSchemaVersion    = 1
	ManifestStatusResolved   = "resolved"
	ManifestStatusUnresolved = "unresolved"
)
View Source
const (
	CacheStateHit  = "hit"
	CacheStateMiss = "miss"
)

Cache-state labels recorded on Resolution.Cache for telemetry.

View Source
const (
	DefaultMaxPackageBytes = 256 * 1024 * 1024
	// Modules can vendor binaries, lambda zips, or docs media; a smaller file
	// cap would reject the whole package over one such file.
	DefaultMaxFileBytes    = DefaultMaxPackageBytes
	DefaultMaxPackageFiles = 100_000
)
View Source
const OriginUnknown = "unknown"

OriginUnknown is the origin label for a resolution whose source type could not be determined. It is distinct from the failure-taxonomy codes: origins and outcomes are different enums and must not drift together.

Variables

View Source
var FetchConcurrency = fetchConcurrencyFromEnv()

FetchConcurrency bounds simultaneous go-getter downloads (IAC_MODULE_FETCH_CONCURRENCY).

View Source
var HostFetchConcurrency = hostFetchConcurrencyFromEnv()

Functions

func CheckPackageUsage added in v1.18.0

func CheckPackageUsage(usage PackageUsage, limits ResourceLimits) error

CheckPackageUsage applies MeasurePackage's package-level limits to a usage the resolver already knows, so callers holding Resolution.Usage need not walk the tree.

func ClassifyFailure added in v1.16.0

func ClassifyFailure(err error) string

ClassifyFailure maps a module resolution error to one of the bounded failure codes above. Typed errors are matched first; the remaining rules match stable substrings of error messages produced by the resolvers. Anything unrecognized maps to FailureUnknown rather than leaking error text.

func ComputePackageDigest added in v1.13.0

func ComputePackageDigest(ctx context.Context, root string) (string, error)

ComputePackageDigest hashes regular files by relative path, size, and content.

func DefaultModuleCacheRoot added in v1.12.0

func DefaultModuleCacheRoot() (string, error)

func GitModuleResolveKey added in v1.8.0

func GitModuleResolveKey(source, version string) (string, bool)

GitModuleResolveKey returns a canonical cache key for pinnable git module sources. Transport scheme is part of the key so HTTPS and SSH spellings cannot coalesce. A missing ref is treated as HEAD so default-branch sources still share an identity.

func LocalModuleClosure added in v1.18.1

func LocalModuleClosure(ctx context.Context, subdir string, read ModuleConfigReader) ([]string, error)

LocalModuleClosure returns subdir followed by every package-relative directory reachable from it through literal local module sources: the directories a package must hold, each with its subtree, for the module at subdir to load. It is the closure BareGitResolver materializes for a git module. Paths use forward slashes and "." is the package root.

func ModuleCacheSubdir added in v1.12.0

func ModuleCacheSubdir(root, name string) string

func NewModuleCache

func NewModuleCache() (*moduleCache, error)

func NewModuleCacheWithDir added in v1.12.0

func NewModuleCacheWithDir(dir string, budget *ModuleCacheBudget) (*moduleCache, error)

func NewRegistryCache added in v1.8.0

func NewRegistryCache(timeout time.Duration, hostAllowlist ...string) *registryCache

func ResolvePathWithinRoot added in v1.12.0

func ResolvePathWithinRoot(ctx context.Context, root, target string) (string, error)

func TerraformRootDirs added in v1.13.0

func TerraformRootDirs(paths []string) []string

TerraformRootDirs returns the Terraform workspace root for each distinct discovery path.

func WalkPackageDigest added in v1.18.0

func WalkPackageDigest(ctx context.Context, root string, visitor PackageVisitor) (string, error)

WalkPackageDigest computes ComputePackageDigest while calling visitor for every directory and regular file of root.

func WithResourceBudget added in v1.12.0

func WithResourceBudget(ctx context.Context, budget *ResourceBudget) context.Context

func WriteManifest added in v1.13.0

func WriteManifest(ctx context.Context, path, root string, modules []ManifestModule) error

WriteManifest writes and validates a versioned prefetched-module manifest atomically.

Types

type AcquisitionLease added in v1.12.0

type AcquisitionLease struct {
	// contains filtered or unexported fields
}

func (*AcquisitionLease) Bytes added in v1.12.0

func (l *AcquisitionLease) Bytes() int64

func (*AcquisitionLease) Release added in v1.12.0

func (l *AcquisitionLease) Release()

type BareGitResolver added in v1.8.0

type BareGitResolver struct {
	// Defaults to <user-cache-dir>/datadog-iac-scanner/git-bare.
	CacheDir string
	Budget   *ModuleCacheBudget
	// contains filtered or unexported fields
}

BareGitResolver keeps one bare clone per repo and extracts refs via git archive.

func NewBareGitResolver added in v1.8.0

func NewBareGitResolver(cacheDir string, hostAllowlist ...string) *BareGitResolver

func (*BareGitResolver) Resolve added in v1.8.0

Resolve implements Resolver for pinnable git:: sources. A missing ref uses HEAD.

type BudgetExceededError added in v1.12.0

type BudgetExceededError struct {
	Gate     string
	Limit    string
	Maximum  int64
	Measured int64
}

func (*BudgetExceededError) Error added in v1.12.0

func (e *BudgetExceededError) Error() string

type ChainResolver

type ChainResolver struct {
	// contains filtered or unexported fields
}

ChainResolver tries each Resolver in order.

func NewChainResolver

func NewChainResolver(resolvers ...Resolver) *ChainResolver

func NewDefaultChain added in v1.13.0

func NewDefaultChain(ctx context.Context, config *DefaultChainConfig) (*ChainResolver, error)

NewDefaultChain builds the standard local, prefetched, injected, git, and registry resolver chain.

func (*ChainResolver) Resolve

type DefaultChainConfig added in v1.13.0

type DefaultChainConfig struct {
	DiscoveryPaths []string
	Manifest       *Manifest
	Additional     []Resolver
	FetchRemote    bool
	FetchTimeout   time.Duration
	HostAllowlist  []string
	CacheRoot      string
	MaxCacheBytes  int64
}

type DotTerraformResolver

type DotTerraformResolver struct {
	RootDirs []string
	// contains filtered or unexported fields
}

DotTerraformResolver reads terraform init output: .terraform/modules/modules.json.

func (*DotTerraformResolver) Resolve

type GoGetterConfig added in v1.8.0

type GoGetterConfig struct {
	Disabled bool

	FetchTimeout time.Duration
	// MaxModuleBytes caps each individual module fetch when no graph budget is attached.
	MaxModuleBytes int64
	// MaxTotalBytes is retained for standalone resolver compatibility.
	MaxTotalBytes int64

	HostAllowlist []string
	Cache         *moduleCache
	RegistryCache *registryCache
	// Git resolves pinnable git:: download URLs (registry X-Terraform-Get and
	// default-branch sources) through BareGit instead of an unpinned git subprocess.
	Git Resolver

	TmpDir string
	// contains filtered or unexported fields
}

GoGetterConfig holds caps and options for GoGetterResolver. It must not be copied after first use (contains atomic.Int64 and a channel); always use a pointer.

func NewGoGetterConfig added in v1.8.0

func NewGoGetterConfig() *GoGetterConfig

NewGoGetterConfig returns default fetch limits and concurrency controls.

type GoGetterResolver added in v1.8.0

type GoGetterResolver struct {
	// contains filtered or unexported fields
}

GoGetterResolver downloads modules via hashicorp/go-getter (registry translation, caps, cache).

func NewGoGetterResolver added in v1.8.0

func NewGoGetterResolver(cfg *GoGetterConfig) *GoGetterResolver

func (*GoGetterResolver) Resolve added in v1.8.0

type LocalGitRefResolver added in v1.8.0

type LocalGitRefResolver struct {
	ScanRoots []string

	// Defaults to <user-cache-dir>/datadog-iac-scanner/git-local.
	CacheDir string
	Budget   *ModuleCacheBudget
	// contains filtered or unexported fields
}

LocalGitRefResolver extracts self-referential git modules from the local checkout.

func NewLocalGitRefResolver added in v1.8.0

func NewLocalGitRefResolver(scanRoots []string, cacheDir string) *LocalGitRefResolver

func (*LocalGitRefResolver) Resolve added in v1.8.0

Resolve implements Resolver for git:: sources that reference the local checkout.

type LocalResolver

type LocalResolver struct{}

LocalResolver resolves local filesystem module sources.

func (LocalResolver) Resolve

type Manifest

type Manifest struct {
	SchemaVersion int                      `json:"schema_version,omitempty"`
	Root          string                   `json:"root,omitempty"`
	Dir           string                   `json:"dir,omitempty"`
	Modules       map[string]ManifestEntry `json:"modules,omitempty"`
	Entries       []ManifestModule         `json:"-"`
	// contains filtered or unexported fields
}

func LoadManifest

func LoadManifest(ctx context.Context, path string) (*Manifest, error)

type ManifestDeclaration added in v1.13.0

type ManifestDeclaration struct {
	Filename   string `json:"filename"`
	LineStart  int    `json:"line_start"`
	LineEnd    int    `json:"line_end"`
	ModuleName string `json:"module_name"`
}

type ManifestEntry

type ManifestEntry struct {
	LocalPath        string `json:"local_path"`
	PackageRoot      string `json:"package_root,omitempty"`
	Version          string `json:"version,omitempty"`
	Origin           string `json:"origin,omitempty"`
	RequestedVersion string `json:"requested_version,omitempty"`
	ResolvedVersion  string `json:"resolved_version,omitempty"`
	ResolvedRef      string `json:"resolved_ref,omitempty"`
	Status           string `json:"status,omitempty"`
	Failure          string `json:"failure,omitempty"`
}

type ManifestModule added in v1.13.0

type ManifestModule struct {
	ID               string                `json:"id,omitempty"`
	Source           string                `json:"source"`
	CanonicalSource  string                `json:"canonical_source,omitempty"`
	SourceType       string                `json:"source_type,omitempty"`
	RegistryScope    string                `json:"registry_scope,omitempty"`
	RequestedVersion string                `json:"requested_version,omitempty"`
	ResolvedVersion  string                `json:"resolved_version,omitempty"`
	ResolvedRef      string                `json:"resolved_ref,omitempty"`
	ContentDigest    string                `json:"content_digest,omitempty"`
	PackageRoot      string                `json:"package_root,omitempty"`
	LocalPath        string                `json:"local_path,omitempty"`
	Status           string                `json:"status"`
	Failure          string                `json:"failure,omitempty"`
	Declarations     []ManifestDeclaration `json:"declarations"`
}

type ModuleCacheBudget added in v1.12.0

type ModuleCacheBudget struct {
	// contains filtered or unexported fields
}

ModuleCacheBudget is a soft byte limit shared across modules/, git-bare/, and git-local/. It bounds disk use only; per-scan package and total limits belong to ResourceBudget.

A modules/ entry's size comes from its size file. Git entries have none and can hold thousands of files, so each is walked once and then kept current by the resolvers writing into it: Grow when the written bytes are known, Invalidate otherwise. Grow can recount bytes that an overlapping walk already saw, which errs toward evicting early.

Sizes and pins are per process. Entries another process adds or removes are picked up by the next Admit, but its growth of an entry already measured here is not, and pins do not stop another process from evicting an entry in use.

func NewModuleCacheBudget added in v1.12.0

func NewModuleCacheBudget(root string, maxBytes int64) (*ModuleCacheBudget, error)

func (*ModuleCacheBudget) Admit added in v1.12.0

func (b *ModuleCacheBudget) Admit(keep string)

func (*ModuleCacheBudget) EnsureEntryFits added in v1.12.0

func (b *ModuleCacheBudget) EnsureEntryFits(path string) error

func (*ModuleCacheBudget) Grow added in v1.18.0

func (b *ModuleCacheBudget) Grow(path string, bytes int64)

Grow adds bytes just written into an entry. An entry that has not been measured yet is left alone; its first check walks it.

func (*ModuleCacheBudget) Invalidate added in v1.18.0

func (b *ModuleCacheBudget) Invalidate(path string)

Invalidate drops an entry's recorded size after a write whose size is not known, so the next check measures it again.

func (*ModuleCacheBudget) Lease added in v1.12.0

func (b *ModuleCacheBudget) Lease(path string) func()

func (*ModuleCacheBudget) MaxBytes added in v1.12.0

func (b *ModuleCacheBudget) MaxBytes() int64

func (*ModuleCacheBudget) Pin added in v1.12.0

func (b *ModuleCacheBudget) Pin(path string)

func (*ModuleCacheBudget) Root added in v1.12.0

func (b *ModuleCacheBudget) Root() string

func (*ModuleCacheBudget) Unpin added in v1.12.0

func (b *ModuleCacheBudget) Unpin(path string)

func (*ModuleCacheBudget) WithPin added in v1.12.0

func (b *ModuleCacheBudget) WithPin(path string, fn func() error) error

type ModuleConfigReader added in v1.18.1

type ModuleConfigReader func(ctx context.Context, dirs []string) (map[string]map[string][]byte, error)

ModuleConfigReader returns the Terraform configuration files directly inside each package-relative directory of dirs, keyed by directory and file name. Missing directories are left out.

type PackageCounter added in v1.12.0

type PackageCounter struct {
	// contains filtered or unexported fields
}

func (*PackageCounter) AddEntry added in v1.12.0

func (c *PackageCounter) AddEntry(size int64) error

func (*PackageCounter) Usage added in v1.12.0

func (c *PackageCounter) Usage() PackageUsage

type PackageUsage added in v1.12.0

type PackageUsage struct {
	Bytes int64
	Files int
}

func MeasurePackage added in v1.12.0

func MeasurePackage(ctx context.Context, root string, limits ResourceLimits) (PackageUsage, error)

type PackageVisitor added in v1.18.0

type PackageVisitor struct {
	Dir func(relative string) error
	// File may read content, which yields the file's bytes as they are hashed;
	// whatever it leaves unread is hashed afterwards.
	File func(relative, path string, info fs.FileInfo, content io.Reader) error
}

PackageVisitor observes a package while its digest is computed, so a caller that also needs the bytes reads each file once. Paths are relative to the package root, which is visited as ".".

type PrefetchedResolver

type PrefetchedResolver struct {
	// contains filtered or unexported fields
}

PrefetchedResolver resolves modules from a --modules-manifest.

func NewPrefetchedResolver

func NewPrefetchedResolver(m *Manifest) *PrefetchedResolver

func (*PrefetchedResolver) Resolve

type Resolution

type Resolution struct {
	LocalPath       string
	PackageRoot     string
	ResolvedVersion string
	ResolvedRef     string
	// Origin identifies which resolver produced this resolution (e.g. "local",
	// "dot_terraform", "prefetched", "git", "registry"). It is telemetry-only:
	// resolution behavior must never depend on it.
	Origin string
	// Cache reports how the bytes were obtained where the resolver knows it
	// ("hit" or "miss" for the on-disk module cache). Empty means unknown.
	Cache string
	// Usage is PackageRoot's exact on-disk usage when the resolver already knows
	// it; nil means callers must measure PackageRoot themselves.
	Usage   *PackageUsage
	Cleanup func() // optional post-scan cleanup
}

Resolution holds an acquired package, its selected module directory, and optional cleanup.

func ConfineResolution added in v1.12.0

func ConfineResolution(ctx context.Context, res *Resolution) (Resolution, error)

type Resolver

type Resolver interface {
	Resolve(ctx context.Context, mod *tfmodules.ParsedModule) (Resolution, error)
}

Resolver maps one module call to disk; errors should wrap *tfmodules.UnresolvedError when appropriate.

type ResourceBudget added in v1.12.0

type ResourceBudget struct {
	// contains filtered or unexported fields
}

func NewResourceBudget added in v1.12.0

func NewResourceBudget(limits ResourceLimits) *ResourceBudget

func ResourceBudgetFromContext added in v1.12.0

func ResourceBudgetFromContext(ctx context.Context) *ResourceBudget

func (*ResourceBudget) AcquireAcquisition added in v1.12.0

func (b *ResourceBudget) AcquireAcquisition(
	ctx context.Context, maximum, requested int64,
) (*AcquisitionLease, bool)

func (*ResourceBudget) AdmitPackage added in v1.12.0

func (b *ResourceBudget) AdmitPackage(identity string, usage PackageUsage) error

func (*ResourceBudget) Limits added in v1.12.0

func (b *ResourceBudget) Limits() ResourceLimits

func (*ResourceBudget) NewPackageCounter added in v1.12.0

func (b *ResourceBudget) NewPackageCounter() *PackageCounter

func (*ResourceBudget) TotalUsage added in v1.12.0

func (b *ResourceBudget) TotalUsage() PackageUsage

func (*ResourceBudget) TryAcquireAcquisition added in v1.12.0

func (b *ResourceBudget) TryAcquireAcquisition(maximum, requested int64) (*AcquisitionLease, bool)

func (*ResourceBudget) Usage added in v1.12.0

func (b *ResourceBudget) Usage(identity string) (PackageUsage, bool)

type ResourceLimits added in v1.12.0

type ResourceLimits struct {
	MaxPackageBytes int64
	MaxFileBytes    int64
	MaxPackageFiles int
	MaxTotalBytes   int64
}

func (ResourceLimits) Enabled added in v1.12.0

func (l ResourceLimits) Enabled() bool

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL