Documentation
¶
Overview ¶
Package filter decides which files a scan should process. Filtering rules are loaded from several sources (built-in defaults, a project's scanoss.json, and the tree's .gitignore), merged into a single deduplicated set, and applied in one pass over the source tree. Skipped files are simply excluded from the scan; the list of them is not tracked, only counted.
The package is standalone: it does not import any other scanoss package, so it can be consumed on its own.
Index ¶
- Constants
- Variables
- type CollectResult
- type Composite
- type Defaults
- type Matcher
- func DefaultSource(d Defaults) []Matcher
- func FileDefaultSource(d Defaults) []Matcher
- func FolderDefaultSource(d Defaults) []Matcher
- func GitIgnoreSource(root string) ([]Matcher, error)
- func HiddenSource() []Matcher
- func SettingsSource(s *Settings) []Matcher
- func SizeSource(min, max int64) []Matcher
- func UnscannableSource() []Matcher
- type Options
- type Settings
- type SizeRule
- type Skip
Constants ¶
const DefaultMaxFileSize int64 = 0
DefaultMaxFileSize is the maximum file size (bytes) to scan. 0 means unlimited, matching scanoss.py's default.
const DefaultMinFileSize int64 = 0
DefaultMinFileSize is the minimum file size (bytes) to scan. 0 means no minimum: a file is collected however small it is, unless another rule skips it. Raise it per run with --min-size, or per pattern with a scanoss.json skip.sizes rule.
Variables ¶
var CommonSkippedDirs = []string{
"__pycache__",
"node_modules",
"vendor",
}
CommonSkippedDirs are skipped by every operation.
This is also the most a pre-filter can safely prune when it does not yet know which operation will consume the result — an archive extractor feeding both a scan and a dependency analysis. Pruning beyond this is irreversible: the files never reach disk.
var DefaultSkippedDirExts = []string{
".egg-info",
}
DefaultSkippedDirExts are directory-name suffixes that are skipped (e.g. a directory named "foo.egg-info").
var DefaultSkippedExts = []string{}/* 154 elements not displayed */
DefaultSkippedExts are file-name extensions (suffixes including the leading dot) that are skipped. Compound extensions such as ".min.js" are matched as a full suffix.
var DefaultSkippedFileEndings = []string{
"-doc",
"changelog",
"config",
"copying",
"license",
"authors",
"news",
"licenses",
"notice",
"readme",
"swiftdoc",
"texidoc",
"todo",
"version",
"ignore",
"manifest",
"sqlite",
"sqlite3",
}
DefaultSkippedFileEndings are file-name suffixes that are not extensions (no leading dot), matched case-insensitively against the whole file name. These correspond to scanoss.py's "file endings" entries.
var DefaultSkippedFiles = []string{
"gradlew",
"gradlew.bat",
"mvnw",
"mvnw.cmd",
"gradle-wrapper.jar",
"maven-wrapper.jar",
"thumbs.db",
"babel.config.js",
"license.txt",
"license.md",
"copying.lib",
"makefile",
}
DefaultSkippedFiles are exact file names that are skipped.
var DependencyOnlySkippedDirs = []string{
"dist",
"build",
"target",
}
DependencyOnlySkippedDirs are skipped when collecting dependencies, on top of CommonSkippedDirs: generated trees, whose manifests are build output rather than declarations. Scanning does not inherit these — the sources under a build tree are still code that can match.
var ScanOnlySkippedDirs = []string{
"nbproject",
"nbbuild",
"nbdist",
"venv",
"_yardoc",
"eggs",
"wheels",
"htmlcov",
"__pypackages__",
"example",
"examples",
}
ScanOnlySkippedDirs are skipped when scanning or fingerprinting, on top of CommonSkippedDirs. Example code is not the product, so its matches are noise; virtualenvs, eggs and wheels hold installed packages whose code is not the project's. Dependency collection does NOT inherit these: a manifest declares real dependencies wherever it lives, examples/ included.
Functions ¶
This section is empty.
Types ¶
type CollectResult ¶
CollectResult is the outcome of a Collect: the absolute paths to scan and how many files were skipped. The skipped files themselves are not retained.
func Collect ¶
func Collect(root string, o Options) (*CollectResult, error)
Collect walks root once, returning the files to scan and a count of those skipped. Rules are loaded from the enabled sources (defaults, scanoss.json, .gitignore), deduplicated, and applied as a single composite — including the hidden-entry rule, unless Options.IncludeHidden says otherwise. Zero-byte files and symbolic links are always skipped (see UnscannableSource). Symlinked directories are not followed. Returned paths are absolute.
type Composite ¶
type Composite struct {
// contains filtered or unexported fields
}
Composite holds many matchers and skips a path if any of them does. It is itself a Matcher, so composites can nest.
func Build ¶
Build concatenates matchers from every source and removes duplicates by Key(), then wraps the result in a single Composite. Sources are processed in the order given, and the first matcher seen for a given Key() wins (later duplicates are dropped) — so a rule shared across sources (e.g. ".png" in both the defaults and a scanoss.json pattern) is applied once, not repeatedly.
type Defaults ¶
type Defaults struct {
Dirs []string // directory names skipped wholesale
DirExts []string // directory-name suffixes skipped (e.g. ".egg-info")
Files []string // exact file names skipped
Exts []string // file extensions skipped (leading dot)
Endings []string // non-extension file-name suffixes skipped
}
Defaults holds the skip lists and size bounds a DefaultSource turns into matchers. Callers normally start from StdDefaults and override fields as needed.
func StdDefaults ¶
func StdDefaults() Defaults
StdDefaults returns the built-in default skip lists. Size bounds are not part of it: they are caller input, not a built-in, and have their own SizeSource.
type Matcher ¶
type Matcher interface {
// Match reports whether the path (rel, relative to the scan root) should be
// skipped. info describes the entry.
Match(rel string, info os.FileInfo) bool
// Key is a stable identity used to deduplicate matchers built from different
// sources (e.g. "ext:.png", "dir:vendor").
Key() string
}
Matcher decides whether a single path should be skipped. It is the leaf of the composite pattern; Composite implements it too, so a set of matchers is itself a Matcher. Match returns true when the path should be skipped. Skipped files are not tracked, so a boolean (no reason) is all that is needed.
func DefaultSource ¶
DefaultSource turns every default skip list into matchers: the directory rules and the file rules together. Callers that want one half without the other use FolderDefaultSource and FileDefaultSource, which is what the two --all-folders / --all-extensions switches select.
func FileDefaultSource ¶ added in v0.5.0
FileDefaultSource turns the default file skip lists into matchers: extensions, non-extension name endings, and exact names.
func FolderDefaultSource ¶ added in v0.5.0
FolderDefaultSource turns the default directory skip lists into matchers: whole directory names and directory-name suffixes. Skipping a directory prunes everything under it.
func GitIgnoreSource ¶
GitIgnoreSource reads the .gitignore at the root of the tree (if present) and returns a matcher for its patterns. If root is a file rather than a directory, its parent directory is used. Missing file is not an error.
func HiddenSource ¶ added in v0.4.0
func HiddenSource() []Matcher
HiddenSource skips entries whose name begins with a dot.
Not part of UnscannableSource: a dotfile has perfectly good content, so this is a policy choice about what belongs to a project, not a statement about the entry. That is why it can be switched off (Options.IncludeHidden) and why it is a source like any other rather than a check buried in the walk — a caller that cannot walk a tree needs to apply it too.
func SettingsSource ¶
SettingsSource turns scanoss.json skip rules into matchers. Returns nil when s is nil.
func SizeSource ¶ added in v0.4.0
SizeSource turns a [min, max] byte range into a matcher. It is a source of its own — not part of DefaultSource — because the bounds come from the caller (--min-size/--max-size), not from the built-in lists: switching the defaults off must not discard a bound the caller asked for. A min of 0 imposes no minimum and a max of 0 no maximum, so 0/0 yields no matcher at all.
func UnscannableSource ¶ added in v0.4.0
func UnscannableSource() []Matcher
UnscannableSource skips entries there is no point fingerprinting
type Options ¶
type Options struct {
// Skip* replace the matching built-in default list when non-nil.
SkipDirs []string
SkipFiles []string
SkipExtensions []string
// Size bounds. The built-in values are DefaultMinFileSize/DefaultMaxFileSize,
// set by DefaultOptions/ScanOptions/DependencyOptions; assign these fields to
// override them. They are applied as their own source, so turning Defaults
// off keeps a bound the caller asked for. A zero-valued Options (built
// literally, without a constructor) means no bound on either side.
MinSize int64 // minimum file size in bytes; 0 imposes no minimum
MaxSize int64 // maximum file size in bytes; 0 imposes no maximum (unlimited)
// FolderDefaults applies the built-in directory skip lists (node_modules, vendor, build output,
// …). FileDefaults applies the built-in file rules — extensions, name endings and exact names —
// which answer one question together: is this file worth fingerprinting.
FolderDefaults bool
FileDefaults bool
GitIgnore bool // honor .gitignore
// IncludeHidden collects entries whose name begins with a dot. They are excluded by default: a
// scan wants the project's source, not its tooling. Setting it reaches version-control
// metadata too — .git and friends are dotted like anything else — which matches the reference
// implementation, where the equivalent flag has the same reach.
IncludeHidden bool
// Settings is the scanoss.json skip/folders rules, already resolved to a
// single operation. Nil when there is no scanoss.json.
Settings *Settings
// PreserveDependencyManifests keeps dependency manifest files
// (package.json, go.mod, pom.xml, … — see pkg/manifests) even when a skip
// rule would otherwise drop them. Use it for stages that consume manifests
// (extraction/upload feeding the dependency parser) while still pruning
// everything else. Fingerprint scanning leaves this false — manifests are
// not useful for matching. Default false → unchanged behavior.
PreserveDependencyManifests bool
}
Options configures a Collect call. Set the Skip* fields to replace the corresponding default list. Use DefaultOptions for the common case where the built-in defaults and .gitignore are applied.
func DefaultOptions ¶
func DefaultOptions() Options
DefaultOptions returns the common-case Options: the built-in default skip lists and .gitignore are applied, with no scanoss.json.
func DependencyOptions ¶ added in v0.4.0
func DependencyOptions() Options
DependencyOptions returns the options for collecting dependency manifests. Three things differ from ScanOptions, and all three are deliberate:
- the directory list adds DependencyOnlySkippedDirs instead of the scanning ones;
- manifests are preserved, since they live behind skipped extensions;
- .gitignore is NOT applied. It answers "should this be versioned", not "is this a dependency": a lock file excluded from git still declares what the project uses, and losing a declaration is worse than analysing one extra.
func FingerprintOptions ¶ added in v0.4.0
func FingerprintOptions() Options
FingerprintOptions returns the options for the fingerprint-only path (the wfp command). Identical to ScanOptions today — the two differ only in which scanoss.json section the caller supplies — but named separately so each layer states which profile it uses, and so the two can diverge without a caller silently inheriting the wrong one.
func ScanOptions ¶
func ScanOptions() Options
ScanOptions returns the options for fingerprint scanning: the built-in defaults and .gitignore. PreserveDependencyManifests stays off, so a manifest the default lists exclude stays excluded — it is a declaration, not a file worth matching. Same values as DefaultOptions today, named separately so each layer states which profile it uses.
type Settings ¶
type Settings struct {
Skip Skip
}
Settings is the local, dependency-free mirror of the scanoss.json bits filter needs. Callers map settings.Settings into this.
Source Files