filter

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 28, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package filter decides which files a scan should process. Filtering rules are loaded from several sources (built-in defaults, a project's scanoss.json, and the tree's .gitignore), merged into a single deduplicated set, and applied in one pass over the source tree. Skipped files are simply excluded from the scan; the list of them is not tracked, only counted.

The package is standalone: it does not import any other scanoss package, so it can be consumed on its own.

Index

Constants

View Source
const DefaultMaxFileSize int64 = 0

DefaultMaxFileSize is the maximum file size (bytes) to scan. 0 means unlimited, matching scanoss.py's default.

View Source
const DefaultMinFileSize int64 = 100

DefaultMinFileSize is the minimum file size (bytes) to scan. Files smaller than this are skipped. Mirrors scanoss's historical 100-byte minimum.

Variables

View Source
var DefaultSkippedDirExts = []string{
	".egg-info",
}

DefaultSkippedDirExts are directory-name suffixes that are skipped (e.g. a directory named "foo.egg-info").

View Source
var DefaultSkippedDirs = []string{
	"nbproject",
	"nbbuild",
	"nbdist",
	"__pycache__",
	"venv",
	"_yardoc",
	"eggs",
	"wheels",
	"htmlcov",
	"__pypackages__",
	"example",
	"examples",

	"node_modules",
	"vendor",
}

DefaultSkippedDirs are directory names that are skipped wholesale (the whole subtree is pruned).

View Source
var DefaultSkippedExts = []string{}/* 153 elements not displayed */

DefaultSkippedExts are file-name extensions (suffixes including the leading dot) that are skipped. Compound extensions such as ".min.js" are matched as a full suffix.

View Source
var DefaultSkippedFileEndings = []string{
	"-doc",
	"changelog",
	"config",
	"copying",
	"license",
	"authors",
	"news",
	"licenses",
	"notice",
	"readme",
	"swiftdoc",
	"texidoc",
	"todo",
	"version",
	"ignore",
	"manifest",
	"sqlite",
	"sqlite3",
}

DefaultSkippedFileEndings are file-name suffixes that are not extensions (no leading dot), matched case-insensitively against the whole file name. These correspond to scanoss.py's "file endings" entries.

View Source
var DefaultSkippedFiles = []string{
	"gradlew",
	"gradlew.bat",
	"mvnw",
	"mvnw.cmd",
	"gradle-wrapper.jar",
	"maven-wrapper.jar",
	"thumbs.db",
	"babel.config.js",
	"license.txt",
	"license.md",
	"copying.lib",
	"makefile",
}

DefaultSkippedFiles are exact file names that are skipped.

Functions

This section is empty.

Types

type CollectResult

type CollectResult struct {
	Files        []string
	SkippedCount int
}

CollectResult is the outcome of a Collect: the absolute paths to scan and how many files were skipped. The skipped files themselves are not retained.

func Collect

func Collect(root string, o Options) (*CollectResult, error)

Collect walks root once, returning the files to scan and a count of those skipped. Rules are loaded from the enabled sources (defaults, scanoss.json, .gitignore), deduplicated, and applied as a single composite. Hidden files and directories (names beginning with ".") are always skipped, preserving prior behavior. Symlinked directories are not followed. Returned paths are absolute.

type Composite

type Composite struct {
	// contains filtered or unexported fields
}

Composite holds many matchers and skips a path if any of them does. It is itself a Matcher, so composites can nest.

func Build

func Build(sources ...[]Matcher) *Composite

Build concatenates matchers from every source and removes duplicates by Key(), then wraps the result in a single Composite. Sources are processed in the order given, and the first matcher seen for a given Key() wins (later duplicates are dropped) — so a rule shared across sources (e.g. ".png" in both the defaults and a scanoss.json pattern) is applied once, not repeatedly.

func (*Composite) Key

func (c *Composite) Key() string

Key identifies the composite.

func (*Composite) Match

func (c *Composite) Match(rel string, info os.FileInfo) bool

Match reports whether any contained matcher skips the path.

func (*Composite) Matchers

func (c *Composite) Matchers() []Matcher

Matchers returns the contained matchers (useful for tests/inspection).

type Defaults

type Defaults struct {
	Dirs    []string // directory names skipped wholesale
	DirExts []string // directory-name suffixes skipped (e.g. ".egg-info")
	Files   []string // exact file names skipped
	Exts    []string // file extensions skipped (leading dot)
	Endings []string // non-extension file-name suffixes skipped
	MinSize int64    // minimum file size; 0 disables
	MaxSize int64    // maximum file size; 0 disables (unlimited)
}

Defaults holds the skip lists and size bounds a DefaultSource turns into matchers. Callers normally start from StdDefaults and override fields as needed.

func StdDefaults

func StdDefaults() Defaults

StdDefaults returns the built-in default skip lists and size bounds.

type Matcher

type Matcher interface {
	// Match reports whether the path (rel, relative to the scan root) should be
	// skipped. info describes the entry.
	Match(rel string, info os.FileInfo) bool
	// Key is a stable identity used to deduplicate matchers built from different
	// sources (e.g. "ext:.png", "dir:vendor").
	Key() string
}

Matcher decides whether a single path should be skipped. It is the leaf of the composite pattern; Composite implements it too, so a set of matchers is itself a Matcher. Match returns true when the path should be skipped. Skipped files are not tracked, so a boolean (no reason) is all that is needed.

func DefaultSource

func DefaultSource(d Defaults) []Matcher

DefaultSource turns the default skip lists and size bounds into matchers.

func GitIgnoreSource

func GitIgnoreSource(root string) ([]Matcher, error)

GitIgnoreSource reads the .gitignore at the root of the tree (if present) and returns a matcher for its patterns. If root is a file rather than a directory, its parent directory is used. Missing file is not an error.

func NewMatcher

func NewMatcher(o Options) Matcher

NewMatcher builds a per-path skip Matcher from o, for callers that evaluate entries one at a time (e.g. a streaming archive extractor) instead of walking a tree with Collect. It applies the default and scanoss.json (Settings) rules and honors PreserveDependencyManifests, so a caller filters exactly the way Collect does — from the same Options. It does NOT apply .gitignore (that needs the whole tree; use Collect). Match returns true when the entry should be skipped.

func SettingsSource

func SettingsSource(s *Settings) []Matcher

SettingsSource turns scanoss.json skip rules into matchers. Returns nil when s is nil.

type Options

type Options struct {
	// Skip* replace the matching built-in default list when non-nil.
	SkipDirs       []string
	SkipFiles      []string
	SkipExtensions []string

	MinSize int64 // minimum file size; 0 uses DefaultMinFileSize
	MaxSize int64 // maximum file size; 0 uses DefaultMaxFileSize (unlimited)

	Defaults  bool // apply the built-in default skip lists
	GitIgnore bool // honor .gitignore

	// Settings is the scanoss.json skip/folders rules, already resolved to a
	// single operation. Nil when there is no scanoss.json.
	Settings *Settings

	// PreserveDependencyManifests keeps dependency manifest files
	// (package.json, go.mod, pom.xml, … — see pkg/manifests) even when a skip
	// rule would otherwise drop them. Use it for stages that consume manifests
	// (extraction/upload feeding the dependency parser) while still pruning
	// everything else. Fingerprint scanning leaves this false — manifests are
	// not useful for matching. Default false → unchanged behavior.
	PreserveDependencyManifests bool

	HFH bool // reserved: high-file-hashing (folder hashing) variants; not yet used
}

Options configures a Collect call. Set the Skip* fields to replace the corresponding default list. Use DefaultOptions for the common case where the built-in defaults and .gitignore are applied.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns the common-case Options: the built-in default skip lists and .gitignore are applied, with no scanoss.json.

func IngestOptions

func IngestOptions() Options

IngestOptions returns the options for materialising files a later stage consumes (extraction/upload feeding the dependency parser): the same prune as ScanOptions, but dependency manifests are preserved.

func ScanOptions

func ScanOptions() Options

ScanOptions returns the options for fingerprint scanning: the built-in defaults and .gitignore, with dependency manifests skipped (they are not useful for matching). Alias of DefaultOptions, named for intent.

type Settings

type Settings struct {
	Skip Skip
}

Settings is the local, dependency-free mirror of the scanoss.json bits filter needs. Callers map settings.Settings into this.

type SizeRule

type SizeRule struct {
	Patterns []string
	Min      int64
	Max      int64
}

SizeRule is one scanoss.json skip.sizes entry: files matching any of Patterns are skipped when smaller than Min or larger than Max (0 disables a bound).

type Skip

type Skip struct {
	Patterns []string   // gitignore-style globs
	Sizes    []SizeRule // per-pattern size limits
}

Skip mirrors the scanoss.json settings.skip subset filter consumes, already resolved to a single operation (scanning, fingerprinting, or dependencies).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL