licenses

licenses

Go library and command for matching license text and scanning repositories against ScanCode's license rule corpus.

The corpus is embedded in the package. Matching needs no network access, cgo, or Python.

Compared to ScanCode

The command is a ~18 MB binary that builds one in-memory index and shares it across goroutines. scancode-toolkit 32.5.0 installs to ~710 MB and loads a separate copy of its 413 MB license index in each worker process.

Scanning a checkout of rust-lang/cargo at a07c49a (2,950 files, 8-core M1 Pro, default flags) takes 0.65 s and 239 MB peak RSS, against 94 s and 4.5 GB across nine processes for scancode -l.

licenses matches whole-text hashes, SPDX tag lines, and exact token sequences. License-text and longer notice rules match across variable words outside the corpus vocabulary, so a differing copyright holder or year does not defeat the match. ScanCode also does approximate matching, so it reports detections in more files (58 vs 51 on the cargo checkout). Differences against ScanCode's own detection suite are tracked in the conformance baseline.

Install

Install the repository scanner with Homebrew:

brew tap git-pkgs/git-pkgs
brew trust --tap git-pkgs/git-pkgs
brew install licenses

Or with Go:

go install github.com/git-pkgs/licenses/cmd/licenses@latest

Add the matching library to a Go module:

go get github.com/git-pkgs/licenses

Scan a repository

licenses .
licenses /path/to/repository
licenses -json /path/to/repository > licenses.json
licenses -scope all -max-files 0 /path/to/repository

The command reports detections by file with the matching rule, expression, rule kind, score, coverage, and byte range. Each JSON file record also reports license_text_coverage, the percentage of decoded file bytes covered by the union of license-text and notice matches. JSON is used when output is redirected; terminals get a text report. Skipped files and directories are named with the reason they were skipped.

Reference rules that join text across document blocks are reported as clues outside files such as LICENSE, COPYING, and NOTICE. Soft-wrapped lines, including lines with a shared source-comment prefix, remain detections. Demoted matches stay visible in the file report but do not contribute to the repository expression totals.

The default project scope skips hidden, dependency, build, cache, and test-data directories. Use -scope all for dependency-license scans. An explicit -skip list applies in either scope. Both scopes exclude .git.

Regular text files are limited to 1 MiB and 32 directory levels. Project scope also has a 10,000-file default limit. All scope requires an explicit -max-files value; use -max-files 0 for an unlimited dependency scan. Setting any limit to zero removes that guard.

The scanner accepts UTF-8, UTF-16LE or UTF-16BE with a byte-order mark, and Latin-1. Reported byte ranges refer to the original file. JSON reports use schema version 2. Schema 2 is additive: consumers must ignore unknown fields and accept file records with empty detections when clues are present. With -matched-text, matched contains decoded UTF-8 rather than the original encoded bytes.

Detection and expression records include identification. Its value is identified, partial, or SPDX's NOASSERTION. Partial expressions contain both concrete identifiers and ScanCode placeholder LicenseRef-* values. NOASSERTION detections confirm license-related text without naming another license identifier.

JSON reports include a declared record for each recognized package manifest that declares license values or a license-file path. Each record preserves the raw manifest values and includes a normalized SPDX expression when all values can be normalized. Multiple values are joined with OR; values that cannot be normalized remain available in raw with an empty normalized_expression.

Reported expressions use canonical SPDX identifiers when ScanCode supplies one. Other ScanCode license keys use LicenseRef-scancode-<key>. ScanCode rule IDs remain in each match for traceability.

The three per-file summary counts overlap when one file contains detections in more than one identification state.

Each match reports the method that produced it. hash is a whole-file match against a rule text, exact is a rule token sequence found within the file, and spdx-id is an SPDX-License-Identifier tag line whose expression bytes are not already covered by a rule match. Tag expression grammar is parsed with github.com/git-pkgs/spdx, then bare identifiers are resolved against the embedded ScanCode mapping and normalized to the primary SPDX identifiers in ScanCode's metadata. Valid custom LicenseRef-* values become LicenseRef-scancode-unknown-spdx and report NOASSERTION. Malformed expressions and unknown bare identifiers do not produce an spdx-id match. Tag matches use the rule id spdx-license-identifier.

Exit status 0 means detections were found, 1 is a fatal command error, 2 means the scan was incomplete because of per-file errors or the file limit, and 3 means no conclusive detections were found.

Use the library

matcher, err := licenses.New()
if err != nil {
	return err
}

result, err := matcher.Match(ctx, text)
if err != nil {
	return err
}

for _, detection := range result.Detections {
	fmt.Println(detection.Expression)
}

Scan a file or directory with the same matcher:

options := licenses.DefaultScanOptions()
options.IncludeLegalFiles = true
report, err := licenses.ScanRepository(ctx, matcher, ".", options)
if err != nil {
	return err
}

for _, file := range report.Files {
	fmt.Println(file.Path, file.Roles, file.Text)
}

IncludeLegalFiles retains recognized license and notice files when the corpus does not produce a match. It also includes their complete text decoded to UTF-8.

Matching uses normalized whole-text hashes, exact token sequences, and SPDX-License-Identifier tag lines. Tokenization omits ScanCode's HTML and entity stopwords. License-text and longer notice rules match across words outside the corpus vocabulary and across omitted stopwords, so variable names or years inside an otherwise verbatim text do not defeat the match. Continuous and required-phrase rules still require the source stopword positions and no unknown words. There is no fuzzy or approximate matching, so other edits within a license text can prevent a match.

Corpus

The ScanCode commit is pinned in CORPUS_VERSION. Regenerate the embedded index and CORPUS_SHA256 with:

scripts/regenerate-corpus /path/to/scancode-toolkit

The path can be any scancode-toolkit checkout; the script mounts a temporary worktree at the pinned commit if the checkout is on a different one. With no argument it reads from .scancode/, matching CI.

Caller-supplied corpus

For applications that need a smaller rule set, build a separate corpus from a clean ScanCode checkout at the commit in CORPUS_VERSION:

go run ./cmd/corpusgen \
  -scancode /path/to/scancode-toolkit \
  -rule-flags text,notice \
  -output /path/to/application/licenses.bin.gz

-rule-flags selects rules with any of the listed flags. Use text for license-text rules alone, or omit the flag (the default is all) for the full corpus. Run go run ./cmd/corpusgen -help for the supported flags. Filtering rebuilds the vocabulary, token IDs, and automaton from the selected rules. The complete stopword list and SPDX mappings are retained so query tokenization and SPDX identifier resolution remain available.

Embed the generated file in your application and pass it to NewFromReader:

import (
    "bytes"
    _ "embed"

    "github.com/git-pkgs/licenses"
)

//go:embed licenses.bin.gz
var licenseCorpus []byte

func newLicenseMatcher() (*licenses.Matcher, error) {
    return licenses.NewFromReader(bytes.NewReader(licenseCorpus))
}

NewFromReader also accepts files and supports the same options as New. It reads the corpus synchronously, leaves the reader open, and builds a separate engine on each call; reuse the matcher for subsequent scans. Generate the corpus with the same package version used by your application, since the binary format can change between releases. Programs that never call New can omit the full embedded corpus from their linked binary.

A filtered corpus trades detection coverage for size. In particular, omitted false-positive rules cannot suppress matches, and a smaller vocabulary changes which query words are treated as unknown. Test the chosen subset against your application's inputs. SPDX tags can still resolve identifiers outside the selected text rules. ScanCode corpus attribution requirements in NOTICE also apply to the generated subset.

Conformance

The exact matcher is evaluated against ScanCode's four active data-driven detection suites. Run the suite against a ScanCode checkout at the commit in CORPUS_VERSION:

SCANCODE_TESTDATA=/path/to/scancode-toolkit/tests/licensedcode/data \
  go test . -run '^TestScanCodeConformanceExact$' -v

Known differences are recorded in the conformance baseline. CI fails if an existing result changes or a new difference appears.

Benchmarks

See docs/benchmarking.md for matcher, repository scanner, and Licensee comparison benchmarks.

License

MIT. ScanCode's license and rule data is licensed under CC-BY-4.0. See NOTICE for attribution and modification details.