secretscan

package
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 3, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package secretscan is vigolium's native, binary-free secret detector. It replaces the former external kingfisher subprocess with an in-process engine driven by the normalized kingfisher rule catalog (see pkg/secretscan/catalog and pkg/secretscan/secretgen). Detection is regex-based with a keyword prefilter, Shannon-entropy and character-class gating, and a benign-placeholder safelist — faithfully mirroring kingfisher's matcher semantics. Live secret VERIFICATION is intentionally not performed.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func LoadCatalog

func LoadCatalog() (*catalog.Catalog, error)

LoadCatalog parses the embedded rule catalog and appends the vigolium-authored supplemental rules (see supplementalRules). The embedded JSON is the generated kingfisher set (regenerated by `make update-secret-rules`); the supplemental rules live in Go so they survive regeneration.

Types

type Detector

type Detector struct {
	// contains filtered or unexported fields
}

Detector holds the compiled rule set and keyword prefilter.

func Default

func Default() (*Detector, error)

Default returns a process-wide Detector built from the embedded catalog with default options (visible rules only). The catalog is parsed and all rule regexes compiled exactly once, then reused across scans. The per-body result cache is enabled (see defaultBodyCacheSize); the experimental Set prefilter is not (opt-in via New with Options.UseRegexSet).

func New

func New(cat *catalog.Catalog, opts Options) (*Detector, error)

New compiles a Detector from a catalog. Regex compilation happens once here; callers should reuse the Detector across scans (see Default).

func (*Detector) Detect

func (d *Detector) Detect(data []byte) []Match

Detect scans data and returns all secrets found. Safe for concurrent use.

When the body-SHA cache is enabled, an identical body short-circuits to its previously computed matches — Detect is a pure function of (rule set, body), so a cached result is always valid. The returned slice is treated as read-only by all callers; it may be shared with concurrent callers of the same body.

func (*Detector) RuleCount

func (d *Detector) RuleCount() int

RuleCount returns the number of active compiled rules.

type Engine

type Engine int

Engine selects the regex backend used to evaluate rule patterns.

const (
	// EngineRE2 (default) uses github.com/wasilibs/go-re2 — Google's RE2 (C++)
	// compiled to WebAssembly (pure Go, no cgo, works on arm64). Identical RE2
	// semantics to stdlib (so it is correct — zero false negatives), and ~10x
	// faster on bodies >~1KB with this complex rule set. A one-time ~350ms
	// compile cost is amortized by the process-wide Default() detector.
	EngineRE2 Engine = iota
	// EngineStdlib uses Go's standard library regexp (RE2). Correctness reference
	// used by tests; ~10x slower than go-re2 here, so not the default.
	EngineStdlib
)

type Match

type Match struct {
	RuleID     string
	RuleName   string
	Source     string  // rule source; always "kingfisher"
	Confidence string  // "high" | "medium" | "low"
	Pattern    string  // the rule's normalized RE2 pattern that fired (for evidence)
	Secret     string  // the selected secret capture
	Start      int     // byte offset of the secret in the scanned input
	End        int     // exclusive end offset
	Entropy    float64 // computed Shannon entropy (bits/byte) of the secret
}

Match is a single detected secret.

type Options

type Options struct {
	// IncludeInvisible enables rules marked visible:false in the catalog
	// (kingfisher's noisy/experimental set). Off by default.
	IncludeInvisible bool
	// MaxMatchesPerRule caps ACCEPTED matches per rule per body (<=0 → default).
	MaxMatchesPerRule int
	// Engine selects the regex backend (zero value = EngineRE2 / go-re2).
	Engine Engine
	// BodyCacheSize bounds the per-body result cache (keyed by body SHA-256).
	// A value >0 enables caching; 0 leaves the caller default (Default enables
	// it, New leaves it off); <0 disables it explicitly. Hashing a body is far
	// cheaper than scanning it, so repeated identical bodies (SPA shells, records
	// re-seen across scan passes) are served from cache. Safe: Detect is a pure
	// function of (rule set, body).
	BodyCacheSize int
	// UseRegexSet enables the experimental compiled-regex Set prefilter
	// (setprefilter.go): the body is scanned once by a sharded go-re2 Set to learn
	// which rule patterns can match at all, and only those rules' capture regexes
	// are run. Off by default — it layers on top of the keyword prefilter and its
	// benefit is workload-dependent, so it ships benchmarked and opt-in. If the
	// set fails to compile, detection falls back to the keyword prefilter alone.
	UseRegexSet bool
	// RegexSetShardSize is the number of patterns per Set shard when UseRegexSet is
	// on (<=0 → default). A large single set may exceed go-re2's program-size
	// limit, so patterns are compiled into shards.
	RegexSetShardSize int
}

Options configures a Detector.

Directories

Path Synopsis
Package catalog defines the on-disk schema for the native secret-detection rule catalog.
Package catalog defines the on-disk schema for the native secret-detection rule catalog.
Command secretgen normalizes the kingfisher secret-detection rule set into a single RE2-compatible catalog embedded by pkg/secretscan.
Command secretgen normalizes the kingfisher secret-detection rule set into a single RE2-compatible catalog embedded by pkg/secretscan.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL