secrets

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 5, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

internal/secrets/confidence.go

Package secrets finds credentials embedded in container images, filesystems, and Dockerfiles. It is a from-scratch implementation of provider-aware secret detection: a data-driven ruleset of provider-specific detectors, an entropy fallback, layer-aware image scanning (including content deleted by a later whiteout, which stays extractable and is a classic leak), a versioned baseline/allowlist, and an opt-in live-verification hook.

Two principles shape the design:

  • Precision is a feature. A noisy secret scanner gets muted, and a muted scanner misses the real leak. The default rule set is high-signal; the broad entropy sweep is gated behind an optional classifier so it never floods the default run.
  • Values never leave the process. A Detection carries a fingerprint (a truncated SHA-256), a type, a length, and a location — never the secret itself. Even the opt-in verifier receives the raw value transiently and the scanner discards it immediately.

The package is deterministic: given the same bytes it emits the same Detections in the same order, with no reliance on the wall clock or a random source. Verification (the one network-touching feature) is strictly opt-in and injected, so tests never reach the network.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SortDetections 

func SortDetections(ds []Detection)

SortDetections orders detections deterministically so identical input yields byte-identical output. The ordering is stable across runs and machines: non-deleted before deleted, then by layer, path, line, rule code, and finally fingerprint to break any remaining ties.

func SupportedVerifiers 

func SupportedVerifiers() []string

SupportedVerifiers returns the detector slugs HTTPVerifier can verify, for diagnostics and docs.

Types

type Baseline 

type Baseline struct {
	Version int             `json:"version"`
	Entries []BaselineEntry `json:"entries"`
	// contains filtered or unexported fields
}

Baseline is a versioned set of accepted findings.

func LoadBaseline 

func LoadBaseline(path string) (*Baseline, error)

LoadBaseline reads and validates a baseline JSON file.

func ParseBaseline 

func ParseBaseline(data []byte) (*Baseline, error)

ParseBaseline parses baseline JSON from memory. It is separate from LoadBaseline so callers (and tests) can supply bytes directly.

func (*Baseline) Allows 

func (b *Baseline) Allows(d Detection) bool

Allows reports whether d has been accepted by this baseline. It matches on rule + fingerprint, optionally scoped to a path: an unscoped entry accepts the secret wherever it appears, a path-scoped entry only at that path.

type BaselineEntry 

type BaselineEntry struct {
	RuleID        string `json:"rule_id"`
	Fingerprint   string `json:"fingerprint"`
	Path          string `json:"path,omitempty"`
	Justification string `json:"justification"`
}

BaselineEntry is a single accepted finding. RuleID and Fingerprint identify it exactly; Path, when present, further scopes the acceptance to one location. Justification is required in spirit (a baseline without reasons rots) and surfaced so reviewers can audit it.

type Candidate 

type Candidate struct {
	Value   string
	Entropy float64
}

Candidate is a high-entropy token offered to a Classifier.

type Classifier 

type Classifier interface {
	Classify(c Candidate) Verdict
}

Classifier decides whether a high-entropy Candidate is plausibly a secret. Implementations must be deterministic and side-effect free.

type Detection 

type Detection struct {
	Code        string          // stable engine RuleID, e.g. "DS-RAT-SEC-001"
	Slug        string          // machine-readable detector id, e.g. "aws-access-key-id"
	Kind        Kind            // coarse category
	Severity    engine.Severity // base severity (verification may raise it)
	Fingerprint string          // truncated SHA-256 of the secret value (never the value)
	Entropy     float64         // Shannon entropy of the secret value
	Confidence  string          // "high" | "medium" | "low" — corroboration grade (see confidence.go)
	Length      int             // length of the secret value in bytes
	Path        string          // where it was found (file path, or a pseudo-path)
	Line        int             // 1-based line within the source, 0 if not line-oriented
	Source      Source          // what kind of location Path refers to
	Deleted     bool            // found only in a layer removed by a later whiteout
	LayerIndex  int             // image layer index; -1 for the effective filesystem / non-layer sources
	LayerDigest string          // image layer digest, when known
	Verify      VerifyState     // live-verification result
	Title       string          // human summary, safe to display
	Remediation string          // structured, agent-consumable fix guidance
	References  []string        // CIS/NIST/vendor references
	// contains filtered or unexported fields
}

Detection is a single secret found by the scanner. It is deliberately value-free: Fingerprint identifies the secret without revealing it, so a Detection can be logged, serialized, and diffed safely.

type HTTPVerifier 

type HTTPVerifier struct {
	// Client is the HTTP client to use. If nil, a client with a short timeout is
	// created per call so a hung provider cannot stall a scan.
	Client *http.Client
	// Endpoints overrides a provider slug's probe URL, for tests. Empty uses the
	// real provider URLs.
	Endpoints map[string]string
}

HTTPVerifier verifies a secret against its provider with the minimal, side-effect-free authenticated request that provider supports (a read like "who am I", never a write). It is only ever constructed when the operator explicitly opts into verification; providers with no safe probe return VerifyUnknown rather than guessing.

Each provider's probe is a probe struct so the set is data-driven and every endpoint is overridable in tests (Endpoints), letting the whole detect→verify loop run against an httptest server with no real network.

func (HTTPVerifier) Verify 

func (v HTTPVerifier) Verify(ctx context.Context, ruleSlug, secret string) VerifyState

Verify implements Verifier for the providers HTTPVerifier understands.

type HeuristicClassifier 

type HeuristicClassifier struct{}

HeuristicClassifier is the built-in, model-free Classifier. It rejects the three big sources of entropy-detector noise — UUIDs, fixed-length hex digests (git SHAs, MD5/SHA hashes), and very long base64 asset blobs — and accepts mixed-alphabet tokens that look generated. It is intentionally conservative: when unsure it declines, because a false "secret" erodes trust faster than a missed context-free token (which the provider rules and assignment detector still have a shot at).

func (HeuristicClassifier) Classify 

func (HeuristicClassifier) Classify(c Candidate) Verdict

Classify implements Classifier.

type Honeytoken 

type Honeytoken struct {
	Label       string
	Value       string
	Fingerprint string
}

Honeytoken is a planted decoy credential. Value is what you embed; Fingerprint is how the scanner recognizes it (matching the fingerprint scheme used for real detections, so the two share one code path).

func GenerateHoneytoken 

func GenerateHoneytoken(label string) Honeytoken

GenerateHoneytoken derives a stable, AWS-access-key-shaped decoy from label. It performs no I/O and reads no clock or randomness, so it is safe to call in deterministic contexts and reproducible across machines.

type Kind 

type Kind string

Kind is a coarse category for a detected secret, used for grouping and for routing verification. It is intentionally small and stable. 

const (
	KindCloud      Kind = "cloud"       // cloud provider keys (AWS, GCP, Azure)
	KindVCS        Kind = "vcs"         // source-forge tokens (GitHub, GitLab)
	KindPrivateKey Kind = "private-key" // PEM private-key blocks
	KindJWT        Kind = "jwt"         // JSON Web Tokens
	KindDatabase   Kind = "database"    // connection strings with credentials
	KindPayment    Kind = "payment"     // payment-processor keys (Stripe, ...)
	KindMessaging  Kind = "messaging"   // Slack, SendGrid, ...
	KindGeneric    Kind = "generic"     // keyword/entropy heuristics
	KindCanary     Kind = "canary"      // a planted honeytoken (not a real leak)
)

type Option 

type Option func(*Scanner)

Option configures a Scanner.

func WithBaseline 

func WithBaseline(b *Baseline) Option

WithBaseline suppresses detections the baseline has accepted, while still reporting anything new.

func WithClassifier 

func WithClassifier(c Classifier) Option

WithClassifier enables the context-free entropy sweep, routing each candidate through c. Off by default; supplying HeuristicClassifier{} is the intended low-cost, offline enablement.

func WithHoneytokens 

func WithHoneytokens(hs ...Honeytoken) Option

WithHoneytokens marks the given canary fingerprints so their appearance is reported as an informational canary rather than a real leak.

func WithMaxFileBytes 

func WithMaxFileBytes(n int64) Option

WithMaxFileBytes overrides the per-file scan cap (bytes).

func WithVerifier 

func WithVerifier(v Verifier) Option

WithVerifier enables opt-in live verification of detected secrets. The verifier receives raw values and must never log them; the scanner discards them after the call. Never wire a network verifier in tests.

type Rule 

type Rule struct {
	Code     string          // stable engine RuleID, e.g. "DS-RAT-SEC-001"
	Slug     string          // machine id, e.g. "aws-access-key-id"
	Title    string          // human summary
	Kind     Kind            // category
	Severity engine.Severity // base severity
	// contains filtered or unexported fields
}

Rule is one provider-specific detector.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner runs the detector set over content. It is safe to reuse across targets and is free of hidden state: two Scanners built with the same options produce identical results. The optional Classifier, Verifier, Baseline, and honeytoken set are all injected, keeping the default scan deterministic and offline.

func New 

func New(opts ...Option) *Scanner

New returns a Scanner with the high-signal defaults: provider rules and the keyword-gated assignment detector on; entropy sweep, verification, baseline, and honeytokens all off.

func (*Scanner) ScanDockerfile 

func (s *Scanner) ScanDockerfile(ctx context.Context, path string, content []byte) []Detection

ScanDockerfile scans Dockerfile text. Beyond the shared detectors (which catch a token pasted into a RUN or ENV value), this is where ARG/ENV/RUN secrets that never reach a layer file still live, so it is a first-class target rather than an afterthought. The Dockerfile module's DS-RAT-DF-006 flags suspicious *key names*; this complements it by fingerprinting the actual *values* with the full provider ruleset.

func (*Scanner) ScanImage 

func (s *Scanner) ScanImage(ctx context.Context, img *oci.Image) []Detection

ScanImage scans a loaded image and returns sorted detections. Any non-fatal problem encountered along the way (currently: an unparsable image config, see scanConfig) is dropped silently here for backward compatibility; use ScanImageDetailed to also receive those as warnings.

func (*Scanner) ScanImageDetailed 

func (s *Scanner) ScanImageDetailed(ctx context.Context, img *oci.Image) ([]Detection, []string)

ScanImageDetailed is ScanImage plus a warnings slice for non-fatal problems (e.g. an image config that failed to parse — pass 3 then silently contributes nothing, which used to be invisible; see scanConfig). A nil/ empty warnings slice means nothing degraded.

func (*Scanner) ScanText 

func (s *Scanner) ScanText(ctx context.Context, path string, data []byte, src Source) []Detection

ScanText scans a single in-memory blob (a Dockerfile, a config file) and returns sorted detections. Source labels the origin for reporting.

func (*Scanner) ScanTree 

func (s *Scanner) ScanTree(ctx context.Context, tree *oci.FileTree) []Detection

ScanTree scans an in-memory file tree (a scanned directory, or any tree built via internal/oci) and returns sorted detections. Files are walked in path order for determinism.

type Source 

type Source string

Source describes what a Detection's Path refers to. 

const (
	SourceFile         Source = "file"          // a file in a filesystem or flattened image
	SourceDeletedLayer Source = "deleted-layer" // a file present only in a superseded layer
	SourceImageEnv     Source = "image-config-env"
	SourceImageHistory Source = "image-history"
	SourceDockerfile   Source = "dockerfile"
)

type Verdict 

type Verdict struct {
	IsSecret   bool
	Confidence float64 // 0..1
	Label      string  // human-readable reason, e.g. "uuid", "hex-digest", "credential"
}

Verdict is a Classifier's judgment of a Candidate.

type Verifier 

type Verifier interface {
	Verify(ctx context.Context, ruleSlug, secret string) VerifyState
}

Verifier confirms whether a secret is live. ruleSlug identifies the detector (e.g. "github-token") so a verifier can route to the right provider check.

type VerifierFunc 

type VerifierFunc func(ctx context.Context, ruleSlug, secret string) VerifyState

VerifierFunc adapts a function to the Verifier interface, convenient for tests and simple cases.

func (VerifierFunc) Verify 

func (f VerifierFunc) Verify(ctx context.Context, ruleSlug, secret string) VerifyState

Verify implements Verifier.

type VerifyState 

type VerifyState string

VerifyState records whether a detected secret was checked against its live provider. Verification is opt-in; the default is VerifySkipped. 

const (
	VerifySkipped  VerifyState = "skipped"  // verification not attempted (default)
	VerifyUnknown  VerifyState = "unknown"  // attempted, provider gave no clear answer
	VerifyActive   VerifyState = "active"   // confirmed live — prioritize
	VerifyInactive VerifyState = "inactive" // confirmed dead — likely already rotated
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.