Documentation
¶
Overview ¶
Package entropy implements deterministic, allocation-conscious heuristics for judging whether a short byte slice "looks like" a secret: order-0 Shannon entropy, alphabet-aware classification (UUID, hex hash, digit run, base64-ish, word-like), and detection of common non-secret placeholder values.
Bounded candidates only ¶
Every function in this package operates on a single, already-bounded candidate slice — typically the value a rule's Validate function just captured from its (already bounded) window. Nothing here scans, tokenizes, or slides a window over arbitrary input, and none of it is safe or meaningful to call on a whole document: entropy measured over megabytes of mixed text is not a useful signal, and the classification heuristics assume they are looking at one token, not a stream. Callers MUST hand this package short, pre-extracted candidates (see Options. MinLen/MaxLen for the bounds the generic contextual rules use) and must never invoke Shannon, BitsTotal, Classify, IsPlaceholder, or Secretlike directly on unbounded input.
All functions are pure, byte-oriented, and safe to call on arbitrary (including non-UTF-8, non-printable) bytes without panicking — including nil or empty slices.
Hashes, UUIDs, IDs, and source examples ¶
Deterministic-looking data — UUIDs, hex hashes (git SHAs, sha256sums, etc.), and pure digit runs (database IDs, timestamps) — is bucketed by Classify and, by default, rejected by Secretlike via the Options. RejectUUID / RejectHexHash / RejectDigits knobs used by the generic contextual rules in internal/rules/validators/generic.go. This is a deliberate false-positive reduction for those CONTEXTUAL/heuristic rules only: git commit SHAs, database primary keys, request IDs, and worked examples in documentation are extremely common near assignment-style trigger words ("token = <sha>", "id = <uuid>") and are not secrets. Word-like values (English-ish lowercase tokens, e.g. config option names) are rejected unconditionally by Secretlike, since a value that classifies as ClassWordlike is — by construction of that class — a low-entropy token, never a plausible secret.
Deterministic PROVIDER rules (e.g. github-pat, slack-bot-token in internal/rules/specs/seed.json) do not use this package at all: their validators confirm an exact, well-known token shape and are entirely unaffected by these heuristics.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var PresetAssignmentValue = Options{ MinLen: 12, MaxLen: 128, MinBitsPerByte: 3.4, RejectUUID: true, RejectHexHash: true, RejectDigits: true, MaxRepeatRun: 4, }
PresetAssignmentValue is the Options used for tight "key = value" style assignments (generic-api-key-assignment, generic-password-assignment): values in this position are typically API keys, tokens, or passwords with a fairly narrow, deliberately-random alphabet (base62/base64-ish), so:
- MinLen/MaxLen of 12-128 excludes short config flags and excludes implausibly long values (embedded certs, multi-line blobs) that a single-line assignment value should not contain.
- MinBitsPerByte of 3.4 sits comfortably above the ~2.7-3.0 bits/byte typical of English words and common placeholder phrases, but below the ~4.5-6 bits/byte typical of real random base62/base64 secrets of this length — chosen empirically against this package's test fixtures rather than derived from a closed-form bound, since a handful of "in-between" alphabets (structured lowercase-hex-like identifiers) are already excluded structurally by RejectHexHash.
- RejectUUID/RejectHexHash/RejectDigits are all true: assignment values matching those shapes are overwhelmingly IDs, hashes, or example data, not secrets, in this context.
- MaxRepeatRun of 4 catches "aaaa..."/"xxxx..." placeholders that slip past the keyword list in IsPlaceholder.
var PresetDeepGenericValue = Options{ MinLen: 16, MaxLen: 128, MinBitsPerByte: 3.7, RejectUUID: true, RejectDigits: true, MaxRepeatRun: 4, }
PresetDeepGenericValue is the Options used by the deep-profile-only generic-secret-assignment rule (GenericSecretAssignment, internal/rules/ validators/genericdeep.go). That rule's keyword surface (bare "key", "secret", "auth", "access", "credential", "creds") is deliberately broader — and so weaker evidence of a secret position — than any of the three balanced-profile generic rules' own keywords: the entire point of the rule is to catch assignments those narrower, specific keywords miss (e.g. "AWS Secret Key = ..." contains no literal "secret_key" substring). To compensate for that weaker context:
- MinBitsPerByte of 3.7 matches PresetLooseToken's own compensation for a weak, generic trigger.
- MinLen of 16 and MaxLen of 128 are the same general bounds PresetAssignmentValue and PresetLooseToken already use for this shape of credential value.
- RejectHexHash is left false, for the same reason PresetLooseToken leaves it false: long hex strings are a common real session/auth token shape, and structurally rejecting all hex here would defeat a meaningful fraction of what this broader rule exists to catch.
- RejectUUID and RejectDigits stay true: a bare UUID or digit run is essentially never the secret value itself in this position.
- MaxRepeatRun of 4 catches placeholder runs the same way PresetAssignmentValue's does.
var PresetLooseToken = Options{ MinLen: 16, MaxLen: 256, MinBitsPerByte: 3.7, RejectUUID: true, RejectDigits: true, }
PresetLooseToken is the Options used for the weaker "token" trigger (generic-bearer-like-token-assignment), which fires on far more non-secret code (`token = csv.next()`, `token: <cursor>`) than a dedicated api_key/password trigger does, so it demands more from the value before calling it secretlike:
- MinLen/MaxLen of 16-256: bearer-like tokens run longer than typical password fields, and the rule additionally requires length >= 20 at the validator level (stricter than this preset alone).
- MinBitsPerByte of 3.7 is higher than PresetAssignmentValue's 3.4, compensating for the trigger being weaker evidence of "this is a secret position" than api_key/password: with a noisier trigger, the value itself has to look more convincingly random.
- RejectUUID and RejectDigits are true (tokens are rarely bare UUIDs or bare digit runs in practice), but RejectHexHash is deliberately left false: long hex strings ("token: deadbeef...") are a common enough real bearer/session token shape that structurally rejecting all hex would defeat the rule's purpose.
- MaxRepeatRun is left at 0 (disabled): unlike the tight assignment preset, this preset relies on IsPlaceholder plus the entropy floor alone to catch placeholder-shaped tokens.
Functions ¶
func BitsTotal ¶
BitsTotal returns Shannon(b) * len(b): the total (not per-byte) entropy of b. It is a convenience for callers that want to reason about absolute information content rather than a per-byte rate — e.g. comparing two candidates of different lengths on the same footing.
func IsModularCryptHash ¶
IsModularCryptHash reports whether b has the modular-crypt-format grammar used by password hashes such as "$2b$12$..." (bcrypt), "$argon2id$v=19$..." (argon2), or "$6$..." (sha512crypt): a leading '$', followed by one of a small set of known algorithm identifiers, followed by at least one more '$'-delimited field.
Counting '$' bytes alone is not enough: a value like "$databaseUser$databasePass$environmentName" — three concatenated shell variables — has just as many '$' separators as a real hash, but its first field is not a recognized algorithm identifier. Validating that field against modularCryptIDs is what tells the two apart.
func IsPlaceholder ¶
IsPlaceholder reports whether b is a well-known non-secret stand-in value rather than real secret material: common keyword placeholders ("example", "changeme", "password", ...), angle-bracket tokens ("<REDACTED>"), template and variable references ("${VAR}", "$VAR", "{token}", "{{ var }}"), a single byte repeated for the whole value, and ascending or keyboard-row runs ("abcdef...", "123456...", "qwerty...").
IsPlaceholder is deliberately permissive about what counts as a placeholder — false positives here (a real secret misclassified as a placeholder) are the cost of catching the extremely common documentation/fixture/config-template values that would otherwise dominate a naive entropy-only detector's false-positive rate. It is byte-oriented and allocation-free — validators call it once per candidate on the streaming hot path — and it never panics regardless of b's contents, including nil or empty.
func Secretlike ¶
Secretlike reports whether b plausibly looks like a secret under o: it checks length bounds, rejects well-known placeholder values (see IsPlaceholder), rejects long identical-byte runs (see Options.MaxRepeatRun), rejects the byte-shape classes o opts into rejecting (see Options.RejectUUID/RejectHexHash/RejectDigits), always rejects word-like values (ClassWordlike, by construction never a plausible secret), and finally requires Shannon(b) to reach o.MinBitsPerByte (when set).
Secretlike is a heuristic, not a proof: it is tuned to a false-positive budget (see internal/rules/validators/generic.go), not to certainty.
func Shannon ¶
Shannon returns the order-0 Shannon entropy of b, in bits per byte: -sum(p(c) * log2(p(c))) over the observed byte value distribution. It ranges from 0 (every byte identical) to 8 (all 256 byte values observed with equal frequency, unreachable for any b shorter than 256 bytes). Shannon(nil) and Shannon of an empty slice are both 0 by convention: an empty candidate carries no information.
This is an empirical, order-0 (single-byte-frequency) measure: it does not detect structure such as repeated substrings, alternating patterns, or bigram statistics. It is intended to run over short (bytes to low hundreds of bytes) candidate slices, never over whole documents.
Types ¶
type Class ¶
type Class int
Class buckets a candidate byte slice by its byte-level shape, used to apply structural (shape-based) rejections that entropy alone cannot: a syntactically perfect UUID or hex hash has exactly the same entropy as a random secret of the same length, so it has to be recognized by shape.
const ( // ClassUnknown is anything that does not match one of the more // specific classes below. ClassUnknown Class = iota // ClassUUID is a canonical 8-4-4-4-12 hyphenated hex UUID, any case. ClassUUID // ClassHexHash is a pure hex string (any case) at one of the common // hash digest lengths: 32 (MD5), 40 (SHA-1, git SHA), 64 (SHA-256), // 96 (SHA-384), 128 (SHA-512). ClassHexHash // ClassDigits is a non-empty run of ASCII decimal digits only. ClassDigits // ClassBase64ish is dominated by the base64/base64url alphabet // (A-Z, a-z, 0-9, +, /, -, _, =) without matching a more specific // class above. ClassBase64ish // ClassWordlike is mostly-lowercase-letters (allowing '_'/'-' // separators) with entropy low enough, relative to that alphabet, to // read as an English-ish word or identifier rather than a random // token. ClassWordlike )
func Classify ¶
Classify buckets b into one of the Class values above. It is deterministic, byte-oriented, and bounded by len(b): it never allocates more than a fixed amount of stack state and never iterates b more than a small constant number of times. Classify(nil) and Classify of an empty slice are both ClassUnknown.
type Options ¶
type Options struct {
// MinLen and MaxLen bound the candidate's byte length. A candidate
// shorter than MinLen or longer than MaxLen (when the respective
// bound is > 0) is never secretlike, regardless of its content.
MinLen, MaxLen int
// MinBitsPerByte is the minimum Shannon(b) (bits per byte) a
// candidate must reach, once it has cleared length, placeholder, and
// classification checks. Because a candidate's alphabet limits its
// achievable entropy (lowercase hex maxes out at 4 bits/byte,
// unpadded base64 at 6), this threshold is chosen per rule relative
// to the alphabets that rule's trigger context tends to produce, not
// as one universal cutoff — see the preset doc comments below for
// the reasoning behind 3.4 and 3.7.
MinBitsPerByte float64
// RejectUUID, RejectHexHash, and RejectDigits reject candidates that
// Classify buckets as ClassUUID, ClassHexHash, or ClassDigits
// respectively, regardless of their entropy. These are structural
// rejections: a syntactically perfect UUID or hex hash is exactly as
// "random-looking" as a real secret of the same length, so entropy
// alone cannot distinguish them — the shape has to.
RejectUUID, RejectHexHash, RejectDigits bool
// MaxRepeatRun rejects candidates containing a run of MaxRepeatRun+1
// or more identical consecutive bytes (0 disables this check). This
// catches placeholder patterns like "aaaaaaaaaaaa" or
// "XXXXXXXXXXXX" that IsPlaceholder's keyword list does not.
MaxRepeatRun int
}
Options configures Secretlike. Every field is opt-in: its zero value disables that check (MinLen/MaxLen/MinBitsPerByte/MaxRepeatRun of 0 mean "no bound"; the Reject* booleans default to false, i.e. "do not reject this class"). Callers construct Options explicitly per rule rather than relying on a single default, since the right thresholds are a function of what a rule's trigger implies about the surrounding text (see PresetAssignmentValue and PresetLooseToken for the two presets used by the generic contextual rules).
Source Files