Documentation
¶
Overview ¶
Package detect decides which columns, meta keys and JSON paths hold personal data or secrets, from their names and from sampled values. It never keeps or returns the values themselves, only counts.
Index ¶
- Constants
- func Analyze(sub Subject, p *Profile) []model.Finding
- func BinaryIP(hexValue string) (ip string, empty, ok bool)
- func BinaryIPColumn(table, column string) bool
- func DataPrefixes() []string
- func Flatten(s string) map[string]string
- func FreeTextKey(key string) bool
- func HasEmail(s string) bool
- func IsEmail(s string) bool
- func JoinPath(path []string) string
- func KeyNameFinding(table, keyColumn, shown string, fam *KeyFamily, partial bool) model.Finding
- func NameWords(name string) []string
- func PeopleWord(w string) bool
- func ShowKey(key string) string
- func ShowPattern(p string) string
- func SortFindings(fs []model.Finding)
- func Tokens(name string) []string
- func Walk(s string, fn func(path []string, value string)) bool
- type FamilyRule
- type Hint
- type KeyFamily
- type KeyTemplate
- type KeyWords
- type NameMatch
- type Profile
- type SecretCount
- type SecretHit
- type Subject
Constants ¶
const HiddenKey = "(key hidden)"
HiddenKey stands for a key name that can't be shown.
const TypeString = tStr
TypeString is the type class of text columns and meta keys, for MatchName.
Variables ¶
This section is empty.
Functions ¶
func Analyze ¶
Analyze judges one subject from its name and the profile of its sampled values. p is nil when no values were read.
func BinaryIP ¶ added in v0.4.0
BinaryIP decodes the hex of a value from a binary IP column. empty reports no bytes or all zero bytes, as unset addresses are stored (MyBB's default is an empty string). ok reports an address in a form it knows: printable text that parses as an IP; 4 bytes, or 4 followed by 12 zero bytes as a BINARY(16) pads them; 16 bytes that are an IPv4-mapped address or fall in the global (2000::/3), unique-local (fc00::/7) or link-local (fe80::/10) IPv6 ranges. Random bytes, such as a hash, land there about 13% of the time. The address only ever reaches a Profile.
func BinaryIPColumn ¶ added in v0.4.0
BinaryIPColumn reports whether a binary column's name says it holds IP addresses, so its bytes are worth reading: the name matches the IP rule and has no word of a hash or a key. visitor_ip_hash isn't read, and keeps its finding by name.
func DataPrefixes ¶ added in v0.3.0
func DataPrefixes() []string
DataPrefixes lists the known prefixes of key names whose rest is data.
func Flatten ¶
Flatten returns the leaves of a JSON or PHP-serialized value by path, keeping the first value of each path. It is for reading settings, such as whether a payment gateway runs in live mode; callers must never print the values.
func FreeTextKey ¶ added in v0.4.0
FreeTextKey reports whether a key, as reports would show it, has a space and doesn't read as a label. In a key/value table found without a preset, such a key may be a person's name, so it isn't shown.
func HasEmail ¶ added in v0.3.0
HasEmail reports whether a value is, or contains, an email address. It is for reading settings; callers must never print the value.
func JoinPath ¶
JoinPath renders a path: billing.email, items[].name, cart.*.line_total, lockouts{} (the keys of a map keyed by data).
func KeyNameFinding ¶ added in v0.3.0
KeyNameFinding reports that the names of a family's keys hold personal data, so a copy has to rewrite the key column as well as the values.
func NameWords ¶ added in v0.4.0
NameWords splits a name that is being judged into words, as Tokens does, and also splits run-together words: regip (reg, ip), loginkey (login, key), lastloginip (last, login, ip). Only judged names go through it; table names, path context, key templates and relationship inference keep the words Tokens gives.
func PeopleWord ¶ added in v0.4.0
PeopleWord reports whether a word names people, as users or members do.
func ShowKey ¶ added in v0.3.0
ShowKey is how a stored key name may appear in reports, warnings and configs: with the parts that look like data replaced by *, or as HiddenKey when even that could carry data.
func ShowPattern ¶ added in v0.3.0
ShowPattern checks a key or pattern for display. A pattern needs three literal letters, so * or *_* never stands for a whole key.
func SortFindings ¶
SortFindings orders findings by severity (worst first), then location.
func Tokens ¶
Tokens splits an identifier into lower-case words. It understands snake_case, kebab-case, camelCase and digits, and splits a few common compounds: "firstname" becomes first, name.
Types ¶
type FamilyRule ¶ added in v0.3.0
type FamilyRule struct {
Prefix string
Carries model.Category // what the rest of the name holds, if personal
Hint *Hint
}
FamilyRule is what a preset knows about keys that share a prefix.
type Hint ¶
type Hint struct {
Category model.Category
Severity model.Severity // zero means the category default
Mask string
Rows []model.RowMask // other masks for some rows of the column
Note string
Source string // e.g. "preset:woocommerce"
Ignore bool // known not to hold personal data
// Delete marks a cache the platform rebuilds, such as WordPress
// transients: a copy should delete these rows whatever they hold. A
// hint with Delete and no Category doesn't say what the data is, so
// the subject is judged as if there were no hint.
Delete bool
}
Hint is what a platform preset knows about a column or meta key.
type KeyFamily ¶ added in v0.3.0
type KeyFamily struct {
Pattern string
Keys []string // sorted
Carries model.Category
Hint *Hint // from a preset rule
// contains filtered or unexported fields
}
KeyFamily is a set of keys whose names differ only in a part that looks like data, such as a user ID, a hash or an email address.
type KeyTemplate ¶ added in v0.3.0
type KeyTemplate struct {
Pattern string // e.g. um_cache_userdata_* or _transient_feed_*
// Short is Pattern with runs of 1 to 3 digits replaced too. Those
// become * only when several keys share the result.
Short string
Carries model.Category // email or IP address when the varying part is one
// contains filtered or unexported fields
}
KeyTemplate is a key name with the parts that look like data replaced by *.
func Template ¶ added in v0.3.0
func Template(key string) KeyTemplate
Template replaces the parts of a key that look like data: email and IP addresses, UUIDs, runs of 4 or more digits, hex hashes of 8 or more characters, long tokens, and whatever follows a known data prefix.
type KeyWords ¶ added in v0.4.0
type KeyWords struct {
// Label says the key names a field: at most one of its words is
// unknown, and it ends in a label word or has a lower-case word the
// rules or field names use. Date of birth and shipping method are
// labels; Jane Doe phone, John Street and Next of kin aren't.
Label bool
Known []string // its words the rules, field names and labels use, lower case, in order
Unknown int // its other words, numbers aside
}
KeyWords is how the key-shape guard of a key/value table found without a preset reads a stored key with a space in it.
type NameMatch ¶
type NameMatch struct {
Category model.Category
Words string // the words that matched, e.g. "first name"
Severity model.Severity // zero means the category default
Note string
// Password marks a name that says a password is stored here: values
// that aren't hashes are then plain text or reversible encryption.
Password bool
// Messages marks text written between people, such as private
// messages and notifications.
Messages bool
}
NameMatch is what a column name, meta key or JSON key says about its data.
type Profile ¶
type Profile struct {
Values int // all values seen, including empty ones
NonEmpty int
Email, EmailReserved int
Phone, PhoneStrict int
IP int
IBAN int
Card, CardTest int
Hash, HexDigest int
HashFormats map[string]int
UserAgent int
PaymentID int
Date int
Secrets map[string]*SecretCount
// SecretValues counts the values with at least one secret, once each.
SecretValues int
// Prose: values with spaces that are longer than a few words.
Text, TextEmail, TextPhone, TextIP, TextCard int
// Values that cannot be names or addresses: plain numbers, and code
// identifiers such as App\Jobs\SendInvoice.
Numeric, CodeLike int
// Values that say a name match is wrong: on/off flags, field names
// (billing_address_1, edit-address), placeholders and ACF field keys,
// and the field's own name (as in translation tables and import
// mappings).
Flag, FieldName, Placeholder, Echo int
// MaxDigits is the most digits any one value holds: an ID number or a
// birth date has at least four.
MaxDigits int
// Values that back or rule out other judgements: hardware IDs
// (Opaque), push notification tokens (Push), email addresses inside
// URLs (Inline), URLs and paths (URL), cryptocurrency addresses
// (Crypto), text with spaces and no merge tags (Spaced), and whole
// numbers (Whole).
Opaque, Push, Inline, URL, Crypto, Spaced, Whole int
Structured int
Paths map[string]*Profile
// contains filtered or unexported fields
}
Profile counts what the sampled values of one column, meta key or JSON path look like. It keeps counts only, never the values.
func NewProfile ¶
func NewProfile() *Profile
NewProfile returns an empty profile. Values that are JSON or PHP-serialized are opened up and counted per path.
func NewProfileNamed ¶ added in v0.3.0
NewProfileNamed returns an empty profile for a field with the given name; values that only repeat the name, such as "First name" under first_name, count as field names.
func (*Profile) Add ¶
Add counts one sampled value. An empty array or object, or one whose leaves are all empty, counts as an empty value: WordPress keeps recovery_keys as a:0:{} on nearly every site.
func (*Profile) SecretKinds ¶
SecretKinds lists the kinds of secrets seen, sorted.
func (*Profile) SecretTotals ¶
SecretTotals sums the secret counts over all kinds.
func (*Profile) WithContext ¶ added in v0.4.0
WithContext records the words of the table a field belongs to, so paths past the path cap are named in the same context as the rest.
type SecretCount ¶
type SecretCount struct {
Live, Test, Unknown int
}
SecretCount counts values holding one kind of secret.
type SecretHit ¶
SecretHit is one secret-looking string. Mode is "live", "test" or "" when the format does not say.
func FindSecrets ¶
FindSecrets returns the kinds of secrets that appear in s, once per kind, each with the strongest mode among its matches: live, then unknown, then test.
type Subject ¶
type Subject struct {
Table string
Column string
MetaKey string
Name string // the name to judge: the column name or the meta key
DataType string // information_schema DATA_TYPE; empty for meta keys
Context []string // words of the table name
Hint *Hint
Sampled bool // false when values were not read (--no-samples)
Keys int // keys a pattern stands for, copied to the findings
KeysPartial bool // Keys is a lower bound
Where string // the subject is these rows only, copied to the findings
// Setting marks a key of a settings table, which holds one value; an
// empty value then means nothing is stored.
Setting bool
// Skipped marks a key left without values by a sampling limit; name
// matches on it stay weak.
Skipped bool
// Labels marks a key of a translation table: it names a UI string, not
// a field, so only its values are judged.
Labels bool
}
Subject is one column, or one key of a key/value table, to judge.