Documentation
¶
Overview ¶
Package detect decides which columns, meta keys and JSON paths hold personal data or secrets, from their names and from sampled values. It never keeps or returns the values themselves, only counts.
Index ¶
- Constants
- Variables
- func Analyze(sub Subject, p *Profile) []model.Finding
- func BinaryIP(hexValue string) (ip string, empty, ok bool)
- func BinaryIPColumn(table, column string) bool
- func DataPrefixes() []string
- func EditStructured(s string, fn func(Leaf) (string, bool)) (out string, edited bool, err error)
- func Flatten(s string) map[string]string
- func FreeTextKey(key string) bool
- func HasEmail(s string) bool
- func IsEmail(s string) bool
- func JoinPath(path []string) string
- func KeyNameFinding(table, keyColumn, shown string, fam *KeyFamily, partial bool) model.Finding
- func KnownWord(w string) bool
- func NameWords(name string) []string
- func PeopleWord(w string) bool
- func ShowKey(key string) string
- func ShowPattern(p string) string
- func SortFindings(fs []model.Finding)
- func Tokens(name string) []string
- func Walk(s string, fn func(path []string, value string)) bool
- type FamilyRule
- type Hint
- type KeyFamily
- type KeyTemplate
- type KeyWords
- type Leaf
- type LeafKind
- type NameMatch
- type Profile
- type SecretCount
- type SecretHit
- type Span
- type SpanFinder
- type Subject
- type TextView
Constants ¶
const ( SpanSecret = "secret" SpanURLPassword = "url_password" SpanPasswordHash = "password_hash" SpanEmail = "email" SpanIBAN = "iban" SpanCard = "card" SpanIP = "ip" SpanPhone = "phone" // SpanLoosePhone is the kind of the finder of loose phone numbers, whose // spans are of kind SpanPhone with Detail "loose". SpanLoosePhone = "loose_phone" )
The kinds of the built-in spans.
const HiddenKey = "(key hidden)"
HiddenKey stands for a key name that can't be shown.
const TypeString = tStr
TypeString is the type class of text columns and meta keys, for MatchName.
Variables ¶
var ( // ErrNotStructured: the value isn't one complete JSON or PHP-serialized // value, such as one cut off or with bytes after it. ErrNotStructured = errors.New("not a complete JSON or PHP-serialized value") // ErrTooLarge: the value is over the editor's limits of size, nodes, // depth or nesting. ErrTooLarge = errors.New("structured value over the editor's limits") // ErrBadEdit: an edit can't be written, such as a key that would equal // another key of its map, or a number that isn't one. ErrBadEdit = errors.New("an edit can't be written into the structured value") )
The errors EditStructured returns. Their text is fixed and never holds any of the value.
Functions ¶
func Analyze ¶
Analyze judges one subject from its name and the profile of its sampled values. p is nil when no values were read.
func BinaryIP ¶ added in v0.4.0
BinaryIP decodes the hex of a value from a binary IP column. empty reports no bytes or all zero bytes, as unset addresses are stored (MyBB's default is an empty string). ok reports an address in a form it knows: printable text that parses as an IP; 4 bytes, or 4 followed by 12 zero bytes as a BINARY(16) pads them; 16 bytes that are an IPv4-mapped address or fall in the global (2000::/3), unique-local (fc00::/7) or link-local (fe80::/10) IPv6 ranges. Random bytes, such as a hash, land there about 13% of the time. The address only ever reaches a Profile.
func BinaryIPColumn ¶ added in v0.4.0
BinaryIPColumn reports whether a binary column's name says it holds IP addresses, so its bytes are worth reading: the name matches the IP rule and has no word of a hash or a key. visitor_ip_hash isn't read, and keeps its finding by name.
func DataPrefixes ¶ added in v0.3.0
func DataPrefixes() []string
DataPrefixes lists the known prefixes of key names whose rest is data.
func EditStructured ¶ added in v0.7.0
EditStructured calls fn for every leaf of a JSON or PHP-serialized value, keys included, in order, and returns the value with the edits fn asked for: fn returns a leaf's new value and true, or false to keep it. Strings holding JSON or PHP-serialized data are opened as Walk opens them, and fn gets their leaves instead.
Bytes that aren't edited are kept exactly: with no edit, out is s. An edited JSON string is escaped as the bytes it replaces are, or else as the value's other strings are, and PHP's lengths are recomputed in bytes, in every string and C: payload that holds an edit. A JSON string holding an edited value has only the characters the edits touch written again. Enum cases can't be edited, PHP integer keys only to integers, numbers and booleans only to valid ones, and keys only to keys unique in their map. On an error, out is empty, so a caller that drops the error writes nothing of the value.
func Flatten ¶
Flatten returns the leaves of a JSON or PHP-serialized value by path, keeping the first value of each path. It is for reading settings, such as whether a payment gateway runs in live mode; callers must never print the values.
func FreeTextKey ¶ added in v0.4.0
FreeTextKey reports whether a key, as reports would show it, has a space and doesn't read as a label. In a key/value table found without a preset, such a key may be a person's name, so it isn't shown.
func HasEmail ¶ added in v0.3.0
HasEmail reports whether a value is, or contains, an email address. It is for reading settings; callers must never print the value.
func JoinPath ¶
JoinPath renders a path: billing.email, items[].name, cart.*.line_total, lockouts{} (the keys of a map keyed by data).
func KeyNameFinding ¶ added in v0.3.0
KeyNameFinding reports that the names of a family's keys hold personal data, so a copy has to rewrite the key column as well as the values.
func KnownWord ¶ added in v0.7.0
KnownWord reports whether a word is one the naming rules, field names or labels use, such as billing, phone or name, in any letter case. The words are fixed in the scan, so a value never makes one known.
func NameWords ¶ added in v0.4.0
NameWords splits a name that is being judged into words, as Tokens does, and also splits run-together words: regip (reg, ip), loginkey (login, key), lastloginip (last, login, ip). Only judged names go through it; table names, path context, key templates and relationship inference keep the words Tokens gives.
func PeopleWord ¶ added in v0.4.0
PeopleWord reports whether a word names people, as users or members do.
func ShowKey ¶ added in v0.3.0
ShowKey is how a stored key name may appear in reports, warnings and configs: with the parts that look like data replaced by *, or as HiddenKey when even that could carry data.
func ShowPattern ¶ added in v0.3.0
ShowPattern checks a key or pattern for display. A pattern needs three literal letters, so * or *_* never stands for a whole key.
func SortFindings ¶
SortFindings orders findings by severity (worst first), then location.
func Tokens ¶
Tokens splits an identifier into lower-case words. It understands snake_case, kebab-case, camelCase and digits, and splits a few common compounds: "firstname" becomes first, name.
Types ¶
type FamilyRule ¶ added in v0.3.0
type FamilyRule struct {
Prefix string
Carries model.Category // what the rest of the name holds, if personal
Hint *Hint
}
FamilyRule is what a preset knows about keys that share a prefix.
type Hint ¶
type Hint struct {
Category model.Category
Severity model.Severity // zero means the category default
Mask string
Rows []model.RowMask // other masks for some rows of the column
Note string
Source string // e.g. "preset:woocommerce"
Ignore bool // known not to hold personal data
// Delete marks a cache the platform rebuilds, such as WordPress
// transients: a copy should delete these rows whatever they hold. A
// hint with Delete and no Category doesn't say what the data is, so
// the subject is judged as if there were no hint.
Delete bool
}
Hint is what a platform preset knows about a column or meta key.
type KeyFamily ¶ added in v0.3.0
type KeyFamily struct {
Pattern string
Keys []string // sorted
Carries model.Category
Hint *Hint // from a preset rule
// contains filtered or unexported fields
}
KeyFamily is a set of keys whose names differ only in a part that looks like data, such as a user ID, a hash or an email address.
type KeyTemplate ¶ added in v0.3.0
type KeyTemplate struct {
Pattern string // e.g. um_cache_userdata_* or _transient_feed_*
// Short is Pattern with runs of 1 to 3 digits replaced too. Those
// become * only when several keys share the result.
Short string
Carries model.Category // email or IP address when the varying part is one
// contains filtered or unexported fields
}
KeyTemplate is a key name with the parts that look like data replaced by *.
func Template ¶ added in v0.3.0
func Template(key string) KeyTemplate
Template replaces the parts of a key that look like data: email and IP addresses, UUIDs, runs of 4 or more digits, hex hashes of 8 or more characters, long tokens, and whatever follows a known data prefix.
type KeyWords ¶ added in v0.4.0
type KeyWords struct {
// Label says the key names a field: at most one of its words is
// unknown, and it ends in a label word or has a lower-case word the
// rules or field names use. Date of birth and shipping method are
// labels; Jane Doe phone, John Street and Next of kin aren't.
Label bool
Known []string // its words the rules, field names and labels use, lower case, in order
Unknown int // its other words, numbers aside
}
KeyWords is how the key-shape guard of a key/value table found without a preset reads a stored key with a space in it.
type Leaf ¶ added in v0.7.0
type Leaf struct {
// Path is the leaf's path as Walk gives it: keys that may be data are
// "*", "[]" or "{}". For a key, it is the path of the object or array
// that holds it. Path is valid only during the call.
Path []string
// Key is a key's own text, for LeafKey: a JSON or PHP string key
// decoded, or a PHP integer key in decimal. Segment is what the key adds
// to the paths below it: the key, a template of it, "*" or "[]".
Key, Segment string
// DataKey marks a key that is itself a value worth checking, such as an
// email or IP address, which Walk reports as a value at path {}.
DataKey bool
// Value is a string decoded, JSON escapes resolved, and other leaves as
// written: 12.50, true, 1 for PHP's b:1;, null or N. For a key, it is
// the key.
Value string
Kind LeafKind
}
Leaf is one scalar or one key of a structured value, as EditStructured hands it out.
type NameMatch ¶
type NameMatch struct {
Category model.Category
Words string // the words that matched, e.g. "first name"
Severity model.Severity // zero means the category default
Note string
// Password marks a name that says a password is stored here: values
// that aren't hashes are then plain text or reversible encryption.
Password bool
// Messages marks text written between people, such as private
// messages and notifications.
Messages bool
}
NameMatch is what a column name, meta key or JSON key says about its data.
type Profile ¶
type Profile struct {
Values int // all values seen, including empty ones
NonEmpty int
Email, EmailReserved int
Phone, PhoneStrict int
IP int
IBAN int
Card, CardTest int
Hash, HexDigest int
HashFormats map[string]int
UserAgent int
PaymentID int
Date int
Secrets map[string]*SecretCount
// SecretValues counts the values with at least one secret, once each.
SecretValues int
// Prose: values with spaces that are longer than a few words.
Text, TextEmail, TextPhone, TextIP, TextCard int
// Values that cannot be names or addresses: plain numbers, and code
// identifiers such as App\Jobs\SendInvoice.
Numeric, CodeLike int
// Values that say a name match is wrong: on/off flags, field names
// (billing_address_1, edit-address), placeholders and ACF field keys,
// and the field's own name (as in translation tables and import
// mappings).
Flag, FieldName, Placeholder, Echo int
// MaxDigits is the most digits any one value holds: an ID number or a
// birth date has at least four.
MaxDigits int
// Values that back or rule out other judgements: hardware IDs
// (Opaque), push notification tokens (Push), email addresses inside
// URLs (Inline), URLs and paths (URL), cryptocurrency addresses
// (Crypto), text with spaces and no merge tags (Spaced), and whole
// numbers (Whole).
Opaque, Push, Inline, URL, Crypto, Spaced, Whole int
Structured int
Paths map[string]*Profile
// contains filtered or unexported fields
}
Profile counts what the sampled values of one column, meta key or JSON path look like. It keeps counts only, never the values.
func NewProfile ¶
func NewProfile() *Profile
NewProfile returns an empty profile. Values that are JSON or PHP-serialized are opened up and counted per path.
func NewProfileNamed ¶ added in v0.3.0
NewProfileNamed returns an empty profile for a field with the given name; values that only repeat the name, such as "First name" under first_name, count as field names.
func (*Profile) Add ¶
Add counts one sampled value. An empty array or object, or one whose leaves are all empty, counts as an empty value: WordPress keeps recovery_keys as a:0:{} on nearly every site.
func (*Profile) SecretKinds ¶
SecretKinds lists the kinds of secrets seen, sorted.
func (*Profile) SecretTotals ¶
SecretTotals sums the secret counts over all kinds.
func (*Profile) WithContext ¶ added in v0.4.0
WithContext records the words of the table a field belongs to, so paths past the path cap are named in the same context as the rest.
type SecretCount ¶
type SecretCount struct {
Live, Test, Unknown int
}
SecretCount counts values holding one kind of secret.
type SecretHit ¶
SecretHit is one secret-looking string. Mode is "live", "test" or "" when the format does not say.
func FindSecrets ¶
FindSecrets returns the kinds of secrets that appear in s, once per kind, each with the strongest mode among its matches: live, then unknown, then test.
type Span ¶ added in v0.7.0
type Span struct {
Start, End int
Kind string // one of the Span* kinds, or a caller's own
// Detail says more: a secret's or a password hash's kind, "strict" or
// "loose" for a phone number, "test" for a published test card number,
// "reserved" for an email address at a domain reserved for examples, and
// "placeholder" for a password in a URL that reads as an example.
Detail string
Mode string // for secrets: live, test or "" when the format does not say
}
Span is one match inside a value, by its byte offsets in the string given. A match found through an escape, such as JSON's \n or \u00e9, or the %40 of a URL, covers the whole escape. Callers must never print the text a span covers.
func FindSpans ¶ added in v0.7.0
func FindSpans(s string, finders []SpanFinder) []Span
FindSpans runs the finders on s and returns their spans sorted by Start, no two overlapping: where spans overlap, the earlier finder's wins, then the longer one. Spans outside s, or empty, are dropped.
The result is final. A caller that keeps only some of a finder's spans, such as loose phone numbers that match a known number, must filter them inside that finder, with SpanFinder.Keep, so the spans it would drop never hide another finder's.
type SpanFinder ¶ added in v0.7.0
type SpanFinder struct {
Kind string
// Find finds spans in a value.
Find func(s string) []Span
// FindView, when set, finds spans in the value's TextView, by offsets of
// its Text. FindSpans prefers it, and builds the view once for every
// finder that has one. A copy of a built-in finder that is given a Find
// of its own must have its FindView set to nil, or wrapped too, as Keep
// wraps both.
FindView func(v *TextView) []Span
}
SpanFinder finds the spans of one kind in a value.
func SpanFinders ¶ added in v0.7.0
func SpanFinders() []SpanFinder
SpanFinders returns the built-in finders in priority order: secrets, passwords in URLs and password hashes; then email addresses, IBANs, card numbers, IP addresses and phone numbers whose layout says phone; and last loose phone numbers, so that a caller can filter them in their finder without their spans hiding others. They use the scan's own patterns and checks, but report every match, where the scan stops counting at 8 per pattern.
func (SpanFinder) Keep ¶ added in v0.7.0
func (f SpanFinder) Keep(keep func(text string, sp Span) bool) SpanFinder
Keep returns the finder with only the spans keep accepts, whichever of Find and FindView FindSpans calls. keep gets the text the finder read, the value or its view's Text, and the span by offsets of that text.
type Subject ¶
type Subject struct {
Table string
Column string
MetaKey string
Name string // the name to judge: the column name or the meta key
DataType string // information_schema DATA_TYPE; empty for meta keys
Context []string // words of the table name
Hint *Hint
Sampled bool // false when values were not read (--no-samples)
Keys int // keys a pattern stands for, copied to the findings
KeysPartial bool // Keys is a lower bound
Where string // the subject is these rows only, copied to the findings
// Setting marks a key of a settings table, which holds one value; an
// empty value then means nothing is stored.
Setting bool
// Skipped marks a key left without values by a sampling limit; name
// matches on it stay weak.
Skipped bool
// Labels marks a key of a translation table: it names a UI string, not
// a field, so only its values are judged.
Labels bool
}
Subject is one column, or one key of a key/value table, to judge.
type TextView ¶ added in v0.7.0
type TextView struct {
// Text is the value's text, as finders read it. Like the value, it
// holds the data: never print it.
Text string
// contains filtered or unexported fields
}
TextView is a value with its escapes read as what they stand for, so finders see text as an application would: JSON's \" \\ \/ \b \f \n \r \t and \uXXXX, surrogate pairs included, and %40 as @. A backslash that starts no such escape is itself. Span maps a span of Text back to the value, widened to whole escapes.
func NewTextView ¶ added in v0.7.0
NewTextView reads a value's escapes.