detect

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package detect decides which columns, meta keys and JSON paths hold personal data or secrets, from their names and from sampled values. It never keeps or returns the values themselves, only counts.

Index

Constants

View Source
const (
	SpanSecret       = "secret"
	SpanURLPassword  = "url_password"
	SpanPasswordHash = "password_hash"
	SpanEmail        = "email"
	SpanIBAN         = "iban"
	SpanCard         = "card"
	SpanIP           = "ip"
	SpanPhone        = "phone"
	// SpanLoosePhone is the kind of the finder of loose phone numbers, whose
	// spans are of kind SpanPhone with Detail "loose".
	SpanLoosePhone = "loose_phone"
)

The kinds of the built-in spans.

View Source
const HiddenKey = "(key hidden)"

HiddenKey stands for a key name that can't be shown.

View Source
const TypeString = tStr

TypeString is the type class of text columns and meta keys, for MatchName.

Variables

View Source
var (
	// ErrNotStructured: the value isn't one complete JSON or PHP-serialized
	// value, such as one cut off or with bytes after it.
	ErrNotStructured = errors.New("not a complete JSON or PHP-serialized value")
	// ErrTooLarge: the value is over the editor's limits of size, nodes,
	// depth or nesting.
	ErrTooLarge = errors.New("structured value over the editor's limits")
	// ErrBadEdit: an edit can't be written, such as a key that would equal
	// another key of its map, or a number that isn't one.
	ErrBadEdit = errors.New("an edit can't be written into the structured value")
)

The errors EditStructured returns. Their text is fixed and never holds any of the value.

Functions

func Analyze

func Analyze(sub Subject, p *Profile) []model.Finding

Analyze judges one subject from its name and the profile of its sampled values. p is nil when no values were read.

func BinaryIP added in v0.4.0

func BinaryIP(hexValue string) (ip string, empty, ok bool)

BinaryIP decodes the hex of a value from a binary IP column. empty reports no bytes or all zero bytes, as unset addresses are stored (MyBB's default is an empty string). ok reports an address in a form it knows: printable text that parses as an IP; 4 bytes, or 4 followed by 12 zero bytes as a BINARY(16) pads them; 16 bytes that are an IPv4-mapped address or fall in the global (2000::/3), unique-local (fc00::/7) or link-local (fe80::/10) IPv6 ranges. Random bytes, such as a hash, land there about 13% of the time. The address only ever reaches a Profile.

func BinaryIPColumn added in v0.4.0

func BinaryIPColumn(table, column string) bool

BinaryIPColumn reports whether a binary column's name says it holds IP addresses, so its bytes are worth reading: the name matches the IP rule and has no word of a hash or a key. visitor_ip_hash isn't read, and keeps its finding by name.

func DataPrefixes added in v0.3.0

func DataPrefixes() []string

DataPrefixes lists the known prefixes of key names whose rest is data.

func EditStructured added in v0.7.0

func EditStructured(s string, fn func(Leaf) (string, bool)) (out string, edited bool, err error)

EditStructured calls fn for every leaf of a JSON or PHP-serialized value, keys included, in order, and returns the value with the edits fn asked for: fn returns a leaf's new value and true, or false to keep it. Strings holding JSON or PHP-serialized data are opened as Walk opens them, and fn gets their leaves instead.

Bytes that aren't edited are kept exactly: with no edit, out is s. An edited JSON string is escaped as the bytes it replaces are, or else as the value's other strings are, and PHP's lengths are recomputed in bytes, in every string and C: payload that holds an edit. A JSON string holding an edited value has only the characters the edits touch written again. Enum cases can't be edited, PHP integer keys only to integers, numbers and booleans only to valid ones, and keys only to keys unique in their map. On an error, out is empty, so a caller that drops the error writes nothing of the value.

func Flatten

func Flatten(s string) map[string]string

Flatten returns the leaves of a JSON or PHP-serialized value by path, keeping the first value of each path. It is for reading settings, such as whether a payment gateway runs in live mode; callers must never print the values.

func FreeTextKey added in v0.4.0

func FreeTextKey(key string) bool

FreeTextKey reports whether a key, as reports would show it, has a space and doesn't read as a label. In a key/value table found without a preset, such a key may be a person's name, so it isn't shown.

func HasEmail added in v0.3.0

func HasEmail(s string) bool

HasEmail reports whether a value is, or contains, an email address. It is for reading settings; callers must never print the value.

func IsEmail

func IsEmail(s string) bool

IsEmail reports whether s is one email address.

func JoinPath

func JoinPath(path []string) string

JoinPath renders a path: billing.email, items[].name, cart.*.line_total, lockouts{} (the keys of a map keyed by data).

func KeyNameFinding added in v0.3.0

func KeyNameFinding(table, keyColumn, shown string, fam *KeyFamily, partial bool) model.Finding

KeyNameFinding reports that the names of a family's keys hold personal data, so a copy has to rewrite the key column as well as the values.

func KnownWord added in v0.7.0

func KnownWord(w string) bool

KnownWord reports whether a word is one the naming rules, field names or labels use, such as billing, phone or name, in any letter case. The words are fixed in the scan, so a value never makes one known.

func NameWords added in v0.4.0

func NameWords(name string) []string

NameWords splits a name that is being judged into words, as Tokens does, and also splits run-together words: regip (reg, ip), loginkey (login, key), lastloginip (last, login, ip). Only judged names go through it; table names, path context, key templates and relationship inference keep the words Tokens gives.

func PeopleWord added in v0.4.0

func PeopleWord(w string) bool

PeopleWord reports whether a word names people, as users or members do.

func ShowKey added in v0.3.0

func ShowKey(key string) string

ShowKey is how a stored key name may appear in reports, warnings and configs: with the parts that look like data replaced by *, or as HiddenKey when even that could carry data.

func ShowPattern added in v0.3.0

func ShowPattern(p string) string

ShowPattern checks a key or pattern for display. A pattern needs three literal letters, so * or *_* never stands for a whole key.

func SortFindings

func SortFindings(fs []model.Finding)

SortFindings orders findings by severity (worst first), then location.

func Tokens

func Tokens(name string) []string

Tokens splits an identifier into lower-case words. It understands snake_case, kebab-case, camelCase and digits, and splits a few common compounds: "firstname" becomes first, name.

func Walk added in v0.3.0

func Walk(s string, fn func(path []string, value string)) bool

Walk feeds every leaf of a JSON or PHP-serialized value to fn, with its path as JoinPath would render it in parts. Keys that may be data are already reduced to "*", "[]" or "{}". It returns false when the value is neither. Callers must never print the values.

Types

type FamilyRule added in v0.3.0

type FamilyRule struct {
	Prefix  string
	Carries model.Category // what the rest of the name holds, if personal
	Hint    *Hint
}

FamilyRule is what a preset knows about keys that share a prefix.

type Hint

type Hint struct {
	Category model.Category
	Severity model.Severity // zero means the category default
	Mask     string
	Rows     []model.RowMask // other masks for some rows of the column
	Note     string
	Source   string // e.g. "preset:woocommerce"
	Ignore   bool   // known not to hold personal data
	// Delete marks a cache the platform rebuilds, such as WordPress
	// transients: a copy should delete these rows whatever they hold. A
	// hint with Delete and no Category doesn't say what the data is, so
	// the subject is judged as if there were no hint.
	Delete bool
}

Hint is what a platform preset knows about a column or meta key.

type KeyFamily added in v0.3.0

type KeyFamily struct {
	Pattern string
	Keys    []string // sorted
	Carries model.Category
	Hint    *Hint // from a preset rule
	// contains filtered or unexported fields
}

KeyFamily is a set of keys whose names differ only in a part that looks like data, such as a user ID, a hash or an email address.

func GroupKeys added in v0.3.0

func GroupKeys(keys []string, rules []FamilyRule, exact func(string) bool) (singles []string, families []KeyFamily)

GroupKeys splits keys into single keys and families. rules come from presets; exact reports keys a preset knows by name, which stay single.

func (*KeyFamily) Known added in v0.3.0

func (f *KeyFamily) Known() bool

Known reports whether the family comes from a preset rule or a known data prefix rather than from the shape of its keys.

type KeyTemplate added in v0.3.0

type KeyTemplate struct {
	Pattern string // e.g. um_cache_userdata_* or _transient_feed_*
	// Short is Pattern with runs of 1 to 3 digits replaced too. Those
	// become * only when several keys share the result.
	Short   string
	Carries model.Category // email or IP address when the varying part is one
	// contains filtered or unexported fields
}

KeyTemplate is a key name with the parts that look like data replaced by *.

func Template added in v0.3.0

func Template(key string) KeyTemplate

Template replaces the parts of a key that look like data: email and IP addresses, UUIDs, runs of 4 or more digits, hex hashes of 8 or more characters, long tokens, and whatever follows a known data prefix.

type KeyWords added in v0.4.0

type KeyWords struct {
	// Label says the key names a field: at most one of its words is
	// unknown, and it ends in a label word or has a lower-case word the
	// rules or field names use. Date of birth and shipping method are
	// labels; Jane Doe phone, John Street and Next of kin aren't.
	Label   bool
	Known   []string // its words the rules, field names and labels use, lower case, in order
	Unknown int      // its other words, numbers aside
}

KeyWords is how the key-shape guard of a key/value table found without a preset reads a stored key with a space in it.

func ReadKey added in v0.4.0

func ReadKey(key string) KeyWords

ReadKey reads a stored key for the key-shape guard.

type Leaf added in v0.7.0

type Leaf struct {
	// Path is the leaf's path as Walk gives it: keys that may be data are
	// "*", "[]" or "{}". For a key, it is the path of the object or array
	// that holds it. Path is valid only during the call.
	Path []string
	// Key is a key's own text, for LeafKey: a JSON or PHP string key
	// decoded, or a PHP integer key in decimal. Segment is what the key adds
	// to the paths below it: the key, a template of it, "*" or "[]".
	Key, Segment string
	// DataKey marks a key that is itself a value worth checking, such as an
	// email or IP address, which Walk reports as a value at path {}.
	DataKey bool
	// Value is a string decoded, JSON escapes resolved, and other leaves as
	// written: 12.50, true, 1 for PHP's b:1;, null or N. For a key, it is
	// the key.
	Value string
	Kind  LeafKind
}

Leaf is one scalar or one key of a structured value, as EditStructured hands it out.

type LeafKind added in v0.7.0

type LeafKind int

LeafKind says what a Leaf is.

const (
	LeafString LeafKind = iota + 1
	LeafNumber
	LeafBool
	LeafNull
	LeafKey
	// LeafEnum is a PHP enum case (E:…), which can't be edited.
	LeafEnum
)

type NameMatch

type NameMatch struct {
	Category model.Category
	Words    string         // the words that matched, e.g. "first name"
	Severity model.Severity // zero means the category default
	Note     string
	// Password marks a name that says a password is stored here: values
	// that aren't hashes are then plain text or reversible encryption.
	Password bool
	// Messages marks text written between people, such as private
	// messages and notifications.
	Messages bool
}

NameMatch is what a column name, meta key or JSON key says about its data.

func MatchName

func MatchName(tokens, ctx []string, tc typeClass) *NameMatch

MatchName applies the naming rules to the words of a column name or key. ctx holds the words of the table name (or parent JSON keys); tc is the column's type class. It returns nil when the name says nothing.

type Profile

type Profile struct {
	Values   int // all values seen, including empty ones
	NonEmpty int

	Email, EmailReserved int
	Phone, PhoneStrict   int
	IP                   int
	IBAN                 int
	Card, CardTest       int
	Hash, HexDigest      int
	HashFormats          map[string]int
	UserAgent            int
	PaymentID            int
	Date                 int
	Secrets              map[string]*SecretCount
	// SecretValues counts the values with at least one secret, once each.
	SecretValues int

	// Prose: values with spaces that are longer than a few words.
	Text, TextEmail, TextPhone, TextIP, TextCard int

	// Values that cannot be names or addresses: plain numbers, and code
	// identifiers such as App\Jobs\SendInvoice.
	Numeric, CodeLike int

	// Values that say a name match is wrong: on/off flags, field names
	// (billing_address_1, edit-address), placeholders and ACF field keys,
	// and the field's own name (as in translation tables and import
	// mappings).
	Flag, FieldName, Placeholder, Echo int
	// MaxDigits is the most digits any one value holds: an ID number or a
	// birth date has at least four.
	MaxDigits int

	// Values that back or rule out other judgements: hardware IDs
	// (Opaque), push notification tokens (Push), email addresses inside
	// URLs (Inline), URLs and paths (URL), cryptocurrency addresses
	// (Crypto), text with spaces and no merge tags (Spaced), and whole
	// numbers (Whole).
	Opaque, Push, Inline, URL, Crypto, Spaced, Whole int

	Structured int
	Paths      map[string]*Profile
	// contains filtered or unexported fields
}

Profile counts what the sampled values of one column, meta key or JSON path look like. It keeps counts only, never the values.

func NewProfile

func NewProfile() *Profile

NewProfile returns an empty profile. Values that are JSON or PHP-serialized are opened up and counted per path.

func NewProfileNamed added in v0.3.0

func NewProfileNamed(name string) *Profile

NewProfileNamed returns an empty profile for a field with the given name; values that only repeat the name, such as "First name" under first_name, count as field names.

func (*Profile) Add

func (p *Profile) Add(v string)

Add counts one sampled value. An empty array or object, or one whose leaves are all empty, counts as an empty value: WordPress keeps recovery_keys as a:0:{} on nearly every site.

func (*Profile) PathNames

func (p *Profile) PathNames() []string

PathNames returns the tracked paths in a stable order.

func (*Profile) SecretKinds

func (p *Profile) SecretKinds() []string

SecretKinds lists the kinds of secrets seen, sorted.

func (*Profile) SecretTotals

func (p *Profile) SecretTotals() (live, test, unknown int)

SecretTotals sums the secret counts over all kinds.

func (*Profile) WithContext added in v0.4.0

func (p *Profile) WithContext(words []string) *Profile

WithContext records the words of the table a field belongs to, so paths past the path cap are named in the same context as the rest.

type SecretCount

type SecretCount struct {
	Live, Test, Unknown int
}

SecretCount counts values holding one kind of secret.

type SecretHit

type SecretHit struct {
	Kind string
	Mode string
}

SecretHit is one secret-looking string. Mode is "live", "test" or "" when the format does not say.

func FindSecrets

func FindSecrets(s string) []SecretHit

FindSecrets returns the kinds of secrets that appear in s, once per kind, each with the strongest mode among its matches: live, then unknown, then test.

type Span added in v0.7.0

type Span struct {
	Start, End int
	Kind       string // one of the Span* kinds, or a caller's own
	// Detail says more: a secret's or a password hash's kind, "strict" or
	// "loose" for a phone number, "test" for a published test card number,
	// "reserved" for an email address at a domain reserved for examples, and
	// "placeholder" for a password in a URL or a Basic credential that reads
	// as an example, or a Basic credential with no password and no key.
	Detail string
	Mode   string // for secrets: live, test or "" when the format does not say
}

Span is one match inside a value, by its byte offsets in the string given. A match found through an escape, such as JSON's \n or \u00e9, or the %40 of a URL, covers the whole escape. Callers must never print the text a span covers.

func FindSpans added in v0.7.0

func FindSpans(s string, finders []SpanFinder) []Span

FindSpans runs the finders on s and returns their spans sorted by Start, no two overlapping: where spans overlap, the earlier finder's wins, then the longer one. Spans outside s, or empty, are dropped.

The result is final. A caller that keeps only some of a finder's spans, such as loose phone numbers that match a known number, must filter them inside that finder, with SpanFinder.Keep, so the spans it would drop never hide another finder's.

type SpanFinder added in v0.7.0

type SpanFinder struct {
	Kind string
	// Find finds spans in a value.
	Find func(s string) []Span
	// FindView, when set, finds spans in the value's TextView, by offsets of
	// its Text. FindSpans prefers it, and builds the view once for every
	// finder that has one. A copy of a built-in finder that is given a Find
	// of its own must have its FindView set to nil, or wrapped too, as Keep
	// wraps both.
	FindView func(v *TextView) []Span
}

SpanFinder finds the spans of one kind in a value.

func SpanFinders added in v0.7.0

func SpanFinders() []SpanFinder

SpanFinders returns the built-in finders in priority order: secrets, passwords in URLs and password hashes; then email addresses, IBANs, card numbers, IP addresses and phone numbers whose layout says phone; and last loose phone numbers, so that a caller can filter them in their finder without their spans hiding others. They use the scan's own patterns and checks, but report every match, where the scan stops counting at 8 per pattern.

func (SpanFinder) Keep added in v0.7.0

func (f SpanFinder) Keep(keep func(text string, sp Span) bool) SpanFinder

Keep returns the finder with only the spans keep accepts, whichever of Find and FindView FindSpans calls. keep gets the text the finder read, the value or its view's Text, and the span by offsets of that text.

type Subject

type Subject struct {
	Table    string
	Column   string
	MetaKey  string
	Name     string   // the name to judge: the column name or the meta key
	DataType string   // information_schema DATA_TYPE; empty for meta keys
	Context  []string // words of the table name
	Hint     *Hint
	Sampled  bool // false when values were not read (--no-samples)

	Keys        int    // keys a pattern stands for, copied to the findings
	KeysPartial bool   // Keys is a lower bound
	Where       string // the subject is these rows only, copied to the findings
	// Setting marks a key of a settings table, which holds one value; an
	// empty value then means nothing is stored.
	Setting bool
	// Skipped marks a key left without values by a sampling limit; name
	// matches on it stay weak.
	Skipped bool
	// Labels marks a key of a translation table: it names a UI string, not
	// a field, so only its values are judged.
	Labels bool
}

Subject is one column, or one key of a key/value table, to judge.

type TextView added in v0.7.0

type TextView struct {
	// Text is the value's text, as finders read it. Like the value, it
	// holds the data: never print it.
	Text string
	// contains filtered or unexported fields
}

TextView is a value with its escapes read as what they stand for, so finders see text as an application would: JSON's \" \\ \/ \b \f \n \r \t and \uXXXX, surrogate pairs included, and %40 as @. A backslash that starts no such escape is itself. Span maps a span of Text back to the value, widened to whole escapes.

func NewTextView added in v0.7.0

func NewTextView(s string) *TextView

NewTextView reads a value's escapes.

func (*TextView) Span added in v0.7.0

func (v *TextView) Span(sp Span) Span

Span maps a span of Text to the value.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL