detect

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 17, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package detect decides which columns, meta keys and JSON paths hold personal data or secrets, from their names and from sampled values. It never keeps or returns the values themselves, only counts.

Index

Constants

View Source
const HiddenKey = "(key hidden)"

HiddenKey stands for a key name that can't be shown.

View Source
const TypeString = tStr

TypeString is the type class of text columns and meta keys, for MatchName.

Variables

This section is empty.

Functions

func Analyze

func Analyze(sub Subject, p *Profile) []model.Finding

Analyze judges one subject from its name and the profile of its sampled values. p is nil when no values were read.

func DataPrefixes added in v0.3.0

func DataPrefixes() []string

DataPrefixes lists the known prefixes of key names whose rest is data.

func Flatten

func Flatten(s string) map[string]string

Flatten returns the leaves of a JSON or PHP-serialized value by path, keeping the first value of each path. It is for reading settings, such as whether a payment gateway runs in live mode; callers must never print the values.

func HasEmail added in v0.3.0

func HasEmail(s string) bool

HasEmail reports whether a value is, or contains, an email address. It is for reading settings; callers must never print the value.

func IsEmail

func IsEmail(s string) bool

IsEmail reports whether s is one email address.

func JoinPath

func JoinPath(path []string) string

JoinPath renders a path: billing.email, items[].name, cart.*.line_total, lockouts{} (the keys of a map keyed by data).

func KeyNameFinding added in v0.3.0

func KeyNameFinding(table, keyColumn, shown string, fam *KeyFamily, partial bool) model.Finding

KeyNameFinding reports that the names of a family's keys hold personal data, so a copy has to rewrite the key column as well as the values.

func ShowKey added in v0.3.0

func ShowKey(key string) string

ShowKey is how a stored key name may appear in reports, warnings and configs: with the parts that look like data replaced by *, or as HiddenKey when even that could carry data.

func ShowPattern added in v0.3.0

func ShowPattern(p string) string

ShowPattern checks a key or pattern for display. A pattern needs three literal letters, so * or *_* never stands for a whole key.

func SortFindings

func SortFindings(fs []model.Finding)

SortFindings orders findings by severity (worst first), then location.

func Tokens

func Tokens(name string) []string

Tokens splits an identifier into lower-case words. It understands snake_case, kebab-case, camelCase and digits, and splits a few common compounds: "firstname" becomes first, name.

func Walk added in v0.3.0

func Walk(s string, fn func(path []string, value string)) bool

Walk feeds every leaf of a JSON or PHP-serialized value to fn, with its path as JoinPath would render it in parts. Keys that may be data are already reduced to "*", "[]" or "{}". It returns false when the value is neither. Callers must never print the values.

Types

type FamilyRule added in v0.3.0

type FamilyRule struct {
	Prefix  string
	Carries model.Category // what the rest of the name holds, if personal
	Hint    *Hint
}

FamilyRule is what a preset knows about keys that share a prefix.

type Hint

type Hint struct {
	Category model.Category
	Severity model.Severity // zero means the category default
	Mask     string
	Rows     []model.RowMask // other masks for some rows of the column
	Note     string
	Source   string // e.g. "preset:woocommerce"
	Ignore   bool   // known not to hold personal data
	// Delete marks a cache the platform rebuilds, such as WordPress
	// transients: a copy should delete these rows whatever they hold. A
	// hint with Delete and no Category doesn't say what the data is, so
	// the subject is judged as if there were no hint.
	Delete bool
}

Hint is what a platform preset knows about a column or meta key.

type KeyFamily added in v0.3.0

type KeyFamily struct {
	Pattern string
	Keys    []string // sorted
	Carries model.Category
	Hint    *Hint // from a preset rule
	// contains filtered or unexported fields
}

KeyFamily is a set of keys whose names differ only in a part that looks like data, such as a user ID, a hash or an email address.

func GroupKeys added in v0.3.0

func GroupKeys(keys []string, rules []FamilyRule, exact func(string) bool) (singles []string, families []KeyFamily)

GroupKeys splits keys into single keys and families. rules come from presets; exact reports keys a preset knows by name, which stay single.

func (*KeyFamily) Known added in v0.3.0

func (f *KeyFamily) Known() bool

Known reports whether the family comes from a preset rule or a known data prefix rather than from the shape of its keys.

type KeyTemplate added in v0.3.0

type KeyTemplate struct {
	Pattern string // e.g. um_cache_userdata_* or _transient_feed_*
	// Short is Pattern with runs of 1 to 3 digits replaced too. Those
	// become * only when several keys share the result.
	Short   string
	Carries model.Category // email or IP address when the varying part is one
	// contains filtered or unexported fields
}

KeyTemplate is a key name with the parts that look like data replaced by *.

func Template added in v0.3.0

func Template(key string) KeyTemplate

Template replaces the parts of a key that look like data: email and IP addresses, UUIDs, runs of 4 or more digits, hex hashes of 8 or more characters, long tokens, and whatever follows a known data prefix.

type NameMatch

type NameMatch struct {
	Category model.Category
	Words    string         // the words that matched, e.g. "first name"
	Severity model.Severity // zero means the category default
	Note     string
}

NameMatch is what a column name, meta key or JSON key says about its data.

func MatchName

func MatchName(tokens, ctx []string, tc typeClass) *NameMatch

MatchName applies the naming rules to the words of a column name or key. ctx holds the words of the table name (or parent JSON keys); tc is the column's type class. It returns nil when the name says nothing.

type Profile

type Profile struct {
	Values   int // all values seen, including empty ones
	NonEmpty int

	Email, EmailReserved int
	Phone, PhoneStrict   int
	IP                   int
	IBAN                 int
	Card, CardTest       int
	Hash, HexDigest      int
	HashFormats          map[string]int
	UserAgent            int
	PaymentID            int
	Date                 int
	Secrets              map[string]*SecretCount

	// Prose: values with spaces that are longer than a few words.
	Text, TextEmail, TextPhone, TextIP, TextCard int

	// Values that cannot be names or addresses: plain numbers, and code
	// identifiers such as App\Jobs\SendInvoice.
	Numeric, CodeLike int

	// Values that say a name match is wrong: on/off flags, field names
	// (billing_address_1, edit-address), placeholders and ACF field keys,
	// and the field's own name (as in translation tables and import
	// mappings).
	Flag, FieldName, Placeholder, Echo int
	// MaxDigits is the most digits any one value holds: an ID number or a
	// birth date has at least four.
	MaxDigits int

	Structured int
	Paths      map[string]*Profile
	// contains filtered or unexported fields
}

Profile counts what the sampled values of one column, meta key or JSON path look like. It keeps counts only, never the values.

func NewProfile

func NewProfile() *Profile

NewProfile returns an empty profile. Values that are JSON or PHP-serialized are opened up and counted per path.

func NewProfileNamed added in v0.3.0

func NewProfileNamed(name string) *Profile

NewProfileNamed returns an empty profile for a field with the given name; values that only repeat the name, such as "First name" under first_name, count as field names.

func (*Profile) Add

func (p *Profile) Add(v string)

Add counts one sampled value. An empty array or object, or one whose leaves are all empty, counts as an empty value: WordPress keeps recovery_keys as a:0:{} on nearly every site.

func (*Profile) PathNames

func (p *Profile) PathNames() []string

PathNames returns the tracked paths in a stable order.

func (*Profile) SecretKinds

func (p *Profile) SecretKinds() []string

SecretKinds lists the kinds of secrets seen, sorted.

func (*Profile) SecretTotals

func (p *Profile) SecretTotals() (live, test, unknown int)

SecretTotals sums the secret counts over all kinds.

type SecretCount

type SecretCount struct {
	Live, Test, Unknown int
}

SecretCount counts values holding one kind of secret.

type SecretHit

type SecretHit struct {
	Kind string
	Mode string
}

SecretHit is one secret-looking string. Mode is "live", "test" or "" when the format does not say.

func FindSecrets

func FindSecrets(s string) []SecretHit

FindSecrets returns the kinds of secrets that appear in s, once per kind.

type Subject

type Subject struct {
	Table    string
	Column   string
	MetaKey  string
	Name     string   // the name to judge: the column name or the meta key
	DataType string   // information_schema DATA_TYPE; empty for meta keys
	Context  []string // words of the table name
	Hint     *Hint
	Sampled  bool // false when values were not read (--no-samples)

	Keys        int    // keys a pattern stands for, copied to the findings
	KeysPartial bool   // Keys is a lower bound
	Where       string // the subject is these rows only, copied to the findings
	// Setting marks a key of a settings table, which holds one value; an
	// empty value then means nothing is stored.
	Setting bool
	// Skipped marks a key left without values by a sampling limit; name
	// matches on it stay weak.
	Skipped bool
}

Subject is one column, or one key of a key/value table, to judge.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL