Documentation
¶
Overview ¶
Package pii is the product-neutral half of personal-data masking: value detectors (email, phone, IBAN, card, national ids), mask modes, the multilingual name heuristics that propose mask rules, and the store of the clear values behind redacted references. Which value is personal (column provenance, JSON paths, headers) is the product's.
Index ¶
- Constants
- Variables
- func CellText(v any) string
- func CutAt(s string, n int) string
- func Detect(v any, ds []Detector) any
- func Detects(s string, ds []Detector) bool
- func LooksPersonal(name string) bool
- func LooksPersonalColumn(name, typ string) bool
- func LooksQuasi(name string) bool
- func LooksSecret(name string) bool
- func Mask(mode, value string) string
- func MaskText(s string) string
- func MaskValue(v any, mode string) any
- func Normalise(kind, value string) string
- func RedactedRef(name string) string
- func Tokens(name string) []string
- func ValidID(s string) bool
- func ValidKind(kind string) bool
- func ValidMode(m string) bool
- type Detector
- type EpochStore
- func (s *EpochStore) Close() error
- func (s *EpochStore) Current() *Pseudonymizer
- func (s *EpochStore) Epochs() []string
- func (s *EpochStore) Get(epoch string) (*Pseudonymizer, error)
- func (s *EpochStore) GoString() string
- func (s *EpochStore) MarshalJSON() ([]byte, error)
- func (s *EpochStore) Prune(now time.Time, keep time.Duration) ([]string, error)
- func (s *EpochStore) Rotate(now time.Time) (*Pseudonymizer, error)
- func (s *EpochStore) RotationDue(now time.Time, every time.Duration) bool
- func (s *EpochStore) String() string
- type Match
- type Pseudonymizer
- func (p *Pseudonymizer) Created() time.Time
- func (p *Pseudonymizer) Epoch() string
- func (p *Pseudonymizer) Forgotten() bool
- func (p *Pseudonymizer) GoString() string
- func (p *Pseudonymizer) ID(kind, value string) (string, error)
- func (p *Pseudonymizer) MarshalJSON() ([]byte, error)
- func (p *Pseudonymizer) String() string
- type RefStore
- type View
Constants ¶
const ( // ModeRedact replaces the whole value: "<redacted>" (the default). ModeRedact = "redact" // ModePartial keeps the first character and the length: "j***(12)". ModePartial = "partial" // ModeEmail keeps the first character and the domain of an address: // "j***@example.com" (other values are masked as partial). ModeEmail = "email" )
Mask modes.
const ( MaxRefResults = 50 MaxRefBytes = 16 << 20 )
Bounds of the reference store: the oldest results are forgotten first.
const ClockSkew = time.Hour
ClockSkew is how far in the future an epoch's creation time may be before the store treats it as wrong (a clock that was ahead).
const CutKind = "cut"
CutKind is the kind of the match that covers the end of a text past DetectLimit (or past where a view reached its cap), masked whole.
const CutMark = "…"
CutMark ends a text value that was cut.
const DetectLimit = 64 << 10
DetectLimit is how much of a text value the detectors scan. A longer value is cut there, and ends with CutMark, before detection: nothing past the scanned part can reach the output. Renderers cut values much earlier, so the cut is normally invisible; it bounds the time a huge value can cost.
const EpochLen = 8
EpochLen is the length of an epoch id: 8 characters of Crockford base32, 40 random bits.
const KeyLen = 32
KeyLen is the length of an epoch key.
const PseudonymLen = 12
PseudonymLen is the length of a pseudonym ID: 12 characters of Crockford base32, 60 bits.
const Redacted = "<redacted>"
Redacted is a value masked in mode redact.
Variables ¶
var ( // ErrNoEpoch: the store holds no epoch and was not asked to create one. ErrNoEpoch = errors.New("pii: no pseudonym epoch key") // ErrClosed: Rotate or Prune on a closed store (it no longer holds the // directory's lock). ErrClosed = errors.New("pii: epoch store closed") // ErrUnknownEpoch: the epoch was never kept here, or was pruned. ErrUnknownEpoch = errors.New("pii: unknown or deleted pseudonym epoch") )
Errors of the epoch store.
var ErrForgottenEpoch = errors.New("pii: pseudonym epoch key deleted")
ErrForgottenEpoch is the error of ID on an epoch whose key was deleted (EpochStore.Prune): the caller must redact the value, never show it.
var ErrInvalidKind = errors.New("pii: invalid pseudonym kind")
ErrInvalidKind is the error of ID on a kind that is not 1 to 32 of [a-z0-9_].
Functions ¶
func CutAt ¶
CutAt cuts s, which must be longer than n bytes, to at most n bytes followed by CutMark: at a rune boundary, then back over up to cutTail value-like characters (valueRune), so that a value the cut splits (a partial e-mail address or number, which no detector would recognise) is dropped rather than shown in part.
func Detect ¶
Detect runs the detectors over a text or integer value (a national number or a card can be stored as one), or an integral float (the same number in a double column), and returns the masked text, or v itself, with its type, when nothing was found. Other values pass.
func LooksPersonal ¶
LooksPersonal reports a field name that looks like personal data.
func LooksPersonalColumn ¶
LooksPersonalColumn is LooksPersonal for a database column of type typ: a network address type is personal whatever its name, and a binary column whose name suggests a document is too.
func LooksQuasi ¶
LooksQuasi reports a field name that looks like a quasi-identifier (birth date, postal code, gender, coordinates, free-text notes).
func LooksSecret ¶ added in v0.8.0
LooksSecret reports a field, label, header or flag name that holds a credential: a password, a token, an API or private key, a cookie, an authorization. Its value is a secret whatever it looks like. Metadata about a credential ("token_type", "password_changed_at") does not count, nor do token counts ("max_tokens", "prompt_tokens"), nor the shell's working directory ("PWD").
func MaskText ¶
MaskText is the partial mask format: first character + "***" + "(length in characters)". An invalid UTF-8 byte counts as one U+FFFD character.
func MaskValue ¶
MaskValue masks one non-nil value in mode. Bytes become "<masked bytes:N>" in modes partial and email; an unknown mode masks as partial.
func Normalise ¶ added in v0.8.0
Normalise returns the form of value (one detected value: digits of two phone numbers would be joined) whose pseudonym is taken, so that one value written differently is one pseudonym: the detection view of the value, or of its decoding (percent, JSON escape, HTML reference) when only that is a whole value of kind; then, by kind:
- email: lower case;
- phone, phone_loose: digits only, with "+" for an international number ("00" becomes "+": E.164 when the country code is written);
- iban: blanks removed, upper case;
- card, be_niss, nl_bsn, us_ssn: digits only; fr_nir: digits and the Corsican letter, upper case;
- ip: the canonical address (an IPv4-mapped IPv6 address is the IPv4 one; zones dropped);
- person_name: lower case, blanks collapsed to one space;
- other kinds: the detection view with surrounding blanks trimmed.
func RedactedRef ¶
RedactedRef is a redacted value that carries a reference to it, which the console alone resolves (see RefStore).
func Tokens ¶
Tokens splits a name on non-alphanumerics and camelCase, lowered, with accents removed from common Latin letters ("prénom" → "prenom").
Types ¶
type Detector ¶
type Detector interface {
Name() string
// Mask returns s with every value the detector recognises replaced by
// its masked form (first character + "***" + "(length)").
Mask(s string) string
}
Detector masks personal values found inside a text, in place.
func AllDetectors ¶ added in v0.3.0
func AllDetectors() []Detector
AllDetectors is every detector but the opt-in ones (phone_loose, ip, secret), in their run order: what masking falls back to when it was given none.
type EpochStore ¶ added in v0.8.0
type EpochStore struct {
// contains filtered or unexported fields
}
EpochStore keeps pseudonym epochs in a directory of their own: one file per epoch ("<epoch>.key", 0600, written once, atomically), the directory 0700, both owned by the process's user. The current epoch is the newest; an epoch is replaced when a newer one is created, and its key is deleted by Prune once it has been replaced for long enough (references minted under it can be re-read until then). A corrupt or exposed key file refuses the store: nothing is repaired silently. Epochs are ordered by a sequence number, not by the clock. The store locks its directory (one process at a time; Close releases it; opening creates the .lock file even when create is false) and is safe for concurrent use within that process. On non-unix platforms there are no permission, owner, symbolic-link or lock checks: keep one process per store there.
func OpenEpochStore ¶ added in v0.8.0
OpenEpochStore opens the epochs kept in dir. When dir holds none, it creates the directory (0700) and a first epoch if create is set, and fails with ErrNoEpoch otherwise.
func (*EpochStore) Close ¶ added in v0.8.0
func (s *EpochStore) Close() error
Close releases the directory's lock and forgets every epoch's key in memory (another process may now prune them): ID then fails with ErrForgottenEpoch, Rotate and Prune with ErrClosed.
func (*EpochStore) Current ¶ added in v0.8.0
func (s *EpochStore) Current() *Pseudonymizer
Current returns the current epoch.
func (*EpochStore) Epochs ¶ added in v0.8.0
func (s *EpochStore) Epochs() []string
Epochs returns the ids of the kept epochs, oldest first.
func (*EpochStore) Get ¶ added in v0.8.0
func (s *EpochStore) Get(epoch string) (*Pseudonymizer, error)
Get returns a kept epoch, current or replaced but not yet pruned.
func (*EpochStore) GoString ¶ added in v0.8.0
func (s *EpochStore) GoString() string
GoString shows the directory and the epochs, never a key.
func (*EpochStore) MarshalJSON ¶ added in v0.8.0
func (s *EpochStore) MarshalJSON() ([]byte, error)
MarshalJSON writes the epochs, never a key.
func (*EpochStore) Prune ¶ added in v0.8.0
Prune deletes the keys of the epochs replaced at least keep ago (an epoch is replaced when the next one is created) and returns their ids. The current epoch is never deleted.
func (*EpochStore) Rotate ¶ added in v0.8.0
func (s *EpochStore) Rotate(now time.Time) (*Pseudonymizer, error)
Rotate creates a new epoch, which becomes current. The previous one is kept until Prune deletes it.
func (*EpochStore) RotationDue ¶ added in v0.8.0
RotationDue reports whether the current epoch began at least every ago, or claims to begin more than ClockSkew in the future (a clock that was ahead must not keep an epoch alive).
func (*EpochStore) String ¶ added in v0.8.0
func (s *EpochStore) String() string
String shows the directory and the epochs, never a key.
type Match ¶ added in v0.8.0
Match is a value a detector recognised: the byte range [Start, End) of the text it was found in, and the name of the detector (its kind: "email", "iban", ...).
func Find ¶ added in v0.8.0
Find returns the values that Detect(s, ds) masks, as byte ranges of s: sorted, non-overlapping, each with the kind of the detector that found it. The detectors run in the order given (Detectors returns their fixed run order), each on the text the previous ones masked, exactly as Detect runs them, so replacing every match with MaskText of its text gives Detect's output. Two exceptions, where Find masks more: a later detector whose match overlaps an earlier mask makes one match of both (Detect would mask the mask), and a Detector that is not one of this package's cannot say where it found a value, so anything it masks makes the whole text one match of its kind.
Past DetectLimit, the rest of s is one more match, of kind CutKind, starting earlier where a value runs across the limit (the detectors scan a little past it to find such values whole): the matches are always safe to apply to s itself.
func FindInViews ¶ added in v0.8.0
FindInViews returns the values found in s and in each of its detection views (DetectionViews), as byte ranges of s: sorted, non-overlapping. A value found only in a view (behind an escape, a look-alike or a zero-width character) is the range of s it was written with, whole. Where matches of different kinds overlap, they make one match, of the kind of the earliest detector in ds (an encoded value next to a plain one may so take a neighbour's kind). Long texts are cut as by Find. Match boundaries are not moved to escape boundaries: a raw match may start inside an escape (the "n" of "\\n" before an address).
type Pseudonymizer ¶ added in v0.8.0
type Pseudonymizer struct {
// contains filtered or unexported fields
}
Pseudonymizer gives each personal value a stable, keyed pseudonym for the duration of one epoch: the same value, however it is spelt (Normalise), gets the same ID; without the key, an ID says nothing about the value beyond what guessing a value and asking for its pseudonym would. The key never leaves the value: there is no accessor, and the fmt and JSON forms show the epoch only. It is safe for concurrent use.
func NewEpoch ¶ added in v0.8.0
func NewEpoch(now time.Time) (*Pseudonymizer, error)
NewEpoch returns a Pseudonymizer for a new epoch, with a random id and key, kept in memory only (a session epoch). EpochStore keeps them on disk.
func NewPseudonymizer ¶ added in v0.8.0
NewPseudonymizer returns the Pseudonymizer of an existing epoch: its id (EpochLen characters of Crockford base32) and its KeyLen-byte key, which it copies.
func (*Pseudonymizer) Created ¶ added in v0.8.0
func (p *Pseudonymizer) Created() time.Time
Created returns when the epoch began (to the second).
func (*Pseudonymizer) Epoch ¶ added in v0.8.0
func (p *Pseudonymizer) Epoch() string
Epoch returns the epoch id: random, it says nothing about the key.
func (*Pseudonymizer) Forgotten ¶ added in v0.8.0
func (p *Pseudonymizer) Forgotten() bool
Forgotten reports whether the epoch's key was deleted.
func (*Pseudonymizer) GoString ¶ added in v0.8.0
func (p *Pseudonymizer) GoString() string
GoString shows the epoch, never the key.
func (*Pseudonymizer) ID ¶ added in v0.8.0
func (p *Pseudonymizer) ID(kind, value string) (string, error)
ID returns the pseudonym of value as a value of kind (a detector name: "email", "phone", ...): the first PseudonymLen characters of the Crockford base32 of HMAC-SHA256(key, "lockkit-pseudonym-v1" || 0x00 || kind || 0x00 || Normalise(kind, value)); kind is 1 to 32 of [a-z0-9_] (ErrInvalidKind otherwise). Different kinds give different IDs for the same text. Once the epoch's key was deleted (EpochStore.Prune), ID fails with ErrForgottenEpoch and never computes an ID: the caller redacts the value.
func (*Pseudonymizer) MarshalJSON ¶ added in v0.8.0
func (p *Pseudonymizer) MarshalJSON() ([]byte, error)
MarshalJSON writes the epoch, never the key.
func (*Pseudonymizer) String ¶ added in v0.8.0
func (p *Pseudonymizer) String() string
String shows the epoch, never the key.
type RefStore ¶
type RefStore struct {
// contains filtered or unexported fields
}
RefStore keeps the clear values behind the redacted references of the last results, so that an agent can name a value ('${rN.K}') without seeing it. It lives in the console's memory and dies with it. The zero value is ready; it is not safe for concurrent use.
A reference is "r<result>.<key>", where key is positive integers separated by dots chosen by the product: locksql uses "<row>.<column>", PutDistinct numbers the distinct values of a result. A key never holds the value.
type View ¶ added in v0.8.0
type View struct {
Text string
// contains filtered or unexported fields
}
View is a text derived from a source text (normalised, decoded) with the map back to it: Source gives the range of the source each range of Text was written with. The zero map means Text is the source itself.
func DetectionView ¶ added in v0.8.0
DetectionView is s as the detectors should see it: the characters fold knows mapped to ASCII (the spaces digits are grouped with, full-width digits), NFKC-normalised (full-width letters, ligatures, other compatibility look-alikes), and without default-ignorable characters (Unicode Cf: zero-width spaces and joiners, bidi controls, the soft hyphen; variation selectors, the combining grapheme joiner, Hangul fillers). An invalid UTF-8 byte is kept as is. Texts longer than DetectLimit are cut as by Find.
func DetectionViews ¶ added in v0.8.0
DetectionViews returns the detection view of s, the views without its terminal sequences, then the detection view of each decoding of s that changes it: percent-decoded once and twice, JSON string escapes, HTML character references, all of them chained (until nothing changes, at most 16 rounds, cleaned up between rounds), \\xHH, \\e and \\033 escapes, and form encoding ("+" as a space) then percent. Texts are distinct. Texts longer than DetectLimit are cut as by Find.
func (View) Source ¶ added in v0.8.0
Source returns the range of the source text that Text[start:end] was written with. It covers every source character dropped inside the range (a zero-width character between two digits) and every escape it decodes whole ("%40", "@", "\u0040"). Out-of-range arguments are clamped.
Source Files