Documentation
¶
Overview ¶
Package pii decides which result cells must be masked: column rules kept in .locksql/pii.toml, rules proposed from the schema, value detectors (email, phone, IBAN, card, national ids) and the masking itself.
Index ¶
- Constants
- func AliasViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect) error
- func MaskOutputs(res *engine.Result, outs []sqlast.Output, r Rules, ds []Detector, tok *Tokens, ...) error
- func MaskResult(res *engine.Result, r Rules, ds []Detector, origin bool)
- func MaskValue(v any, mode string, tok *Tokens) any
- func NeedsAliasCheck(res engine.Result, origin bool) bool
- func PlanCheck(st sqlclass.Statement, r Rules, d sqlclass.Dialect, origin bool) error
- func Propose(cols []engine.ColumnInfo) []string
- func RedactMessage(msg, sql string, ds []Detector) string
- func ResultAliasViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect, cols []engine.ResultColumn) error
- func SaveRules(root string, r Rules) error
- func StatsViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect) error
- func ValidMode(m string) bool
- type Detector
- type Rules
- func (r *Rules) Add(pattern string) error
- func (r *Rules) AddMode(pattern, mode string) error
- func (r Rules) Covered(db, table, column string) bool
- func (r Rules) Matches(db, table, column string) bool
- func (r Rules) MatchesName(column string) bool
- func (r Rules) Mode(db, table, column string) (string, bool)
- func (r Rules) ModeByName(column string) (string, bool)
- type Tokens
Constants ¶
const ( // ModeRedact replaces the whole value: "<redacted>". ModeRedact = "redact" // ModePartial keeps the first character and the length: "j***(12)". ModePartial = "partial" // ModeEmail keeps the first character and the domain of an address: // "j***@example.com" (other values are masked as partial). ModeEmail = "email" // ModeHash replaces the value with a token, the same for the same value // within one console session: "tok_...". Joins, grouping, counting and // equality filters on tokens keep working; the value never leaves the // console. The token key changes with every session. ModeHash = "hash" )
Mask modes.
const DetectLimit = 64 << 10
DetectLimit is how much of a text cell the detectors scan. A longer cell is cut there, and ends with "…", before detection: nothing past the scanned part can reach the output. The renderer cuts cells much earlier (max_cell_chars), so the cut is normally invisible; it bounds the time a huge value can cost.
const Redacted = "<redacted>"
Redacted is the cell of a column masked in mode redact.
const RulesFile = ".locksql/pii.toml"
RulesFile is the rules file, relative to the project root.
Variables ¶
This section is empty.
Functions ¶
func AliasViolation ¶
AliasViolation is the fallback for result columns without an origin: it refuses a READ select in which a rule-matched column (matched by name, see Rules.MatchesName) is renamed or used inside an expression anywhere in a select list, since its values would then reach a label that no rule matches. Renaming covers "AS x", an implicit alias, a later UNION arm whose first arm labels the position differently, CTE column lists over a body that mentions a matched column or selects * from a source that may hold one, a * in a later arm over such a source, and table alias column lists ("t(x, y)"), which can rename any column. A PostgreSQL whole-row reference to a table that may hold such a column ("SELECT u", "row_to_json(u)", "to_jsonb(users.*)") is refused too, see wholeRow. A matched column or whole-row reference inside a VALUES list or the arguments of a FROM-clause function, and a TABLE arm after the first, are refused as well. Every named relation counts as a source that may hold a matched column, since it may be a view over a rule table. COUNT(...) is allowed. The error is a *sqlclass.Refusal that explains how to write the query.
A statement that is not a plain READ select is checked the same way when it may return rows: a RETURNING list counts as a select list, and a statement led by SELECT, WITH, VALUES or TABLE that the classifier made WRITE (a data-modifying CTE) is checked whole. A write without RETURNING is only checked by writtenValues. Other statements (SHOW, DESCRIBE, PRAGMA, DDL) pass.
On PostgreSQL, "rel.f" may be the call f(rel) on the whole row of rel when rel has no column f (attribute notation: b.row_to_json, t.name, big.record_out). Since a statement alone cannot tell such a call from a column, a qualified reference "rel.x" to a FROM-clause relation is refused wherever its value may reach the result (select lists, RETURNING, VALUES, FROM-clause function arguments), see attrCall. ResultAliasViolation accepts a plain qualified item of the outer select list whose result column carries an origin, which proves it is a base-table column.
A write, with or without RETURNING, must not copy a rule-matched column's values into another column: a rule-matched column, a whole row or a star over a source that may hold one is refused anywhere its value may be written (SET values, the select list or VALUES of an INSERT source), see writtenValues.
func MaskOutputs ¶
func MaskOutputs(res *engine.Result, outs []sqlast.Output, r Rules, ds []Detector, tok *Tokens, origin bool) error
MaskOutputs masks res in place following the analysed outputs of its statement: each column whose resolved sources are under a rule is masked in the output's mode. As a second line of defence a column whose engine- reported origin matches a rule is masked too. The other text and integer cells go through the detectors.
The result must have exactly the analysed columns, and the columns the analysis expects by name (plain references, aliases, star expansions) must carry that label: otherwise masking by position could hit the wrong column, and the result is refused.
func MaskResult ¶
MaskResult masks res in place.
A column is masked whole when its origin matches a rule. Engines report an origin only when the catalog confirms it is a base-table column; a view, a derived table or a CTE alias never counts. A view column is therefore masked only by its own name: a view that renames a rule column (CREATE VIEW v AS SELECT firstname AS contact ...) needs a rule for that name (*.*.contact or db.v.contact). When origin is false (the session reports no origins), or a column has none (an expression, a UNION, an untrusted origin the engine blanked), the column label is matched by name instead, see Rules.MatchesName; the console must then also refuse renamed uses with AliasViolation (see NeedsAliasCheck). Binary cells under a rule become "<masked bytes:N>"; NULL stays NULL.
Every other text cell, and every integer (a national number or a card can be stored as one), goes through the detectors in place.
func NeedsAliasCheck ¶
NeedsAliasCheck reports whether the statement behind res must pass AliasViolation (ResultAliasViolation, given res.Columns) before its rows may be shown: the session reports no origins, or some result column came back without one.
func PlanCheck ¶
PlanCheck is the PII check a statement must pass before it runs, unless it is an unmask run: StatsViolation, then AliasViolation for a statement that is not a plain READ select (it runs before the write, whose result may carry no origin) or when the session reports no origins. A plain READ select on an origin-reporting session is checked after it runs, only when some result column has no origin (see NeedsAliasCheck).
func Propose ¶
func Propose(cols []engine.ColumnInfo) []string
Propose returns the "db.table.column" patterns for the columns that look like personal data from their name or type, sorted and de-duplicated. A name that would break the pattern syntax becomes '*'.
func RedactMessage ¶
RedactMessage removes from a database error message the row values it may quote: a failed cast or function call in a WHERE clause echoes the value it choked on ("invalid input syntax for type integer: \"Smith\""), and that value reaches no result row, so no rule or detector masks it.
Everything from the first quote character to the last one is replaced, unless each quoted part is text of the statement itself (a syntax error, an unknown column) or a known server phrase, joined only by known server words: a value cannot shift the pairing of quotes, since its text would then sit in a gap. A single stray quote redacts the rest of the message. The detectors then run over what is left; an unquoted value they cannot recognise is not caught.
func ResultAliasViolation ¶
func ResultAliasViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect, cols []engine.ResultColumn) error
ResultAliasViolation is AliasViolation once the statement has run, with the result columns it returned: a plain qualified PostgreSQL item of the outer select list ("SELECT b.id, ...") is accepted when its result column has an origin. With cols nil, every such item is refused.
func SaveRules ¶
SaveRules writes the rules to <root>/.locksql/pii.toml, sorted and de-duplicated, replacing the file atomically.
func StatsViolation ¶
StatsViolation refuses, while mask rules exist, a statement that names a planner statistics relation (pg_stats, pg_statistic, mysql.column_stats, information_schema.COLUMN_STATISTICS, sqlite_stat4, ...): they hold real values of the columns, rule columns included, which no rule can match.
Types ¶
type Detector ¶
type Detector interface {
Name() string
// Mask returns s with every value the detector recognises replaced by
// its masked form (first character + "***" + "(length)").
Mask(s string) string
}
Detector masks personal values found inside a text, in place.
type Rules ¶
type Rules struct {
Mask []string
Allow []string
// Modes maps a Mask pattern to its mode when it is not ModePartial.
Modes map[string]string
}
Rules are the column rules. Each pattern is "db.table.column", where any segment may be '*'. For PostgreSQL the db segment is the schema. Matching ignores case. An Allow pattern beats every Mask pattern.
func LoadRules ¶
LoadRules reads <projectRoot>/.locksql/pii.toml. A missing file gives empty rules and no error.
func (*Rules) Add ¶
Add adds a mask pattern after validating it. Adding an existing pattern is a no-op.
func (*Rules) AddMode ¶
AddMode adds a mask pattern with a mode ("" for the default). Adding an existing pattern sets its mode.
func (Rules) Covered ¶
Covered reports whether a mask or an allow pattern names the column: the human already decided about it.
func (Rules) Matches ¶
Matches reports whether the column db.table.column is masked: some Mask pattern matches it and no Allow pattern does.
func (Rules) MatchesName ¶
MatchesName reports whether a column known only by its name may be under a rule: some Mask pattern has that column segment (whatever its db and table) and no Allow pattern exempts the name everywhere ("*.*.name"). It is the conservative test used when a result column has no origin.
Servers resolve identifiers with their own collation (MySQL compares column names by collation weight, so "fírstname" and "fİrstname" name the column firstname), which no Go case folding reproduces. A name with a non-ASCII character therefore matches as soon as any Mask pattern exists, and a non-ASCII character of a pattern matches any character.
type Tokens ¶
type Tokens struct {
// contains filtered or unexported fields
}
Tokens is the token table of one console session (mask mode "hash"): a keyed HMAC of the value, so the same value always gives the same token within the session, and a random key per session, so tokens cannot be correlated across sessions. It remembers which value each token stands for, so that an agent may filter on a token (WHERE email = 'tok_...'): the console substitutes the value in the statement that runs. Only equality leaks: no order, no prefix, no length.
Source Files