Documentation
¶
Overview ¶
Package pii decides which result cells must be masked: column rules kept in .locksql/pii.toml, rules proposed from the schema, value detectors (email, phone, IBAN, card, national ids) and the masking itself.
Index ¶
- Constants
- func AliasViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect) error
- func CellText(v any) string
- func Detects(s string, ds []Detector) bool
- func MaskOutputs(res *engine.Result, outs []sqlast.Output, r Rules, ds []Detector, ...) error
- func MaskResult(res *engine.Result, r Rules, ds []Detector, origin bool)
- func MaskValue(v any, mode string) any
- func NeedsAliasCheck(res engine.Result, origin bool) bool
- func PlanCheck(st sqlclass.Statement, r Rules, d sqlclass.Dialect, origin bool) error
- func Propose(cols []engine.ColumnInfo) []string
- func ProposeQuasi(cols []engine.ColumnInfo) []string
- func RedactMessage(msg, sql string, ds []Detector) string
- func RedactedRef(name string) string
- func ResultAliasViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect, cols []engine.ResultColumn) error
- func SaveRules(root string, r Rules) error
- func SaveRulesFile(path string, r Rules) error
- func StatementTextRelation(d sqlclass.Dialect, rel string) bool
- func StatsViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect) error
- func ValidMode(m string) bool
- func WriteValuesViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect) error
- type Detector
- type Rules
- func (r *Rules) Add(pattern string) error
- func (r *Rules) AddAllow(pattern string) error
- func (r *Rules) AddMode(pattern, mode string) error
- func (r Rules) Covered(db, table, column string) bool
- func (r Rules) Matches(db, table, column string) bool
- func (r Rules) MatchesName(column string) bool
- func (r Rules) Mode(db, table, column string) (string, bool)
- func (r Rules) ModeByName(column string) (string, bool)
Constants ¶
const ( // ModeRedact replaces the whole value: "<redacted>" (the default). ModeRedact = "redact" // ModePartial keeps the first character and the length: "j***(12)". ModePartial = "partial" // ModeEmail keeps the first character and the domain of an address: // "j***@example.com" (other values are masked as partial). ModeEmail = "email" )
Mask modes.
const DetectLimit = 64 << 10
DetectLimit is how much of a text cell the detectors scan. A longer cell is cut there, and ends with "…", before detection: nothing past the scanned part can reach the output. The renderer cuts cells much earlier (max_cell_chars), so the cut is normally invisible; it bounds the time a huge value can cost. The engines already cut a cell at engine.MaxCellBytes, which is no more.
const QuasiLimit = "masking blocks range filters, LIKE and ORDER BY on this column"
QuasiLimit is what masking a quasi-identifier costs, shown when the scan proposes one.
const Redacted = "<redacted>"
Redacted is the cell of a column masked in mode redact.
const RulesFile = ".locksql/pii.toml"
RulesFile is the rules file, relative to the project root.
Variables ¶
This section is empty.
Functions ¶
func AliasViolation ¶
AliasViolation is the fallback for result columns without an origin: it refuses a READ select in which a rule-matched column (matched by name, see Rules.MatchesName) is renamed or used inside an expression anywhere in a select list, since its values would then reach a label that no rule matches. Renaming covers "AS x", an implicit alias, a later UNION arm whose first arm labels the position differently, CTE column lists over a body that mentions a matched column or selects * from a source that may hold one, a * in a later arm over such a source, and table alias column lists ("t(x, y)"), which can rename any column. A PostgreSQL whole-row reference to a table that may hold such a column ("SELECT u", "row_to_json(u)", "to_jsonb(users.*)") is refused too, see wholeRow. A matched column or whole-row reference inside a VALUES list or the arguments of a FROM-clause function, and a TABLE arm after the first, are refused as well. Every named relation counts as a source that may hold a matched column, since it may be a view over a rule table. COUNT(...) is allowed. The error is a *sqlclass.Refusal that explains how to write the query.
A statement that is not a plain READ select is checked the same way when it may return rows: a RETURNING list counts as a select list, and a statement led by SELECT, WITH, VALUES or TABLE that the classifier made WRITE (a data-modifying CTE) is checked whole. A write without RETURNING is only checked by writtenValues and columnLists. Other statements (SHOW, DESCRIBE, PRAGMA, DDL) pass. The console runs the write analysis of package sqlast instead of the select-list checks when sqlast.NeedsWriteAnalysis says so (see WriteValuesViolation).
On PostgreSQL, "rel.f" may be the call f(rel) on the whole row of rel when rel has no column f (attribute notation: b.row_to_json, t.name, big.record_out). Since a statement alone cannot tell such a call from a column, a qualified reference "rel.x" to a FROM-clause relation is refused wherever its value may reach the result (select lists, RETURNING, VALUES, FROM-clause function arguments), see attrCall. ResultAliasViolation accepts a plain qualified item of the outer select list whose result column carries an origin, which proves it is a base-table column.
A write, with or without RETURNING, must not copy a rule-matched column's values into another column: a rule-matched column, a whole row or a star over a source that may hold one is refused anywhere its value may be written (SET values, the select list or VALUES of an INSERT source), see writtenValues.
func MaskOutputs ¶
func MaskOutputs(res *engine.Result, outs []sqlast.Output, r Rules, ds []Detector, cell func(row, col int, v any) string, origin bool) error
MaskOutputs masks res in place following the analysed outputs of its statement: each column whose resolved sources are under a rule is masked in the output's mode. As a second line of defence a column whose engine- reported origin matches a rule is masked too. The other text and integer cells go through the detectors.
cell, when not nil, stores the value of a cell masked in mode redact and returns its reference name, or "" for none. It is called only for a plain column the analysis masks in mode redact, whose values cannot be literals of the statement and all of whose sources are under a mask rule (referenceable): anywhere else the agent may know the value behind the reference (a constant, an unmasked UNION arm, an aggregate) and could look up one row on it without the k-anonymity check. A column masked by its engine-reported origin alone gets plain <redacted> too.
The result must have exactly the analysed columns, and the columns the analysis expects by name (plain references, aliases, star expansions) must carry that label: otherwise masking by position could hit the wrong column, and the result is refused.
func MaskResult ¶
MaskResult masks res in place.
A column is masked whole when its origin matches a rule. Engines report an origin only when the catalog confirms it is a base-table column; a view, a derived table or a CTE alias never counts. A view column is therefore masked only by its own name: a view that renames a rule column (CREATE VIEW v AS SELECT firstname AS contact ...) needs a rule for that name (*.*.contact or db.v.contact). When origin is false (the session reports no origins), or a column has none (an expression, a UNION, an untrusted origin the engine blanked), the column label is matched by name instead, see Rules.MatchesName; the console must then also refuse renamed uses with AliasViolation (see NeedsAliasCheck). A column under a rule is masked in the rule's mode (Rules.Mode, Rules.ModeByName; several rules with different modes give redact), as MaskValue does: redact gives plain "<redacted>" with no reference, since a write keeps no result in the reference store. A binary cell under a partial or email rule becomes "<masked bytes:N>"; NULL stays NULL.
Every other text cell, and every integer (a national number or a card can be stored as one), goes through the detectors in place.
func NeedsAliasCheck ¶
NeedsAliasCheck reports whether the statement behind res must pass AliasViolation (ResultAliasViolation, given res.Columns) before its rows may be shown: the session reports no origins, or some result column came back without one.
func PlanCheck ¶
PlanCheck is the PII check a statement must pass before it runs, unless it is an unmask run: StatsViolation, then AliasViolation for a statement that is not a plain READ select (it runs before the write, whose result may carry no origin) or when the session reports no origins. A plain READ select on an origin-reporting session is checked after it runs, only when some result column has no origin (see NeedsAliasCheck).
func Propose ¶
func Propose(cols []engine.ColumnInfo) []string
Propose returns the "db.table.column" patterns for the columns that look like personal data from their name or type, sorted and de-duplicated. A name that would break the pattern syntax becomes '*'.
func ProposeQuasi ¶ added in v0.1.2
func ProposeQuasi(cols []engine.ColumnInfo) []string
ProposeQuasi returns the "db.table.column" patterns for the columns that look like quasi-identifiers and not like personal data, sorted and de-duplicated.
func RedactMessage ¶
RedactMessage removes from a database error message the row values it may quote: a failed cast or function call in a WHERE clause echoes the value it choked on ("invalid input syntax for type integer: \"Smith\""), and that value reaches no result row, so no rule or detector masks it.
Everything from the first quote character to the last one is replaced, unless each quoted part is text of the statement itself (a syntax error, an unknown column) or a known server phrase, joined only by known server words: a value cannot shift the pairing of quotes, since its text would then sit in a gap. A single stray quote redacts the rest of the message. The detectors then run over what is left; an unquoted value they cannot recognise is not caught.
func RedactedRef ¶ added in v0.1.2
RedactedRef is a redacted cell that carries a reference to its value, which the console alone keeps.
func ResultAliasViolation ¶
func ResultAliasViolation(st sqlclass.Statement, r Rules, d sqlclass.Dialect, cols []engine.ResultColumn) error
ResultAliasViolation is AliasViolation once the statement has run, with the result columns it returned: a plain qualified PostgreSQL item of the outer select list ("SELECT b.id, ...") is accepted when its result column has an origin. With cols nil, every such item is refused.
func SaveRules ¶
SaveRules writes the rules to <root>/.locksql/pii.toml, sorted and de-duplicated, replacing the file atomically.
func SaveRulesFile ¶ added in v0.1.2
SaveRulesFile writes the rules to path, sorted and de-duplicated, replacing the file atomically. Missing directories are created.
func StatementTextRelation ¶ added in v0.1.2
StatementTextRelation reports whether rel, a relation name as written ("name", "schema.name"), holds the text of past statements, and so the values the console substituted into them. On PostgreSQL these are pg_stat_statements and pg_stat_activity. On MySQL and MariaDB they are information_schema.PROCESSLIST, every performance_schema and sys relation (threads, events_statements_*, session, statement_analysis, x$...), and mysql.general_log and slow_log.
func StatsViolation ¶
StatsViolation refuses, while mask rules exist, a statement that names a planner statistics relation (pg_stats, pg_statistic, mysql.column_stats, information_schema.COLUMN_STATISTICS, sqlite_stat4, ...): they hold real values of the columns, rule columns included, which no rule can match. It refuses a statement-text relation (see StatementTextRelation) the same way.
func WriteValuesViolation ¶ added in v0.5.0
WriteValuesViolation is the check by column name of a write that the console also analyses in full (sqlast.NeedsWriteAnalysis): it refuses statistics and statement-text relations (StatsViolation), copies and planted values (writtenValues) and column lists that rename columns by position (columnLists). Its RETURNING list and the outer SELECT of its data-modifying CTEs are left to the analysis, which resolves and masks them like a read's outputs.
Types ¶
type Detector ¶
type Detector interface {
Name() string
// Mask returns s with every value the detector recognises replaced by
// its masked form (first character + "***" + "(length)").
Mask(s string) string
}
Detector masks personal values found inside a text, in place.
type Rules ¶
type Rules struct {
Mask []string
Allow []string
// Modes maps a Mask pattern to its mode when it is not ModeRedact.
Modes map[string]string
}
Rules are the column rules. Each pattern is "db.table.column", where any segment may be '*'. For PostgreSQL the db segment is the schema. Matching ignores case. An Allow pattern beats every Mask pattern.
func LoadRules ¶
LoadRules reads <projectRoot>/.locksql/pii.toml. A missing file gives empty rules and no error.
func LoadRulesFile ¶ added in v0.1.2
LoadRulesFile reads the rules file at path. A missing file gives empty rules and no error.
func (*Rules) Add ¶
Add adds a mask pattern after validating it. Adding an existing pattern is a no-op.
func (*Rules) AddAllow ¶ added in v0.1.2
AddAllow adds an allow pattern after validating it. Adding an existing pattern is a no-op.
func (*Rules) AddMode ¶
AddMode adds a mask pattern with a mode ("" for the default). Adding an existing pattern sets its mode.
func (Rules) Covered ¶
Covered reports whether a mask or an allow pattern names the column: the human already decided about it.
func (Rules) Matches ¶
Matches reports whether the column db.table.column is masked: some Mask pattern matches it and no Allow pattern does.
func (Rules) MatchesName ¶
MatchesName reports whether a column known only by its name may be under a rule: some Mask pattern has that column segment (whatever its db and table) and no Allow pattern exempts the name everywhere ("*.*.name"). It is the conservative test used when a result column has no origin.
Servers resolve identifiers with their own collation (MySQL compares column names by collation weight, so "fírstname" and "fİrstname" name the column firstname), which no Go case folding reproduces. A name with a non-ASCII character therefore matches as soon as any Mask pattern exists, and a non-ASCII character of a pattern matches any character.
Source Files