mask

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 6, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package mask turns raw events into masked records, the only form of log content that leaves the console (design §4). Every string of an event passes the same fixed order: secrets first, and final; then rules, personal field names, value detectors and the structured/text mode defaults, all on the string step 1 produced.

A Record has no exported field and refuses to be encoded unless the masker built it: typed responses that hold Records cannot carry raw content by mistake.

Index

Constants

View Source
const (
	StructuredRedactUnknown = "redact-unknown"
	StructuredDetect        = "detect"
)

Structured modes (design §4.2), as the profile names them.

View Source
const (
	Open  = "⟦" // ⟦
	Close = "⟧" // ⟧
	// Secret replaces a secret: never unmaskable, never pseudonymised.
	Secret = Open + "secret" + Close
	// Redacted replaces a value withheld whole (redact-unknown, a redact
	// rule, a forgotten epoch).
	Redacted = Open + "redacted" + Close
)

Placeholders written in place of masked content. Source text never contains the brackets: they are replaced by U+FFFD before masking, so a placeholder in a record was always written by the masker.

View Source
const (
	// ModeRedact withholds the value whole (⟦redacted⟧).
	ModeRedact = "redact"
	// ModePseudonym replaces the value whole by its pseudonym.
	ModePseudonym = "pseudonym"
	// ModeAllow shows the value after the secret step and the detectors,
	// also under redact-unknown.
	ModeAllow = "allow"
)

Rule modes.

View Source
const KeyKind = "key"

KeyKind is the kind of a field name that was masked (⟦key:ID⟧).

View Source
const MaxRegex = 256

MaxRegex bounds a value rule's expression (RE2).

View Source
const PersonalKind = "personal"

PersonalKind is the kind of a value pseudonymised because of its field name (LooksPersonal, LooksQuasi) or a rule, when no detector recognises it whole.

View Source
const SecretDetectorName = "secret"

SecretDetectorName is lockkit's secret detector: step 1, always on, never a value detector a profile names.

Variables

View Source
var (
	ErrUnknownField = errors.New("mask: unknown field")
	ErrPIIInQuery   = errors.New("mask: the rule holds a personal value or a secret (use locklog_lookup)")
	ErrRuleCap      = errors.New("mask: agent rule cap reached")
	ErrRuleExists   = errors.New("mask: a rule already decides this field")
	ErrRuleMode     = errors.New("mask: an agent rule is redact or pseudonym")
)

Refusals of an agent rule (design §4.4 rule 7). The engine maps them to E_UNKNOWN_FIELD, E_PII_IN_QUERY and E_OVER_BUDGET.

View Source
var (
	// ErrLookupKind: the kind is not one of the masker's value detectors.
	ErrLookupKind = errors.New("mask: not a detector kind")
	// ErrLookupSecret: the value reads as a secret (never looked up).
	ErrLookupSecret = errors.New("mask: the value reads as a secret")
	// ErrLookupValue: the value is not, whole, a value of that kind.
	ErrLookupValue = errors.New("mask: not a value of that kind")
)

Lookup errors.

View Source
var ErrUnmasked = errors.New("mask: record not built by the masker")

ErrUnmasked is the error of encoding a Record the masker did not build.

Functions

func Change

func Change(old, new Rules) (loosenings []string)

Change classifies a change of the project rules (design §4.4 rule 7): the loosenings, which go to the human's review; a change without any only tightens and may apply at once. A new pseudonym rule is a loosening: under redact-unknown it turns ⟦redacted⟧ into a linkable pseudonym (and, with a detector kind, opens lookup).

func Clean

func Clean(s string) string

cleanSource prepares source text for masking: the placeholder brackets become U+FFFD (design §4.4 rule 6: only the masker writes them), and controls (but tab and newline) and invisible characters are dropped (§5.4), as are invalid bytes and U+FFFD, before detection, so that the detectors see exactly the text that will be shown. Clean is the text the masker reads of a clear string before masking it: invalid bytes, U+FFFD, controls and invisible runes dropped (which can join text that is apart in the clear), brackets made U+FFFD. Every unmasked segment of a masked string is a substring of Clean of the clear string (the engine's pre-filter relies on it).

func DefaultValueDetectors

func DefaultValueDetectors() []string

DefaultValueDetectors names the value detectors of a profile that sets none, and of a masker given no Values: lockkit's default set (pii.AllDetectors) and ip (public addresses; private and loopback ones are infrastructure, never found).

func Token

func Token(kind, id string) string

Token is the placeholder of a pseudonym: ⟦kind:ID⟧.

Types

type AgentLimits

type AgentLimits struct {
	PerHour int
	Total   int
}

AgentLimits cap the agent's rules: per hour and in total. Both must be positive: a zero limit refuses every rule (it never means unlimited).

type AgentRules

type AgentRules struct {
	// contains filtered or unexported fields
}

AgentRules are the rules the agent added: they only tighten. They live only in the console's PII directory, each field named by a keyed digest of (source, clear path), never by its name: neither the project's pii.toml nor this directory holds a name that was shown masked (round-6 amendment). The key is the directory's rules.key (0600).

func OpenAgentRules

func OpenAgentRules(dir string) (*AgentRules, error)

OpenAgentRules opens (creating if needed) the agent rules of a PII directory, which must be private to the console (0700).

func (*AgentRules) Len

func (a *AgentRules) Len() int

Len returns the number of agent rules.

type Detectors

type Detectors []pii.Detector

Detectors finds what lockkit detectors find, in the text and in its detection views (encodings, look-alikes, invisible characters).

func SecretDetectors

func SecretDetectors() Detectors

SecretDetectors finds secrets with lockkit's secret detector (credentials, token shapes, private keys, URL userinfo), in the text and in its detection views. Every masker runs it in step 1, whatever Options.Secrets adds.

func ValueDetectors

func ValueDetectors(names []string) (Detectors, error)

ValueDetectors returns the value detectors a profile names (step 4). The secret detector is refused: secrets are step 1, always on.

func (Detectors) Find

func (ds Detectors) Find(s string) []pii.Match

Find implements Finder.

type Field

type Field struct {
	// contains filtered or unexported fields
}

Field is one masked member: its name as shown (clear, or ⟦key:ID⟧ when the name itself was masked) and its value.

func (Field) Name

func (f Field) Name() string

Name returns the name as shown to the agent.

func (Field) Value

func (f Field) Value() Value

Value returns the masked value.

type Finder

type Finder interface {
	Find(s string) []pii.Match
}

Finder finds values in a text, as byte ranges of it.

type Kind

type Kind uint8

Kind is the kind of a masked value.

const (
	Null Kind = iota
	Bool
	Number
	String
	Object
	Array
)

Value kinds. A masked number or bool becomes a string.

type Masker

type Masker struct {
	// contains filtered or unexported fields
}

Masker masks events. It is safe for concurrent use.

func New

func New(o Options) (*Masker, error)

New returns a masker; Pseudonyms is required.

func (*Masker) AgentAdd

func (m *Masker) AgentAdd(source, shownPath, mode string, now time.Time, lim AgentLimits) error

AgentAdd adds an agent rule (locklog_pii_add): the field named by its shown path (clear names as shown, masked names as ⟦key:ID⟧, translated here, in the console, to the clear name). It only tightens: the field's names must all have been shown (clear, or masked under this epoch), the path holds no glob, no personal value and no secret, no human rule decides the field, and the mode is redact or pseudonym. The rule is saved as a keyed digest in the PII directory only.

func (*Masker) DetectorKinds

func (m *Masker) DetectorKinds() []string

DetectorKinds are the kinds the masker's value detectors find (sorted): the only kinds Lookup resolves.

func (*Masker) Epoch

func (m *Masker) Epoch() string

Epoch returns the pseudonym epoch records are masked under.

func (*Masker) Lookup

func (m *Masker) Lookup(kind, value string) (string, error)

Lookup returns the pseudonym a clear value of a detector kind has in the logs (equality only): the token masking the value as free text gives, when that is one whole token of that kind. A value a secret detector flags, a kind no value detector finds (field names, template tokens, rule kinds) and a value that is not wholly of the kind are refused.

func (*Masker) Mask

func (m *Masker) Mask(source string, ev parse.Event) (rec Record)

Mask masks one event of source. A panic in masking withholds the whole record (Failed), never part of it.

func (*Masker) MaskClear

func (m *Masker) MaskClear(source string, ev parse.Event) (Record, map[string]string)

MaskClear masks ev as Mask does and also returns, for every pseudonym of a value the record shows (⟦kind:ID⟧, field names excepted), the clear value it stands for: what an approved unmask reveals. Never return the map to anyone but the approved unmask's own spans.

func (*Masker) RedactText

func (m *Masker) RedactText(s string) (out string)

RedactText masks text the agent wrote before it is echoed back (refusal echoes): secrets as ⟦secret⟧, personal values as ⟦redacted⟧, never as a pseudonym (no oracle, design §4.4).

func (*Masker) Text

func (m *Masker) Text(s string) (out string)

Text masks a string read from the logs that leaves the console other than as a record (names in `sources`, bucket keys, errors, audit causes): secrets, then personal values as pseudonyms. Never pass it text the agent wrote: the pseudonym of a guessed value would answer whether that value is in the logs (design §4.4); use RedactText.

func (*Masker) TrustedValue

func (m *Masker) TrustedValue(source, path, v string) (out string)

TrustedValue masks a trusted value read outside a record (an index value, a value compared with a trusted field) as Mask masks a trusted leaf at path of source: secret names, human and agent rules (an ancestor's too), then the detectors, a number in exponent or decimal form read as the integer it is. Its JSON kind and how path splits into keys are unknown, so every reading is tried (string and, for a number literal, number; the dotted path as nested keys and as one key): the first that changes the value is returned, else the value. A value it changes must never be shown or matched clear.

type Options

type Options struct {
	// Structured is StructuredRedactUnknown (default) or StructuredDetect.
	Structured string
	// Values are the value detectors of step 4 (personal data); left
	// nil, DefaultValueDetectors.
	Values Finder
	// Secrets finds secret values (step 1); SecretName reports a field
	// name whose value is a secret whole. Both add to lockkit's secret
	// detector (SecretDetectors) and pii.LooksSecret, which every masker
	// runs: they can widen step 1, never replace it.
	Secrets    Finder
	SecretName func(name string) bool
	// Trusted are the clear paths of trusted fields: shown after the
	// secret step and the detectors, never redacted by redact-unknown nor
	// pseudonymised for their name.
	Trusted []string
	// AttrKeys are the frame and static attribute keys of the source
	// (parse.FrameKeys for a framed format, and the docker source's
	// "container"): its attributes (parse.Event.Attrs) are trusted on
	// their own, and a Trusted path under one of these keys names an
	// attribute, never a payload field. Field trust is not applied to a
	// payload path whose first key is one of them, so a line's own
	// {"container": ...} is masked as any untrusted field.
	AttrKeys []string
	// Rules are the human and agent rules (step 2).
	Rules *Rules
	// Pseudonyms is the current epoch.
	Pseudonyms *pii.Pseudonymizer
	// contains filtered or unexported fields
}

Options configure a Masker.

type Record

type Record struct {
	// contains filtered or unexported fields
}

Record is one masked event. The zero value is invalid: only the masker builds valid records.

func (Record) Attrs

func (r Record) Attrs() []Field

Attrs returns the masked container-frame fields.

func (Record) Failed

func (r Record) Failed() bool

Failed reports whether masking failed on the record, which then holds nothing but its position (masked_failure).

func (Record) Fields

func (r Record) Fields() []Field

Fields returns the masked structured part.

func (Record) Format

func (r Record) Format() string

Format returns the format that decoded the event.

func (Record) HadSecret

func (r Record) HadSecret(path string) bool

HadSecret reports whether the leaf at the shown path held a secret.

func (Record) Kinds

func (r Record) Kinds() []string

Kinds returns the kinds of what was masked.

func (Record) Lookup

func (r Record) Lookup(path string) (Value, bool)

Lookup returns the first leaf or object at a shown path ("user.email", "⟦key:ID⟧.city"); arrays are crossed without an index.

func (Record) MarshalJSON

func (r Record) MarshalJSON() ([]byte, error)

MarshalJSON writes the masked record: {"ts", "fields", "attrs", "msg"}, or {"masked_failure": true}. A record the masker did not build is refused.

func (Record) Message

func (r Record) Message() string

Message returns the masked free text.

func (Record) Position

func (r Record) Position() (offset, length int64)

Position returns the event's place in its stream (byte offset, bytes consumed): plan metadata, not content.

func (Record) Time

func (r Record) Time() (time.Time, bool)

Time returns the record's timestamp and whether it has one.

func (Record) Valid

func (r Record) Valid() bool

Valid reports whether the masker built r.

type Rule

type Rule struct {
	Sources []string // source names; empty: every source
	Field   string
	Mode    string
	Kind    string // pseudonym rules: the pseudonym kind ("customer")
}

Rule is a human rule of .locklog/pii.toml: field names and modes only. Field is a clear dotted path; "*" matches one segment (in a mask rule, also a key holding dots: one or more segments, so a mask rule covers more, never less).

type Rules

type Rules struct {
	Mask   []Rule
	Allow  []Rule
	Values []ValueRule
	Agent  *AgentRules
}

Rules are the masking rules in force: the project's (names and modes), the console's value rules, and the agent's (tighten-only, keyed digests in the PII directory).

func LoadProjectRules

func LoadProjectRules(file string) (Rules, error)

LoadProjectRules reads .locklog/pii.toml. A missing file is no rules. Only field (or label) names and modes may appear: value lists and expressions belong to the console's PII directory.

func ParseProjectRules

func ParseProjectRules(data []byte) (Rules, error)

ParseProjectRules parses the content of a pii.toml.

func (*Rules) Find

func (r *Rules) Find(s string) []pii.Match

Find implements Finder for the value rules: their matches, as ranges.

func (*Rules) Fingerprint

func (r *Rules) Fingerprint() string

Fingerprint identifies the rules in force (part of the policy fingerprint): human rules by their text, agent rules by their keyed digests.

func (*Rules) Texts

func (r *Rules) Texts() []string

Texts returns the canonical form of each rule, sorted.

type Value

type Value struct {
	// contains filtered or unexported fields
}

Value is one masked node of a structured record.

func (Value) Items

func (v Value) Items() []Value

Items returns an array's masked items.

func (Value) Kind

func (v Value) Kind() Kind

Kind returns the value's kind.

func (Value) MarshalJSON

func (v Value) MarshalJSON() ([]byte, error)

MarshalJSON writes the masked value.

func (Value) Members

func (v Value) Members() []Field

Members returns an object's masked members, in order, duplicates kept.

func (Value) Text

func (v Value) Text() string

Text returns a scalar's masked text ("true", an exact number literal, a masked string).

type ValueRule

type ValueRule struct {
	Kind  string
	Regex string
	// contains filtered or unexported fields
}

ValueRule is a human rule of the console's PII directory: a regular expression whose matches are pseudonymised as Kind.

func LoadValueRules

func LoadValueRules(dir string) ([]ValueRule, error)

LoadValueRules reads values.toml in the console's PII directory: value expressions are human rules, never the project's.

Directories

Path Synopsis
Package masktest generates seeded log corpora with planted canaries (synthetic personal values and secrets, in every position and encoding) and scans bytes for any of them: the leak harness of design §4.7.
Package masktest generates seeded log corpora with planted canaries (synthetic personal values and secrets, in every position and encoding) and scans bytes for any of them: the leak harness of design §4.7.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL