Documentation
¶
Overview ¶
Package mask turns raw events into masked records, the only form of log content that leaves the console (design §4). Every string of an event passes the same fixed order: secrets first, and final; then rules, personal field names, value detectors and the structured/text mode defaults, all on the string step 1 produced.
A Record has no exported field and refuses to be encoded unless the masker built it: typed responses that hold Records cannot carry raw content by mistake.
Index ¶
- Constants
- Variables
- func Change(old, new Rules) (loosenings []string)
- func Clean(s string) string
- func DefaultValueDetectors() []string
- func Token(kind, id string) string
- type AgentLimits
- type AgentRules
- type Detectors
- type Field
- type Finder
- type Kind
- type Masker
- func (m *Masker) AgentAdd(source, shownPath, mode string, now time.Time, lim AgentLimits) error
- func (m *Masker) DetectorKinds() []string
- func (m *Masker) Epoch() string
- func (m *Masker) Lookup(kind, value string) (string, error)
- func (m *Masker) Mask(source string, ev parse.Event) (rec Record)
- func (m *Masker) MaskClear(source string, ev parse.Event) (Record, map[string]string)
- func (m *Masker) RedactText(s string) (out string)
- func (m *Masker) Text(s string) (out string)
- func (m *Masker) TrustedValue(source, path, v string) (out string)
- type Options
- type Record
- func (r Record) Attrs() []Field
- func (r Record) Failed() bool
- func (r Record) Fields() []Field
- func (r Record) Format() string
- func (r Record) HadSecret(path string) bool
- func (r Record) Kinds() []string
- func (r Record) Lookup(path string) (Value, bool)
- func (r Record) MarshalJSON() ([]byte, error)
- func (r Record) Message() string
- func (r Record) Position() (offset, length int64)
- func (r Record) Time() (time.Time, bool)
- func (r Record) Valid() bool
- type Rule
- type Rules
- type Value
- type ValueRule
Constants ¶
const ( StructuredRedactUnknown = "redact-unknown" StructuredDetect = "detect" )
Structured modes (design §4.2), as the profile names them.
const ( Open = "⟦" // ⟦ Close = "⟧" // ⟧ // Secret replaces a secret: never unmaskable, never pseudonymised. Secret = Open + "secret" + Close // Redacted replaces a value withheld whole (redact-unknown, a redact // rule, a forgotten epoch). Redacted = Open + "redacted" + Close )
Placeholders written in place of masked content. Source text never contains the brackets: they are replaced by U+FFFD before masking, so a placeholder in a record was always written by the masker.
const ( // ModeRedact withholds the value whole (⟦redacted⟧). ModeRedact = "redact" // ModePseudonym replaces the value whole by its pseudonym. ModePseudonym = "pseudonym" // ModeAllow shows the value after the secret step and the detectors, // also under redact-unknown. ModeAllow = "allow" )
Rule modes.
const KeyKind = "key"
KeyKind is the kind of a field name that was masked (⟦key:ID⟧).
const MaxRegex = 256
MaxRegex bounds a value rule's expression (RE2).
const PersonalKind = "personal"
PersonalKind is the kind of a value pseudonymised because of its field name (LooksPersonal, LooksQuasi) or a rule, when no detector recognises it whole.
const SecretDetectorName = "secret"
SecretDetectorName is lockkit's secret detector: step 1, always on, never a value detector a profile names.
Variables ¶
var ( ErrUnknownField = errors.New("mask: unknown field") ErrPIIInQuery = errors.New("mask: the rule holds a personal value or a secret (use locklog_lookup)") ErrRuleCap = errors.New("mask: agent rule cap reached") ErrRuleExists = errors.New("mask: a rule already decides this field") ErrRuleMode = errors.New("mask: an agent rule is redact or pseudonym") )
Refusals of an agent rule (design §4.4 rule 7). The engine maps them to E_UNKNOWN_FIELD, E_PII_IN_QUERY and E_OVER_BUDGET.
var ( // ErrLookupKind: the kind is not one of the masker's value detectors. ErrLookupKind = errors.New("mask: not a detector kind") // ErrLookupSecret: the value reads as a secret (never looked up). ErrLookupSecret = errors.New("mask: the value reads as a secret") // ErrLookupValue: the value is not, whole, a value of that kind. ErrLookupValue = errors.New("mask: not a value of that kind") )
Lookup errors.
var ErrUnmasked = errors.New("mask: record not built by the masker")
ErrUnmasked is the error of encoding a Record the masker did not build.
Functions ¶
func Change ¶
Change classifies a change of the project rules (design §4.4 rule 7): the loosenings, which go to the human's review; a change without any only tightens and may apply at once. A new pseudonym rule is a loosening: under redact-unknown it turns ⟦redacted⟧ into a linkable pseudonym (and, with a detector kind, opens lookup).
func Clean ¶
cleanSource prepares source text for masking: the placeholder brackets become U+FFFD (design §4.4 rule 6: only the masker writes them), and controls (but tab and newline) and invisible characters are dropped (§5.4), as are invalid bytes and U+FFFD, before detection, so that the detectors see exactly the text that will be shown. Clean is the text the masker reads of a clear string before masking it: invalid bytes, U+FFFD, controls and invisible runes dropped (which can join text that is apart in the clear), brackets made U+FFFD. Every unmasked segment of a masked string is a substring of Clean of the clear string (the engine's pre-filter relies on it).
func DefaultValueDetectors ¶
func DefaultValueDetectors() []string
DefaultValueDetectors names the value detectors of a profile that sets none, and of a masker given no Values: lockkit's default set (pii.AllDetectors) and ip (public addresses; private and loopback ones are infrastructure, never found).
Types ¶
type AgentLimits ¶
AgentLimits cap the agent's rules: per hour and in total. Both must be positive: a zero limit refuses every rule (it never means unlimited).
type AgentRules ¶
type AgentRules struct {
// contains filtered or unexported fields
}
AgentRules are the rules the agent added: they only tighten. They live only in the console's PII directory, each field named by a keyed digest of (source, clear path), never by its name: neither the project's pii.toml nor this directory holds a name that was shown masked (round-6 amendment). The key is the directory's rules.key (0600).
func OpenAgentRules ¶
func OpenAgentRules(dir string) (*AgentRules, error)
OpenAgentRules opens (creating if needed) the agent rules of a PII directory, which must be private to the console (0700).
type Detectors ¶
Detectors finds what lockkit detectors find, in the text and in its detection views (encodings, look-alikes, invisible characters).
func SecretDetectors ¶
func SecretDetectors() Detectors
SecretDetectors finds secrets with lockkit's secret detector (credentials, token shapes, private keys, URL userinfo), in the text and in its detection views. Every masker runs it in step 1, whatever Options.Secrets adds.
func ValueDetectors ¶
ValueDetectors returns the value detectors a profile names (step 4). The secret detector is refused: secrets are step 1, always on.
type Field ¶
type Field struct {
// contains filtered or unexported fields
}
Field is one masked member: its name as shown (clear, or ⟦key:ID⟧ when the name itself was masked) and its value.
type Masker ¶
type Masker struct {
// contains filtered or unexported fields
}
Masker masks events. It is safe for concurrent use.
func (*Masker) AgentAdd ¶
AgentAdd adds an agent rule (locklog_pii_add): the field named by its shown path (clear names as shown, masked names as ⟦key:ID⟧, translated here, in the console, to the clear name). It only tightens: the field's names must all have been shown (clear, or masked under this epoch), the path holds no glob, no personal value and no secret, no human rule decides the field, and the mode is redact or pseudonym. The rule is saved as a keyed digest in the PII directory only.
func (*Masker) DetectorKinds ¶
DetectorKinds are the kinds the masker's value detectors find (sorted): the only kinds Lookup resolves.
func (*Masker) Lookup ¶
Lookup returns the pseudonym a clear value of a detector kind has in the logs (equality only): the token masking the value as free text gives, when that is one whole token of that kind. A value a secret detector flags, a kind no value detector finds (field names, template tokens, rule kinds) and a value that is not wholly of the kind are refused.
func (*Masker) Mask ¶
Mask masks one event of source. A panic in masking withholds the whole record (Failed), never part of it.
func (*Masker) MaskClear ¶
MaskClear masks ev as Mask does and also returns, for every pseudonym of a value the record shows (⟦kind:ID⟧, field names excepted), the clear value it stands for: what an approved unmask reveals. Never return the map to anyone but the approved unmask's own spans.
func (*Masker) RedactText ¶
RedactText masks text the agent wrote before it is echoed back (refusal echoes): secrets as ⟦secret⟧, personal values as ⟦redacted⟧, never as a pseudonym (no oracle, design §4.4).
func (*Masker) Text ¶
Text masks a string read from the logs that leaves the console other than as a record (names in `sources`, bucket keys, errors, audit causes): secrets, then personal values as pseudonyms. Never pass it text the agent wrote: the pseudonym of a guessed value would answer whether that value is in the logs (design §4.4); use RedactText.
func (*Masker) TrustedValue ¶
TrustedValue masks a trusted value read outside a record (an index value, a value compared with a trusted field) as Mask masks a trusted leaf at path of source: secret names, human and agent rules (an ancestor's too), then the detectors, a number in exponent or decimal form read as the integer it is. Its JSON kind and how path splits into keys are unknown, so every reading is tried (string and, for a number literal, number; the dotted path as nested keys and as one key): the first that changes the value is returned, else the value. A value it changes must never be shown or matched clear.
type Options ¶
type Options struct {
// Structured is StructuredRedactUnknown (default) or StructuredDetect.
Structured string
// Values are the value detectors of step 4 (personal data); left
// nil, DefaultValueDetectors.
Values Finder
// Secrets finds secret values (step 1); SecretName reports a field
// name whose value is a secret whole. Both add to lockkit's secret
// detector (SecretDetectors) and pii.LooksSecret, which every masker
// runs: they can widen step 1, never replace it.
Secrets Finder
SecretName func(name string) bool
// Trusted are the clear paths of trusted fields: shown after the
// secret step and the detectors, never redacted by redact-unknown nor
// pseudonymised for their name.
Trusted []string
// AttrKeys are the frame and static attribute keys of the source
// (parse.FrameKeys for a framed format, and the docker source's
// "container"): its attributes (parse.Event.Attrs) are trusted on
// their own, and a Trusted path under one of these keys names an
// attribute, never a payload field. Field trust is not applied to a
// payload path whose first key is one of them, so a line's own
// {"container": ...} is masked as any untrusted field.
AttrKeys []string
// Rules are the human and agent rules (step 2).
Rules *Rules
// Pseudonyms is the current epoch.
Pseudonyms *pii.Pseudonymizer
// contains filtered or unexported fields
}
Options configure a Masker.
type Record ¶
type Record struct {
// contains filtered or unexported fields
}
Record is one masked event. The zero value is invalid: only the masker builds valid records.
func (Record) Failed ¶
Failed reports whether masking failed on the record, which then holds nothing but its position (masked_failure).
func (Record) Lookup ¶
Lookup returns the first leaf or object at a shown path ("user.email", "⟦key:ID⟧.city"); arrays are crossed without an index.
func (Record) MarshalJSON ¶
MarshalJSON writes the masked record: {"ts", "fields", "attrs", "msg"}, or {"masked_failure": true}. A record the masker did not build is refused.
func (Record) Position ¶
Position returns the event's place in its stream (byte offset, bytes consumed): plan metadata, not content.
type Rule ¶
type Rule struct {
Sources []string // source names; empty: every source
Field string
Mode string
Kind string // pseudonym rules: the pseudonym kind ("customer")
}
Rule is a human rule of .locklog/pii.toml: field names and modes only. Field is a clear dotted path; "*" matches one segment (in a mask rule, also a key holding dots: one or more segments, so a mask rule covers more, never less).
type Rules ¶
type Rules struct {
Mask []Rule
Allow []Rule
Values []ValueRule
Agent *AgentRules
}
Rules are the masking rules in force: the project's (names and modes), the console's value rules, and the agent's (tighten-only, keyed digests in the PII directory).
func LoadProjectRules ¶
LoadProjectRules reads .locklog/pii.toml. A missing file is no rules. Only field (or label) names and modes may appear: value lists and expressions belong to the console's PII directory.
func ParseProjectRules ¶
ParseProjectRules parses the content of a pii.toml.
func (*Rules) Fingerprint ¶
Fingerprint identifies the rules in force (part of the policy fingerprint): human rules by their text, agent rules by their keyed digests.
type Value ¶
type Value struct {
// contains filtered or unexported fields
}
Value is one masked node of a structured record.
func (Value) MarshalJSON ¶
MarshalJSON writes the masked value.
type ValueRule ¶
ValueRule is a human rule of the console's PII directory: a regular expression whose matches are pseudonymised as Kind.
func LoadValueRules ¶
LoadValueRules reads values.toml in the console's PII directory: value expressions are human rules, never the project's.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package masktest generates seeded log corpora with planted canaries (synthetic personal values and secrets, in every position and encoding) and scans bytes for any of them: the leak harness of design §4.7.
|
Package masktest generates seeded log corpora with planted canaries (synthetic personal values and secrets, in every position and encoding) and scans bytes for any of them: the leak harness of design §4.7. |