Documentation
¶
Overview ¶
Package datamasking erases, encrypts and decrypts sensitive data: the Go port of Powertools for AWS Lambda (Python) data masking utility.
Erasing replaces values irreversibly with "*****" (Mask) or with a mask of your own; it needs nothing but memory. Encrypting turns the whole data into a ciphertext through a Provider; package datamasking/kms is the AWS Encryption SDK provider, with AWS KMS keys, that Powertools uses, and its messages decrypt in Powertools Python and the other AWS Encryption SDK implementations, and the other way round.
Erasing ¶
Build one DataMasking and share it:
var masker = datamasking.New()
erased, err := masker.Erase(event.Body, datamasking.WithFields("email", "address.street"))
Erase accepts raw JSON ([]byte, jsontext.Value), any Go value that marshals to JSON with encoding/json/v2 (structs through their JSON field names, maps, aws-lambda-go events) and scalars. With field selection or masking rules a string is a JSON document, as in Python. The caller's data is never modified. The result is a decoded JSON value (nil, bool, int64, float64, string, []any, map[string]any); EraseAs decodes it into a type of your choice instead, raw JSON included.
What gets erased:
- without options, the whole data becomes Mask; an array becomes as many Mask strings as it has elements;
- WithFields selects values with JSONPath expressions, applied in order: each selected value is erased the same way (an object becomes Mask, an array a list of Mask), the rest of the data stays;
- WithDynamicMask, WithCustomMask and WithRegex mask instead of erasing: objects and arrays keep their shape, every scalar inside is masked as text (see Rule);
- WithMaskingRules gives each expression masks of its own, in order, and takes precedence over the other options.
Scalars are masked as the text Python's str() writes, so that masks have the same length as in Python: 30 is "30", 4.2 is "4.2", 1e20 is "1e+20", true is "True". JSON integers stay int64 and decimals float64, as Python keeps int and float apart. With WithMaskingRules every selected value is masked as text, objects and arrays as their Python repr ("['Pat', 'powertools']"), and null values are left alone.
An expression that matches nothing fails with ErrFieldNotFound, unless WithRaiseOnMissingField(false) is set: then it is skipped, and reported on the Powertools debug log (POWERTOOLS_DEBUG=true).
Field expressions ¶
Powertools Python parses expressions with jsonpath-ng's extended parser; ojg (github.com/ohler55/ojg) evaluates them here. No Go library speaks jsonpath-ng's dialect, so the package parses the part Powertools documents and tests, and rejects everything else with ErrInvalidPath rather than guess. Accepted:
- an optional root: card_number and $.card_number are the same; $ alone selects the whole data;
- members: name, a.b, names with letters, digits, '_', '@' and '-' (non-existing), quoted names ('1', "a b", with \ escaping the next character), numbers used as names (a.1 is the member "1"), lists of names (a.b,c and ['b','c']);
- * and [*] for every member of an object or element of an array;
- descendants: category..name, a..'4', a..[0];
- indexes and slices: [1], [-1], [0,2], [1:3], [-2:], [::2];
- filters on the elements of an array (or the members of an object): [?(@.postcode > 12000)], [?postcode > 12000], [?(@.kind)] (present), [?(!@.kind)] (absent), conditions joined with & and grouped with parentheses; the operators ==, = (the same), !=, <, <=, >, >= and =~ (a regular expression, searched anywhere in the text); the values are integers, decimals, true, false, quoted strings and bare words (home is the string "home", as in jsonpath-ng). Inside a filter, @ is the element, and the path after it may use the forms above.
Rejected, because jsonpath-ng extensions have no ojg equivalent: | and & between paths, where and wherenot, `this`, `len` and the other named operators, arithmetic, sorting, || and && in filters, $ after the start, parenthesized paths.
Where both accept an expression, they can still disagree on data of an unexpected shape: jsonpath-ng turns a string into an integer to compare it with an integer ("12345" > 12000 holds there, not here) and raises an error comparing incompatible types, where ojg simply does not match; jsonpath-ng's .* skips arrays, and its [*] treats an object as a list of one element.
Masks ¶
See Rule. Regular expressions use Go's RE2 syntax, which has no lookarounds and no backreferences; named groups work as (?P<name>...). The mask format follows Python's re.sub: \1 to \99, \g<1> and \g<name> refer to groups, \0 and three-digit octal escapes are characters, \n and the other C escapes are themselves, and $ has no special meaning. Empty matches follow Go's rules, which skip an empty match right after a non-empty one where Python does not. Patterns are compiled once per process.
Encrypting ¶
provider, err := kms.NewFromConfig(cfg, []string{keyARN})
masker := datamasking.New(datamasking.WithProvider(provider))
ciphertext, err := masker.Encrypt(ctx, order, datamasking.WithEncryptionContext(map[string]string{
"tenant_id": tenantID,
}))
order, err := masker.DecryptAs[Order](ctx, ciphertext, datamasking.WithEncryptionContext(...))
Encrypt marshals the data to JSON (raw JSON is encrypted as the document it is, a string as a JSON string) and returns the provider's ciphertext. Decrypt parses the decrypted JSON; DecryptAs decodes it into T. The encryption context must be exactly the same on both sides, or decryption fails with ErrContextMismatch. WithProviderOptions passes per-call settings to the provider, as Python's provider_options.
A Provider deals with bytes; one that also implements Eraser masks values its own way, and can fall back to MaskValue.
Options ¶
New takes WithProvider, WithRaiseOnMissingField (default true) and WithJSONOptions, the encoding/json/v2 options for marshalling Go values and decoding results (Python's json_serializer and json_deserializer). The package reads no environment variable of its own; POWERTOOLS_DEBUG turns on the debug log.
Errors ¶
Every error the package detects matches ErrDataMasking, and one of ErrUnsupportedType, ErrEmptyFields, ErrFieldNotFound, ErrInvalidPath, ErrMaskField, ErrInvalidRegex, ErrNoProvider, ErrEncryptKey, ErrDecryptKey, ErrDecryptValue and ErrContextMismatch. A *PathError names the expression that failed.
Differences from Python ¶
- Raw JSON ([]byte, jsontext.Value) is the document it holds, never bytes to erase; Go values are seen through encoding/json/v2, not dataclasses, Pydantic or __dict__.
- Masking rules are an ordered slice, not a dict, and accept a JSON string as fields do (Python searches the string itself and finds nothing).
- $ selects the whole data; in jsonpath-ng it matches but updates nothing.
- Field selection needs an object or an array; Python accepts a top level array only inside a JSON string.
- The Python repr of an object lists its keys in sorted order, Python in the document's; masks of the same object can differ in where the kept characters fall, not in length.
- Regular expressions are compiled when Erase starts, so an invalid one fails even when nothing matches.
- Errors of a custom Eraser are tied to the expression with ErrMaskField for fields too, not only for masking rules.
- Python raises NotImplementedError when encrypting without a provider; here it is ErrNoProvider. Python's ValueError for empty fields is ErrEmptyFields.
- Encryption context values are strings by type, so Python's check that they are is not needed.
- Go reads integers beyond int64 as float64, Python keeps them exact.
Index ¶
- Constants
- Variables
- func MaskValue(value any, rule Rule) (any, error)
- type CryptoOption
- type DataMasking
- func (m *DataMasking) Decrypt(ctx context.Context, ciphertext string, opts ...CryptoOption) (any, error)
- func (m *DataMasking) DecryptAs[T any](ctx context.Context, ciphertext string, opts ...CryptoOption) (T, error)
- func (m *DataMasking) Encrypt(ctx context.Context, data any, opts ...CryptoOption) (string, error)
- func (m *DataMasking) Erase(data any, opts ...EraseOption) (any, error)
- func (m *DataMasking) EraseAs[T any](data any, opts ...EraseOption) (T, error)
- type EraseOption
- type Eraser
- type Option
- type PathError
- type Provider
- type ProviderOption
- type Request
- type Rule
Examples ¶
Constants ¶
const Mask = "*****"
Mask is the string that replaces an erased value: Python's DATA_MASKING_STRING.
Variables ¶
var ( // ErrUnsupportedType reports data that cannot be masked or encrypted: // a value encoding/json/v2 cannot marshal, JSON that does not parse, or // a scalar given to field selection, which needs an object or an array. ErrUnsupportedType = fmt.Errorf("%w: unsupported data type", ErrDataMasking) // ErrEmptyFields reports WithFields called with no expression. ErrEmptyFields = fmt.Errorf("%w: fields parameter cannot be empty", ErrDataMasking) // ErrFieldNotFound reports an expression that matches nothing, when the // DataMasking raises on missing fields (the default). ErrFieldNotFound = fmt.Errorf("%w: field or expression not found", ErrDataMasking) // ErrInvalidPath reports an expression outside the accepted JSONPath // dialect (see the package documentation). ErrInvalidPath = fmt.Errorf("%w: invalid masking path", ErrDataMasking) // ErrMaskField reports an Eraser that failed on a selected value with // an error of its own. ErrMaskField = fmt.Errorf("%w: failed to mask field at path", ErrDataMasking) // ErrInvalidRegex reports a regular expression, or a mask format, that // does not compile. ErrInvalidRegex = fmt.Errorf("%w: invalid regex pattern", ErrDataMasking) // ErrNoProvider reports Encrypt or Decrypt on a DataMasking built // without a provider: erasing needs none, encrypting does. ErrNoProvider = fmt.Errorf("%w: no encryption provider configured", ErrDataMasking) // ErrEncryptKey reports a provider that could not obtain or wrap a data // key, typically a KMS key that is not a symmetric key ARN the function // may use. ErrEncryptKey = fmt.Errorf("%w: failed to encrypt data, use a valid symmetric AWS KMS key ARN, not a key ID or alias", ErrDataMasking) // ErrDecryptKey reports a provider that could not decrypt any data key of // the message with the keys it was given. ErrDecryptKey = fmt.Errorf("%w: failed to decrypt data, use a valid symmetric AWS KMS key ARN, not a key ID or alias", ErrDataMasking) // ErrDecryptValue reports ciphertext that is not a message the provider // produced: bad base64, a malformed or tampered message, or a plaintext // that is not JSON. ErrDecryptValue = fmt.Errorf("%w: data decryption failed, decrypt only data that was previously encrypted", ErrDataMasking) // ErrContextMismatch reports a decryption whose encryption context is // not exactly the one the data was encrypted with. ErrContextMismatch = fmt.Errorf("%w: encryption context does not match, use the exact context used during encryption", ErrDataMasking) )
The failures the package and its providers report. Each one matches ErrDataMasking too.
var ErrDataMasking = errors.New("datamasking")
ErrDataMasking is what every error the package detects matches with errors.Is, as Python's DataMaskingError is the base of its exceptions.
Functions ¶
Types ¶
type CryptoOption ¶
type CryptoOption func(*Request)
CryptoOption configures one Encrypt or Decrypt.
func WithEncryptionContext ¶
func WithEncryptionContext(ec map[string]string) CryptoOption
WithEncryptionContext binds non-secret key-value pairs to the ciphertext (additional authenticated data). Decrypt must be given exactly the same pairs, or it fails with ErrContextMismatch. Never put secrets in it: AWS KMS logs it in CloudTrail.
func WithProviderOptions ¶
func WithProviderOptions(opts ...ProviderOption) CryptoOption
WithProviderOptions passes per-call settings to the provider: the Go form of Python's provider_options.
type DataMasking ¶
type DataMasking struct {
// contains filtered or unexported fields
}
DataMasking erases, encrypts and decrypts data. It holds no state of its own: build one at package level, or in main, and share it.
func New ¶
func New(opts ...Option) *DataMasking
New returns a DataMasking. By default it erases only, and fails on expressions that match nothing.
func (*DataMasking) Decrypt ¶
func (m *DataMasking) Decrypt(ctx context.Context, ciphertext string, opts ...CryptoOption) (any, error)
Decrypt decrypts what Encrypt returned and parses the JSON document into a decoded JSON value (nil, bool, int64, float64, string, []any, map[string]any).
func (*DataMasking) DecryptAs ¶
func (m *DataMasking) DecryptAs[T any](ctx context.Context, ciphertext string, opts ...CryptoOption) (T, error)
DecryptAs is Decrypt, with the document decoded into T; raw JSON targets ([]byte, jsontext.Value) receive the document itself.
func (*DataMasking) Encrypt ¶
func (m *DataMasking) Encrypt(ctx context.Context, data any, opts ...CryptoOption) (string, error)
Encrypt encrypts the JSON document of data with the provider and returns the provider's ciphertext (for the AWS Encryption SDK provider, a base64 encrypted message). Data is a Go value that marshals to JSON, or raw JSON ([]byte, jsontext.Value) encrypted as the document it is; a string is a JSON string, as in Python.
Example ¶
package main
import (
"context"
"fmt"
"github.com/alessandrolattao/powertools-lambda-go/datamasking"
)
// reverseProvider stands in for a real one, such as datamasking/kms.
type reverseProvider struct{}
func (reverseProvider) Encrypt(_ context.Context, plaintext []byte, _ datamasking.Request) (string, error) {
out := []byte(string(plaintext))
for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
out[i], out[j] = out[j], out[i]
}
return string(out), nil
}
func (p reverseProvider) Decrypt(ctx context.Context, ciphertext string, req datamasking.Request) ([]byte, error) {
plaintext, err := p.Encrypt(ctx, []byte(ciphertext), req)
return []byte(plaintext), err
}
func main() {
masker := datamasking.New(datamasking.WithProvider(reverseProvider{}))
ctx := context.Background()
ciphertext, err := masker.Encrypt(ctx, map[string]any{"card_number": "1111"})
if err != nil {
panic(err)
}
type card struct {
Number string `json:"card_number"`
}
c, err := masker.DecryptAs[card](ctx, ciphertext)
if err != nil {
panic(err)
}
fmt.Println(c.Number)
}
Output: 1111
func (*DataMasking) Erase ¶
func (m *DataMasking) Erase(data any, opts ...EraseOption) (any, error)
Erase masks data irreversibly and returns the result as a decoded JSON value (nil, bool, int64, float64, string, []any, map[string]any).
Data is raw JSON ([]byte, jsontext.Value), a Go value that marshals to JSON (a struct, a map, an aws-lambda-go event), or a scalar. A string is a JSON document when fields or rules are given, a plain value otherwise, as in Python.
Without options the whole data becomes Mask (an array becomes as many Mask strings). WithFields and WithMaskingRules select what to mask; the mask options say how (see Rule).
Example ¶
package main
import (
"encoding/json/v2"
"fmt"
"github.com/alessandrolattao/powertools-lambda-go/datamasking"
)
// jsonText renders a result as JSON with sorted keys.
func jsonText(v any) string {
out, err := json.Marshal(v, json.Deterministic(true))
if err != nil {
panic(err)
}
return string(out)
}
func main() {
masker := datamasking.New()
body := []byte(`{
"name": "John Doe",
"email": "johndoe@example.com",
"address": {"street": "123 Main St", "zip": "12345"},
"company_address": {"street": "456 ACME Ave", "zip": "12345"}
}`)
erased, err := masker.Erase(body, datamasking.WithFields("email", "address.street", "company_address"))
if err != nil {
panic(err)
}
fmt.Println(jsonText(erased))
}
Output: {"address":{"street":"*****","zip":"12345"},"company_address":"*****","email":"*****","name":"John Doe"}
Example (CustomMasks) ¶
package main
import (
"encoding/json/v2"
"fmt"
"github.com/alessandrolattao/powertools-lambda-go/datamasking"
)
// jsonText renders a result as JSON with sorted keys.
func jsonText(v any) string {
out, err := json.Marshal(v, json.Deterministic(true))
if err != nil {
panic(err)
}
return string(out)
}
func main() {
masker := datamasking.New()
body := `{"email": "johndoe@example.com", "address": {"street": "123 Main St", "zip": "12345"}}`
for _, mask := range []datamasking.EraseOption{
datamasking.WithDynamicMask(),
datamasking.WithCustomMask("XX"),
datamasking.WithRegex(`(.)(.*)(@.*)`, `\1****\3`),
} {
erased, err := masker.Erase(body, datamasking.WithFields("address.street", "address.zip", "email"), mask)
if err != nil {
panic(err)
}
fmt.Println(jsonText(erased))
}
}
Output: {"address":{"street":"*** **** **","zip":"*****"},"email":"***************.***"} {"address":{"street":"XX","zip":"XX"},"email":"XX"} {"address":{"street":"123 Main St","zip":"12345"},"email":"j****@example.com"}
Example (WholeData) ¶
package main
import (
"encoding/json/v2"
"fmt"
"github.com/alessandrolattao/powertools-lambda-go/datamasking"
)
// jsonText renders a result as JSON with sorted keys.
func jsonText(v any) string {
out, err := json.Marshal(v, json.Deterministic(true))
if err != nil {
panic(err)
}
return string(out)
}
func main() {
masker := datamasking.New()
erased, err := masker.Erase([]string{"name", "phone"})
if err != nil {
panic(err)
}
fmt.Println(jsonText(erased))
}
Output: ["*****","*****"]
func (*DataMasking) EraseAs ¶
func (m *DataMasking) EraseAs[T any](data any, opts ...EraseOption) (T, error)
EraseAs is Erase, with the result decoded into T: a struct whose erased fields are strings, a map, or raw JSON ([]byte, jsontext.Value), which receives the document itself.
Example ¶
package main
import (
"encoding/json/jsontext"
"fmt"
"github.com/alessandrolattao/powertools-lambda-go/datamasking"
)
func main() {
type customer struct {
Name string `json:"name"`
Email string `json:"email"`
}
masker := datamasking.New()
c, err := masker.EraseAs[customer](customer{Name: "Pat", Email: "pat@example.com"}, datamasking.WithFields("email"))
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", c)
raw, err := masker.EraseAs[jsontext.Value](`{"card_number": "1111 2222 3333 4444"}`, datamasking.WithFields("card_number"))
if err != nil {
panic(err)
}
fmt.Println(string(raw))
}
Output: {Name:Pat Email:*****} {"card_number":"*****"}
type EraseOption ¶
type EraseOption func(*eraseOptions)
EraseOption configures one Erase.
func WithCustomMask ¶
func WithCustomMask(mask string) EraseOption
WithCustomMask replaces the text with mask, cut to the text's length when longer.
func WithDynamicMask ¶
func WithDynamicMask() EraseOption
WithDynamicMask masks every character but '-', '_', '.' and space, keeping the length of the text.
func WithFields ¶
func WithFields(paths ...string) EraseOption
WithFields erases only the values the expressions select, in order, and keeps the rest of the data. An expression that selects an object or an array erases it whole, unless a mask option is set too.
func WithMaskingRules ¶
func WithMaskingRules(rules ...Rule) EraseOption
WithMaskingRules masks each rule's Path with the rule's own masks, rules applied in order. It takes precedence over every other option, as Python's masking_rules does. Values that are null are left alone.
Example ¶
package main
import (
"encoding/json/v2"
"fmt"
"github.com/alessandrolattao/powertools-lambda-go/datamasking"
)
// jsonText renders a result as JSON with sorted keys.
func jsonText(v any) string {
out, err := json.Marshal(v, json.Deterministic(true))
if err != nil {
panic(err)
}
return string(out)
}
func main() {
masker := datamasking.New()
body := []byte(`{
"age": 30,
"email": "janedoe@example.com",
"address": {"zip": "12345"},
"other_address": [{"postcode": 11345}, {"postcode": 67890}]
}`)
erased, err := masker.Erase(body, datamasking.WithMaskingRules(
datamasking.Rule{Path: "email", RegexPattern: `(.)(.*)(@.*)`, MaskFormat: `\1****\3`},
datamasking.Rule{Path: "age", DynamicMask: true},
datamasking.Rule{Path: "address.zip", CustomMask: "xxx"},
datamasking.Rule{Path: "$.other_address[?(@.postcode > 12000)]", CustomMask: "Masked"},
))
if err != nil {
panic(err)
}
fmt.Println(jsonText(erased))
}
Output: {"address":{"zip":"xxx"},"age":"**","email":"j****@example.com","other_address":[{"postcode":11345},"Masked"]}
func WithRegex ¶
func WithRegex(pattern, format string) EraseOption
WithRegex replaces what pattern (Go RE2 syntax) matches with format, where \1 or \g<name> refer to the pattern's groups, as in Python.
type Eraser ¶
Eraser is implemented by a Provider that masks values its own way: the Go form of a Python provider overriding erase. DataMasking hands it every selected value (with WithMaskingRules, the value as text, as Python does) and the rule that applies; MaskValue is the default it can fall back to.
type Option ¶
type Option func(*DataMasking)
Option configures a DataMasking.
func WithJSONOptions ¶
WithJSONOptions sets the encoding/json/v2 options used to marshal Go values (structs, maps) into JSON and to decode results into the caller's type: the Go form of Python's json_serializer and json_deserializer.
func WithProvider ¶
WithProvider sets the encryption provider, typically the AWS Encryption SDK one of package datamasking/kms. Without one, DataMasking only erases. A provider that also implements Eraser masks values its own way.
func WithRaiseOnMissingField ¶
WithRaiseOnMissingField says what to do when an expression of WithFields or WithMaskingRules matches nothing: fail with ErrFieldNotFound (true, the default), or skip the expression (false). Skipped expressions are reported on the Powertools debug log (POWERTOOLS_DEBUG), where Python emits a warning.
type PathError ¶
type PathError struct {
// Path is the expression, as the caller wrote it.
Path string
// Err is ErrFieldNotFound, ErrInvalidPath or ErrMaskField.
Err error
// Cause is what went wrong underneath, when there is more to say.
Cause error
}
PathError reports a failure tied to one field expression or masking rule.
type Provider ¶
type Provider interface {
// Encrypt encrypts plaintext and returns the ciphertext as text.
Encrypt(ctx context.Context, plaintext []byte, req Request) (string, error)
// Decrypt reverses Encrypt. It fails with ErrContextMismatch when
// req.EncryptionContext is not the one the data was encrypted with.
Decrypt(ctx context.Context, ciphertext string, req Request) ([]byte, error)
}
Provider encrypts and decrypts for DataMasking: the Go form of Python's BaseProvider. DataMasking turns the data into its JSON document before Encrypt and parses the document Decrypt returns, so a provider deals with bytes and its own ciphertext encoding only. Implementations must be safe for concurrent use.
type ProviderOption ¶
type ProviderOption interface {
// Provider names the provider the option is meant for.
Provider() string
}
ProviderOption is a per-call setting only one provider understands: the Go form of Python's provider_options. Each provider documents its own.
type Request ¶
type Request struct {
// EncryptionContext is the non-secret data bound to the ciphertext,
// which decryption must present again.
EncryptionContext map[string]string
// Options are the per-call settings of WithProviderOptions.
Options []ProviderOption
}
Request is what an Encrypt or a Decrypt hands the provider besides the data.
type Rule ¶
type Rule struct {
// Path selects the values to mask, in the dialect the package
// documentation describes. WithMaskingRules needs it; elsewhere it is
// the expression that selected the value, or empty for the whole data.
Path string
// DynamicMask masks every character but '-', '_', '.' and space.
DynamicMask bool
// CustomMask replaces the text, cut to the text's length.
CustomMask string
// RegexPattern is a regular expression in Go's RE2 syntax.
RegexPattern string
// MaskFormat replaces each match of RegexPattern; \1 and \g<name> refer
// to its groups.
MaskFormat string
}
Rule says how to mask the values an expression selects. With none of the masking fields set, a value is erased whole: it becomes Mask, and an array becomes as many Mask strings as it had elements. With any of them set, objects and arrays keep their shape and every scalar inside is masked as text:
- RegexPattern and MaskFormat together replace what the pattern matches with the format, which refers to groups as Python does (\1, \g<name>);
- otherwise CustomMask replaces the text, cut to the text's length when it is longer;
- otherwise each character becomes '*', except '-', '_', '.' and space (DynamicMask, or a lone RegexPattern or MaskFormat).
Scalars become text the way Python's str() writes them: 30 is "30", 4.2 is "4.2", true is "True"; null becomes Mask.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package kms is the AWS Encryption SDK provider of package datamasking, with AWS KMS keys: the Go port of Powertools for AWS Lambda (Python) AWSEncryptionSDKProvider.
|
Package kms is the AWS Encryption SDK provider of package datamasking, with AWS KMS keys: the Go port of Powertools for AWS Lambda (Python) AWSEncryptionSDKProvider. |