rules

package
v0.0.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2025 License: Apache-2.0 Imports: 10 Imported by: 0

README

Transformation Rules

There are two types of transformation rules available:

  1. Regex Transform

    • Type: "regex"
    • Parameters:
      • pattern: The regular expression pattern to match
      • replace: The replacement string
    • Description: Applies a regular expression replacement on string values in the specified column.

  2. Mask Transform

    • Type: "mask"
    • Parameters:
      • mask_char: The character to use for masking
    • Description: Masks the content of string values, keeping the first and last characters visible and replacing the rest with the specified mask character.

Filtering Rules

Filtering rules use various comparison operators to determine whether a row should be included in the output. The available operators are:

  1. Equality ("eq")
  2. Inequality ("ne")
  3. Greater Than ("gt")
  4. Less Than ("lt")
  5. Greater Than or Equal To ("gte")
  6. Less Than or Equal To ("lte")
  7. Contains ("contains")

Rule Properties

Both transformation and filtering rules share these common properties:

  • type: Specifies whether it's a "transform" or "filter" rule.
  • column: The name of the column to apply the rule to.
  • operations: An array of operations (INSERT, UPDATE, DELETE) to which the rule should be applied. If not specified, it applies to all operations.
  • allow_empty_deletes: A boolean flag that, when set to true, allows the rule to process delete operations even if the column value is empty.

Additional Notes

  • The rules support various data types, including integers, floats, strings, timestamps, booleans, and numeric (decimal) values.
  • For filtering rules, the comparison is type-aware, ensuring that values are compared appropriately based on their data type.
  • The contains operator for filtering only works on string values.
  • Transformation rules currently only work on string values. If a non-string value is encountered, the transformation is skipped and a warning is logged.

To use these rules, you would define them in a YAML configuration file and specify the path to this file using the --rules-config flag when running pg_flo. The exact structure of the YAML file should match the rule properties and parameters described above.

Documentation

Overview

Package rules provides filtering and transformation rules for CDC messages.

Index

Constants

View Source 
const (
	// OpEqual represents the equality comparison operator
	OpEqual = "eq"
	// OpNotEqual represents the not-equal comparison operator
	OpNotEqual = "ne"
	// OpGreaterThan represents the greater-than comparison operator
	OpGreaterThan = "gt"
	// OpLessThan represents the less-than comparison operator
	OpLessThan = "lt"
	// OpGreaterThanOrEqual represents the greater-than-or-equal comparison operator
	OpGreaterThanOrEqual = "gte"
	// OpLessThanOrEqual represents the less-than-or-equal comparison operator
	OpLessThanOrEqual = "lte"
	// OpContains represents the contains comparison operator
	OpContains = "contains"
)

Variables

This section is empty.

Functions

func NewComparisonCondition 

func NewComparisonCondition(column, operator string, value interface{}) func(*utils.CDCMessage) bool

NewComparisonCondition creates a new comparison condition function using PostgreSQL semantics

func NewContainsCondition 

func NewContainsCondition(column string, value interface{}) func(*utils.CDCMessage) bool

NewContainsCondition creates a new contains condition function

Types

type Config 

type Config struct {
	Tables map[string][]RuleConfig `yaml:"tables"`
}

Config represents the overall configuration for rules

type ExcludeColumnRule added in v0.0.14 

type ExcludeColumnRule struct {
	TableName  string
	ColumnName string
}

ExcludeColumnRule removes specified columns from CDC messages

func (*ExcludeColumnRule) Apply added in v0.0.14 

func (r *ExcludeColumnRule) Apply(message *utils.CDCMessage) (*utils.CDCMessage, error)

Apply applies the exclude_column rule to the provided data

type FilterRule 

type FilterRule struct {
	TableName         string
	ColumnName        string
	Condition         func(*utils.CDCMessage) bool
	Operations        []utils.OperationType
	AllowEmptyDeletes bool
}

FilterRule represents a rule that filters data

func (*FilterRule) Apply 

func (r *FilterRule) Apply(message *utils.CDCMessage) (*utils.CDCMessage, error)

Apply applies the filter rule to the provided data

type Rule 

type Rule interface {
	Apply(message *utils.CDCMessage) (*utils.CDCMessage, error)
}

Rule interface defines the methods that all rules must implement

func NewExcludeColumnRule added in v0.0.14 

func NewExcludeColumnRule(table, column string) (Rule, error)

NewExcludeColumnRule creates a new exclude column rule for the specified table and column

func NewFilterRule 

func NewFilterRule(table, column string, params map[string]interface{}) (Rule, error)

NewFilterRule creates a new filter rule based on the provided parameters

func NewTransformRule 

func NewTransformRule(table, column string, params map[string]interface{}) (Rule, error)

NewTransformRule creates a new transform rule based on the provided parameters

type RuleConfig 

type RuleConfig struct {
	Type              string                 `yaml:"type"`
	Column            string                 `yaml:"column"`
	Parameters        map[string]interface{} `yaml:"parameters"`
	Operations        []utils.OperationType  `yaml:"operations,omitempty"`
	AllowEmptyDeletes bool                   `yaml:"allow_empty_deletes,omitempty"`
}

RuleConfig represents the configuration for a single rule

type RuleEngine 

type RuleEngine struct {
	Rules map[string][]Rule // map of table name to slice of rules
	// contains filtered or unexported fields
}

RuleEngine manages and applies rules to data

func NewRuleEngine 

func NewRuleEngine() *RuleEngine

NewRuleEngine creates a new RuleEngine instance

func (*RuleEngine) AddRule 

func (re *RuleEngine) AddRule(tableName string, rule Rule)

AddRule adds a new rule for the specified table

func (*RuleEngine) ApplyRules 

func (re *RuleEngine) ApplyRules(message *utils.CDCMessage) (*utils.CDCMessage, error)

ApplyRules applies all rules for the specified table to the given data

func (*RuleEngine) LoadRules 

func (re *RuleEngine) LoadRules(config Config) error

LoadRules loads rules from the provided configuration

type TransformRule 

type TransformRule struct {
	TableName         string
	ColumnName        string
	Transform         func(*utils.CDCMessage) (*utils.CDCMessage, error)
	Operations        []utils.OperationType
	AllowEmptyDeletes bool
}

TransformRule represents a rule that transforms data

func NewMaskTransformRule 

func NewMaskTransformRule(table, column string, params map[string]interface{}) (*TransformRule, error)

NewMaskTransformRule creates a new mask transform rule

func NewRegexTransformRule 

func NewRegexTransformRule(table, column string, params map[string]interface{}) (*TransformRule, error)

NewRegexTransformRule creates a new regex transform rule

func (*TransformRule) Apply 

func (r *TransformRule) Apply(message *utils.CDCMessage) (*utils.CDCMessage, error)

Apply applies the transform rule to the provided data

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.