README
¶
Cardinality limit plugin
Limits the cardinality of fields on events, drops events or just do nothing.
Examples
Discarding events with high cardinality field:
pipelines:
example_pipeline:
...
- type: cardinality
limit: 2
action: discard
ttl: 1m
metric_prefix: service_client
key:
- service
fields:
- client_id
...
Events:
{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "2"}
{"service": "registration", "client_id": "3"} // will be discarded
Remove high cardinality fields:
pipelines:
example_pipeline:
...
- type: cardinality
limit: 2
action: remove_fields
ttl: 1m
metric_prefix: service_client
key:
- service
fields:
- client_id
...
The original events:
{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "2"}
{"service": "registration", "client_id": "3"}
The resulting events:
{"service": "registration", "client_id": "1"}
{"service": "registration", "client_id": "2"}
{"service": "registration"}
Config params
key []cfg.FieldSelector required
Fields used to group events before calculating cardinality. Events with the same key values are aggregated together. Required for proper cardinality tracking per logical group.
fields []cfg.FieldSelector required
Target fields whose unique values are counted within each key group. The plugin monitors how many distinct values these fields contain. Required to define what constitutes high cardinality.
action string default=nothing options=discard|remove_fields|nothing
Action to perform when cardinality limit is exceeded. Determines whether to discard events, remove fields, or just monitor. Choose based on whether you need to preserve other event data.
metric_prefix string
Prefix added to metric names for better organization. Useful when running multiple instances to avoid metric name collisions. Leave empty for default metric naming.
limit int default=10000
Maximum allowed number of unique values for monitored fields. When exceeded within a key group, the configured action triggers. Set based on expected diversity and system capacity.
ttl cfg.Duration default=1h
Time-to-live for cardinality tracking cache entries. Prevents unbounded memory growth by forgetting old unique values. Should align with typical patterns of field value changes.
Generated using insane-doc
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
func (*Cache) CountPrefix ¶
type Config ¶
type Config struct {
// > @3@4@5@6
// >
// > Fields used to group events before calculating cardinality.
// > Events with the same key values are aggregated together.
// > Required for proper cardinality tracking per logical group.
KeyFields []cfg.FieldSelector `json:"key" slice:"true" required:"true"` // *
// > @3@4@5@6
// >
// > Target fields whose unique values are counted within each key group.
// > The plugin monitors how many distinct values these fields contain.
// > Required to define what constitutes high cardinality.
Fields []cfg.FieldSelector `json:"fields" slice:"true" required:"true"` // *
// > @3@4@5@6
// >
// > Action to perform when cardinality limit is exceeded.
// > Determines whether to discard events, remove fields, or just monitor.
// > Choose based on whether you need to preserve other event data.
Action string `json:"action" default:"nothing" options:"discard|remove_fields|nothing"` // *
// > @3@4@5@6
// >
// > Prefix added to metric names for better organization.
// > Useful when running multiple instances to avoid metric name collisions.
// > Leave empty for default metric naming.
MetricPrefix string `json:"metric_prefix" default:""` // *
// > @3@4@5@6
// >
// > Maximum allowed number of unique values for monitored fields.
// > When exceeded within a key group, the configured action triggers.
// > Set based on expected diversity and system capacity.
Limit int `json:"limit" default:"10000"` // *
// > @3@4@5@6
// >
// > Time-to-live for cardinality tracking cache entries.
// > Prevents unbounded memory growth by forgetting old unique values.
// > Should align with typical patterns of field value changes.
TTL cfg.Duration `json:"ttl" default:"1h" parse:"duration"` // *
TTL_ time.Duration
}
! config-params ^ config-params
Source Files