Documentation
¶
Overview ¶
Package kprom provides prometheus plug-in metrics for a kgo client.
This package tracks the following metrics under the following names, all metrics being counter vecs:
#{ns}_connects_total{node_id="#{node}"}
#{ns}_connect_errors_total{node_id="#{node}"}
#{ns}_write_errors_total{node_id="#{node}"}
#{ns}_write_bytes_total{node_id="#{node}"}
#{ns}_read_errors_total{node_id="#{node}"}
#{ns}_read_bytes_total{node_id="#{node}"}
#{ns}_produce_bytes_total{node_id="#{node}",topic="#{topic}"}
#{ns}_fetch_bytes_total{node_id="#{node}",topic="#{topic}"}
#{ns}_buffered_produce_records_total
#{ns}_buffered_produce_bytes_total
#{ns}_buffered_fetch_records_total
#{ns}_buffered_fetch_bytes_total
The above metrics can be expanded considerably with options in this package, allowing timings, uncompressed and compressed bytes, and different labels.
The labels on broker-level metrics can be configured with the BrokerLabels option to reduce cardinality or add dimensions like host or rack.
This can be used in a client like so:
m := kprom.NewMetrics("my_namespace")
cl, err := kgo.NewClient(
kgo.WithHooks(m),
// ...other opts
)
More examples are linked in the main project readme: https://github.com/twmb/franz-go/#metrics--logging
By default, metrics are installed under the a new prometheus registry, but this can be overridden with the Registry option.
Note that seed brokers use broker IDs prefixed with "seed_", with the number corresponding to which seed it is.
Index ¶
- Constants
- Variables
- type BrokerLabel
- type Detail
- type Histogram
- type HistogramOpts
- type Metrics
- func (m *Metrics) Collect(ch chan<- prometheus.Metric)
- func (m *Metrics) Describe(ch chan<- *prometheus.Desc)
- func (m *Metrics) Handler() http.Handler
- func (m *Metrics) OnBrokerConnect(meta kgo.BrokerMetadata, _ time.Duration, _ net.Conn, err error)
- func (m *Metrics) OnBrokerDisconnect(meta kgo.BrokerMetadata, _ net.Conn)
- func (m *Metrics) OnBrokerE2E(meta kgo.BrokerMetadata, _ int16, e2e kgo.BrokerE2E)
- func (m *Metrics) OnBrokerRead(meta kgo.BrokerMetadata, _ int16, bytesRead int, _, _ time.Duration, err error)
- func (m *Metrics) OnBrokerThrottle(meta kgo.BrokerMetadata, throttleInterval time.Duration, _ bool)
- func (m *Metrics) OnBrokerWrite(meta kgo.BrokerMetadata, _ int16, bytesWritten int, _, _ time.Duration, ...)
- func (m *Metrics) OnClientClosed(*kgo.Client)
- func (m *Metrics) OnFetchBatchRead(meta kgo.BrokerMetadata, topic string, _ int32, metrics kgo.FetchBatchMetrics)
- func (m *Metrics) OnNewClient(client *kgo.Client)
- func (m *Metrics) OnProduceBatchWritten(meta kgo.BrokerMetadata, topic string, _ int32, ...)
- func (m *Metrics) Registry() prometheus.Registerer
- type Opt
- func BrokerLabels(labels ...BrokerLabel) Opt
- func Buckets(buckets []float64) Opt
- func FetchAndProduceDetail(details ...Detail) Opt
- func Gatherer(gatherer prometheus.Gatherer) Opt
- func GoCollectors() Opt
- func HandlerOpts(opts promhttp.HandlerOpts) Opt
- func Histograms(hs ...Histogram) Opt
- func HistogramsFromOpts(hs ...HistogramOpts) Opt
- func NativeBucketFactor(factor float64) Opt
- func NativeBucketMinReset(d time.Duration) Opt
- func NativeBucketsOnly() Opt
- func NativeMaxBuckets(n uint32) Opt
- func Registerer(reg prometheus.Registerer) Opt
- func Registry(rg RegistererGatherer) Opt
- func Subsystem(ss string) Opt
- func WithClientLabel() Opt
- func WithStaticLabel(labels prometheus.Labels) Opt
- type RegistererGatherer
Constants ¶
const ( DefNativeBucketFactor = 1.1 DefNativeMaxBuckets = 100 DefNativeBucketMinReset = time.Hour )
Default native histogram parameters, used by all kprom histograms unless overridden with the Native* options.
Variables ¶
var DefBuckets = []float64{0.001, 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1.024, 2.048}
DefBuckets are the default Histogram buckets. The default buckets are tailored to broadly measure the kafka timings (in seconds).
Functions ¶
This section is empty.
Types ¶
type BrokerLabel ¶ added in v1.4.0
type BrokerLabel uint8
A BrokerLabel is a label that can be set on broker-level metrics.
const ( BrokerNodeID BrokerLabel = iota // Include "node_id" label on broker metrics. BrokerHost // Include "host" label on broker metrics. BrokerRack // Include "rack" label on broker metrics. )
type Detail ¶ added in v1.1.0
type Detail uint8
A Detail is a label that can be set on fetch/produce metrics.
const ( ByNode Detail = iota // Include label "node_id" for fetch and produce metrics. ByTopic // Include label "topic" for fetch and produce metrics. Batches // Report number of fetched and produced batches. Records // Report the number of fetched and produced records. CompressedBytes // Report the number of fetched and produced compressed bytes. UncompressedBytes // Report the number of fetched and produced uncompressed bytes. ConsistentNaming // Renames {fetch,produce}_bytes_total to {fetch,produce}_uncompressed_bytes_total, making the names consistent with the CompressedBytes detail. )
type Histogram ¶ added in v1.1.0
type Histogram uint8
A Histogram is an identifier for a kprom histogram that can be enabled
const ( ReadWait Histogram = iota // Enables {ns}_{ss}_read_wait_seconds. ReadTime // Enables {ns}_{ss}_read_time_seconds. WriteWait // Enables {ns}_{ss}_write_wait_seconds. WriteTime // Enables {ns}_{ss}_write_time_seconds. RequestDurationE2E // Enables {ns}_{ss}_request_durationE2E_seconds. RequestThrottled // Enables {ns}_{ss}_request_throttled_seconds. )
type HistogramOpts ¶ added in v1.1.0
HistogramOpts allows histograms to be enabled with custom buckets
type Metrics ¶
type Metrics struct {
// contains filtered or unexported fields
}
Metrics provides prometheus metrics
func NewMetrics ¶
NewMetrics returns a new Metrics that adds prometheus metrics to the registry under the given namespace.
func (*Metrics) Collect ¶ added in v1.2.0
func (m *Metrics) Collect(ch chan<- prometheus.Metric)
Collect returns the current state of all metrics of the collector. Collect & Describe will allow applications that use kprom to register multiple collectors to the same registry by inverting the dependency. This follows the recommended approach of collector implementation https://github.com/prometheus/client_golang/blob/main/prometheus/collector.go#L16-L18
func (*Metrics) Describe ¶ added in v1.2.0
func (m *Metrics) Describe(ch chan<- *prometheus.Desc)
Describe returns all descriptions of the collector.
func (*Metrics) OnBrokerConnect ¶
OnBrokerConnect implements the HookBrokerConnect interface for metrics gathering. This method is meant to be called by the hook system and not by the user
func (*Metrics) OnBrokerDisconnect ¶
func (m *Metrics) OnBrokerDisconnect(meta kgo.BrokerMetadata, _ net.Conn)
OnBrokerDisconnect implements the HookBrokerDisconnect interface for metrics gathering. This method is meant to be called by the hook system and not by the user
func (*Metrics) OnBrokerE2E ¶ added in v1.1.0
OnBrokerE2E implements the HookBrokerE2E interface for metrics gathering This method is meant to be called by the hook system and not by the user
func (*Metrics) OnBrokerRead ¶
func (m *Metrics) OnBrokerRead(meta kgo.BrokerMetadata, _ int16, bytesRead int, _, _ time.Duration, err error)
// Nop hook for compat, logic moved to OnBrokerE2E
func (*Metrics) OnBrokerThrottle ¶ added in v1.1.0
OnBrokerThrottle implements the HookBrokerThrottle interface for metrics gathering. This method is meant to be called by the hook system and not by the user
func (*Metrics) OnBrokerWrite ¶
func (m *Metrics) OnBrokerWrite(meta kgo.BrokerMetadata, _ int16, bytesWritten int, _, _ time.Duration, err error)
Nop hook for compat, logic moved to OnBrokerE2E
func (*Metrics) OnClientClosed ¶ added in v1.1.0
OnClientClosed will unregister kprom metrics from kprom registerer
func (*Metrics) OnFetchBatchRead ¶
func (m *Metrics) OnFetchBatchRead(meta kgo.BrokerMetadata, topic string, _ int32, metrics kgo.FetchBatchMetrics)
OnFetchBatchRead implements the HookFetchBatchRead interface for metrics gathering. This method is meant to be called by the hook system and not by the user
func (*Metrics) OnNewClient ¶ added in v0.3.0
OnNewClient implements the HookNewClient interface for metrics gathering. This method is meant to be called by the hook system and not by the user
func (*Metrics) OnProduceBatchWritten ¶
func (m *Metrics) OnProduceBatchWritten(meta kgo.BrokerMetadata, topic string, _ int32, metrics kgo.ProduceBatchMetrics)
OnProduceBatchWritten implements the HookProduceBatchWritten interface for metrics gathering. This method is meant to be called by the hook system and not by the user
func (*Metrics) Registry ¶
func (m *Metrics) Registry() prometheus.Registerer
Registry returns the prometheus registry that metrics were added to.
This is useful if you want the Metrics type to create its own registry for you to add additional metrics to.
type Opt ¶
type Opt interface {
// contains filtered or unexported methods
}
Opt is an option to configure Metrics.
func BrokerLabels ¶ added in v1.4.0
func BrokerLabels(labels ...BrokerLabel) Opt
BrokerLabels configures which labels are included on broker-level connection, read, write, and request metrics, overriding the default of (BrokerNodeID). Calling with no arguments removes all broker-level labels, aggregating metrics across all brokers:
kprom.BrokerLabels( /* none */ )
This is useful for reducing metrics cardinality in environments with many or frequently changing broker IDs.
func Buckets ¶ added in v1.1.0
Buckets sets the buckets to be used with Histograms, overriding the default of kprom.DefBuckets. If custom buckets per histogram is needed, HistogramOpts can be used.
func FetchAndProduceDetail ¶ added in v1.1.0
FetchAndProduceDetail determines details for fetch/produce metrics, overriding the default of (UncompressedBytes, ByTopic, ByNode).
func Gatherer ¶ added in v0.2.0
func Gatherer(gatherer prometheus.Gatherer) Opt
Gatherer sets the gatherer to add gather to, rather than a new registry.
func GoCollectors ¶
func GoCollectors() Opt
GoCollectors adds the prometheus.NewProcessCollector and prometheus.NewGoCollector collectors the Metric's registry.
func HandlerOpts ¶
func HandlerOpts(opts promhttp.HandlerOpts) Opt
HandlerOpts sets handler options to use if you wish you use the Metrics.Handler function.
This is only useful if you both (a) do not want to provide your own registry and (b) want to override the default handler options.
func Histograms ¶ added in v1.1.0
Histograms sets the histograms to be enabled for kprom, overiding the default of disabling all histograms.
metrics, _ := kprom.NewMetrics( kprom.Histograms( kprom.RequestDurationE2E, ), )
func HistogramsFromOpts ¶ added in v1.1.0
func HistogramsFromOpts(hs ...HistogramOpts) Opt
HistogramsFromOpts allows the user full control of what histograms to enable and define buckets to be used with each histogram.
metrics, _ := kprom.NewMetrics(
kprom.HistogramsFromOpts(
kprom.HistogramOpts{
Enable: kprom.ReadWait,
Buckets: prometheus.LinearBuckets(10, 10, 8),
},
kprom.HistogramOpts{
Enable: kprom.ReadTime,
// kprom default bucket will be used
},
),
)
func NativeBucketFactor ¶ added in v1.5.0
NativeBucketFactor sets the bucket factor used by all kprom native histograms, overriding the default of DefNativeBucketFactor. It controls the resolution of native histogram buckets: the growth factor between consecutive buckets is at most this value. It must be greater than 1 to enable native histograms; a value of 1 or less disables native histograms, leaving only the classic histograms.
func NativeBucketMinReset ¶ added in v1.5.0
NativeBucketMinReset sets the minimum duration that must pass before kprom native histogram buckets may be reset to keep them within the max bucket number, overriding the default of DefNativeBucketMinReset.
func NativeBucketsOnly ¶ added in v1.5.0
func NativeBucketsOnly() Opt
NativeBucketsOnly stops kprom from emitting classic (fixed-bucket) histograms, leaving only native histograms. Native histograms must be enabled for this to work: if the native bucket factor is 1 or less (see NativeBucketFactor), prometheus silently falls back to its own default classic buckets (prometheus.DefBuckets), which are neither the kprom defaults nor tailored to Kafka timings.
func NativeMaxBuckets ¶ added in v1.5.0
NativeMaxBuckets sets the maximum number of buckets used by all kprom native histograms, overriding the default of DefNativeMaxBuckets. Once exceeded, buckets are widened to keep cardinality bounded. A value of 0 means no limit.
func Registerer ¶ added in v0.2.0
func Registerer(reg prometheus.Registerer) Opt
Registerer sets the registerer to add register to, rather than a new registry.
func Registry ¶
func Registry(rg RegistererGatherer) Opt
Registry sets the registerer and gatherer to add metrics to, rather than a new registry. Use this option if you want to configure both Gatherer and Registerer with the same object. Additionally, passing a `nil` RegistererGatherer allows registering an external registry, exposing kprom as a custom prometheus collector, this option is mutually exclusive with GoCollectors
func Subsystem ¶ added in v1.1.0
Subsystem sets the subsystem for the kprom metrics, overriding the default empty string.
func WithClientLabel ¶ added in v1.1.0
func WithClientLabel() Opt
WithClientLabel adds a "cliend_id" label to all metrics.
func WithStaticLabel ¶ added in v1.2.0
func WithStaticLabel(labels prometheus.Labels) Opt
WithStaticLabel adds a static label to all metrics.
type RegistererGatherer ¶ added in v0.2.0
type RegistererGatherer interface {
prometheus.Registerer
prometheus.Gatherer
}