kprom

package module
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 1, 2026 License: BSD-3-Clause Imports: 10 Imported by: 17

README

kprom

kprom is a plug-in package to provide prometheus metrics through a kgo.Hook.

This package tracks the following metrics under the following names, all metrics being counter vecs:

#{ns}_connects_total{node_id="#{node}"}
#{ns}_connect_errors_total{node_id="#{node}"}
#{ns}_write_errors_total{node_id="#{node}"}
#{ns}_write_bytes_total{node_id="#{node}"}
#{ns}_read_errors_total{node_id="#{node}"}
#{ns}_read_bytes_total{node_id="#{node}"}
#{ns}_produce_bytes_total{node_id="#{node}",topic="#{topic}"}
#{ns}_fetch_bytes_total{node_id="#{node}",topic="#{topic}"}
#{ns}_buffered_produce_records_total
#{ns}_buffered_fetch_records_total

The above metrics can be expanded considerably with options in this package, allowing timings, uncompressed and compressed bytes, and different labels.

Note that seed brokers use broker IDs prefixed with "seed_", with the number corresponding to which seed it is.

To use,

metrics := kprom.NewMetrics("namespace")
cl, err := kgo.NewClient(
	kgo.WithHooks(metrics),
	// ...other opts
)

You can use your own prometheus registry, as well as a few other options. See the package documentation for more info!

Documentation

Overview

Package kprom provides prometheus plug-in metrics for a kgo client.

This package tracks the following metrics under the following names, all metrics being counter vecs:

#{ns}_connects_total{node_id="#{node}"}
#{ns}_connect_errors_total{node_id="#{node}"}
#{ns}_write_errors_total{node_id="#{node}"}
#{ns}_write_bytes_total{node_id="#{node}"}
#{ns}_read_errors_total{node_id="#{node}"}
#{ns}_read_bytes_total{node_id="#{node}"}
#{ns}_produce_bytes_total{node_id="#{node}",topic="#{topic}"}
#{ns}_fetch_bytes_total{node_id="#{node}",topic="#{topic}"}
#{ns}_buffered_produce_records_total
#{ns}_buffered_produce_bytes_total
#{ns}_buffered_fetch_records_total
#{ns}_buffered_fetch_bytes_total

The above metrics can be expanded considerably with options in this package, allowing timings, uncompressed and compressed bytes, and different labels.

The labels on broker-level metrics can be configured with the BrokerLabels option to reduce cardinality or add dimensions like host or rack.

This can be used in a client like so:

m := kprom.NewMetrics("my_namespace")
cl, err := kgo.NewClient(
        kgo.WithHooks(m),
        // ...other opts
)

More examples are linked in the main project readme: https://github.com/twmb/franz-go/#metrics--logging

By default, metrics are installed under the a new prometheus registry, but this can be overridden with the Registry option.

Note that seed brokers use broker IDs prefixed with "seed_", with the number corresponding to which seed it is.

Index

Constants

View Source
const (
	DefNativeBucketFactor   = 1.1
	DefNativeMaxBuckets     = 100
	DefNativeBucketMinReset = time.Hour
)

Default native histogram parameters, used by all kprom histograms unless overridden with the Native* options.

Variables

View Source
var DefBuckets = []float64{0.001, 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1.024, 2.048}

DefBuckets are the default Histogram buckets. The default buckets are tailored to broadly measure the kafka timings (in seconds).

Functions

This section is empty.

Types

type BrokerLabel added in v1.4.0

type BrokerLabel uint8

A BrokerLabel is a label that can be set on broker-level metrics.

const (
	BrokerNodeID BrokerLabel = iota // Include "node_id" label on broker metrics.
	BrokerHost                      // Include "host" label on broker metrics.
	BrokerRack                      // Include "rack" label on broker metrics.
)

type Detail added in v1.1.0

type Detail uint8

A Detail is a label that can be set on fetch/produce metrics.

const (
	ByNode            Detail = iota // Include label "node_id" for fetch and produce metrics.
	ByTopic                         // Include label "topic" for fetch and produce metrics.
	Batches                         // Report number of fetched and produced batches.
	Records                         // Report the number of fetched and produced records.
	CompressedBytes                 // Report the number of fetched and produced compressed bytes.
	UncompressedBytes               // Report the number of fetched and produced uncompressed bytes.
	ConsistentNaming                // Renames {fetch,produce}_bytes_total to {fetch,produce}_uncompressed_bytes_total, making the names consistent with the CompressedBytes detail.
)

type Histogram added in v1.1.0

type Histogram uint8

A Histogram is an identifier for a kprom histogram that can be enabled

const (
	ReadWait           Histogram = iota // Enables {ns}_{ss}_read_wait_seconds.
	ReadTime                            // Enables {ns}_{ss}_read_time_seconds.
	WriteWait                           // Enables {ns}_{ss}_write_wait_seconds.
	WriteTime                           // Enables {ns}_{ss}_write_time_seconds.
	RequestDurationE2E                  // Enables {ns}_{ss}_request_durationE2E_seconds.
	RequestThrottled                    // Enables {ns}_{ss}_request_throttled_seconds.
)

type HistogramOpts added in v1.1.0

type HistogramOpts struct {
	Enable  Histogram
	Buckets []float64
}

HistogramOpts allows histograms to be enabled with custom buckets

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics provides prometheus metrics

func NewMetrics

func NewMetrics(namespace string, opts ...Opt) *Metrics

NewMetrics returns a new Metrics that adds prometheus metrics to the registry under the given namespace.

func (*Metrics) Collect added in v1.2.0

func (m *Metrics) Collect(ch chan<- prometheus.Metric)

Collect returns the current state of all metrics of the collector. Collect & Describe will allow applications that use kprom to register multiple collectors to the same registry by inverting the dependency. This follows the recommended approach of collector implementation https://github.com/prometheus/client_golang/blob/main/prometheus/collector.go#L16-L18

func (*Metrics) Describe added in v1.2.0

func (m *Metrics) Describe(ch chan<- *prometheus.Desc)

Describe returns all descriptions of the collector.

func (*Metrics) Handler

func (m *Metrics) Handler() http.Handler

Handler returns an http.Handler providing prometheus metrics.

func (*Metrics) OnBrokerConnect

func (m *Metrics) OnBrokerConnect(meta kgo.BrokerMetadata, _ time.Duration, _ net.Conn, err error)

OnBrokerConnect implements the HookBrokerConnect interface for metrics gathering. This method is meant to be called by the hook system and not by the user

func (*Metrics) OnBrokerDisconnect

func (m *Metrics) OnBrokerDisconnect(meta kgo.BrokerMetadata, _ net.Conn)

OnBrokerDisconnect implements the HookBrokerDisconnect interface for metrics gathering. This method is meant to be called by the hook system and not by the user

func (*Metrics) OnBrokerE2E added in v1.1.0

func (m *Metrics) OnBrokerE2E(meta kgo.BrokerMetadata, _ int16, e2e kgo.BrokerE2E)

OnBrokerE2E implements the HookBrokerE2E interface for metrics gathering This method is meant to be called by the hook system and not by the user

func (*Metrics) OnBrokerRead

func (m *Metrics) OnBrokerRead(meta kgo.BrokerMetadata, _ int16, bytesRead int, _, _ time.Duration, err error)

// Nop hook for compat, logic moved to OnBrokerE2E

func (*Metrics) OnBrokerThrottle added in v1.1.0

func (m *Metrics) OnBrokerThrottle(meta kgo.BrokerMetadata, throttleInterval time.Duration, _ bool)

OnBrokerThrottle implements the HookBrokerThrottle interface for metrics gathering. This method is meant to be called by the hook system and not by the user

func (*Metrics) OnBrokerWrite

func (m *Metrics) OnBrokerWrite(meta kgo.BrokerMetadata, _ int16, bytesWritten int, _, _ time.Duration, err error)

Nop hook for compat, logic moved to OnBrokerE2E

func (*Metrics) OnClientClosed added in v1.1.0

func (m *Metrics) OnClientClosed(*kgo.Client)

OnClientClosed will unregister kprom metrics from kprom registerer

func (*Metrics) OnFetchBatchRead

func (m *Metrics) OnFetchBatchRead(meta kgo.BrokerMetadata, topic string, _ int32, metrics kgo.FetchBatchMetrics)

OnFetchBatchRead implements the HookFetchBatchRead interface for metrics gathering. This method is meant to be called by the hook system and not by the user

func (*Metrics) OnNewClient added in v0.3.0

func (m *Metrics) OnNewClient(client *kgo.Client)

OnNewClient implements the HookNewClient interface for metrics gathering. This method is meant to be called by the hook system and not by the user

func (*Metrics) OnProduceBatchWritten

func (m *Metrics) OnProduceBatchWritten(meta kgo.BrokerMetadata, topic string, _ int32, metrics kgo.ProduceBatchMetrics)

OnProduceBatchWritten implements the HookProduceBatchWritten interface for metrics gathering. This method is meant to be called by the hook system and not by the user

func (*Metrics) Registry

func (m *Metrics) Registry() prometheus.Registerer

Registry returns the prometheus registry that metrics were added to.

This is useful if you want the Metrics type to create its own registry for you to add additional metrics to.

type Opt

type Opt interface {
	// contains filtered or unexported methods
}

Opt is an option to configure Metrics.

func BrokerLabels added in v1.4.0

func BrokerLabels(labels ...BrokerLabel) Opt

BrokerLabels configures which labels are included on broker-level connection, read, write, and request metrics, overriding the default of (BrokerNodeID). Calling with no arguments removes all broker-level labels, aggregating metrics across all brokers:

kprom.BrokerLabels( /* none */ )

This is useful for reducing metrics cardinality in environments with many or frequently changing broker IDs.

func Buckets added in v1.1.0

func Buckets(buckets []float64) Opt

Buckets sets the buckets to be used with Histograms, overriding the default of kprom.DefBuckets. If custom buckets per histogram is needed, HistogramOpts can be used.

func FetchAndProduceDetail added in v1.1.0

func FetchAndProduceDetail(details ...Detail) Opt

FetchAndProduceDetail determines details for fetch/produce metrics, overriding the default of (UncompressedBytes, ByTopic, ByNode).

func Gatherer added in v0.2.0

func Gatherer(gatherer prometheus.Gatherer) Opt

Gatherer sets the gatherer to add gather to, rather than a new registry.

func GoCollectors

func GoCollectors() Opt

GoCollectors adds the prometheus.NewProcessCollector and prometheus.NewGoCollector collectors the Metric's registry.

func HandlerOpts

func HandlerOpts(opts promhttp.HandlerOpts) Opt

HandlerOpts sets handler options to use if you wish you use the Metrics.Handler function.

This is only useful if you both (a) do not want to provide your own registry and (b) want to override the default handler options.

func Histograms added in v1.1.0

func Histograms(hs ...Histogram) Opt

Histograms sets the histograms to be enabled for kprom, overiding the default of disabling all histograms.

metrics, _ := kprom.NewMetrics(
	kprom.Histograms(
		kprom.RequestDurationE2E,
	),
)

func HistogramsFromOpts added in v1.1.0

func HistogramsFromOpts(hs ...HistogramOpts) Opt

HistogramsFromOpts allows the user full control of what histograms to enable and define buckets to be used with each histogram.

metrics, _ := kprom.NewMetrics(
 kprom.HistogramsFromOpts(
 	kprom.HistogramOpts{
 		Enable:  kprom.ReadWait,
 		Buckets: prometheus.LinearBuckets(10, 10, 8),
 	},
 	kprom.HistogramOpts{
 		Enable: kprom.ReadTime,
 		// kprom default bucket will be used
 	},
 ),
)

func NativeBucketFactor added in v1.5.0

func NativeBucketFactor(factor float64) Opt

NativeBucketFactor sets the bucket factor used by all kprom native histograms, overriding the default of DefNativeBucketFactor. It controls the resolution of native histogram buckets: the growth factor between consecutive buckets is at most this value. It must be greater than 1 to enable native histograms; a value of 1 or less disables native histograms, leaving only the classic histograms.

func NativeBucketMinReset added in v1.5.0

func NativeBucketMinReset(d time.Duration) Opt

NativeBucketMinReset sets the minimum duration that must pass before kprom native histogram buckets may be reset to keep them within the max bucket number, overriding the default of DefNativeBucketMinReset.

func NativeBucketsOnly added in v1.5.0

func NativeBucketsOnly() Opt

NativeBucketsOnly stops kprom from emitting classic (fixed-bucket) histograms, leaving only native histograms. Native histograms must be enabled for this to work: if the native bucket factor is 1 or less (see NativeBucketFactor), prometheus silently falls back to its own default classic buckets (prometheus.DefBuckets), which are neither the kprom defaults nor tailored to Kafka timings.

func NativeMaxBuckets added in v1.5.0

func NativeMaxBuckets(n uint32) Opt

NativeMaxBuckets sets the maximum number of buckets used by all kprom native histograms, overriding the default of DefNativeMaxBuckets. Once exceeded, buckets are widened to keep cardinality bounded. A value of 0 means no limit.

func Registerer added in v0.2.0

func Registerer(reg prometheus.Registerer) Opt

Registerer sets the registerer to add register to, rather than a new registry.

func Registry

func Registry(rg RegistererGatherer) Opt

Registry sets the registerer and gatherer to add metrics to, rather than a new registry. Use this option if you want to configure both Gatherer and Registerer with the same object. Additionally, passing a `nil` RegistererGatherer allows registering an external registry, exposing kprom as a custom prometheus collector, this option is mutually exclusive with GoCollectors

func Subsystem added in v1.1.0

func Subsystem(ss string) Opt

Subsystem sets the subsystem for the kprom metrics, overriding the default empty string.

func WithClientLabel added in v1.1.0

func WithClientLabel() Opt

WithClientLabel adds a "cliend_id" label to all metrics.

func WithStaticLabel added in v1.2.0

func WithStaticLabel(labels prometheus.Labels) Opt

WithStaticLabel adds a static label to all metrics.

type RegistererGatherer added in v0.2.0

type RegistererGatherer interface {
	prometheus.Registerer
	prometheus.Gatherer
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL