monitoring

package
v1.39.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 3, 2026 License: BSD-3-Clause Imports: 24 Imported by: 1

Documentation

Index

Constants

View Source 
const (
	DefaultMetricsNamespace = "weaviate"
)

Variables

View Source 
var (

	// LatencyBuckets is default histogram bucket for response time (in seconds).
	// It also includes request that served *very* fast and *very* slow
	LatencyBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 25, 50, 100}

	// RequestLatencyBuckets prepends sub-ms boundaries (down to 100µs) to
	// LatencyBuckets for the HTTP/gRPC request-duration histograms, so
	// histogram_quantile resolves sub-5ms requests instead of collapsing them into
	// one bucket. Derived from LatencyBuckets so the shared tail can't drift.
	RequestLatencyBuckets = append([]float64{.0001, .00025, .0005, .001, .0025}, LatencyBuckets...)
)
View Source 
var NoopRegisterer prometheus.Registerer = noopRegisterer{}

NoopRegisterer is a no-op Prometheus register.

Functions

func AddTracingToGRPCOptions added in v1.35.0 

func AddTracingToGRPCOptions(options []grpc.ServerOption, logger logrus.FieldLogger) []grpc.ServerOption

AddTracingToGRPCOptions adds tracing interceptors to gRPC server options

func AddTracingToHTTPMiddleware added in v1.35.0 

func AddTracingToHTTPMiddleware(next http.Handler, logger logrus.FieldLogger) http.Handler

AddTracingToHTTPMiddleware adds tracing to the existing HTTP middleware chain

func CountingListener added in v1.27.12 

func CountingListener(l net.Listener, g prometheus.Gauge) net.Listener

func EnsureRegisteredMetric added in v1.31.17 

func EnsureRegisteredMetric[T prometheus.Collector](reg prometheus.Registerer, metric T) (T, bool, error)

EnsureRegisteredMetric tries to register the given metric with the given registerer. If the metric is already registered, it returns the existing metric.

func GRPCStreamTracingInterceptor added in v1.35.0 

func GRPCStreamTracingInterceptor() grpc.StreamServerInterceptor

GRPCStreamTracingInterceptor creates a gRPC stream interceptor that adds OpenTelemetry tracing

func GRPCTracingInterceptor added in v1.35.0 

func GRPCTracingInterceptor() grpc.UnaryServerInterceptor

GRPCTracingInterceptor creates a gRPC interceptor that adds OpenTelemetry tracing

func HTTPTracingMiddleware added in v1.35.0 

func HTTPTracingMiddleware(next http.Handler) http.Handler

HTTPTracingMiddleware creates a middleware that adds OpenTelemetry tracing to HTTP requests

func InitConfig added in v1.19.7 

func InitConfig(cfg Config)

func InitCounterVec added in v1.31.17 

func InitCounterVec(vec *prometheus.CounterVec, labelNames [][]string)

func InitGaugeVec added in v1.31.17 

func InitGaugeVec(vec *prometheus.GaugeVec, labelNames [][]string)

func InstrumentGrpc added in v1.27.10 

func InstrumentGrpc(svrMetrics *GRPCServerMetrics) []grpc.ServerOption

InstrumentGrpc accepts server metrics and returns the few `[]grpc.ServerOption` which you can then wrap it with any `grpc.Server` to get these metrics instrumented automatically.

``` 

svrMetrics := monitoring.NewGRPCServerMetrics(metrics, prometheus.DefaultRegisterer)
grpcServer := grpc.NewServer(monitoring.InstrumentGrpc(*svrMetrics)...)

grpcServer.Serve(listener)

```

func NewTracingTransport added in v1.35.0 

func NewTracingTransport(base http.RoundTripper) http.RoundTripper

NewTracingTransport creates an HTTP transport that injects OpenTelemetry trace context

func StreamServerInstrument added in v1.27.10 

func StreamServerInstrument(hist *prometheus.HistogramVec) grpc.StreamServerInterceptor

func UnaryServerInstrument added in v1.27.10 

func UnaryServerInstrument(hist *prometheus.HistogramVec) grpc.UnaryServerInterceptor

Types

type BackgroundProcess added in v1.37.14 

type BackgroundProcess string

BackgroundProcess is the value of the closed `process` label — the only label, so cardinality stays constant regardless of shard/tenant count. 

const (
	ProcessBackup           BackgroundProcess = "backup"
	ProcessRestore          BackgroundProcess = "restore"
	ProcessOffload          BackgroundProcess = "offload"
	ProcessUsageCollection  BackgroundProcess = "usage_collection"
	ProcessCompaction       BackgroundProcess = "compaction"
	ProcessTombstoneCleanup BackgroundProcess = "tombstone_cleanup"
	ProcessAsyncReplication BackgroundProcess = "async_replication"
	ProcessAsyncIndexing    BackgroundProcess = "async_indexing"
	ProcessTTLDeletion      BackgroundProcess = "ttl_deletion"
	ProcessReplicaMovement  BackgroundProcess = "replica_movement"
)

type BackgroundProcessMetrics added in v1.37.14 

type BackgroundProcessMetrics struct {
	// contains filtered or unexported fields
}

BackgroundProcessMetrics reports heavy background processes per node, keyed by `process`. Nil-safe and concurrency-safe.

func GetBackgroundProcessMetrics added in v1.37.14 

func GetBackgroundProcessMetrics() *BackgroundProcessMetrics

GetBackgroundProcessMetrics returns the singleton on the default registerer.

func (*BackgroundProcessMetrics) DecActive added in v1.37.14 

func (m *BackgroundProcessMetrics) DecActive(process BackgroundProcess)

DecActive is the counterpart of IncActive.

func (*BackgroundProcessMetrics) Failed added in v1.37.14 

func (m *BackgroundProcessMetrics) Failed(process BackgroundProcess)

Failed records one errored run. Call it from a genuine-failure branch, not on cancellation. Independent of Started/Inc-Dec, which track liveness regardless of outcome.

func (*BackgroundProcessMetrics) IncActive added in v1.37.14 

func (m *BackgroundProcessMetrics) IncActive(process BackgroundProcess)

IncActive/DecActive raise and lower the active gauge — the primitive for split-callback sites (e.g. the replication engine) that can't hold a closure. Pair each IncActive with one DecActive; elsewhere prefer Started.

func (*BackgroundProcessMetrics) Started added in v1.37.14 

func (m *BackgroundProcessMetrics) Started(process BackgroundProcess) func()

Started marks the process active and returns a done-callback (call once, e.g. `defer Started(p)()`) that clears it and records the run's duration. Wrap the working burst, not an idle loop, so the gauge reflects real activity.

type Config added in v1.24.8 

type Config struct {
	Enabled                    bool   `json:"enabled" yaml:"enabled" long:"enabled"`
	Tool                       string `json:"tool" yaml:"tool"`
	Port                       int    `json:"port" yaml:"port" long:"port" default:"8081"`
	Group                      bool   `json:"group_classes" yaml:"group_classes"`
	MonitorCriticalBucketsOnly bool   `json:"monitor_critical_buckets_only" yaml:"monitor_critical_buckets_only"`

	// Metrics namespace group the metrics with common prefix.
	MetricsNamespace string `json:"metrics_namespace" yaml:"metrics_namespace" long:"metrics_namespace" default:""`
}

type GRPCServerMetrics added in v1.27.12 

type GRPCServerMetrics struct {
	RequestDuration  *prometheus.HistogramVec
	RequestBodySize  *prometheus.HistogramVec
	ResponseBodySize *prometheus.HistogramVec
	InflightRequests *prometheus.GaugeVec
}

GRPCServerMetrics exposes set of prometheus metrics for grpc servers.

func NewGRPCServerMetrics added in v1.27.12 

func NewGRPCServerMetrics(namespace string, reg prometheus.Registerer) *GRPCServerMetrics

type GrpcStatsHandler added in v1.27.10 

type GrpcStatsHandler struct {
	// contains filtered or unexported fields
}

func NewGrpcStatsHandler added in v1.27.10 

func NewGrpcStatsHandler(inflight *prometheus.GaugeVec, requestSize *prometheus.HistogramVec, responseSize *prometheus.HistogramVec) *GrpcStatsHandler

func (*GrpcStatsHandler) HandleConn added in v1.27.10 

func (g *GrpcStatsHandler) HandleConn(_ context.Context, _ stats.ConnStats)

func (*GrpcStatsHandler) HandleRPC added in v1.27.10 

func (g *GrpcStatsHandler) HandleRPC(ctx context.Context, rpcStats stats.RPCStats)

func (*GrpcStatsHandler) TagConn added in v1.27.10 

func (g *GrpcStatsHandler) TagConn(ctx context.Context, _ *stats.ConnTagInfo) context.Context

func (*GrpcStatsHandler) TagRPC added in v1.27.10 

func (g *GrpcStatsHandler) TagRPC(ctx context.Context, info *stats.RPCTagInfo) context.Context

type HTTPServerMetrics added in v1.27.12 

type HTTPServerMetrics struct {
	TCPActiveConnections *prometheus.GaugeVec
	RequestDuration      *prometheus.HistogramVec
	RequestBodySize      *prometheus.HistogramVec
	ResponseBodySize     *prometheus.HistogramVec
	InflightRequests     *prometheus.GaugeVec
}

HTTPServerMetrics exposes set of prometheus metrics for http servers.

func NewHTTPServerMetrics added in v1.27.12 

func NewHTTPServerMetrics(namespace string, reg prometheus.Registerer) *HTTPServerMetrics

NewHTPServerMetrics return the ServerMetrics that can be used in any of the grpc or http servers.

type InstrumentHandler added in v1.27.12 

type InstrumentHandler struct {
	// contains filtered or unexported fields
}

func InstrumentHTTP added in v1.27.12 

func InstrumentHTTP(
	next http.Handler,
	routeLabel StaticRouteLabel,
	inflight *prometheus.GaugeVec,
	duration *prometheus.HistogramVec,
	requestSize *prometheus.HistogramVec,
	responseSize *prometheus.HistogramVec,
) *InstrumentHandler

func (*InstrumentHandler) ServeHTTP added in v1.27.12 

func (i *InstrumentHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type OnceUponATimer 

type OnceUponATimer struct {
	sync.Once
	Timer *prometheus.Timer
}

func NewOnceTimer 

func NewOnceTimer(promTimer *prometheus.Timer) *OnceUponATimer

func (*OnceUponATimer) ObserveDurationOnce 

func (o *OnceUponATimer) ObserveDurationOnce()

type PrometheusMetrics 

type PrometheusMetrics struct {
	Registerer prometheus.Registerer

	BatchTime                           *prometheus.HistogramVec
	BatchSizeBytes                      *prometheus.SummaryVec
	BatchSizeObjects                    prometheus.Summary
	BatchSizeTenants                    prometheus.Summary
	BatchDeleteTime                     *prometheus.SummaryVec
	BatchCount                          *prometheus.CounterVec
	BatchCountBytes                     *prometheus.CounterVec
	ObjectsTime                         *prometheus.SummaryVec
	AsyncOperations                     *prometheus.GaugeVec
	LSMSegmentCount                     *prometheus.GaugeVec
	LSMObjectsBucketSegmentCount        *prometheus.GaugeVec
	LSMCompressedVecsBucketSegmentCount *prometheus.GaugeVec
	LSMSegmentCountByLevel              *prometheus.GaugeVec
	LSMSegmentUnloaded                  *prometheus.GaugeVec
	LSMSegmentObjects                   *prometheus.GaugeVec
	LSMSegmentSize                      *prometheus.GaugeVec
	LSMMemtableSize                     *prometheus.GaugeVec
	LSMMemtableDurations                *prometheus.SummaryVec
	LSMBitmapBuffersUsage               *prometheus.CounterVec
	ObjectCount                         *prometheus.GaugeVec
	QueriesCount                        *prometheus.GaugeVec
	RequestsTotal                       *prometheus.GaugeVec
	QueriesDurations                    *prometheus.HistogramVec
	QueriesFilteredVectorDurations      *prometheus.SummaryVec
	QueryDimensions                     *prometheus.CounterVec
	QueryDimensionsCombined             prometheus.Counter
	GoroutinesCount                     *prometheus.GaugeVec
	FileIOWrites                        *prometheus.SummaryVec
	FileIOReads                         *prometheus.SummaryVec
	MmapOperations                      *prometheus.CounterVec
	MmapProcMaps                        prometheus.Gauge

	// Reindex metrics
	RangeableInMemoryRebuildDegraded *prometheus.CounterVec

	// Backup/Restore metrics
	BackupRestoreDurations            *prometheus.SummaryVec
	BackupStoreDurations              *prometheus.SummaryVec
	BucketPauseDurations              *prometheus.SummaryVec
	BackupRestoreClassDurations       *prometheus.SummaryVec
	BackupRestoreBackupInitDurations  *prometheus.SummaryVec
	BackupRestoreFromStorageDurations *prometheus.SummaryVec
	BackupRestoreDataTransferred      *prometheus.CounterVec
	BackupStoreDataTransferred        *prometheus.CounterVec
	RestorePhaseDurations             *prometheus.HistogramVec

	// offload metric
	QueueSize                        *prometheus.GaugeVec
	QueueDiskUsage                   *prometheus.GaugeVec
	QueuePaused                      prometheus.Gauge
	QueueCount                       prometheus.Gauge
	QueuePartitionProcessingDuration *prometheus.HistogramVec

	VectorIndexQueueInsertCount *prometheus.CounterVec
	VectorIndexQueueDeleteCount *prometheus.CounterVec

	VectorIndexTombstones              *prometheus.GaugeVec
	VectorIndexTombstoneCleanupThreads *prometheus.GaugeVec
	VectorIndexTombstoneCleanedCount   *prometheus.CounterVec
	VectorIndexTombstoneUnexpected     *prometheus.CounterVec
	VectorIndexTombstoneCycleStart     *prometheus.GaugeVec
	VectorIndexTombstoneCycleEnd       *prometheus.GaugeVec
	VectorIndexTombstoneCycleProgress  *prometheus.GaugeVec
	VectorIndexOperations              *prometheus.GaugeVec
	VectorIndexDurations               *prometheus.SummaryVec
	VectorIndexSize                    *prometheus.GaugeVec
	VectorIndexMaintenanceDurations    *prometheus.SummaryVec
	// IVF
	VectorIndexPostings                      *prometheus.GaugeVec
	VectorIndexPostingSize                   *prometheus.HistogramVec
	VectorIndexPendingBackgroundOperations   *prometheus.GaugeVec
	VectorIndexBackgroundOperationsDurations *prometheus.SummaryVec
	VectorIndexBackgroundOperationsCount     *prometheus.GaugeVec
	VectorIndexStoreOperationsDurations      *prometheus.SummaryVec

	VectorDimensionsSum                 *prometheus.GaugeVec
	VectorSegmentsSum                   *prometheus.GaugeVec
	VectorIndexMemoryAllocationRejected prometheus.Counter

	StartupProgress  *prometheus.GaugeVec
	StartupDurations *prometheus.SummaryVec
	StartupDiskIO    *prometheus.SummaryVec

	StartupShardsLoaded prometheus.Gauge
	StartupShardsToLoad prometheus.Gauge

	ShardsLoaded    prometheus.Gauge
	ShardsUnloaded  prometheus.Gauge
	ShardsLoading   prometheus.Gauge
	ShardsUnloading prometheus.Gauge

	// ShardHaltForTransferForceResume: non-zero means a transfer was
	// force-resumed mid-stream — compaction may have raced the transfer.
	ShardHaltForTransferForceResume *prometheus.CounterVec

	// RAFT-based schema metrics
	SchemaWrites         *prometheus.SummaryVec
	SchemaReadsLocal     *prometheus.SummaryVec
	SchemaReadsLeader    *prometheus.SummaryVec
	SchemaWaitForVersion *prometheus.SummaryVec

	TombstoneFindLocalEntrypoint  *prometheus.CounterVec
	TombstoneFindGlobalEntrypoint *prometheus.CounterVec
	TombstoneReassignNeighbors    *prometheus.CounterVec
	TombstoneDeleteListSize       *prometheus.GaugeVec

	Group bool
	// Keeping metering to only the critical buckets (objects, vectors_compressed)
	// helps cut down on noise when monitoring
	LSMCriticalBucketsOnly bool

	// Vectorization
	T2VBatches            *prometheus.GaugeVec
	T2VBatchQueueDuration *prometheus.HistogramVec
	T2VRequestDuration    *prometheus.HistogramVec
	T2VTokensInBatch      *prometheus.HistogramVec
	T2VTokensInRequest    *prometheus.HistogramVec
	T2VRateLimitStats     *prometheus.GaugeVec
	T2VRepeatStats        *prometheus.GaugeVec
	T2VRequestsPerBatch   *prometheus.HistogramVec

	TokenizerDuration           *prometheus.HistogramVec
	TokenizerRequests           *prometheus.CounterVec
	TokenizerInitializeDuration *prometheus.HistogramVec
	TokenCount                  *prometheus.CounterVec
	TokenCountPerRequest        *prometheus.HistogramVec

	// Currently targeted at OpenAI, the metrics will have to be added to every vectorizer for complete coverage
	ModuleExternalRequests           *prometheus.CounterVec
	ModuleExternalRequestDuration    *prometheus.HistogramVec
	ModuleExternalBatchLength        *prometheus.HistogramVec
	ModuleExternalRequestSingleCount *prometheus.CounterVec
	ModuleExternalRequestBatchCount  *prometheus.CounterVec
	ModuleExternalRequestSize        *prometheus.HistogramVec
	ModuleExternalResponseSize       *prometheus.HistogramVec
	ModuleExternalResponseStatus     *prometheus.CounterVec
	VectorizerRequestTokens          *prometheus.HistogramVec
	ModuleExternalError              *prometheus.CounterVec
	ModuleCallError                  *prometheus.CounterVec
	ModuleBatchError                 *prometheus.CounterVec

	ModuleExternalRequestResends prometheus.Counter

	// Checksum metrics
	ChecksumValidationDuration prometheus.Summary
	ChecksumBytesRead          prometheus.Summary
	// contains filtered or unexported fields
}

NOTE: Do not add any new metrics to this global `PrometheusMetrics` struct. Instead add your metrics close the corresponding component.

func GetMetrics 

func GetMetrics() *PrometheusMetrics

func (*PrometheusMetrics) AddObjectsTtlObjectsDeleted added in v1.36.0 

func (m *PrometheusMetrics) AddObjectsTtlObjectsDeleted(count float64)

func (*PrometheusMetrics) DecObjectsTtlRunning added in v1.36.0 

func (m *PrometheusMetrics) DecObjectsTtlRunning()

func (*PrometheusMetrics) DeleteClass added in v1.21.7 

func (pm *PrometheusMetrics) DeleteClass(className string) error

DeleteClass deletes all metrics that match the class name, but do not have a shard-specific label. See [DeleteShard] for more information.

func (*PrometheusMetrics) DeleteLoadedShard added in v1.36.6 

func (pm *PrometheusMetrics) DeleteLoadedShard()

Unregister a loaded shard (when it's deleted, not unloaded)

func (*PrometheusMetrics) DeleteShard added in v1.21.7 

func (pm *PrometheusMetrics) DeleteShard(className, shardName string) error

Delete Shard deletes existing label combinations that match both the shard and class name. If a metric is not collected at the shard level it is unaffected. This is to make sure that deleting a single shard (e.g. multi-tenancy) does not affect metrics for existing shards.

In addition, there are some metrics that we explicitly keep, such as vector_dimensions_sum as they can be used in billing decisions.

func (*PrometheusMetrics) DeleteUnloadedShard added in v1.36.6 

func (pm *PrometheusMetrics) DeleteUnloadedShard()

Unregister an unloaded shard (when it's deleted without ever being loaded)

func (*PrometheusMetrics) FailLoadingShard added in v1.36.6 

func (pm *PrometheusMetrics) FailLoadingShard()

Revert shard from loading back to unloaded (when loading fails)

func (*PrometheusMetrics) FinishLoadingShard added in v1.23.0 

func (pm *PrometheusMetrics) FinishLoadingShard()

Move the shard from in progress to loaded

func (*PrometheusMetrics) FinishUnloadingShard added in v1.23.0 

func (pm *PrometheusMetrics) FinishUnloadingShard()

Move the shard from in progress to unloaded

func (*PrometheusMetrics) IncObjectsTtlCount added in v1.36.0 

func (m *PrometheusMetrics) IncObjectsTtlCount()

func (*PrometheusMetrics) IncObjectsTtlFailureCount added in v1.36.0 

func (m *PrometheusMetrics) IncObjectsTtlFailureCount()

func (*PrometheusMetrics) IncObjectsTtlRunning added in v1.36.0 

func (m *PrometheusMetrics) IncObjectsTtlRunning()

func (*PrometheusMetrics) IncRangeableInMemoryRebuildDegraded added in v1.38.7 

func (m *PrometheusMetrics) IncRangeableInMemoryRebuildDegraded(className, shardName, propName string)

func (*PrometheusMetrics) NewLoadedShard added in v1.36.6 

func (pm *PrometheusMetrics) NewLoadedShard()

Register a new shard that is immediately loaded (for non-lazy loading path)

func (*PrometheusMetrics) NewUnloadedshard added in v1.23.0 

func (pm *PrometheusMetrics) NewUnloadedshard()

Register a new, unloaded shard

func (*PrometheusMetrics) ObserveObjectsTtlDuration added in v1.36.0 

func (m *PrometheusMetrics) ObserveObjectsTtlDuration(d time.Duration)

func (*PrometheusMetrics) SetStartupShardProgress added in v1.37.12 

func (pm *PrometheusMetrics) SetStartupShardProgress(loaded, total int64)

SetStartupShardProgress publishes the latest eager shard-loading progress (loaded so far / expected to load eagerly) computed during startup.

func (*PrometheusMetrics) StartLoadingShard added in v1.23.0 

func (pm *PrometheusMetrics) StartLoadingShard()

Move the shard from unloaded to in progress

func (*PrometheusMetrics) StartUnloadingShard added in v1.23.0 

func (pm *PrometheusMetrics) StartUnloadingShard()

Move the shard from loaded to in progress

type StaticRouteLabel added in v1.27.14 

type StaticRouteLabel func(r *http.Request) (*http.Request, string)

Examples: `/schema/Movies/properties` -> `/schema/{className}` `/replicas/indices/Movies/shards/hello0/objects` -> `/replicas/indices`

type TenantOffloadMetrics added in v1.28.0 

type TenantOffloadMetrics struct {
	// NOTE: These ops are not GET or PUT requests to object storage.
	// these are one of the `download`, `upload` or `delete`. Because we use s5cmd to talk
	// to object storage currently. Which supports these operations at high level.
	FetchedBytes     prometheus.Counter
	TransferredBytes prometheus.Counter
	OpsDuration      *prometheus.HistogramVec
}

func NewTenantOffloadMetrics added in v1.28.0 

func NewTenantOffloadMetrics(cfg Config, reg prometheus.Registerer) *TenantOffloadMetrics

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.