metric

package
v0.1.0-alpha.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 15, 2026 License: MIT Imports: 19 Imported by: 0

README

Metric

Prometheus-based metrics collection and HTTP server for SemEngine monitoring and observability.

Overview

The metric package provides comprehensive metrics collection capabilities for SemEngine using Prometheus as the underlying metrics system. It includes core platform metrics, service-specific metric registration, and an HTTP metrics server for monitoring and alerting integration.

The package provides a centralized metrics registry that manages both core platform metrics (service status, message processing, NATS connection health) and custom service-specific metrics. The HTTP server exposes metrics in Prometheus format for collection by monitoring systems.

The design follows Prometheus best practices with proper metric naming, labeling, and types (counters, gauges, histograms) to provide comprehensive operational visibility into SemEngine.

Installation

import (
    "log"

    "github.com/c360studio/semengine/metric"
    "github.com/c360studio/semengine/pkg/security"
)

core Concepts

Metrics Registry

Centralized registry that manages Prometheus metric registration, provides core platform metrics, and allows services to register custom metrics with conflict detection and lifecycle management.

Core metrics (Metrics)

Pre-defined platform-level metrics covering service status, message processing throughput, error rates, health check status, and NATS connection monitoring for consistent operational visibility.

Metrics Server

HTTP server that exposes collected metrics in Prometheus format with configurable port and path, a /health endpoint that answers 200 with the plain-text body OK, and OpenMetrics support for integration with monitoring systems.

Service Metrics

Services register their own counters, gauges and histograms with RegisterOrGet, keyed by service and metric name. It refuses a registration that conflicts with one already held; it does not add a namespace or prefix to the metric name.

Usage

Basic Example
import (
    "context"
    "log"
    "time"

    "github.com/c360studio/semengine/metric"
    "github.com/c360studio/semengine/pkg/security"
)

// Create metrics registry with core platform metrics
registry := metric.NewMetricsRegistry()
securityCfg := security.Config{}

// Start metrics HTTP server
server := metric.NewServer(9090, "/metrics", registry, securityCfg)
runtimeCtx := context.Background()
if err := server.Start(runtimeCtx); err != nil {
    log.Fatalf("Failed to start metrics server: %v", err)
}
defer func() {
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := server.Stop(shutdownCtx); err != nil {
        log.Printf("Failed to stop metrics server: %v", err)
    }
}()

// Record core platform metrics
coreMetrics := registry.CoreMetrics()
coreMetrics.RecordServiceStatus("my-service", 2) // 2=running
coreMetrics.RecordMessageReceived("my-service", "gps.position")
coreMetrics.RecordHealthStatus("my-service", true)

// Access metrics at http://localhost:9090/metrics
Advanced Usage - Custom Service Metrics
// Create custom metrics for a service
messageCounter := prometheus.NewCounter(prometheus.CounterOpts{
    Namespace: "semengine",
    Subsystem: "gps_service",
    Name:      "coordinates_processed_total",
    Help:      "Total GPS coordinates processed",
})

latencyHistogram := prometheus.NewHistogram(prometheus.HistogramOpts{
    Namespace: "semengine",
    Subsystem: "gps_service",
    Name:      "processing_duration_seconds",
    Help:      "GPS coordinate processing duration",
    Buckets:   []float64{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0},
})

queueGauge := prometheus.NewGauge(prometheus.GaugeOpts{
    Namespace: "semengine",
    Subsystem: "gps_service",
    Name:      "queue_size",
    Help:      "Current GPS processing queue size",
})

// Register custom metrics with the registry, keeping the collectors it returns. On an
// error the returned collector is nil: stop rather than use it.
messageCounter, err := metric.RegisterOrGet(registry, "gps-service", "coordinates_processed", messageCounter)
if err != nil {
    return fmt.Errorf("register counter: %w", err)
}

latencyHistogram, err = metric.RegisterOrGet(registry, "gps-service", "processing_duration", latencyHistogram)
if err != nil {
    return fmt.Errorf("register histogram: %w", err)
}

queueGauge, err = metric.RegisterOrGet(registry, "gps-service", "queue_size", queueGauge)
if err != nil {
    return fmt.Errorf("register gauge: %w", err)
}

// Use the custom metrics
messageCounter.Inc()
queueGauge.Set(25)

start := time.Now()
processGPSCoordinate()
latencyHistogram.Observe(time.Since(start).Seconds())

// Cleanup on service shutdown
registry.Unregister("gps-service", "coordinates_processed")
registry.Unregister("gps-service", "processing_duration")
registry.Unregister("gps-service", "queue_size")

API Reference

Types
MetricsRegistry

Central registry for managing all platform metrics.

type MetricsRegistry struct {
    Metrics *Metrics // the core platform metrics; also returned by CoreMetrics
    // other fields are private
}

func NewMetricsRegistry() *MetricsRegistry                 // Create new registry
func (r *MetricsRegistry) PrometheusRegistry() *prometheus.Registry  // Get Prometheus registry
func (r *MetricsRegistry) CoreMetrics() *Metrics          // Get core platform metrics

// Service metric registration: one collector per service/metric key
func RegisterOrGet[C prometheus.Collector](r *MetricsRegistry, serviceName, metricName string, candidate C) (C, error)
func (r *MetricsRegistry) Unregister(serviceName, metricName string) bool  // Remove metric
Metrics

Pre-defined platform-level metrics for consistent monitoring, returned by CoreMetrics().

type Metrics struct {
    // Service metrics
    ServiceStatus      *prometheus.GaugeVec    // Service status (0=stopped, 1=starting, 2=running, 3=stopping, 4=failed)
    MessagesReceived   *prometheus.CounterVec  // Total messages received by service and type
    MessagesProcessed  *prometheus.CounterVec  // Total messages processed by service, type, and status
    MessagesPublished  *prometheus.CounterVec  // Total messages published by service and subject
    ProcessingDuration *prometheus.HistogramVec // Processing duration by service and operation
    ErrorsTotal        *prometheus.CounterVec  // Total errors by service and type
    HealthCheckStatus  *prometheus.GaugeVec    // Health status by service (0=unhealthy, 1=healthy)
    LogEntriesTotal    *prometheus.CounterVec  // WARN+ log records by component and level (no Record method)

    // NATS metrics
    NATSConnected      prometheus.Gauge        // NATS connection status (0=disconnected, 1=connected)
    NATSRTT            prometheus.Gauge        // NATS round-trip time in seconds
    NATSReconnects     prometheus.Counter      // Total NATS reconnections
    NATSCircuitBreaker prometheus.Gauge        // Circuit breaker status (0=closed, 1=open, 2=half-open)
}

// Recording methods
func (c *Metrics) RecordServiceStatus(service string, status int)
func (c *Metrics) RecordMessageReceived(service, messageType string)
func (c *Metrics) RecordMessageProcessed(service, messageType, status string)
func (c *Metrics) RecordMessagePublished(service, subject string)
func (c *Metrics) RecordProcessingDuration(service, operation string, duration time.Duration)
func (c *Metrics) RecordError(service, errorType string)
func (c *Metrics) RecordHealthStatus(service string, healthy bool)
func (c *Metrics) RecordNATSStatus(connected bool)
func (c *Metrics) RecordNATSRTT(rtt time.Duration)
func (c *Metrics) RecordNATSReconnect()
func (c *Metrics) RecordCircuitBreakerState(state int)
Server

HTTP server for exposing metrics to Prometheus.

type Server struct {
    // private fields
}

func NewServer(port int, path string, registry *MetricsRegistry, security security.Config) *Server
func (s *Server) Start(context.Context) error // One-shot synchronous bind; context owns served requests
func (s *Server) StartWithListener(context.Context, net.Listener) error // Start on a caller-bound listener
func (s *Server) Stop(context.Context) error  // Caller-bounded graceful attempt, then bounded force-close/join
func (s *Server) Address() string              // Get server address
Functions
RegisterOrGet

RegisterOrGet[C prometheus.Collector](r *MetricsRegistry, serviceName, metricName string, candidate C) (C, error)

Registers candidate under the key serviceName.metricName, or returns the collector already registered under that key when it has the same concrete type and descriptors. Use the returned collector, never your own candidate. A nil candidate, a type or descriptor mismatch, and a descriptor another key already owns are refused with a fatal error, and nothing is stored.

NewMetricsRegistry() *MetricsRegistry

Creates a new metrics registry with core platform metrics and Go runtime metrics.

NewServer

NewServer(port int, path string, registry *MetricsRegistry, security security.Config) *Server

Creates a new HTTP metrics server with default values for zero port (9090) and empty path ("/metrics").

Architecture

Design Decisions

Prometheus Integration: Chose Prometheus as the metrics backend because it's the industry standard for container and microservice monitoring with excellent tooling ecosystem.

  • Trade-off: Gained mature monitoring ecosystem but coupled to Prometheus API
  • Alternative considered: Custom metrics format (lacks tooling and standardization)

Centralized Registry: Implemented single MetricsRegistry to manage all metrics and prevent conflicts between services registering the same metric names.

  • Rationale: Prevents metric naming conflicts and provides consistent lifecycle management
  • Trade-off: Gained consistency but requires services to coordinate through registry

core Platform Metrics: Pre-defined core metrics for all platform services to ensure consistent monitoring across services without each service needing to define standard metrics.

  • Chose standardized metrics over service-specific only because operational consistency is critical
  • Trade-off: Gained consistency but reduced flexibility for unique service requirements
Integration Points
  • Dependencies: Prometheus Go client library for metrics implementation and HTTP handlers
  • Used By: natsclient (WithMetrics, JetStream metrics) and internal/cache (WithMetrics, WithCoalescingMetrics)
  • Data Flow: Service Operations → Metric Recording → Registry Storage → HTTP Endpoint → Prometheus Scraping

Configuration

The package reads no configuration file. The server's port and path are NewServer arguments, its TLS and mTLS settings come from the security.Config passed to it, and the Go runtime and process collectors are always registered by NewMetricsRegistry. Service metrics are declared in code with RegisterOrGet.

Error Handling

Metric Registration Errors
// Registering the same key again with an identical collector returns the
// registered collector; a conflicting registration is a fatal error
counter, err := metric.RegisterOrGet(registry, "service", "metric", counter)
if err != nil {
    log.Printf("Failed to register metric: %v", err)
    return err
}

// Safe metric unregistration
if !registry.Unregister("service", "metric") {
    log.Printf("Metric was not registered or already removed")
}
Server Lifecycle Errors
// Start server with error handling
server := metric.NewServer(9090, "/metrics", registry, security.Config{})
if err := server.Start(ctx); err != nil {
    log.Fatalf("Metrics server failed: %v", err)
}

// Graceful shutdown
if err := server.Stop(shutdownCtx); err != nil {
    log.Printf("Error stopping metrics server: %v", err)
}

Stop first attempts a graceful HTTP shutdown within shutdownCtx. If that attempt fails or its budget expires, the server and listener are force-closed and Stop waits for its exact serving goroutine, which returns once its listener is closed, and for every request the server admitted, including the metrics collection it runs. Stop returns the context's error whenever shutdownCtx has ended, and a later Stop waits again for what is still running. Once Stop has begun, a request that reaches the handler is refused with 503 Service Unavailable. The instance is terminal after that attempt; a completed repeated Stop returns nil.

Best Practices
// DO: Use consistent metric naming
counter := prometheus.NewCounter(prometheus.CounterOpts{
    Namespace: "semengine",           // Always use "semengine"
    Subsystem: "service_name",         // Use service name
    Name:      "operations_total",     // Descriptive name with units
    Help:      "Total operations processed",
})

// DO: Handle registration errors and keep the returned collector
counter, err := metric.RegisterOrGet(registry, "service", "operations", counter)
if err != nil {
    return fmt.Errorf("metric registration failed: %w", err)
}

// DO: Use appropriate metric types
counter.Inc()                          // For monotonically increasing values
gauge.Set(42)                         // For values that can go up and down
histogram.Observe(duration.Seconds()) // For distributions and timings

// DON'T: Ignore registration errors or the returned collector
metric.RegisterOrGet(registry, "service", "metric", counter) // Missing error check; writes to counter may be lost

// DON'T: Use inconsistent naming
prometheus.NewCounter(prometheus.CounterOpts{
    Name: "my_custom_metric", // Missing namespace and subsystem
})

Testing

Test Utilities
// Test metrics registry
func TestMetricsRegistry(t *testing.T) {
    registry := metric.NewMetricsRegistry()

    // Test counter registration
    counter := prometheus.NewCounter(prometheus.CounterOpts{
        Name: "test_counter",
        Help: "Test counter",
    })

    registered, err := metric.RegisterOrGet(registry, "test-service", "test_counter", counter)
    assert.NoError(t, err)

    // Registering the same key again returns the registered collector
    again, err := metric.RegisterOrGet(registry, "test-service", "test_counter", counter)
    assert.NoError(t, err)
    assert.Same(t, registered, again)

    // Test unregistration
    success := registry.Unregister("test-service", "test_counter")
    assert.True(t, success)
}

// Test core metrics recording
func TestCoreMetrics(t *testing.T) {
    registry := metric.NewMetricsRegistry()
    coreMetrics := registry.CoreMetrics()

    // Test service status recording
    coreMetrics.RecordServiceStatus("test-service", 2)

    // Verify metric was recorded (would require prometheus test utilities)
    // This is a simplified example - actual testing would use prometheus/testutil
}
Testing Patterns
  • Use prometheus/testutil package for metric value verification
  • Test metric registration success and failure scenarios
  • Verify metric cleanup during service shutdown
  • Test concurrent metric access with multiple goroutines

Performance Considerations

  • Metric Recording: O(1) operation for most Prometheus metric types
  • Registry Operations: Thread-safe with read-write mutex - concurrent reads allowed
  • Memory Usage: Metrics consume memory proportional to label cardinality (avoid high-cardinality labels)
  • HTTP Server: net/http serves each request on its own goroutine; Stop waits for every admitted request

Examples

Example 1: Service Integration
package main

import (
    "context"
    "log"
    "math/rand"
    "time"

    "github.com/c360studio/semengine/metric"
    "github.com/c360studio/semengine/pkg/security"
    "github.com/prometheus/client_golang/prometheus"
)

type GPSService struct {
    name         string
    registry     *metric.MetricsRegistry
    coreMetrics  *metric.Metrics

    // Custom metrics
    coordinatesProcessed prometheus.Counter
    processingLatency    prometheus.Histogram
    queueSize           prometheus.Gauge
}

func NewGPSService(registry *metric.MetricsRegistry) (*GPSService, error) {
    service := &GPSService{
        name:        "gps-service",
        registry:    registry,
        coreMetrics: registry.CoreMetrics(),
    }

    // Create custom metrics
    service.coordinatesProcessed = prometheus.NewCounter(prometheus.CounterOpts{
        Namespace: "semengine",
        Subsystem: "gps",
        Name:      "coordinates_processed_total",
        Help:      "Total GPS coordinates processed",
    })

    service.processingLatency = prometheus.NewHistogram(prometheus.HistogramOpts{
        Namespace: "semengine",
        Subsystem: "gps",
        Name:      "processing_duration_seconds",
        Help:      "GPS coordinate processing duration",
        Buckets:   []float64{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0},
    })

    service.queueSize = prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "semengine",
        Subsystem: "gps",
        Name:      "queue_size",
        Help:      "Current GPS processing queue size",
    })

    // Register custom metrics, keeping the collectors the registry returns
    var err error
    service.coordinatesProcessed, err = metric.RegisterOrGet(registry, "gps-service", "coordinates_processed",
        service.coordinatesProcessed)
    if err != nil {
        return nil, err
    }

    service.processingLatency, err = metric.RegisterOrGet(registry, "gps-service", "processing_latency",
        service.processingLatency)
    if err != nil {
        return nil, err
    }

    service.queueSize, err = metric.RegisterOrGet(registry, "gps-service", "queue_size", service.queueSize)
    if err != nil {
        return nil, err
    }

    return service, nil
}

func (g *GPSService) Start(ctx context.Context) error {
    // Record service startup
    g.coreMetrics.RecordServiceStatus(g.name, 1) // starting

    // Simulate startup delay
    time.Sleep(100 * time.Millisecond)

    g.coreMetrics.RecordServiceStatus(g.name, 2) // running
    g.coreMetrics.RecordHealthStatus(g.name, true)

    // Start processing loop
    go g.processingLoop(ctx)

    log.Printf("GPS service started")
    return nil
}

func (g *GPSService) Stop() error {
    g.coreMetrics.RecordServiceStatus(g.name, 3) // stopping

    // Cleanup custom metrics
    g.registry.Unregister("gps-service", "coordinates_processed")
    g.registry.Unregister("gps-service", "processing_latency")
    g.registry.Unregister("gps-service", "queue_size")

    g.coreMetrics.RecordServiceStatus(g.name, 0) // stopped
    log.Printf("GPS service stopped")
    return nil
}

func (g *GPSService) processingLoop(ctx context.Context) {
    ticker := time.NewTicker(1 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            g.processCoordinates()
        case <-ctx.Done():
            return
        }
    }
}

func (g *GPSService) processCoordinates() {
    // Simulate processing
    start := time.Now()

    // Random processing time
    processingTime := time.Duration(rand.Intn(50)) * time.Millisecond
    time.Sleep(processingTime)

    // Record metrics
    g.coordinatesProcessed.Inc()
    g.processingLatency.Observe(time.Since(start).Seconds())
    g.queueSize.Set(float64(rand.Intn(100)))

    // Record core metrics
    g.coreMetrics.RecordMessageReceived(g.name, "gps.coordinates")
    g.coreMetrics.RecordMessageProcessed(g.name, "gps.coordinates", "success")
    g.coreMetrics.RecordProcessingDuration(g.name, "coordinates", time.Since(start))

    // Simulate occasional errors
    if rand.Float32() < 0.05 { // 5% error rate
        g.coreMetrics.RecordError(g.name, "processing_error")
    }
}

func main() {
    // Create metrics registry
    registry := metric.NewMetricsRegistry()

    // Start metrics server
    server := metric.NewServer(9090, "/metrics", registry, security.Config{})
    log.Printf("Starting metrics server on %s", server.Address())
    runtimeCtx := context.Background()
    if err := server.Start(runtimeCtx); err != nil {
        log.Fatalf("Failed to start metrics server: %v", err)
    }

    // Create and start GPS service
    gpsService, err := NewGPSService(registry)
    if err != nil {
        log.Fatalf("Failed to create GPS service: %v", err)
    }

    if err := gpsService.Start(runtimeCtx); err != nil {
        log.Fatalf("Failed to start GPS service: %v", err)
    }

    // Run for demo period
    time.Sleep(30 * time.Second)

    // Cleanup
    gpsService.Stop()
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := server.Stop(shutdownCtx); err != nil {
        log.Printf("Failed to stop metrics server: %v", err)
    }
}
Example 2: Multi-Service Monitoring
package main

import (
    "context"
    "fmt"
    "log"
    "math/rand"
    "sync"
    "time"

    "github.com/c360studio/semengine/metric"
    "github.com/c360studio/semengine/pkg/security"
    "github.com/prometheus/client_golang/prometheus"
)

type MonitoredService struct {
    name        string
    registry    *metric.MetricsRegistry
    coreMetrics *metric.Metrics

    // Service-specific metrics
    requestCounter  prometheus.Counter
    responseTime    prometheus.Histogram
    activeUsers     prometheus.Gauge
    errorRate       prometheus.Counter

    // Control
    stopCh chan struct{}
    wg     sync.WaitGroup
}

func NewMonitoredService(name string, registry *metric.MetricsRegistry) (*MonitoredService, error) {
    service := &MonitoredService{
        name:        name,
        registry:    registry,
        coreMetrics: registry.CoreMetrics(),
        stopCh:      make(chan struct{}),
    }

    // Create metrics
    service.requestCounter = prometheus.NewCounter(prometheus.CounterOpts{
        Namespace: "semengine",
        Subsystem: name,
        Name:      "requests_total",
        Help:      fmt.Sprintf("Total requests processed by %s", name),
    })

    service.responseTime = prometheus.NewHistogram(prometheus.HistogramOpts{
        Namespace: "semengine",
        Subsystem: name,
        Name:      "response_time_seconds",
        Help:      fmt.Sprintf("Response time for %s", name),
    })

    service.activeUsers = prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "semengine",
        Subsystem: name,
        Name:      "active_users",
        Help:      fmt.Sprintf("Active users for %s", name),
    })

    service.errorRate = prometheus.NewCounter(prometheus.CounterOpts{
        Namespace: "semengine",
        Subsystem: name,
        Name:      "errors_total",
        Help:      fmt.Sprintf("Total errors for %s", name),
    })

    // Register metrics, keeping the collectors the registry returns
    var err error
    if service.requestCounter, err = metric.RegisterOrGet(registry, name, "requests", service.requestCounter); err != nil {
        return nil, fmt.Errorf("failed to register request counter: %w", err)
    }

    if service.responseTime, err = metric.RegisterOrGet(registry, name, "response_time", service.responseTime); err != nil {
        return nil, fmt.Errorf("failed to register response time histogram: %w", err)
    }

    if service.activeUsers, err = metric.RegisterOrGet(registry, name, "active_users", service.activeUsers); err != nil {
        return nil, fmt.Errorf("failed to register active users gauge: %w", err)
    }

    if service.errorRate, err = metric.RegisterOrGet(registry, name, "errors", service.errorRate); err != nil {
        return nil, fmt.Errorf("failed to register error counter: %w", err)
    }

    return service, nil
}

func (s *MonitoredService) Start() error {
    s.coreMetrics.RecordServiceStatus(s.name, 1) // starting

    // Simulate startup
    time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)

    s.coreMetrics.RecordServiceStatus(s.name, 2) // running
    s.coreMetrics.RecordHealthStatus(s.name, true)

    // Start metrics collection
    s.wg.Add(1)
    go s.metricsLoop()

    log.Printf("Service %s started", s.name)
    return nil
}

func (s *MonitoredService) Stop() error {
    s.coreMetrics.RecordServiceStatus(s.name, 3) // stopping

    close(s.stopCh)
    s.wg.Wait()

    // Unregister metrics
    s.registry.Unregister(s.name, "requests")
    s.registry.Unregister(s.name, "response_time")
    s.registry.Unregister(s.name, "active_users")
    s.registry.Unregister(s.name, "errors")

    s.coreMetrics.RecordServiceStatus(s.name, 0) // stopped
    log.Printf("Service %s stopped", s.name)
    return nil
}

func (s *MonitoredService) metricsLoop() {
    defer s.wg.Done()

    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            s.generateMetrics()
        case <-s.stopCh:
            return
        }
    }
}

func (s *MonitoredService) generateMetrics() {
    // Simulate request processing
    numRequests := rand.Intn(10) + 1

    for i := 0; i < numRequests; i++ {
        start := time.Now()

        // Simulate work
        workTime := time.Duration(rand.Intn(100)) * time.Millisecond
        time.Sleep(workTime)

        // Record metrics
        s.requestCounter.Inc()
        s.responseTime.Observe(time.Since(start).Seconds())

        // Simulate errors (5% rate)
        if rand.Float32() < 0.05 {
            s.errorRate.Inc()
            s.coreMetrics.RecordError(s.name, "processing_error")
        }

        // Record core metrics
        messageType := fmt.Sprintf("%s.request", s.name)
        s.coreMetrics.RecordMessageReceived(s.name, messageType)
        s.coreMetrics.RecordMessageProcessed(s.name, messageType, "success")
        s.coreMetrics.RecordProcessingDuration(s.name, "request", time.Since(start))
    }

    // Update active users (random walk)
    currentUsers := rand.Intn(1000)
    s.activeUsers.Set(float64(currentUsers))
}

func main() {
    // Create metrics infrastructure
    registry := metric.NewMetricsRegistry()

    server := metric.NewServer(9090, "/metrics", registry, security.Config{})
    log.Printf("Metrics server running at %s", server.Address())
    runtimeCtx := context.Background()
    if err := server.Start(runtimeCtx); err != nil {
        log.Fatalf("Failed to start metrics server: %v", err)
    }

    // Create multiple services
    serviceNames := []string{"auth", "api", "processor", "storage", "notification"}
    var services []*MonitoredService

    for _, name := range serviceNames {
        service, err := NewMonitoredService(name, registry)
        if err != nil {
            log.Fatalf("Failed to create service %s: %v", name, err)
        }

        if err := service.Start(); err != nil {
            log.Fatalf("Failed to start service %s: %v", name, err)
        }

        services = append(services, service)
    }

    log.Printf("All services started. Metrics available at http://localhost:9090/metrics")

    // Run services
    time.Sleep(60 * time.Second)

    // Shutdown all services
    log.Printf("Shutting down services...")
    for _, service := range services {
        service.Stop()
    }

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := server.Stop(shutdownCtx); err != nil {
        log.Printf("Failed to stop metrics server: %v", err)
    }
    log.Printf("All services stopped")
}
  • natsclient: NATS client that records JetStream stream and consumer metrics in a registry
  • internal/cache: caches that export hit, miss and size metrics
  • pkg/errs: error classification used by this package's refusals

License

MIT

Documentation

Overview

Package metric provides Prometheus-based metrics collection and an HTTP server for monitoring and observability.

The package offers a centralized metrics registry managing both core platform metrics (service status, message processing, NATS health) and custom service-specific metrics. It includes an HTTP server exposing metrics in Prometheus format for monitoring system integration.

Architecture

The package follows a three-layer design:

  1. Core Metrics: Platform-level metrics automatically registered (Metrics type)
  2. Service Registry: Extensible registration for service-specific metrics (RegisterOrGet)
  3. HTTP Server: Metrics endpoint with health checks (Server type)

This architecture separates infrastructure concerns (core metrics) from application concerns (service-specific metrics) while providing a unified metrics endpoint for monitoring systems.

Basic Usage

Setting up metrics collection and HTTP server:

registry := metric.NewMetricsRegistry()
securityCfg := security.Config{} // Platform security config
server := metric.NewServer(9090, "/metrics", registry, securityCfg)

if err := server.Start(ctx); err != nil {
    log.Fatalf("Failed to start metrics server: %v", err)
}
defer server.Stop(shutdownCtx)

// Record core platform metrics
coreMetrics := registry.CoreMetrics()
coreMetrics.RecordServiceStatus("my-service", 2)
coreMetrics.RecordMessageProcessed("my-service", "gps.position", "success")
coreMetrics.RecordNATSStatus(true)

The metrics server will expose Prometheus-formatted metrics at http://localhost:9090/metrics and a health check at http://localhost:9090/health, which answers 200 with the plain-text body OK.

Core Metrics

The package automatically registers core platform metrics (the Metrics type), all in the "semengine" namespace:

  • Service lifecycle: semengine_service_status{service} (0=stopped, 1=starting, 2=running, 3=stopping, 4=failed)
  • Message flow: semengine_messages_received_total{service,type}, semengine_messages_processed_total{service,type,status}, semengine_messages_published_total{service,subject}
  • Processing performance: semengine_processing_duration_seconds{service,operation}
  • Errors and health: semengine_errors_total{service,type}, semengine_health_status{service}
  • Logging: semengine_log_entries_total{component,level}, which has no recording method; write it through the LogEntriesTotal field
  • NATS connectivity: semengine_nats_connected, semengine_nats_rtt_seconds, semengine_nats_reconnects_total, semengine_nats_circuit_breaker

Record core metrics through the registry:

coreMetrics := registry.CoreMetrics()

// Service lifecycle tracking
coreMetrics.RecordServiceStatus("processor", 2) // 2 = running
coreMetrics.RecordHealthStatus("processor", true)

// Message processing metrics
coreMetrics.RecordMessageReceived("processor", "sensor.reading")
coreMetrics.RecordMessageProcessed("processor", "sensor.reading", "success")
coreMetrics.RecordMessagePublished("processor", "output.subject")
coreMetrics.RecordProcessingDuration("processor", "transform", 150*time.Millisecond)

// NATS connectivity
coreMetrics.RecordNATSStatus(true)
coreMetrics.RecordNATSRTT(2 * time.Millisecond)
coreMetrics.RecordNATSReconnect()
coreMetrics.RecordCircuitBreakerState(0) // 0 = closed

// Error tracking
coreMetrics.RecordError("processor", "validation")

Service-Specific Metrics

Services register custom metrics with RegisterOrGet. There is one collector per service/metric key: RegisterOrGet returns it, and the caller uses the returned collector, never its own candidate. When RegisterOrGet returns an error the collector it returns is the zero value (nil for these types), so stop on the error rather than use it; each example below continues with `if err != nil { return err }`.

// Register a counter
requestCounter, err := metric.RegisterOrGet(registry, "api-service", "api_requests_total",
    prometheus.NewCounter(prometheus.CounterOpts{
        Name: "api_requests_total",
        Help: "Total number of API requests",
    }))

// Register a gauge
activeConnections, err := metric.RegisterOrGet(registry, "websocket-service", "active_connections",
    prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "active_connections",
        Help: "Number of active client connections",
    }))

// Register a histogram
queryDuration, err := metric.RegisterOrGet(registry, "database-service", "query_duration_seconds",
    prometheus.NewHistogram(prometheus.HistogramOpts{
        Name:    "query_duration_seconds",
        Help:    "Time spent executing queries",
        Buckets: prometheus.DefBuckets,
    }))

Vector Metrics with Labels

Register metrics with labels for multi-dimensional data:

// Counter with labels
httpRequestsVec := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by status and method",
    },
    []string{"status", "method"},
)
httpRequestsVec, err := metric.RegisterOrGet(registry, "api-service", "http_requests_total", httpRequestsVec)
if err != nil {
    return err
}

// Use the metric with specific label values
httpRequestsVec.WithLabelValues("200", "GET").Inc()
httpRequestsVec.WithLabelValues("404", "POST").Inc()

// Gauge with labels
cacheItemsVec := prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: "cache_items",
        Help: "Number of items in cache by type",
    },
    []string{"cache_type"},
)
cacheItemsVec, err = metric.RegisterOrGet(registry, "cache-service", "cache_items", cacheItemsVec)

// Histogram with labels
requestDurationVec := prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "request_duration_seconds",
        Help:    "Request duration by endpoint",
        Buckets: []float64{.001, .01, .1, 1, 10},
    },
    []string{"endpoint"},
)
requestDurationVec, err = metric.RegisterOrGet(registry, "api-service", "request_duration_seconds",
    requestDurationVec)

HTTP Server

The metrics server provides three endpoints:

  • GET / - HTML page with links to metrics and health endpoints
  • GET /metrics - Prometheus-formatted metrics (default path, configurable)
  • GET /health - 200 with the plain-text body OK

Server configuration:

// Default configuration (port 9090, path /metrics)
securityCfg := security.Config{} // Platform security config
server := metric.NewServer(0, "", registry, securityCfg)

// Custom configuration
server := metric.NewServer(8080, "/prometheus", registry, securityCfg)

// Start server (binds synchronously, then serves in the background)
if err := server.Start(ctx); err != nil {
    log.Fatalf("Failed to start metrics server: %v", err)
}

// Stop server and wait for its serving goroutine to exit
if err := server.Stop(shutdownCtx); err != nil {
    log.Printf("Error stopping server: %v", err)
}

The health endpoint answers every request that reaches it with status 200 and the plain-text body OK; it reports that the server is serving, nothing more. Once Stop has begun, a request is refused with 503 instead.

Prometheus Integration

The package uses the official Prometheus Go client library and exposes metrics in OpenMetrics format. Configure Prometheus to scrape the endpoint:

# prometheus.yml
scrape_configs:
  - job_name: 'semengine'
    static_configs:
      - targets: ['localhost:9090']
    metrics_path: '/metrics'
    scrape_interval: 15s

All core metrics use the namespace "semengine" and appropriate subsystems:

  • semengine_service_status{service="..."}
  • semengine_messages_processed_total{service="...",type="...",status="..."}
  • semengine_nats_connected

Service-specific metrics use the metric name as provided during registration.

Registering From a Service

A service receives the *MetricsRegistry and keeps the collectors RegisterOrGet returns. A recreated service registering the same keys gets the same collectors back, so its writes are gathered:

type MyService struct {
    operations prometheus.Counter
}

func NewMyService(registry *metric.MetricsRegistry) (*MyService, error) {
    operations, err := metric.RegisterOrGet(registry, "my-service", "operations_total",
        prometheus.NewCounter(prometheus.CounterOpts{
            Name: "operations_total",
            Help: "Total operations",
        }))
    if err != nil {
        return nil, err
    }
    return &MyService{operations: operations}, nil
}

A test gives the service its own registry from NewMetricsRegistry and reads what the service recorded through PrometheusRegistry().Gather().

Thread Safety

All registry operations are thread-safe:

  • Registration methods use mutex protection
  • Metric recording is lock-free (Prometheus guarantee)
  • CoreMetrics() returns a thread-safe shared instance
  • PrometheusRegistry() is safe for concurrent access

Example concurrent usage:

registry := metric.NewMetricsRegistry()
coreMetrics := registry.CoreMetrics()

// Safe to call from multiple goroutines
go coreMetrics.RecordMessageProcessed("service-1", "event", "success")
go coreMetrics.RecordMessageProcessed("service-2", "event", "success")
go coreMetrics.RecordMessageProcessed("service-3", "event", "failed")

Error Handling

RegisterOrGet returns a fatal error, stores nothing and leaves the canonical collector untouched for:

  • A nil or typed-nil candidate
  • A key already registered with another concrete type, help text or labels
  • A descriptor another key, a core metric or a directly registered collector already owns

Registering the same key again with an identical collector is not an error: it returns the collector already registered.

Example error handling:

counter, err := metric.RegisterOrGet(registry, "service", "test",
    prometheus.NewCounter(prometheus.CounterOpts{Name: "test", Help: "Test counter"}))
if err != nil {
    log.Fatalf("Failed to register metric: %v", err)
}
counter.Inc()

The Server.Start(ctx) method returns errors for:

  • A nil or already-ended context
  • Server instance already used (Server is one-shot)
  • Nil registry
  • mTLS enabled while server TLS is disabled
  • TLS configuration that cannot be loaded
  • HTTP server failures (port in use, permission denied)

Testing

The package includes comprehensive tests:

  • Unit tests: Core metrics recording, registry operations
  • Integration tests: Full registry lifecycle, Prometheus gathering
  • Race detection: Concurrent access patterns verified

Example test using the registry:

func TestMyService_Metrics(t *testing.T) {
    registry := metric.NewMetricsRegistry()
    service, err := NewMyService(registry)
    require.NoError(t, err)

    // Perform operations
    service.DoWork()

    // Verify metrics through what the registry gathers
    families, err := registry.PrometheusRegistry().Gather()
    require.NoError(t, err)
    // Find operations_total in families and check its value
}

Performance Considerations

Metric recording performance:

  • Counter.Inc(): ~100ns per operation (lock-free)
  • Gauge.Set(): ~100ns per operation (lock-free)
  • Histogram.Observe(): ~150ns per operation (bucket lookup)

Registry operations:

  • Registration: O(1) map insert with mutex
  • Gathering: O(n) for n registered metrics

Memory usage:

  • Core metrics: ~2KB base overhead
  • Per service metric: ~200 bytes
  • Vector metrics: ~200 bytes + (100 bytes × number of label combinations)

The HTTP server adds minimal overhead (~1MB base) and handles Prometheus scraping efficiently with streaming responses.

Architecture Integration

Within this repository, natsclient (WithMetrics, for JetStream metrics) and internal/cache (WithMetrics, WithCoalescingMetrics) register their collectors in a MetricsRegistry.

Data flow:

Component → Core Metrics → Prometheus Registry → HTTP Server → Prometheus

Design Decisions

Centralized Registry: Chose centralized registry over distributed collectors to ensure consistent metric namespace, prevent duplication, and enable runtime metric discovery.

Core vs Service Metrics: Separated platform-level metrics (core) from service-specific metrics to distinguish infrastructure health from application health.

Prometheus Direct Integration: Used official Prometheus client rather than abstraction to leverage native features, avoid wrapper overhead, and ensure compatibility with Prometheus ecosystem.

Synchronous Bind in Server.Start(ctx): Start reports listener ownership or bind failure before returning. Server instances are one-shot. Stop(ctx) attempts graceful shutdown within the caller's budget; failure or expiry triggers a force-close, after which Stop waits for the exact serving goroutine and for every request the server admitted, and it returns the context's error whenever its context has ended; a later Stop then waits again. A request that reaches the handler after Stop has begun is refused with 503. Restart uses a freshly constructed Server.

Examples

Complete service integration:

package main

import (
    "context"
    "log"
    "time"

    "github.com/c360studio/semengine/metric"
    "github.com/c360studio/semengine/pkg/security"
    "github.com/prometheus/client_golang/prometheus"
)

func main() {
    // Create metrics registry
    registry := metric.NewMetricsRegistry()

    // Start metrics server
    securityCfg := security.Config{} // Platform security config
    server := metric.NewServer(9090, "/metrics", registry, securityCfg)
    runtimeCtx := context.Background()
    if err := server.Start(runtimeCtx); err != nil {
        log.Fatalf("Failed to start metrics server: %v", err)
    }
    defer func() {
        shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        if err := server.Stop(shutdownCtx); err != nil {
            log.Printf("Failed to stop metrics server: %v", err)
        }
    }()

    // Get core metrics
    coreMetrics := registry.CoreMetrics()

    // Register service-specific metric
    operationCounter, err := metric.RegisterOrGet(registry, "my-service", "operations_total",
        prometheus.NewCounter(prometheus.CounterOpts{
            Name: "operations_total",
            Help: "Total operations performed",
        }))
    if err != nil {
        log.Fatal(err)
    }

    // Record service status
    coreMetrics.RecordServiceStatus("my-service", 2) // running

    // Simulate work
    for i := 0; i < 100; i++ {
        operationCounter.Inc()
        coreMetrics.RecordMessageProcessed("my-service", "operation", "success")
        time.Sleep(100 * time.Millisecond)
    }
}

For more examples and detailed usage, see the README.md in this directory.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RegisterOrGet

func RegisterOrGet[C prometheus.Collector](r *MetricsRegistry, serviceName, metricName string, candidate C) (C, error)

RegisterOrGet registers candidate under the logical key serviceName.metricName, or returns the collector already registered under that key. There is one canonical collector per key: the caller must use the collector this returns, never its own candidate, which is not gathered when another collector already owns the key.

Same key: the existing collector is returned only when it has the exact concrete type of candidate (a gauge satisfies prometheus.Counter, so the interface alone is not enough) and the same descriptors. A nil or typed-nil candidate, a type or descriptor mismatch, and a candidate whose descriptor another key, a core metric or a collector registered directly through PrometheusRegistry already owns, are refused with a fatal error; nothing is stored and the canonical collector is untouched.

Types

type Metrics

type Metrics struct {
	// Service metrics
	ServiceStatus      *prometheus.GaugeVec
	MessagesReceived   *prometheus.CounterVec
	MessagesProcessed  *prometheus.CounterVec
	MessagesPublished  *prometheus.CounterVec
	ProcessingDuration *prometheus.HistogramVec
	ErrorsTotal        *prometheus.CounterVec
	HealthCheckStatus  *prometheus.GaugeVec

	// LogEntriesTotal counts slog records at WARN level and above, keyed by
	// the component attribute. Downstream products compute windowed rates
	// via PromQL (rate/increase) rather than in-process windowing.
	LogEntriesTotal *prometheus.CounterVec

	// NATS metrics
	NATSConnected      prometheus.Gauge
	NATSRTT            prometheus.Gauge
	NATSReconnects     prometheus.Counter
	NATSCircuitBreaker prometheus.Gauge
}

Metrics contains all platform-level metrics (not domain-specific)

func NewMetrics

func NewMetrics() *Metrics

NewMetrics creates a new Metrics instance with all platform metrics

func (*Metrics) RecordCircuitBreakerState

func (c *Metrics) RecordCircuitBreakerState(state int)

RecordCircuitBreakerState updates circuit breaker status

func (*Metrics) RecordError

func (c *Metrics) RecordError(service, errorType string)

RecordError increments error counter

func (*Metrics) RecordHealthStatus

func (c *Metrics) RecordHealthStatus(service string, healthy bool)

RecordHealthStatus updates health check status

func (*Metrics) RecordMessageProcessed

func (c *Metrics) RecordMessageProcessed(service, messageType, status string)

RecordMessageProcessed increments processed message counter

func (*Metrics) RecordMessagePublished

func (c *Metrics) RecordMessagePublished(service, subject string)

RecordMessagePublished increments published message counter

func (*Metrics) RecordMessageReceived

func (c *Metrics) RecordMessageReceived(service, messageType string)

RecordMessageReceived increments received message counter

func (*Metrics) RecordNATSRTT

func (c *Metrics) RecordNATSRTT(rtt time.Duration)

RecordNATSRTT updates NATS round-trip time

func (*Metrics) RecordNATSReconnect

func (c *Metrics) RecordNATSReconnect()

RecordNATSReconnect increments reconnection counter

func (*Metrics) RecordNATSStatus

func (c *Metrics) RecordNATSStatus(connected bool)

RecordNATSStatus updates NATS connection status

func (*Metrics) RecordProcessingDuration

func (c *Metrics) RecordProcessingDuration(service, operation string, duration time.Duration)

RecordProcessingDuration records processing time

func (*Metrics) RecordServiceStatus

func (c *Metrics) RecordServiceStatus(service string, status int)

RecordServiceStatus updates service status metric

type MetricsRegistry

type MetricsRegistry struct {
	Metrics *Metrics
	// contains filtered or unexported fields
}

MetricsRegistry manages the registration and lifecycle of metrics

func NewMetricsRegistry

func NewMetricsRegistry() *MetricsRegistry

NewMetricsRegistry creates a new metrics registry with core platform metrics

func (*MetricsRegistry) CoreMetrics

func (r *MetricsRegistry) CoreMetrics() *Metrics

CoreMetrics returns the core platform metrics

func (*MetricsRegistry) PrometheusRegistry

func (r *MetricsRegistry) PrometheusRegistry() *prometheus.Registry

PrometheusRegistry returns the underlying Prometheus registry

func (*MetricsRegistry) Unregister

func (r *MetricsRegistry) Unregister(serviceName, metricName string) bool

Unregister removes a metric from the registry

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server represents the metrics HTTP server

func NewServer

func NewServer(port int, path string, registry *MetricsRegistry, securityCfg security.Config) *Server

NewServer creates a new metrics server with the provided registry

func (*Server) Address

func (s *Server) Address() string

Address reports the accepted listener's endpoint while Server owns it. Before startup and after completed Stop, it reports the configured endpoint. The address does not assert that Serve is ready to accept requests.

func (*Server) Start

func (s *Server) Start(ctx context.Context) error

Start starts the one-shot metrics HTTP server. It binds synchronously and returns only after this Server owns its listener. ctx is the exact base context for served requests. Restart requires a freshly constructed Server.

func (*Server) StartWithListener

func (s *Server) StartWithListener(ctx context.Context, listener net.Listener) error

StartWithListener starts the one-shot metrics HTTP server on a caller-bound raw TCP listener. A successful return transfers listener ownership to Server; every returned error leaves it with the caller. Server applies configured TLS itself, so callers must not pass a TLS-wrapped listener.

func (*Server) Stop

func (s *Server) Stop(ctx context.Context) error

Stop attempts graceful shutdown within ctx. If that budget ends or graceful shutdown fails, Stop force-closes the server and listener and then waits for the exact serving goroutine, which returns once its listener is closed. Stop returns ctx's error whenever ctx has ended, even if shutdown completed. A request the server admitted, with the metrics collection it runs, is the server's work until its handler returns: Stop returns nil only after every such request has finished, and when ctx ends first it returns ctx's error and a later Stop waits again. Once Stop has begun, a request that reaches the handler is refused with 503 Service Unavailable and its handler never runs. A completed repeat is a nil no-op. Concurrent Stop is unsupported and returns a typed transient error.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL