observability

Observability Package

The observability package provides Prometheus metrics collection and OpenTelemetry tracing for cd-operator.

Features

• Prometheus Metrics: Collect and export metrics about log entries, errors, and log write duration
• Pull Mode: HTTP endpoint for Prometheus scraping (/metrics)
• Push Mode: Push metrics to Prometheus Pushgateway on exit or periodically
• OpenTelemetry Tracing: Distributed tracing with trace ID correlation
• Metrics Writer: Automatic metrics collection from log output

Components

1. Configuration (config.go)

Define how metrics and tracing are configured:

// Metrics configuration
config := observability.DefaultMetricsConfig()
config.Enabled = true
config.Mode = "pull"           // "pull", "push", or "disabled"
config.PullPort = ":8081"      // HTTP port for /metrics endpoint
config.JobName = "cd-operator"

// Tracing configuration
tracingConfig := observability.DefaultTracingConfig()
tracingConfig.Enabled = true
tracingConfig.ServiceName = "cd-operator"

2. Metrics Collection (metrics.go)

Prometheus metrics for logging observability:

• cd_operator_log_total - Counter: Total log entries by level
• cd_operator_error_total - Counter: Total errors by level
• cd_operator_log_duration_seconds - Histogram: Time spent writing logs

registry := prometheus.NewRegistry()
metrics := observability.NewMetricsWithRegistry(registry)

// Wrap your writer to collect metrics automatically
writer := observability.NewMetricsWriter(os.Stdout, metrics)

3. Metrics Export (exporter.go)

Export metrics via pull or push mode:

Pull Mode (Prometheus scrapes):

config := observability.MetricsConfig{
    Mode:     "pull",
    PullPort: ":8081",
}
exporter := observability.NewMetricsExporter(config, registry)
// Metrics available at http://localhost:8081/metrics

Push Mode (Push to Pushgateway):

config := observability.MetricsConfig{
    Mode:           "push",
    PushGatewayURL: "http://localhost:9091",
    PushOnExit:     true,
    JobName:        "cd-operator",
    InstanceID:     "instance-123",
}
exporter := observability.NewMetricsExporter(config, registry)

4. Tracing (tracer.go, span.go)

OpenTelemetry distributed tracing:

// Initialize tracer
tp, err := observability.InitTracer("cd-operator")
if err != nil {
    return err
}
defer tp.Shutdown(ctx)

// Create spans
ctx, span := observability.StartSpan(ctx, "deploy-application")
defer span.End()

// Extract trace ID for log correlation
traceID := observability.TraceIDFromContext(ctx)
log.Info(ctx, "deploying", "trace_id", traceID)

Usage Example

Complete Setup

package main

import (
    "context"
    "os"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/grhili/cd-operator/pkg/observability"
)

func main() {
    ctx := context.Background()

    // Initialize tracing
    tp, err := observability.InitTracer("cd-operator")
    if err != nil {
        panic(err)
    }
    defer tp.Shutdown(ctx)

    // Configure metrics (pull mode)
    config := observability.DefaultMetricsConfig()
    config.Enabled = true
    config.Mode = "pull"
    config.PullPort = ":8081"

    // Create metrics and exporter
    registry := prometheus.NewRegistry()
    metrics := observability.NewMetricsWithRegistry(registry)
    exporter := observability.NewMetricsExporter(config, registry)
    metrics.SetExporter(exporter)

    // Wrap stdout with metrics collection
    writer := observability.NewMetricsWriter(os.Stdout, metrics)

    // Create your logger with this writer
    // logger := logger.New(logger.Config{Writer: writer})

    // Start a traced operation
    ctx, span := observability.StartSpan(ctx, "main-operation")
    defer span.End()

    // Get trace ID for correlation
    traceID := observability.TraceIDFromContext(ctx)
    _ = traceID

    // Do work...

    // Shutdown exporter
    defer exporter.Shutdown(context.Background())
}

Integration with Logger

// Create metrics
registry := prometheus.NewRegistry()
metrics := observability.NewMetricsWithRegistry(registry)

// Wrap writer
writer := observability.NewMetricsWriter(os.Stdout, metrics)

// Use with logger
logger := logger.New(logger.Config{
    Writer: writer,
    Level:  "info",
})

// All log writes will now record metrics
logger.Info(ctx, "application started")

Metrics Details

Counters

• cd_operator_log_total{level="info"} - Total INFO logs
• cd_operator_log_total{level="error"} - Total ERROR logs
• cd_operator_log_total{level="warn"} - Total WARN logs
• cd_operator_log_total{level="debug"} - Total DEBUG logs
• cd_operator_error_total{level="error"} - Total write errors

Histograms

• cd_operator_log_duration_seconds - Time to write logs (seconds)
  • Buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10

Architecture

Application Code
      |
      v
  Logger/Writer
      |
      v
MetricsWriter (records metrics)
      |
      +---> Prometheus Registry
      |           |
      |           v
      |     MetricsExporter
      |           |
      |           +---> Pull: HTTP Server (:8081/metrics)
      |           +---> Push: Pushgateway
      |
      v
  Underlying Writer (os.Stdout)

Design Principles

Following code-convention.md:

1. Value types for config - MetricsConfig and TracingConfig are value types
2. Pointer types for services - Metrics and MetricsExporter are pointer types
3. Context-aware - All operations accept context.Context
4. Typed parameters - No raw strings, everything is strongly typed
5. Best-effort metrics - Metrics failures don't crash the application
6. Clean shutdown - Exporter.Shutdown() pushes final metrics

Dependencies

• github.com/prometheus/client_golang - Prometheus metrics
• go.opentelemetry.io/otel - OpenTelemetry tracing
• go.opentelemetry.io/otel/exporters/stdout/stdouttrace - Stdout trace exporter

Testing

Run tests:

go test -v ./pkg/observability/...

All tests use isolated registries to avoid conflicts.