monitor

Monitor SDK

Monitor is a lightweight Go metrics SDK with Prometheus Remote Write and an optional Prometheus HTTP pull exporter. It provides counters, labeled counters/gauges, official Prometheus histograms, system metrics, bounded snapshot construction, and DNS-aware Remote Write delivery.

Installation

go get github.com/nikiz24/monitor

Push-only quick start

The pull exporter is disabled by default. Existing push-only configurations remain valid and do not start an HTTP listener or allocate a snapshot cache.

package main

import (
    "log"
    "time"

    "github.com/nikiz24/monitor"
)

func main() {
    cfg := monitor.Config{
        Namespace:           "myapp",
        Subsystem:           "prod",
        ServiceName:         "api",
        RemoteWriteURL:      "http://prometheus:9090/api/v1/write",
        RemoteWriteInterval: 15 * time.Second,
    }
    if err := monitor.Init(cfg); err != nil {
        log.Fatal(err)
    }
    defer monitor.Shutdown()

    monitor.IncrementCounter("requests_total")
    monitor.SetLabeledCounter("connections", 42, "type", "websocket")
    monitor.ObserveHistogram("response_time_seconds", 0.123)
}

Prometheus pull exporter

Set PrometheusExporter to enable pull. The following configuration enables both Remote Write and /metrics:

cfg := monitor.Config{
    Namespace: "myapp", Subsystem: "prod", ServiceName: "api",
    RemoteWriteURL: "http://prometheus:9090/api/v1/write",
    RemoteWriteInterval: 15*time.Second,
    PrometheusExporter: &monitor.PrometheusExporterConfig{ListenAddress: ":9091", Path: "/metrics"},
}

Use this Prometheus scrape configuration:

scrape_configs:
  - job_name: monitor-sdk
    scrape_interval: 15s
    honor_labels: true
    static_configs:
      - targets:
          - myapp:9091

Warning: push and pull may be enabled together, but they must never feed the same logical backend. The SDK does not deduplicate the two delivery paths and cannot make them transactional; sending both to one backend produces duplicate or inconsistent samples.

For pull-only operation, leave RemoteWriteURL empty. Leaving PrometheusExporter nil preserves push-only behavior.

Effective defaults

Zero values below select these defaults after the exporter is enabled. ListenAddress is the exception: it is required and has no default.

The default 1-second cache is appropriate for 15-, 30-, and 60-second scrape intervals. For a scrape interval below 10 seconds, set SnapshotCacheTTL <= scrape_interval / 10. Increasing the TTL reduces rebuilds but deliberately serves older values; it is a freshness tradeoff, not free capacity.

MaxStoredSeries=1,000,000 is an emergency ceiling for the active metric store. It is not the 1-second snapshot cache. Byte and per-series label limits usually reject new cardinality first. Admission may clean expired series when approaching a limit, but the global guard does not evict unexpired old series. The standard labeled collector TTL remains 60 minutes unless explicitly changed.

HTTP behavior and failure responses

The endpoint supports GET and HEAD; GET negotiates gzip from Accept-Encoding, and HEAD returns the same representation headers without a body. Other methods return 405. The exporter owns a dedicated http.Server and http.ServeMux; it never registers with http.DefaultServeMux.

The listener provides no TLS and no authentication. Bind it to a protected interface or place an authenticated TLS reverse proxy in front of it. The path must be an absolute, clean, canonical literal ASCII path; escaped or alternate spellings are rejected so the dedicated mux has one route identity.

• 503 means scrape concurrency exhaustion, a snapshot/resource byte limit, or a busy retired-payload generation.
• 500 means validation, collector, or output-build failure before a response is committed.
• A response write failure after headers are committed is reported through observability and ErrorHandler; it cannot change the already committed 200.
• A failed refresh never serves an expired or stale snapshot as fallback.

Labels and collisions

The SDK's raw common labels are _instance_, instance, _target_, and every entry in Config.CustomLabels. Business labels supplied to a metric are merged with those labels consistently in push and pull. __name__ is special to Remote Write and is not emitted as an ordinary label in text exposition.

Keep honor_labels: true in Prometheus. With honor_labels: false, Prometheus renames the SDK's instance label to exported_instance when it supplies its own target label, so pull labels no longer match Remote Write labels.

The system-reserved labels are __name__, _instance_, instance, _target_, le, and quantile. When PrometheusExporter is enabled, CustomLabels additionally cannot use the fixed self-metric dimensions result, kind, consumer, code, or reason. Those names remain valid business labels on other metric families. Duplicate labels and conflicts between custom and business labels are rejected.

When PrometheusExporter is enabled, the following 11 self-metric families are SDK-owned and cannot be registered by business collectors:

monitor_snapshot_builds_total
monitor_snapshot_build_duration_seconds
monitor_snapshot_series
monitor_snapshot_bytes
monitor_snapshot_cache_requests_total
monitor_exporter_requests_total
monitor_exporter_requests_in_flight
monitor_metric_errors_total
monitor_dropped_series_total
monitor_remote_write_batches_total
monitor_remote_write_batch_series

Push-only managers preserve the legacy Remote Write profile: they do not emit SDK self metrics, so the raw family and dimension names above remain available to business collectors and CustomLabels in that mode.

Checked APIs such as TryAddCounter, TryIncrementLabeledCounter, TrySetLabeledCounter, and RegisterCollector return validation or ownership errors synchronously. Legacy void update APIs preserve their signatures: they drop an invalid operation and record a fixed-cardinality monitor_metric_errors_total{reason=...} sample plus a rate-limited log rather than creating a dynamic error series.

Collectors and histograms

The legacy Collector interface remains supported:

type MyCollector struct {
    monitor.BaseCollector
}

func (c *MyCollector) Collect() []monitor.Metric {
    return []monitor.Metric{{
        Name: "queue_depth", Value: 3, MetricType: monitor.Gauge,
        Labels: map[string]string{"queue": "default"},
        Timestamp: time.Now(),
    }}
}

Register global collectors with monitor.RegisterCollector, which returns any checked registration error. A custom manager also exposes CheckedCollectorRegistrar.RegisterCollectorChecked without changing the legacy Manager interface.

SnapshotCollector is the allocation-efficient fast path. Its CollectTo method receives a synchronous SnapshotAppender; the collector must not retain the appender, use it concurrently, mutate submitted histogram DTOs, or claim unvalidated capacity. This is a trust boundary for custom code. The SDK bounds its own snapshot and payload allocations, but it cannot bound arbitrary memory allocated inside a legacy Collector.Collect() before that method returns.

Histograms use the official prometheus.Histogram implementation internally. They are intentionally unregistered and never touch prometheus.DefaultRegistry or prometheus.DefaultGatherer.

monitor.RegisterHistogramBuckets("response_time_seconds", []float64{0.01, 0.1, 1, 5})
monitor.ObserveHistogram("response_time_seconds", 0.125)

Cardinality and 100K operation

Labeled collectors support explicit TTL and local max-series controls:

collector := monitor.NewApplicationCollector("http_requests", logger)
collector.SetTTL(30 * time.Minute)
collector.SetMaxSeries(50_000)
collector.Inc("total", "method", "GET", "route", "/v1/users")

The exporter is designed to build and expose 100,000 ordinary labeled series when their actual label mix fits the configured store, per-series, and snapshot budgets. The 1,000,000-series default is not a promise that every label shape fits in 256 MiB or that a snapshot fits in 128 MiB. Benchmark representative labels, avoid unbounded IDs, and monitor the SDK self metrics.

Remote Write expands at most RemoteWriteBatchSeries (10,000 by default) at a time, so a 100K snapshot does not become one 100K protocol object graph. Batches are sent sequentially. Delivery is best effort and may be partial: accepted batches cannot be rolled back if a later batch fails, and the next interval attempts a fresh complete snapshot. There is no WAL or unbounded retry queue. ForceWrite uses the same non-blocking single-flight gate as the periodic writer. If a write is already active, another call returns ErrRemoteWriteBusy immediately, before snapshot cache lookup, snapshot build, or network request work begins.

Status, health, and error callbacks

monitor.HealthCheck() reports an enabled exporter that has not started, lost its listener, or failed at runtime. A disabled exporter does not make an otherwise initialized manager unhealthy.

monitor.GetStatus() includes the existing initialization fields and these exporter fields:

exporter_enabled
exporter_listening
exporter_address
exporter_path
snapshot_cached
snapshot_age
stored_series
estimated_store_bytes
last_exporter_error
last_exporter_error_time

Config.ErrorHandler receives exporter lifecycle, snapshot-build, and response output failures. Dispatch is asynchronous from the failing operation, serialized to one callback at a time, bounded in retained error state, and panic-safe. The callback must return promptly: an indefinitely blocked callback delays all later error notifications. Hot-path metric validation uses the fixed self metric and logging path rather than invoking this callback.

Basic metric APIs

monitor.IncrementCounter("requests_total")
monitor.AddCounter("bytes_processed_total", 1024)
monitor.SetCounter("active_connections", 42)

monitor.IncrementLabeledCounter("http_requests_total",
    "method", "GET", "status", "200")
monitor.SetLabeledCounter("connected_users", 5,
    "region", "us-west", "type", "premium")
monitor.DeleteLabeledCounter("connected_users",
    "region", "us-west", "type", "premium")

Prometheus sample values are float64. Integer and internal counters remain monotonic according to their APIs, but after 2^53 the exported Prometheus value cannot represent every one-unit increment. Do not rely on unit precision past that point or interpret the implementation's integer storage as a promise that a counter can never wrap.

System metrics and DNS

if err := monitor.RegisterSystemMetricsCollector(logger); err != nil {
    log.Fatal(err)
}

alloc, sys, heapInUse := monitor.GetProcessMemory()
_ = alloc
_ = sys
_ = heapInUse

Advanced Remote Write DNS resolution remains optional:

cfg := monitor.Config{
    Namespace: "myapp", Subsystem: "prod", ServiceName: "api",
    RemoteWriteURL: "http://prometheus:9090/api/v1/write",
    RemoteWriteInterval: 15 * time.Second,
    DNSEnable: true,
    DNSCacheTTL: 10 * time.Minute,
    DNSRefreshInterval: 5 * time.Minute,
    DNSTimeout: 800 * time.Millisecond,
    DNSUDPServers: []string{"1.1.1.1:53", "8.8.8.8:53"},
    DNSTLSServers: []string{"1.1.1.1:853"},
    DNSDoHEndpoints: []string{"https://cloudflare-dns.com/dns-query"},
}

Always call monitor.Shutdown() so the exporter, cache, Remote Write, and DNS workers release their resources.

Verification

Run the complete functional suite with strict pointer checking through the formal target:

make test-checkptr

The target executes this exact command:

go test -tags=checkptr_full -gcflags=all=-d=checkptr=2 ./... -count=1

The checkptr_full tag disables only numeric testing.AllocsPerRun assertions, because checkptr intentionally adds per-operation allocations. Each measured operation still runs once, and all functional, protocol, GC, finalizer, memory-budget, and ownership tests remain enabled.

License

MIT License. See LICENSE.