monitor

Monitor Package

The Monitor Package is a high-performance, production-ready health monitoring system for Go applications. It provides a robust framework for tracking the operational status of internal components and external dependencies using an intelligent state machine with built-in hysteresis and lock-free metrics reporting.

Table of Contents

• Overview
• Architecture
• Performance
• Subpackages
  • info
  • pool
  • status
• Use Cases
• Quick Start
• Best Practices
• API Reference
• Contributing
• Resources

Overview

The monitor package is designed to provide "observation without interference". It allows developers to register periodic health checks that automatically transition through health states (OK, Warn, KO) based on configurable failure and recovery thresholds.

Design Philosophy

1. Lock-Free Hot Path: Reading the status or metrics of a monitor is architected using atomic operations, ensuring zero contention even under thousands of concurrent requests.
2. Dampened Transitions: Hysteresis logic prevents "alert flapping" by requiring consecutive successes or failures before triggering a state change.
3. Context-Aware: Every health check execution is bounded by a context timeout, ensuring that hanging diagnostics do not block the system.
4. Middleware-First: Execution is wrapped in a LIFO stack, allowing for easy injection of tracing, logging, or custom metrics logic.

Key Features

• ✅ Three-State Machine: Full lifecycle tracking (OK ↔ Warn ↔ KO).
• ✅ Adaptive Ticker: Dynamically adjusts polling frequency during transition phases (Rise/Fall).
• ✅ Atomic Metrics: High-precision tracking of Latency, Uptime, Downtime, and Transition times.
• ✅ Prometheus Integration: Built-in dispatching logic for automated metrics exporting.
• ✅ Metadata Management: Dynamic runtime information through the info subpackage.
• ✅ Zero-Allocation Reads: Optimized memory path for high-frequency status polling.

Key Benefits

• vs Standard Tickers: Provides a complete state machine and metrics container out-of-the-box, rather than just a periodic trigger.
• vs Basic Maps: Thread-safety is guaranteed through atomic primitives rather than global mutexes, offering superior scalability on multi-core systems.

Architecture

Package Structure

monitor/
├── monitor.go               # Implementation of the core Monitor orchestrator
├── interface.go             # Public interface definitions and factory
├── model.go                 # Internal structures and atomic containers
├── last.go                  # High-performance metrics & status storage
├── server.go                # Ticker runner and periodic execution logic
├── internalConfig.go        # Configuration normalization and validation
├── middleware.go            # Execution pipeline implementation
├── encode.go                # JSON/Text serialization logic
├── doc.go                   # GoDoc package documentation
│
├── info/                    # Metadata management sub-package
├── pool/                    # Group management and batch operations
├── status/                  # Status enumeration and multi-format parsing
└── types/                   # Cross-package shared interfaces

Package Architecture

The monitor uses a Split-State Architecture. Configuration and Metadata are stored in thread-safe but high-level containers, while operational metrics are stored in a dedicated lastRun structure using low-level sync/atomic primitives.

[ Monitor Instance ]
       |
       +--- [ Config Context ] ---> (Logger, Ticker Intervals, Thresholds)
       |
       +--- [ Metadata Container ] ---> (Atomic Name, Version, Custom Data)
       |
       +--- [ Background Runner ] ---> (Ticker Goroutine)
       |
       +--- [ Performance Metrics ] ---> (Atomic Status, Latency, Uptime Counters)

Dataflow

The periodic check cycle follows a structured pipeline:

1. Ticker Tick --------> 2. Interval Resolver ----> 3. Middleware Stack
                               |                          |
   (Adjusts speed if           |                          |-- [ mdlStatus ] (Start Time)
    Rising or Falling)         |                          |-- [ User Function ] (Diagnostic)
                               |                          |-- [ mdlStatus ] (Set Result)
                               |                          |
4. Metrics Export <---- 5. State Transition <-------------+
      |                 (Update Counters & Status)
      v
[ Prometheus / Logs ]

Performance

The monitor is optimized for zero-contention on the read path. The following benchmarks were captured on an Intel Core i7-4700HQ.

Note: Status and Metric reads are lock-free and do not produce pressure on the Garbage Collector.

Subpackages

info

Dynamic metadata management. It allows attaching functions to retrieve runtime data (like version or git hash) only when requested.

• Documentation: info/README.md

pool

Manages monitor groups. Provides batch control (Start/Stop all) and aggregated Prometheus exporters.

• Documentation: pool/README.md

status

Type-safe status enumeration. Handles conversions and multi-format marshalling (JSON/YAML/TOML).

• Documentation: status/README.md

Use Cases

1. External API Resilience

Monitor third-party services with "dampened" transitions to avoid false alarms on transient glitches.

cfg := types.Config{
    FallCountWarn: 3, // Ignore isolated failures
    IntervalCheck: duration.ParseDuration("30s"),
}

2. High-Frequency Telemetry

Feed Prometheus scrapers or liveness probes using the lock-free read path without impacting diagnostic performance.

// Atomic read (~3ns) - No impact on system latency
status := mon.Status() 

Quick Start

import (
    "context"
    "github.com/nabbar/golib/monitor"
    "github.com/nabbar/golib/monitor/info"
    "github.com/nabbar/golib/monitor/types"
    "github.com/nabbar/golib/duration"
)

func main() {
    inf, _ := info.New("api-service")
    mon, _ := monitor.New(context.Background(), inf)
    
    _ = mon.SetConfig(context.Background(), types.Config{
        IntervalCheck: duration.ParseDuration("10s"),
        FallCountKO: 3,
    })
    
    mon.SetHealthCheck(func(ctx context.Context) error {
        return db.PingContext(ctx)
    })
    
    _ = mon.Start(context.Background())
    defer mon.Stop(context.Background())
}

Best Practices

✅ DO

• Use Eventually in tests: Since monitoring is asynchronous, use non-blocking matchers.
• Respect Context: Ensure your diagnostic function honors the ctx provided to handle timeouts.
• Register Metrics Early: Association with Prometheus should be done during initialization.

❌ DON'T

• Don't use time.Sleep: The monitor orchestrator already handles intervals.
• Don't block the Read Path: The package provides atomic counters; do not wrap them in heavy mutex-guarded logic.

API Reference

1. Primary Factory

2. Monitor Interface

The Monitor interface aggregates multiple specialized behaviors.

Lifecycle Methods

Configuration & Core Logic

Status & State (MonitorStatus)

Metrics & Prometheus (MonitorMetrics)

Metadata Management (MonitorInfo)

3. Config Structure (types.Config)

Contributing

Contributions are welcome! Please follow these guidelines:

1. Code Quality

• Follow Go best practices and idioms
• Maintain or improve code coverage (target: >80%)
• Pass all tests including race detector
• Use gofmt, golangci-lint and gosec

1. AI Usage Policy

• ❌ AI must NEVER be used to generate package code or core functionality
• ✅ AI assistance is limited to:
  • Testing (writing and improving tests)
  • Debugging (troubleshooting and bug resolution)
  • Documentation (comments, README, TESTING.md)
• All AI-assisted work must be reviewed and validated by humans

1. Testing

• Add tests for new features
• Use Ginkgo v2 / Gomega for test framework
• Ensure zero race conditions
• Maintain coverage above 80%

1. Documentation

• Update GoDoc comments for public APIs
• Add examples for new features
• Update README.md and TESTING.md if needed

1. Pull Request Process

• Fork the repository
• Create a feature branch
• Write clear commit messages
• Ensure all tests pass
• Update documentation
• Submit PR with description of changes

Resources

Documentation

• TESTING.md: Exhaustive test inventory, performance benchmarks, and CPU/Memory profiling data.

Monitoring Standards & Industry References

• Google SRE Book: Monitoring Distributed Systems: Core theoretical framework for service monitoring, explaining the difference between symptoms and causes, and the "Four Golden Signals".
• Kubernetes Probes Documentation: Industry standard for container orchestration probes. This package's transition logic (Fall/Rise counts) is designed to align with Kubernetes liveness and readiness probe behaviors.
• HAProxy Health Check Configuration: Operational reference for load-balancer health monitoring, detailing intervals and thresholds for high-availability systems.
• Traefik Backend Health Monitoring: Configuration standards for dynamic proxy health checks used in modern microservices architectures.
• Prometheus Metric Naming Best Practices: Essential guide for ensuring that metrics registered via RegisterMetricsName follow standard conventions for dashboards and alerting.

Summary

These resources provide the context for why this package exists. The Monitor Package provides the Go implementation of the Hysteresis and State Machine patterns used by orchestrators like Kubernetes, with the Atomic-Read performance required for SRE-grade telemetry.

AI Transparency

In compliance with EU AI Act Article 50.4: AI assistance was used for performance profiling, test inventory generation, and documentation synchronization under human supervision. Core monitoring logic is human-designed and validated.

License

MIT License - See LICENSE file for details.

Copyright (c) 2022-2025 Nicolas JUHEL