monitoring

Monitoring System for Seshat

A comprehensive monitoring and observability system for Seshat providing metrics, logging, and instrumentation.

Components

1. Metrics System

Provides structured metrics collection with multiple metric types:

Counters

Monotonically increasing metrics for event counting:

counter := monitoring.NewCounter("api_requests_total", map[string]string{
    "subsystem": "api",
    "type":      "requests",
})
counter.Inc()
counter.Add(5)

Gauges

Point-in-time metrics for current state:

gauge := monitoring.NewGauge("active_sessions", map[string]string{
    "subsystem": "system",
    "type":      "sessions",
})
gauge.Set(42.5)
gauge.Add(10.0)

Timers

Duration metrics with percentile calculations:

timer := monitoring.NewTimer("api_latency_ms", map[string]string{
    "subsystem": "api",
    "type":      "latency",
})
timer.Record(150 * time.Millisecond)

// Or using Start/Stop pattern
stop := timer.Start()
// ... perform operation
stop()

Histograms

Value distribution metrics:

histogram := monitoring.NewHistogram("request_duration_ms", labels, []float64{
    1, 5, 10, 25, 50, 100, 250, 500, 1000,
})
histogram.Observe(42.5)

2. Logging System

Structured logging with context awareness:

logger := monitoring.NewLoggerWithConfig(&monitoring.LoggerConfig{
    Level:         monitoring.LogLevelInfo,
    Output:        "stdout",
    Format:        "text",
    ContextFields: []string{"session_id", "turn_id"},
})

// Simple logging
logger.Info("Message", nil)

// Context-aware logging
ctx := context.Background()
logger.InfoWithContext(ctx, "Contextual message", nil)

// Logging with fields
logger.ErrorWithStack("Error occurred", err, map[string]interface{}{
    "operation": "api_call",
    "user_id":   "user-123",
})

3. Monitoring System

Centralized monitoring that aggregates all subsystem metrics:

monitoringSystem := monitoring.NewSystem(logger)

// Record API metrics
monitoringSystem.RecordAPIRequest()
monitoringSystem.RecordAPISuccess(duration)
monitoringSystem.RecordAPIFailure(err, duration)

// Record tool metrics
monitoringSystem.RecordToolCall("file_read")
monitoringSystem.RecordToolSuccess("file_read", duration)
monitoringSystem.RecordToolFailure("file_read", err, duration)

// Record query metrics
monitoringSystem.RecordQueryTurn()
monitoringSystem.RecordQuerySuccess(duration)
monitoringSystem.RecordQueryFailure(err, duration)

// Get all metrics
metrics := monitoringSystem.GetAllMetrics()

// Export metrics
snapshot, _ := monitoringSystem.GetMetricsSnapshot("prometheus")
fmt.Println(snapshot)

Metric Types Available

API Metrics

• api_requests_total - Total API requests
• api_requests_success - Successful API requests
• api_requests_failure - Failed API requests
• api_latency_ms - API request latency (with percentiles)
• api_rate_limit_hits - Rate limit events
• api_circuit_breaker_trips - Circuit breaker activations

Tool Metrics

• tool_calls_total - Total tool calls
• tool_calls_success - Successful tool calls
• tool_calls_failure - Failed tool calls
• tool_latency_ms - Tool execution latency
• tool_permission_denials - Permission denial events

Query Loop Metrics

• query_turns_total - Total query turns
• query_iterations_total - Total query loop iterations
• query_latency_ms - Query turn latency
• query_errors - Query errors

Circuit Breaker Metrics

• circuit_breaker_state_transitions - State change events
• circuit_breaker_current_state - Current state (0=closed, 1=open, 2=half-open)

System Metrics

• active_sessions - Currently active sessions
• active_connections - Active network connections
• memory_usage_mb - Memory usage in MB
• total_errors - Total error count

Configuration

Logger Configuration

type LoggerConfig struct {
    Level         LogLevel           // Minimum log level
    Output        string             // stdout, stderr, or file
    FilePath     string             // File path when Output="file"
    Format        string             // text or json
    ContextFields []string           // Context fields to extract
}

Metric Configuration

Metrics can have custom labels for filtering and grouping:

labels := map[string]string{
    "subsystem": "api",
    "provider":  "anthropic",
    "model":      "claude-3-5-sonnet",
}

Testing

Run Tests

# Run all monitoring tests
go test ./internal/monitoring/... -v

# Run specific test suite
go test ./internal/monitoring/metrics_test.go -v
go test ./internal/monitoring/logger_test.go -v
go test ./internal/monitoring/system_test.go -v

Test Coverage

# Generate coverage report
go test ./internal/monitoring/... -coverprofile=coverage.out
go tool cover -html=coverage.html

Integration Examples

1. Basic Setup

import "github.com/EngineerProjects/Nexus_ai/apps/seshat/internal/monitoring"

func main() {
    // Create logger
    logger := monitoring.NewLogger()
    logger.SetLevel(monitoring.LogLevelInfo)

    // Create monitoring system
    monitoringSystem := monitoring.NewSystem(logger)

    // Use monitoring
    monitoringSystem.RecordAPIRequest()
    // ... operation ...
    monitoringSystem.RecordAPISuccess(duration)
}

2. Advanced Setup with Context

func handleRequest(ctx context.Context, request Request) error {
    // Record request start
    monitoringSystem.RecordAPIRequest()

    start := time.Now()

    // Handle request
    err := processRequest(ctx, request)

    duration := time.Since(start)

    if err != nil {
        monitoringSystem.RecordAPIFailure(err, duration)
        logger.LogErrorWithContext(ctx, err, "process_request")
        return err
    }

    monitoringSystem.RecordAPISuccess(duration)
    logger.InfoWithContext(ctx, "Request completed", nil)
    return nil
}

3. Custom Metrics

func setupCustomMetrics(monitoringSystem *monitoring.System) {
    // Create custom counter
    customCounter := monitoring.NewCounter("custom_events", map[string]string{
        "category": "business",
        "type":      "events",
    })

    // Register custom metric
    monitoringSystem.RegisterCustomMetric("custom_events", customCounter)

    // Use custom metric
    customCounter.Inc()
}

Performance Characteristics

Memory Usage

• Counter: ~48 bytes per instance
• Gauge: ~56 bytes per instance
• Timer: ~128 bytes per instance + bucket storage
• Histogram: ~200 bytes + bucket storage
• Logger: ~4KB default buffer
• System: ~10KB base + metrics storage

CPU Usage

• Counter operations: ~10ns per increment
• Gauge operations: ~15ns per set
• Timer operations: ~50-100ns per record
• Logger operations: ~100-500ns per log entry
• System queries: ~1-2µs for GetAllMetrics()

Thread Safety

All metric types are thread-safe using mutexes:

• Counters: Atomic operations where possible
• Gauges: Full mutex protection
• Timers: Mutex for histogram operations
• System: RWMutex for metrics map access

Best Practices

1. Label Strategy

Use consistent, meaningful labels:

// Good
labels := map[string]string{
    "subsystem": "api",
    "provider":  "anthropic",
    "model":      "claude-3-5-sonnet",
}

// Bad - too specific
labels := map[string]string{
    "request_id": "req-12345",
    "timestamp":   "2023-11-15T10:30:00Z",
}

// Bad - too generic
labels := map[string]string{
    "env": "production",
    "app": "seshat",
}

2. Metric Granularity

Balance between detail and cardinality:

// Good - reasonable cardinality
metric := NewTimer("api_latency_ms", map[string]string{
    "subsystem": "api",
    "endpoint":  "/v1/messages", // Low cardinality
})

// Bad - high cardinality (many unique values)
metric := NewTimer("api_latency_ms", map[string]string{
    "subsystem": "api",
    "user_id":   "{user_id}", // Very high cardinality
})

3. Log Level Strategy

Use appropriate log levels:

• Debug: Detailed development information
• Info: Normal operational information
• Warn: Warning conditions that don't stop operation
• Error: Error conditions that affect operation
• Fatal: Fatal errors that require immediate termination

4. Context Propagation

Always propagate context for distributed tracing:

func handleRequest(ctx context.Context) {
    // Add request ID to context
    requestID := generateRequestID()
    ctx = context.WithValue(ctx, "request_id", requestID)

    // All operations use context
    processRequest(ctx)
    logRequest(ctx, requestID)
}

Troubleshooting

Metrics Not Updating

Problem: Metrics show zero values

Solutions:

1. Verify monitoring system is initialized
2. Check that Record* methods are being called
3. Ensure no error suppression in metric code
4. Review log output for initialization errors

High Memory Usage

Problem: Memory grows continuously

Solutions:

1. Review histogram bucket configurations
2. Check for metric leaks (unbounded growth)
3. Implement metric sampling for high-cardinality metrics
4. Consider metric expiration policies

Poor Performance

Problem: Monitoring overhead is high

Solutions:

1. Reduce log verbosity
2. Sample high-frequency metrics
3. Use appropriate metric types (counter vs timer)
4. Optimize metric recording frequency

File Structure

internal/monitoring/
├── metrics.go           # Core metric types
├── logger.go           # Logging system
├── system.go           # Centralized monitoring
├── metrics_test.go      # Unit tests for metrics
├── logger_test.go      # Unit tests for logger
└── system_test.go      # Unit tests for system

License

This monitoring system is part of Seshat and follows the same license terms.

Contributing

For bug reports, feature requests, or contributions:

1. Check existing issues and pull requests
2. Follow the coding style and conventions
3. Add tests for new features
4. Update documentation as needed