log

Lynx Logging Framework

A high-performance, production-ready logging framework for Go applications, built on top of zerolog and integrated with Kratos.

Features

Core Features

• ✁EMultiple Log Levels: Debug, Info, Warn, Error, Fatal
• ✁EStructured Logging: JSON format with custom fields
• ✁EContext Support: Context-aware logging with trace/span IDs
• ✁EMultiple Outputs: Console and file output simultaneously
• ✁EHot Reload: Dynamic configuration updates without restart

Advanced Features

• ✁ETime-based Rotation: Daily, hourly, or weekly log rotation
• ✁ESize-based Rotation: Automatic rotation when file size limit reached
• ✁ETotal Size Limit: Automatic cleanup of old log files
• ✁ECompression: Automatic compression of rotated logs
• ✁EBatch Writing: Reduces system calls by 90%+
• ✁EAsync Writing: Non-blocking log writes
• ✁ESampling & Rate Limiting: Control log volume
• ✁EStack Traces: Configurable stack trace capture
• ✁EPerformance Metrics: Built-in performance monitoring

Format Support

• ✁EJSON Format: Structured logging (default for files)
• ✁EPretty Format: Human-readable console output
• ✁EText Format: Simple text format
• ✁EColor Output: Colored console output (configurable)

Quick Start

Basic Usage

package main

import (
    "github.com/go-lynx/lynx/log"
)

func main() {
    // Log messages
    log.Info("Application started")
    log.Infof("User %s logged in", "john")
    log.Error("Failed to connect to database")
    
    // Structured logging
    log.Infow("key1", "value1", "key2", "value2")
    
    // With context
    ctx := context.Background()
    log.InfoCtx(ctx, "Processing request")
}

Initialization

The logger is automatically initialized by the Lynx framework. For manual initialization:

import (
    "github.com/go-lynx/lynx/log"
    kconf "github.com/go-kratos/kratos/v2/config"
)

func initLogger(cfg kconf.Config) error {
    return log.InitLogger(
        "my-service",      // service name
        "host-123",        // host identifier
        "1.0.0",           // version
        cfg,               // config instance
    )
}

Configuration

Configuration File (YAML)

lynx:
  log:
    # Basic configuration
    level: info                 # debug/info/warn/error
    console_output: true        # output to console
    file_path: logs/app.log     # log file path (empty = no file output)
    
    # File rotation
    max_size_mb: 128            # Maximum file size in MB before rotation
    max_backups: 10             # Maximum number of backup files to keep
    max_age_days: 7             # Maximum age of log files in days
    compress: true              # Compress rotated log files
    
    # Time-based rotation (NEW)
    rotation_strategy: "both"   # "size" | "time" | "both"
    rotation_interval: "daily"  # "hourly" | "daily" | "weekly"
    
    # Total size limit (NEW)
    max_total_size_mb: 1024     # Maximum total size of all log files (0 = unlimited)
    
    # Format configuration (NEW)
    format:
      type: "json"              # File format: "json" | "text" | "pretty"
      console_format: "pretty"   # Console format
      console_color: true        # Enable color output for console
    
    # Timezone
    timezone: Asia/Shanghai     # Timezone for timestamps (e.g., "UTC", "America/New_York")
    caller_skip: 5              # Number of stack frames to skip for caller info
    
    # Stack trace configuration
    stack:
      enable: true
      level: error              # Minimum level to capture stack: debug|info|warn|error|fatal
      skip: 6                   # Number of frames to skip
      max_frames: 32            # Maximum frames to capture
      filter_prefixes:          # Prefixes to filter out
        - github.com/go-kratos/kratos
        - github.com/rs/zerolog
        - github.com/go-lynx/lynx/log
    
    # Sampling and rate limiting
    sampling:
      enable: true
      info_ratio: 0.5           # Fraction of info logs to keep (0.0-1.0)
      debug_ratio: 0.2          # Fraction of debug logs to keep (0.0-1.0)
      max_info_per_sec: 50      # Maximum info logs per second (0 = unlimited)
      max_debug_per_sec: 20     # Maximum debug logs per second (0 = unlimited)

Configuration Options

Basic Options

Option	Type	Default	Description
level	string	info	Log level: debug, info, warn, error
console_output	bool	true	Enable console output
file_path	string	""	Log file path (empty = disabled)
timezone	string	Local	Timezone for timestamps (IANA timezone name)
caller_skip	int	5	Number of stack frames to skip for caller info

Rotation Options

Option	Type	Default	Description
max_size_mb	int	100	Maximum file size in MB before rotation
max_backups	int	0	Maximum number of backup files (0 = unlimited)
max_age_days	int	0	Maximum age in days (0 = keep forever)
compress	bool	false	Compress rotated log files
rotation_strategy	string	size	Rotation strategy: size, time, or both
rotation_interval	string	daily	Time rotation interval: hourly, daily, or weekly
max_total_size_mb	int	0	Maximum total size of all log files (0 = unlimited)

Format Options

Option	Type	Default	Description
format.type	string	json	File format: json, text, or pretty
format.console_format	string	json	Console format (can differ from file)
format.console_color	bool	true	Enable color output for console

Stack Trace Options

Option	Type	Default	Description
stack.enable	bool	true	Enable stack trace capture
stack.level	string	error	Minimum level to capture stack
stack.skip	int	6	Number of frames to skip
stack.max_frames	int	32	Maximum frames to capture
stack.filter_prefixes	[]string	[]	Package prefixes to filter out

Sampling Options

Option	Type	Default	Description
sampling.enable	bool	false	Enable sampling/rate limiting
sampling.info_ratio	float	1.0	Fraction of info logs to keep (0.0-1.0)
sampling.debug_ratio	float	1.0	Fraction of debug logs to keep (0.0-1.0)
sampling.max_info_per_sec	int	0	Maximum info logs per second (0 = unlimited)
sampling.max_debug_per_sec	int	0	Maximum debug logs per second (0 = unlimited)

Usage Examples

Basic Logging

import "github.com/go-lynx/lynx/log"

// Simple messages
log.Debug("Debug message")
log.Info("Info message")
log.Warn("Warning message")
log.Error("Error message")
log.Fatal("Fatal error") // Exits program

// Formatted messages
log.Infof("User %s logged in with ID %d", "john", 12345)
log.Errorf("Failed to connect: %v", err)

// Structured logging
log.Infow(
    "user_id", 12345,
    "action", "login",
    "ip", "192.168.1.1",
    "duration", time.Second,
)

Context-Aware Logging

import (
    "context"
    "github.com/go-lynx/lynx/log"
)

func handleRequest(ctx context.Context) {
    // Context automatically includes trace/span IDs
    log.InfoCtx(ctx, "Processing request")
    log.InfofCtx(ctx, "User %s accessed resource", userID)
    log.ErrorwCtx(ctx, "error", err, "resource", resourceID)
}

Error Logging with Stack Trace

if err != nil {
    // Stack trace is automatically captured for error level
    log.Errorw(
        "error", err,
        "operation", "database_query",
        "query", sql,
    )
}

Dynamic Level Control

// Change log level at runtime
log.SetLevel(log.DebugLevel)

// Get current level
currentLevel := log.GetLevel()

Advanced Features

Time-Based Rotation

Time-based rotation automatically rotates logs at specified intervals:

lynx:
  log:
    rotation_strategy: "time"    # or "both" for size + time
    rotation_interval: "daily"    # "hourly" | "daily" | "weekly"

File Naming:

• Hourly: app.log.2024010115 (YYYYMMDDHH)
• Daily: app.log.20240101 (YYYYMMDD)
• Weekly: app.log.20240101 (Monday's date)

Total Size Limit

Automatically manages total log file size:

lynx:
  log:
    max_total_size_mb: 1024  # 1GB total limit

When the total size exceeds the limit, oldest files are automatically deleted (active file is protected).

Format Configuration

Console Pretty Format

lynx:
  log:
    format:
      console_format: "pretty"
      console_color: true

Output Example:

14:30:45.123 INF User logged in user_id=12345 action=login
14:30:46.456 ERR Database connection failed error="timeout"

File JSON Format (Default)

lynx:
  log:
    format:
      type: "json"  # Default for files

Output Example:

{"time":"14:30:45.123","level":"info","caller":"app/handler.go:42","msg":"User logged in","user_id":12345}

Sampling and Rate Limiting

Control log volume to prevent log storms:

lynx:
  log:
    sampling:
      enable: true
      info_ratio: 0.5        # Keep 50% of info logs
      debug_ratio: 0.2        # Keep 20% of debug logs
      max_info_per_sec: 50    # Max 50 info logs/second
      max_debug_per_sec: 20   # Max 20 debug logs/second

Note: Sampling only applies to info and debug levels. warn, error, and fatal are never sampled.

Stack Trace Configuration

Configure when and how stack traces are captured:

lynx:
  log:
    stack:
      enable: true
      level: error            # Capture stack for error and above
      skip: 6                 # Skip 6 frames
      max_frames: 32          # Maximum frames to capture
      filter_prefixes:
        - github.com/go-kratos/kratos
        - github.com/rs/zerolog

Performance Optimization

Built-in Optimizations

The framework includes several performance optimizations:

1. Batch Writing: Collects multiple logs and writes in batches (64KB default)

   • Reduces system calls by 90%+
   • Improves write throughput by 2-5x

2. Async Writing: Non-blocking log writes

   • Queue size: 2000 logs (configurable)
   • Never blocks application code

3. Buffered Writing: Reduces I/O operations

   • Console buffer: 32KB
   • File buffer: 64KB

4. Fast Path Optimization:

   • Sampling check: ~1ns when disabled (98% faster)
   • Stack cache: 90%+ faster for repeated errors

Performance Metrics

Access performance metrics:

import "github.com/go-lynx/lynx/log"

// Get performance metrics
metrics := log.GetLogPerformanceMetrics()
for name, m := range metrics {
    fmt.Printf("%s: logs=%d, dropped=%d, avg_write=%v\n",
        name, m.TotalLogs, m.DroppedLogs, m.AvgWriteTime)
}

// Reset metrics
log.ResetLogPerformanceMetrics()

Best Practices

1. Log Level Selection

• Debug: Detailed information for debugging (disabled in production)
• Info: General informational messages (normal operation)
• Warn: Warning messages (unusual but recoverable)
• Error: Error messages (failures that need attention)
• Fatal: Critical errors (application will exit)

2. Structured Logging

Prefer structured logging for better searchability:

// Good: Structured logging
log.Infow(
    "user_id", userID,
    "action", "purchase",
    "amount", amount,
    "currency", "USD",
)

// Avoid: Unstructured messages
log.Infof("User %d purchased %f USD", userID, amount)

3. Error Logging

Always include context with errors:

if err != nil {
    log.Errorw(
        "error", err,
        "operation", "database_query",
        "query", sql,
        "params", params,
    )
}

4. Context Usage

Use context for request-scoped logging:

func handleRequest(ctx context.Context) {
    // Trace/span IDs are automatically included
    log.InfoCtx(ctx, "Request started")
    // ... processing
    log.InfoCtx(ctx, "Request completed")
}

5. Configuration Recommendations

Development:

level: debug
console_format: pretty
console_color: true

Production:

level: info
console_output: false
file_path: logs/app.log
rotation_strategy: both
max_total_size_mb: 1024
sampling:
  enable: true
  info_ratio: 0.5

Hot Reload

The logger supports hot reloading configuration without restart:

# Change log level
level: debug  # Automatically applied within 2 seconds

# Enable/disable sampling
sampling:
  enable: true

Changes are automatically detected and applied.

Troubleshooting

Logs Not Appearing

1. Check log level: Ensure your log level is appropriate

   log.SetLevel(log.DebugLevel)  // Enable debug logs
   

2. Check initialization: Ensure logger is initialized

   if log.Logger == nil {
       // Logger not initialized
   }
   

3. Check file permissions: Ensure write permissions for log directory

High Memory Usage

1. Reduce queue size: Lower async queue size
2. Enable sampling: Reduce log volume
3. Check buffer sizes: Reduce buffer sizes if needed

Logs Being Dropped

1. Check queue utilization: Monitor DroppedLogs metric
2. Increase queue size: Adjust async queue size
3. Check disk space: Ensure sufficient disk space

Performance Issues

1. Enable sampling: Reduce log volume
2. Disable stack traces: Only enable for errors
3. Use async writing: Ensure async writer is enabled
4. Monitor metrics: Check performance metrics regularly

API Reference

Log Functions

// Basic logging
log.Debug(args ...any)
log.Info(args ...any)
log.Warn(args ...any)
log.Error(args ...any)
log.Fatal(args ...any)

// Formatted logging
log.Debugf(format string, args ...any)
log.Infof(format string, args ...any)
log.Warnf(format string, args ...any)
log.Errorf(format string, args ...any)
log.Fatalf(format string, args ...any)

// Structured logging
log.Debugw(keyvals ...any)
log.Infow(keyvals ...any)
log.Warnw(keyvals ...any)
log.Errorw(keyvals ...any)
log.Fatalw(keyvals ...any)

// Context-aware logging
log.DebugCtx(ctx context.Context, args ...any)
log.InfoCtx(ctx context.Context, args ...any)
// ... (same pattern for all levels)

Utility Functions

// Level control
log.SetLevel(level Level)
log.GetLevel() Level

// Performance metrics
log.GetLogPerformanceMetrics() map[string]LogPerformanceMetrics
log.ResetLogPerformanceMetrics()
log.EnablePerformanceMonitoring(enabled bool)

// Cleanup
log.CleanupLoggers()  // Called automatically on shutdown

Migration Guide

From Standard Log

// Before
log.Println("Message")

// After
log.Info("Message")

From logrus

// Before
logrus.WithFields(logrus.Fields{
    "user_id": 123,
}).Info("User logged in")

// After
log.Infow("user_id", 123, "msg", "User logged in")

From zap

// Before
logger.Info("User logged in", zap.Int("user_id", 123))

// After
log.Infow("user_id", 123, "msg", "User logged in")

Complete Configuration Example

lynx:
  log:
    # Basic settings
    level: info
    console_output: true
    file_path: logs/app.log
    
    # File rotation
    max_size_mb: 128
    max_backups: 10
    max_age_days: 7
    compress: true
    
    # Time-based rotation
    rotation_strategy: "both"
    rotation_interval: "daily"
    max_total_size_mb: 1024
    
    # Format
    format:
      type: "json"
      console_format: "pretty"
      console_color: true
    
    # Timezone
    timezone: Asia/Shanghai
    caller_skip: 5
    
    # Stack traces
    stack:
      enable: true
      level: error
      skip: 6
      max_frames: 32
      filter_prefixes:
        - github.com/go-kratos/kratos
        - github.com/rs/zerolog
    
    # Sampling
    sampling:
      enable: true
      info_ratio: 0.5
      debug_ratio: 0.2
      max_info_per_sec: 50
      max_debug_per_sec: 20

Real-World Examples

Web Server Logging

func handleHTTPRequest(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    
    log.InfoCtx(ctx, "Request received",
        "method", r.Method,
        "path", r.URL.Path,
        "ip", r.RemoteAddr,
    )
    
    // Process request...
    
    log.InfoCtx(ctx, "Request completed",
        "status", 200,
        "duration", time.Since(start),
    )
}

Database Operations

func queryDatabase(ctx context.Context, sql string) error {
    log.DebugCtx(ctx, "Executing query", "sql", sql)
    
    result, err := db.Exec(sql)
    if err != nil {
        log.ErrorwCtx(ctx,
            "error", err,
            "sql", sql,
            "operation", "database_query",
        )
        return err
    }
    
    log.InfowCtx(ctx,
        "query_success",
        "rows_affected", result.RowsAffected(),
    )
    return nil
}

Error Handling

func processPayment(amount float64) error {
    if amount <= 0 {
        log.Warnw(
            "invalid_amount",
            "amount", amount,
        )
        return ErrInvalidAmount
    }
    
    err := chargeCard(amount)
    if err != nil {
        log.Errorw(
            "payment_failed",
            "error", err,
            "amount", amount,
        )
        return err
    }
    
    log.Infow(
        "payment_success",
        "amount", amount,
    )
    return nil
}

Examples

See the examples directory for complete usage examples.

Contributing

Contributions are welcome! Please read the contributing guidelines first.

License

See the main project LICENSE file.

Support

For issues and questions:

• GitHub Issues: Create an issue
• Documentation: See docs directory

Changelog

Latest Updates

• ✁EAdded time-based rotation (daily/hourly/weekly)
• ✁EAdded total size limit for log files
• ✁EAdded batch writing optimization (90%+ system call reduction)
• ✁EAdded format configuration (JSON/Pretty/Text)
• ✁EAdded color output support
• ✁EOptimized sampling check (98% performance improvement)
• ✁EOptimized stack trace collection (caching)
• ✁EFixed metrics race conditions

---

Note: After adding new configuration fields to log.proto, regenerate the proto files:

make config

Or manually:

protoc --proto_path=./app/log/conf \
       -I ./third_party -I ./boot -I ./app \
       --go_out=paths=source_relative:./app/log/conf \
       ./app/log/conf/log.proto