logging

gokit/logging

Production-ready structured logging built on zerolog.

Features

• Structured JSON / console output
• Sensitive data masking (on by default)
• Rate-based log sampling (burst + thereafter)
• Per-module log level overrides
• OpenTelemetry Logs bridge (OTLP export)
• Unified log schema (consistent across gokit, pykit, rskit)
• Context propagation (trace ID, span ID, correlation ID)

Quick Start

package main

import "github.com/kbukum/gokit/logging"

func main() {
    // Default logger — masking enabled, console format, info level
    log := logging.NewDefault("my-service")
    log.Info("server started", logging.Fields("port", 8080))

    // Component-scoped logger
    dbLog := log.WithComponent("database")
    dbLog.Debug("query executed", logging.DurationFields("select", elapsed))

    // Package-level functions use the immutable process default (logging.Default())
    logging.Info("using package-level default logger")

    // Sensitive data is automatically redacted
    log.Info("user login", logging.Fields("password", "hunter2"))
    // output: password=***REDACTED***
}

Configuration

logging:
  level: info              # debug | info | warn | error | fatal | trace
  format: json             # json | console | text
  output: stdout           # stdout | stderr
  no_color: false
  timestamp: true
  caller: false
  stacktrace: false
  service_name: my-service

  # File rotation (optional)
  max_size: 100            # megabytes
  max_backups: 3
  max_age: 28              # days
  compress: false

  # Sensitive data masking
  masking:
    enabled: true           # on by default
    field_names:             # additional field names to redact
      - my_secret_field
    value_patterns:          # additional regex patterns
      - 'MYSECRET_[A-Z0-9]{20}'
    replacement: "***REDACTED***"
    preserve_last: 0         # preserve last N chars (0 = full redaction)

  # Rate-based sampling
  sampling:
    enabled: false
    initial_rate: 100        # allow first N per second per level
    thereafter_rate: 100     # then keep every Nth

  # Per-module log level overrides
  module_levels:
    database: debug
    kafka: warn
    auth: trace

  # OpenTelemetry OTLP export
  otlp:
    enabled: false
    endpoint: "localhost:4317"
    protocol: grpc           # grpc | http
    insecure: false
    headers:
      x-api-key: "my-key"

Masking

Masking is enabled by default. Every log field is checked against sensitive field names (case-insensitive) and value patterns (regex). If a match is found, the value is replaced before it reaches any output sink.

Default Masked Fields

Value Patterns

These patterns detect sensitive data regardless of field name:

Adding Custom Fields and Patterns

masking:
  field_names:
    - my_internal_token
    - employee_id
  value_patterns:
    - 'MYSVC_[A-Za-z0-9]{32}'

Partial Masking

Use preserve_last to keep the last N characters visible:

masking:
  preserve_last: 4

This turns "password": "hunter2" into "password": "***REDACTED***ter2".

Sampling

Sampling reduces log volume in high-throughput services. When enabled, each log level gets an independent counter per one-second window:

1. Burst — the first initial_rate messages per second per level pass through unconditionally.
2. Thereafter — after the burst, only every thereafter_rate-th message is kept.

sampling:
  enabled: true
  initial_rate: 100     # allow first 100/sec per level
  thereafter_rate: 10   # then keep every 10th

When to use: Enable sampling on hot-path services producing thousands of log lines per second. Leave disabled for low-volume services or during debugging.

Sampling uses zerolog's BurstSampler under the hood:

// Equivalent to:
&zerolog.BurstSampler{
    Burst:       100,
    Period:      time.Second,
    NextSampler: &zerolog.BasicSampler{N: 100},
}

Module Levels

Override the global log level for specific components. Useful for silencing noisy dependencies or enabling debug output for a single subsystem.

logging:
  level: info
  module_levels:
    database: debug     # verbose DB logs
    kafka: warn         # suppress Kafka noise
    auth: trace         # detailed auth tracing

// Programmatic usage
log := logging.New(cfg, "my-service")

// WithComponent applies the module-level override automatically
dbLog := log.WithComponent("database")   // → debug level
kafkaLog := log.WithComponent("kafka")   // → warn level

Module levels are configured via the module_levels map (config or logging.Config) and applied automatically by WithComponent. The underlying ModuleLevelManager is thread-safe; its SetLevel() is used internally when levels are established from configuration.

OTLP Export

The OpenTelemetry Logs bridge sends log records to an OTLP collector alongside your local output. Logs are emitted via the OTel SDK LoggerProvider with batch processing.

Setup

otlp:
  enabled: true
  endpoint: "otel-collector:4317"
  protocol: grpc        # grpc | http
  insecure: true        # skip TLS for dev
  headers:
    Authorization: "Bearer my-token"

Programmatic Usage

cfg := &logging.Config{
    Level:       "info",
    Format:      "json",
    ServiceName: "my-service",
    OTLP: logging.OTLPConfig{
        Enabled:  true,
        Endpoint: "localhost:4317",
        Protocol: "grpc",
        Insecure: true,
    },
}
log := logging.New(cfg, "my-service")
defer log.Close()  // flush pending OTLP logs on shutdown

log.Info("order created", logging.Fields("order_id", "abc-123"))

Graceful Shutdown

Always call Close() before process exit to flush buffered log records:

log := logging.New(cfg, "my-service")
defer log.Close()

Unified Schema

All three kits (gokit, pykit, rskit) share the same structured field names:

ServiceFields Helper

Attach standard service identification to any log entry:

svcFields := logging.ServiceFields("order-svc", "production", "1.2.3")
log.Info("service started", svcFields)
// → {"service":"order-svc","environment":"production","version":"1.2.3",...}

Field Helpers

// Build fields from key-value pairs
logging.Fields("op", "save", "id", 42)

// Error fields
logging.ErrorFields("db.connect", err)

// Duration fields
logging.DurationFields("query", 150*time.Millisecond)

// Merge helpers
fields := logging.Fields("op", "save")
logging.MergeWithError(fields, err)
logging.MergeWithDuration(fields, elapsed)

Custom Masker

Implement the Masker interface to provide your own masking logic:

type Masker interface {
    MaskValue(key string, value string) string
}

type MyMasker struct{}

func (m *MyMasker) MaskValue(key, value string) string {
    if key == "internal_id" {
        return "***"
    }
    return value
}

log := logging.New(cfg, "my-service")
log = log.WithMasker(&MyMasker{})
log.Info("event", logging.Fields("internal_id", "secret-123"))
// → internal_id=***

API Reference

⬅ Back to main README