Documentation
¶
Overview ¶
Package runtime provides panic recovery utilities for services with full observability integration.
This package offers policy-based panic recovery primitives that integrate with lib-uncommons logging, OpenTelemetry metrics/tracing, and optional error tracking services like Sentry.
Panic Policies ¶
Two panic policies are supported:
KeepRunning: Log the panic and stack trace, then continue execution. Use this for worker goroutines and HTTP/gRPC handlers.
CrashProcess: Log the panic and stack trace, then re-panic to crash the process. Use this for critical invariant violations where continuing would cause data corruption.
Safe Goroutine Launching ¶
Use SafeGo and SafeGoWithContext to launch goroutines with automatic panic recovery and observability:
// Basic (no observability)
runtime.SafeGo(logger, "background-task", runtime.KeepRunning, func() {
doWork()
})
// With full observability (recommended)
runtime.SafeGoWithContextAndComponent(ctx, logger, "transaction", "balance-sync", runtime.KeepRunning,
func(ctx context.Context) {
syncBalances(ctx)
})
Deferred Recovery ¶
Use RecoverAndLog, RecoverAndCrash, or RecoverWithPolicy in defer statements. Context-aware variants provide full observability:
func handler(ctx context.Context) {
defer runtime.RecoverAndLogWithContext(ctx, logger, "transaction", "handler")
// Panics here will be logged, recorded as metrics, and added to the trace
}
Observability Integration ¶
The package integrates with three observability systems:
Metrics: Records panic_recovered_total counter with component and goroutine_name labels. Initialize with InitPanicMetrics(metricsFactory).
Tracing: Records panic.recovered span events with stack traces and sets span status to Error. Automatically uses the span from the context.
Error Reporting: Optionally reports panics to services like Sentry. Configure with SetErrorReporter(reporter).
Initialization ¶
During application startup, initialize the observability integrations:
tl, err := opentelemetry.NewTelemetry(cfg)
if err != nil {
return err
}
runtime.InitPanicMetrics(tl.MetricsFactory)
// Optional: Configure Sentry or other error reporter
runtime.SetErrorReporter(mySentryReporter)
Stack Traces ¶
All recovery functions capture and log the full stack trace using runtime/debug.Stack() for debugging purposes.
Index ¶
- Constants
- Variables
- func HandlePanicValue(ctx context.Context, logger Logger, panicValue any, component, name string)
- func InitPanicMetrics(factory *metrics.MetricsFactory, logger ...Logger)
- func IsProductionMode() bool
- func RecordPanicToSpan(ctx context.Context, panicValue any, stack []byte, goroutineName string)
- func RecordPanicToSpanWithComponent(ctx context.Context, panicValue any, stack []byte, ...)
- func RecoverAndCrash(logger Logger, name string)
- func RecoverAndCrashWithContext(ctx context.Context, logger Logger, component, name string)
- func RecoverAndLog(logger Logger, name string)
- func RecoverAndLogWithContext(ctx context.Context, logger Logger, component, name string)
- func RecoverWithPolicy(logger Logger, name string, policy PanicPolicy)
- func RecoverWithPolicyAndContext(ctx context.Context, logger Logger, component, name string, policy PanicPolicy)
- func ResetPanicMetrics()
- func SafeGo(logger Logger, name string, policy PanicPolicy, fn func())
- func SafeGoWithContext(ctx context.Context, logger Logger, name string, policy PanicPolicy, ...)
- func SafeGoWithContextAndComponent(ctx context.Context, logger Logger, component, name string, policy PanicPolicy, ...)
- func SetErrorReporter(reporter ErrorReporter)
- func SetProductionMode(enabled bool)
- type ErrorReporter
- type Logger
- type PanicMetrics
- type PanicPolicy
Constants ¶
const PanicSpanEventName = constant.EventPanicRecovered
PanicSpanEventName is the event name used when recording panic events on spans.
Variables ¶
var ErrPanic = errors.New("panic")
ErrPanic is the sentinel error for recovered panics recorded to spans.
Functions ¶
func HandlePanicValue ¶
HandlePanicValue processes a panic value that was already recovered by an external mechanism (e.g., Fiber's recover middleware). This function logs and records observability data without calling recover() itself.
Use this when integrating with frameworks that provide their own panic recovery but still need our observability pipeline.
Parameters:
- ctx: Context for observability (metrics, tracing, error reporting)
- logger: Logger for structured logging
- panicValue: The panic value recovered by the external mechanism
- component: The service component (e.g., "matcher", "ingestion")
- name: Descriptive name for the handler (e.g., "http_handler")
Example (Fiber middleware):
recover.New(recover.Config{
StackTraceHandler: func(c *fiber.Ctx, panicValue any) {
ctx := extractContext(c)
runtime.HandlePanicValue(ctx, logger, panicValue, "matcher", "http_handler")
},
})
func InitPanicMetrics ¶
func InitPanicMetrics(factory *metrics.MetricsFactory, logger ...Logger)
InitPanicMetrics initializes panic metrics with the provided MetricsFactory.
Backward compatibility:
- InitPanicMetrics(factory)
- InitPanicMetrics(factory, logger)
The logger is optional and used only for metric recording diagnostics. This should be called once during application startup after telemetry is initialized. It is safe to call multiple times; subsequent calls are no-ops.
Example:
tl, err := opentelemetry.NewTelemetry(cfg)
if err != nil {
log.Fatalf("failed to init telemetry: %v", err)
}
tl.ApplyGlobals()
runtime.InitPanicMetrics(tl.MetricsFactory)
func IsProductionMode ¶
func IsProductionMode() bool
IsProductionMode returns whether production mode is enabled.
func RecordPanicToSpan ¶
RecordPanicToSpan records a recovered panic as an error event on the current span. This enriches distributed traces with panic information for debugging.
The function:
- Adds a "panic.recovered" event with panic value, stack trace, and goroutine name
- Records the panic as an error using span.RecordError
- Sets the span status to Error with a descriptive message
Parameters:
- ctx: Context containing the active span
- panicValue: The value passed to panic()
- stack: The stack trace captured via debug.Stack()
- goroutineName: The name of the goroutine where the panic occurred
If there is no active span in the context, this function is a no-op.
func RecordPanicToSpanWithComponent ¶
func RecordPanicToSpanWithComponent( ctx context.Context, panicValue any, stack []byte, component, goroutineName string, )
RecordPanicToSpanWithComponent is like RecordPanicToSpan but also includes the component name. This is useful for HTTP/gRPC handlers where both component and handler name are relevant.
Parameters:
- ctx: Context containing the active span
- panicValue: The value passed to panic()
- stack: The stack trace captured via debug.Stack()
- component: The service component (e.g., "transaction", "onboarding")
- goroutineName: The name of the handler or goroutine
func RecoverAndCrash ¶
RecoverAndCrash recovers from a panic, logs it with the stack trace, and re-panics to crash the process. Use this in defer statements for critical operations where continuing after a panic would be dangerous.
Example:
func criticalOperation() {
defer runtime.RecoverAndCrash(logger, "critical-op")
// ...
}
func RecoverAndCrashWithContext ¶
RecoverAndCrashWithContext is like RecoverAndCrash but with full observability integration. It records metrics and span events before re-panicking.
Parameters:
- ctx: Context for observability (metrics, tracing, error reporting)
- logger: Logger for structured logging
- component: The service component (e.g., "transaction", "onboarding")
- name: Descriptive name for the goroutine or handler
func RecoverAndLog ¶
RecoverAndLog recovers from a panic, logs it with the stack trace, and continues execution. Use this in defer statements for handlers and workers where you want to prevent crashes.
Note: This function does not record metrics or span events because it lacks context. For observability integration, use RecoverAndLogWithContext instead.
Example:
func worker() {
defer runtime.RecoverAndLog(logger, "worker")
// ...
}
func RecoverAndLogWithContext ¶
RecoverAndLogWithContext is like RecoverAndLog but with full observability integration. It records metrics, span events, and reports to error tracking services.
Parameters:
- ctx: Context for observability (metrics, tracing, error reporting)
- logger: Logger for structured logging
- component: The service component (e.g., "transaction", "onboarding")
- name: Descriptive name for the goroutine or handler
Example:
func handler(ctx context.Context) {
defer runtime.RecoverAndLogWithContext(ctx, logger, "transaction", "create_handler")
// ...
}
func RecoverWithPolicy ¶
func RecoverWithPolicy(logger Logger, name string, policy PanicPolicy)
RecoverWithPolicy recovers from a panic and handles it according to the specified policy. Use this when the recovery behavior needs to be determined at runtime.
Note: This function does not record metrics or span events because it lacks context. For observability integration, use RecoverWithPolicyAndContext instead.
Example:
func flexibleHandler(policy runtime.PanicPolicy) {
defer runtime.RecoverWithPolicy(logger, "handler", policy)
// ...
}
func RecoverWithPolicyAndContext ¶
func RecoverWithPolicyAndContext( ctx context.Context, logger Logger, component, name string, policy PanicPolicy, )
RecoverWithPolicyAndContext is like RecoverWithPolicy but with full observability integration. It records metrics, span events, and reports to error tracking services.
Parameters:
- ctx: Context for observability (metrics, tracing, error reporting)
- logger: Logger for structured logging
- component: The service component (e.g., "transaction", "onboarding")
- name: Descriptive name for the goroutine or handler
- policy: How to handle the panic after logging/recording
Example:
func worker(ctx context.Context, policy runtime.PanicPolicy) {
defer runtime.RecoverWithPolicyAndContext(ctx, logger, "transaction", "balance_worker", policy)
// ...
}
func ResetPanicMetrics ¶
func ResetPanicMetrics()
ResetPanicMetrics clears the panic metrics singleton. This is primarily intended for testing to ensure test isolation. In production, this should generally not be called.
func SafeGo ¶
func SafeGo(logger Logger, name string, policy PanicPolicy, fn func())
SafeGo launches a goroutine with panic recovery. If the goroutine panics, the panic is handled according to the specified policy.
Note: This function does not record metrics or span events because it lacks context. For observability integration, use SafeGoWithContext instead.
Parameters:
- logger: Logger for recording panic information
- name: Descriptive name for the goroutine (used in logs)
- policy: How to handle panics (KeepRunning or CrashProcess)
- fn: The function to execute in the goroutine
Example:
runtime.SafeGo(logger, "email-sender", runtime.KeepRunning, func() {
sendEmail(to, subject, body)
})
func SafeGoWithContext ¶
func SafeGoWithContext( ctx context.Context, logger Logger, name string, policy PanicPolicy, fn func(context.Context), )
SafeGoWithContext launches a goroutine with panic recovery and context propagation.
Note: For better observability labeling, prefer SafeGoWithContextAndComponent.
func SafeGoWithContextAndComponent ¶
func SafeGoWithContextAndComponent( ctx context.Context, logger Logger, component, name string, policy PanicPolicy, fn func(context.Context), )
SafeGoWithContextAndComponent is like SafeGoWithContext but also records the provided component name in observability signals.
Parameters:
- ctx: Context for cancellation, values, and observability
- logger: Logger for recording panic information
- component: The service component (e.g., "transaction", "onboarding")
- name: Descriptive name for the goroutine (used in logs and metrics)
- policy: How to handle panics (KeepRunning or CrashProcess)
- fn: The function to execute, receiving the context
func SetErrorReporter ¶
func SetErrorReporter(reporter ErrorReporter)
SetErrorReporter configures the global error reporter for panic reporting. Pass nil to disable error reporting.
This should be called once during application startup if an external error tracking service is desired.
Example with structured logging:
type logReporter struct {
logger *slog.Logger
}
func (r *logReporter) CaptureException(ctx context.Context, err error, tags map[string]string) {
attrs := make([]any, 0, len(tags)*2)
for k, v := range tags {
attrs = append(attrs, k, v)
}
r.logger.ErrorContext(ctx, "panic recovered", append(attrs, "error", err)...)
}
runtime.SetErrorReporter(&logReporter{logger: slog.Default()})
func SetProductionMode ¶
func SetProductionMode(enabled bool)
SetProductionMode enables or disables production mode for error reporting. In production mode, stack traces and potentially sensitive panic details are redacted.
Types ¶
type ErrorReporter ¶
type ErrorReporter interface {
// CaptureException reports a panic/exception to the error tracking service.
// The tags map can include metadata like "component", "goroutine_name", etc.
CaptureException(ctx context.Context, err error, tags map[string]string)
}
ErrorReporter defines an interface for external error reporting services. This abstraction allows integration with error tracking services (e.g., logging to Grafana Loki, sending to an alerting system) without creating a hard dependency on any specific SDK.
Implementations should:
- Handle nil contexts gracefully
- Be safe for concurrent use
- Not panic themselves
func GetErrorReporter ¶
func GetErrorReporter() ErrorReporter
GetErrorReporter returns the currently configured error reporter. Returns nil if no reporter has been configured.
type Logger ¶
type Logger interface {
Log(ctx context.Context, level log.Level, msg string, fields ...log.Field)
}
Logger defines the minimal logging interface required by runtime. This interface is satisfied by github.com/LerianStudio/lib-uncommons/v2/uncommons/log.Logger.
type PanicMetrics ¶
type PanicMetrics struct {
// contains filtered or unexported fields
}
PanicMetrics provides panic-related metrics using OpenTelemetry. It wraps lib-uncommons' MetricsFactory for consistent metric handling.
func GetPanicMetrics ¶
func GetPanicMetrics() *PanicMetrics
GetPanicMetrics returns the singleton PanicMetrics instance. Returns nil if InitPanicMetrics has not been called.
func (*PanicMetrics) RecordPanicRecovered ¶
func (pm *PanicMetrics) RecordPanicRecovered(ctx context.Context, component, goroutineName string)
RecordPanicRecovered increments the panic_recovered_total counter with the given labels. If metrics are not initialized, this is a no-op.
Parameters:
- ctx: Context for metric recording (may contain trace correlation)
- component: The component where the panic occurred (e.g., "transaction", "onboarding", "crm")
- goroutineName: The name of the goroutine or handler (e.g., "http_handler", "rabbitmq_worker")
type PanicPolicy ¶
type PanicPolicy int
PanicPolicy determines how a recovered panic should be handled.
const ( // KeepRunning logs the panic and stack trace, then continues execution. // Use for HTTP/gRPC handlers and worker goroutines where crashing would // affect other requests or tasks. KeepRunning PanicPolicy = iota // CrashProcess logs the panic and stack trace, then re-panics to crash // the process. Use for critical invariant violations where continuing // would cause data corruption or undefined behavior. CrashProcess )
func (PanicPolicy) String ¶
func (p PanicPolicy) String() string
String returns the string representation of the PanicPolicy.