traceinspector

Go Pprof Trace Analyzer

A lightweight command-line tool for analyzing Go execution traces (trace.out). This tool processes traces generated by the Go runtime to identify performance bottlenecks, scheduler latency issues, contention, and potential goroutine leaks.

It uses the modern golang.org/x/exp/trace API (compatible with Go 1.22+) to parse and aggregate event data.

Features

• Scheduler Latency Analysis: Detects goroutines waiting too long in the global run queue before executing.
• Blocking/Contention Analysis: Identifies goroutines blocked on synchronization primitives (Mutex, Channels, WaitGroups) longer than a specified threshold.
• Goroutine Leak Detection: Reports goroutines that remain active/running at the very end of the trace.
• CPU & Lifecycle Statistics: Aggregates CPU time, system call time, and lifetime for top resource-consuming goroutines.

Requirements

• Go 1.22 or higher (required for golang.org/x/exp/trace)

Installation

1. Clone or copy the source code.
2. Build the binary:

make traceinspector 

Usage

1. Generate a Trace

First, generate a trace file from your Go application. You can do this using runtime/trace in your code or via tests:

In code:

f, _ := os.Create("trace.out")
trace.Start(f)
defer trace.Stop()

Via tests:

go test -trace=trace.out

2. Run the Analyzer

Run the tool against your generated trace file.

trace-analyzer -file trace.out

Command-Line Flags

Examples

Analyze with custom thresholds: Detect blocking events longer than 50ms and scheduler delays longer than 20ms:

./trace-analyzer -file mytrace.out -block 50ms -latency 20ms

Understanding the Output

The tool prints a summary report to standard output containing four sections:

1. Suspiciously Active Goroutines (Potential Leaks)

   • Lists goroutines that were still in a Running state when the trace ended.
   • Useful for finding background workers that didn't shut down gracefully.

2. Scheduler Latency Summary

   • Avg Scheduler Latency: Average time goroutines spent waiting in the run queue.
   • Max Scheduler Latency: The worst-case wait time observed.

3. Top 5 Most Blocked Goroutines

   • Goroutines that spent the most time in a Waiting state due to synchronization (e.g., chan receive, Mutex.Lock).
   • Shows the specific reason and the stack trace where the goroutine was created.

4. Top 5 CPU-Heavy Goroutines

   • Goroutines with the highest total CPUTime (User + Syscall).
   • Includes lifetime duration and average runnable wait time.