README
¶
Metrics Utilities (platform/metrics)
The metrics package provides reusable helpers for emitting counters, gauges, and histograms on a tally.Scope.
Design
Free functions on tally.Scope — no wrapper types. Existing constructors accept tally.Scope and do not need to change.
Operation lifecycle — Begin and Complete tie operation metrics together. Begin captures the start time and emits {name}.start; Complete records duration and count on {name}.finish.
Result tagging — the finish histogram is tagged with result=success, result=error, or result=cancel. Cancellation is detected with errors.Is(err, context.Canceled). Callers that accumulate tags while the operation runs can pass them to Complete.
Consistent naming — named helpers follow the {name}.{sub} sub-scope pattern, producing metric paths such as process.start and publish.attempts.
Operation Lifecycle
For any operation with a clear start and end, use Begin and Complete:
| Function | Emits |
|---|---|
Begin(scope, name, buckets, ...tags) |
{name}.start counter +1 and returns an Op |
op.Complete(err, ...tags) |
{name}.finish histogram tagged with result=success|error|cancel and any completion tags |
op.CompleteAs(result, ...tags) |
{name}.finish histogram tagged with the given result and any completion tags, for outcomes the error alone does not classify (for example result=not_found) |
buckets is required at Begin because operations differ widely in expected latency. The finish histogram records both the duration distribution and the number of completed operations, so Complete does not emit a separate counter.
func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) (retErr error) {
op := metrics.Begin(c.scope, "process", metrics.LongLatencyBuckets)
defer func() { op.Complete(retErr) }()
// ... business logic ...
return nil
}
Tags passed to Begin apply to both lifecycle metrics. Tags known only after execution, such as an error classification, can be attached to the finish histogram:
err := controller.Process(ctx, delivery)
err = classifier.Process(err)
op.Complete(err, metrics.NewTag("origin", "infra_retryable"))
Named Helpers
For ad-hoc metrics that do not fit the operation lifecycle:
| Function | Emits | Example |
|---|---|---|
NamedCounter(scope, name, counter, value, ...tags) |
{name}.{counter} counter |
publish.attempts |
NamedGauge(scope, name, gauge, value, ...tags) |
{name}.{gauge} gauge |
record.last_green_timestamp_seconds |
NamedHistogram(scope, name, histogram, buckets, ...tags) |
{name}.{histogram} histogram |
process.duration |
metrics.NamedCounter(c.scope, "publish", "attempts", 1)
h := metrics.NamedHistogram(c.scope, "process", "duration", metrics.FastLatencyBuckets)
h.RecordDuration(elapsed)
Use gauges only for state whose latest value is the whole answer, such as a bookmark timestamp. A gauge is reported once per update rather than continuously, so a gauge set on a discrete event produces a sparse series: it carries no value between updates or after a restart, and each replica reports only the updates it made, so queries must aggregate across replicas with max or last-value. State that must be readable at any moment needs a periodic re-emit rather than an event-driven one.
Represent operation latency and completion count with lifecycle histograms; do not emit timers.
Why histograms, not timers
Durations are recorded as histograms rather than timers. Timer percentiles cannot be combined accurately across time series, while bucketed histogram counts can be summed to reconstruct a combined distribution for correct aggregate percentiles.
Tags
Use NewTag to pass dimensional tags to a helper:
op := metrics.Begin(c.scope, "process", metrics.LongLatencyBuckets, metrics.NewTag("queue", req.Queue))
defer func() { op.Complete(retErr) }()
metrics.NamedCounter(c.scope, "publish", "attempts", 1, metrics.NewTag("topic", c.topic))
Duration Buckets
There is no default bucket set. The package exports five common sets:
| Set | Range | Use for |
|---|---|---|
FastLatencyBuckets |
~100µs – 5s | Fast in-process work such as scoring, cache lookups, and CPU-bound operations |
StorageLatencyBuckets |
~1ms – 1m | Storage and message-queue round trips such as database reads, writes, publishing, and consuming |
LongLatencyBuckets |
~5ms – 4h | Long-running pipeline work and external calls such as builds, lands, pushes, and provider calls |
ChangeAgeBuckets |
~1m – 30d | Elapsed time measured from a source-control change's commit timestamp rather than from work this system started |
LandLatencyBuckets |
~1m – 24h | End-to-end request lifecycles, from accepting a land request to its terminal outcome |
Pass one of these sets or a custom tally.DurationBuckets to Begin or NamedHistogram.
For counts recorded with RecordValue, such as how many items a unit of work carries or depends on, use CountBuckets (0 – 144, Fibonacci-spaced, with zero in its own bucket).
Documentation
¶
Index ¶
- Variables
- func NamedCounter(scope tally.Scope, name string, counter string, value int64, tags ...Tag)
- func NamedGauge(scope tally.Scope, name string, gauge string, value float64, tags ...Tag)
- func NamedHistogram(scope tally.Scope, name string, histogram string, buckets tally.Buckets, ...) tally.Histogram
- func WithContextTags(ctx context.Context, tags ...Tag) context.Context
- type Op
- type Tag
Constants ¶
This section is empty.
Variables ¶
var ( // FastLatencyBuckets suits fast in-process operations (~microseconds to // seconds): scoring, cache lookups, and other CPU-bound work. FastLatencyBuckets = tally.DurationBuckets{ 100 * time.Microsecond, 250 * time.Microsecond, 500 * time.Microsecond, 1 * time.Millisecond, 2500 * time.Microsecond, 5 * time.Millisecond, 10 * time.Millisecond, 25 * time.Millisecond, 50 * time.Millisecond, 100 * time.Millisecond, 250 * time.Millisecond, 500 * time.Millisecond, 1 * time.Second, 2500 * time.Millisecond, 5 * time.Second, } // StorageLatencyBuckets suits storage and message-queue round-trips // (~1ms to a minute): database reads/writes, publish/consume, and RPC // handlers whose latency is dominated by such calls. StorageLatencyBuckets = tally.DurationBuckets{ 1 * time.Millisecond, 2500 * time.Microsecond, 5 * time.Millisecond, 10 * time.Millisecond, 25 * time.Millisecond, 50 * time.Millisecond, 100 * time.Millisecond, 250 * time.Millisecond, 500 * time.Millisecond, 1 * time.Second, 2500 * time.Millisecond, 5 * time.Second, 10 * time.Second, 30 * time.Second, 1 * time.Minute, } // LongLatencyBuckets suits long-running pipeline work and external calls // (~5ms to hours): builds, lands, git pushes, and external provider calls. LongLatencyBuckets = tally.DurationBuckets{ 5 * time.Millisecond, 10 * time.Millisecond, 25 * time.Millisecond, 50 * time.Millisecond, 100 * time.Millisecond, 250 * time.Millisecond, 500 * time.Millisecond, 1 * time.Second, 2500 * time.Millisecond, 5 * time.Second, 10 * time.Second, 30 * time.Second, 1 * time.Minute, 2 * time.Minute, 5 * time.Minute, 10 * time.Minute, 30 * time.Minute, 1 * time.Hour, 2 * time.Hour, 4 * time.Hour, } // ChangeAgeBuckets suits durations measured from a source-control change's // commit timestamp. They span minutes to a month because that is the honest // range of such a signal: a break caught in minutes and one that survived a // fortnight are both ordinary observations, and collapsing the tail would hide // exactly the cases worth seeing. ChangeAgeBuckets = tally.DurationBuckets{ 1 * time.Minute, 5 * time.Minute, 15 * time.Minute, 30 * time.Minute, 1 * time.Hour, 2 * time.Hour, 4 * time.Hour, 8 * time.Hour, 12 * time.Hour, 24 * time.Hour, 48 * time.Hour, 7 * 24 * time.Hour, 14 * 24 * time.Hour, 30 * 24 * time.Hour, } // LandLatencyBuckets suits end-to-end request lifecycles, from accepting a // land request to its terminal outcome. Resolution is densest between 5 and // 60 minutes, where typical lands fall, and the tail reaches a day so that // requests stuck overnight stay distinguishable instead of overflowing. LandLatencyBuckets = tally.DurationBuckets{ 1 * time.Minute, 2 * time.Minute, 3 * time.Minute, 5 * time.Minute, 7*time.Minute + 30*time.Second, 10 * time.Minute, 15 * time.Minute, 20 * time.Minute, 30 * time.Minute, 45 * time.Minute, 1 * time.Hour, 90 * time.Minute, 2 * time.Hour, 3 * time.Hour, 4 * time.Hour, 6 * time.Hour, 8 * time.Hour, 12 * time.Hour, 24 * time.Hour, } )
Common duration bucket sets for latency histograms. Operations differ widely in expected latency, so there is no single default — pick the set whose range matches the operation and pass it to Begin or NamedHistogram. Buckets far outside an operation's real latency waste series cardinality and lose resolution where the data actually lands.
var CountBuckets = tally.ValueBuckets{0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144}
CountBuckets suits small cardinalities recorded with RecordValue, such as how many items a unit of work carries or depends on. Zero has its own bucket because "none" is a meaningful answer, and the Fibonacci spacing keeps resolution where counts usually fall while still separating outliers.
Functions ¶
func NamedCounter ¶
NamedCounter increments the {name}.{counter} counter by value.
func NamedGauge ¶
NamedGauge sets the {name}.{gauge} gauge to value.
func NamedHistogram ¶
func NamedHistogram(scope tally.Scope, name string, histogram string, buckets tally.Buckets, tags ...Tag) tally.Histogram
NamedHistogram returns a tally.Histogram at {name}.{histogram} with the given bucket configuration. Store the returned histogram and call RecordDuration or RecordValue on each invocation.
Types ¶
type Op ¶
type Op struct {
// contains filtered or unexported fields
}
Op tracks the lifecycle of a named operation. It captures the start time on creation, emits a {name}.start counter, and records the duration and result on a {name}.finish histogram when Complete is called.
Usage:
func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) (retErr error) {
op := metrics.Begin(c.scope, "process", metrics.StorageLatencyBuckets)
defer func() { op.Complete(retErr) }()
// ... business logic ...
}
func Begin ¶
Begin starts a new operation. It emits a {name}.start counter, captures the start time, and retains the buckets used by Complete.
func (Op) Complete ¶
Complete records elapsed time on the {name}.finish histogram, tagged with result=success|error|cancel and any additional tags accumulated while the operation ran. The histogram records both duration and count. Cancellation is detected through the error chain.
func (Op) CompleteAs ¶
CompleteAs records elapsed time on the {name}.finish histogram like Complete, but with a caller-chosen result tag. It serves outcomes that the error alone does not classify, such as a lookup that found no record.
Source Files