Documentation
¶
Overview ¶
Package otel initializes the OpenTelemetry trace, metric, and log pipelines. It uses autoexport for environment-driven configuration of exporters, supporting all standard OTEL_* env vars.
Package otel — sampler.go provides perRouteSampler, a composite sdktrace.Sampler that dispatches sampling decisions based on the HTTP route prefix read from SamplingParameters.Attributes (url.path / http.target). Used to keep /healthz + /readyz at low ratio while /api/* /rest/v1/* /peeringdb.v1.* stay at full ratio. Wrapped in sdktrace.ParentBased at provider.go so child span decisions inherit from the root.
Index ¶
- Constants
- Variables
- func BindInstruments()
- func InitFreshnessGauge(lastSyncFn func(ctx context.Context) (time.Time, bool)) error
- func InitMemoryGauges() error
- func InitObjectCountGauges(countsFn func() map[string]int64) error
- func NewDualLogger(stdout io.Writer, logProvider *sdklog.LoggerProvider) *slog.Logger
- func NewPerRouteSampler(in PerRouteSamplerInput) sdktrace.Sampler
- func PrewarmCounters(ctx context.Context)
- type PerRouteSamplerInput
- type SetupInput
- type SetupOutput
Constants ¶
const ( AttrSyncOrigin = "pdbplus.origin" SyncOriginValue = "sync" AttrForceSample = "pdbplus.force_sample" )
Span-start attributes the sync worker stamps on its root span so this sampler can gate sync traces independently of HTTP route. AttrSyncOrigin set to SyncOriginValue marks a scheduled sync cycle (dropped by default so it emits no trace — and, with PDBPLUS_OTEL_SQL on, no per-query DB spans); AttrForceSample set true marks a manually-triggered sync (POST /sync) and forces the trace to be sampled. Untyped string constants so they compare directly against attribute.Key in the ShouldSample scan.
Variables ¶
var PeeringDBRateLimitWaitMS metric.Float64Histogram
PeeringDBRateLimitWaitMS is a histogram of per-request rate-limiter wait durations in milliseconds. Replaces the span-event-only signal with a metric so operators can see p50/p95/p99 without enabling sampled tracing. Bucket boundaries cover 0 (always- available bursts) up to 5s (the biggest gap a 1/3s limiter can impose on a single Wait call).
var PeeringDBRequests metric.Int64Counter
PeeringDBRequests counts outbound HTTP requests to the PeeringDB API by status_class ∈ {2xx, 3xx, 4xx, 5xx, network_error}. The sync-level fk_backfill counter only sees post-decision events; this counter sees every request the transport makes (including retries). Cardinality: 5 values × 1 metric = 5 series (well under any concern).
var PeeringDBRetries metric.Int64Counter
PeeringDBRetries counts in-transport retries broken down by cause ∈ {429, 5xx, network_error}. The 429 axis catches upstream rate-limit pressure that the limiter under-provisioned for; 5xx catches upstream instability; network_error catches conn/DNS failures. The application-level 5xx ladder in doWithRetry also bumps the 5xx counter — both are intentional (every retry is a retry).
var ResponseHeapDeltaBytes metric.Int64Histogram
ResponseHeapDeltaBytes records the per-request Go heap HeapInuse delta (exit - entry) for pdbcompat list handlers, in bytes. Populated by internal/pdbcompat.recordResponseHeapDelta via defer at the top of serveList.
Attributes: endpoint (e.g. "/api/net"), entity (e.g. "net"). Low-cardinality by construction — 1 endpoint per type × 13 types = 13 label combinations.
var RoleTransitions metric.Int64Counter
RoleTransitions counts LiteFS role transition events (promoted/demoted).
var SyncDuration metric.Float64Histogram
SyncDuration records the duration of sync operations in seconds.
var SyncFKBackfill metric.Int64Counter
SyncFKBackfill counts live FK-backfill attempts during sync, broken down by {type, parent_type, result} where result ∈ {hit, miss, ratelimited, error}. When fkCheckParent finds a missing parent, sync attempts one upstream fetch via ?since=1&id__in=N before declaring the child an orphan. Cap-hit (per-cycle budget exhausted), HTTP failures, and "parent truly absent upstream" each get their own result label so dashboards can split successful recoveries from rate-limit pressure.
var SyncOperations metric.Int64Counter
SyncOperations counts sync operations by status (success/failed).
var SyncPeakHeapBytes atomic.Int64
SyncPeakHeapBytes holds the most-recent end-of-sync-cycle Go runtime HeapInuse in bytes. Exposed via pdbplus.sync.peak_heap ObservableGauge (Prom name: pdbplus_sync_peak_heap_bytes) for Grafana / Prometheus dashboards to plot over time. Zero means "no sync has completed yet" — the gauge suppresses observation until the first store. Bytes is the canonical Prom unit; dashboards format MiB / GiB at render time.
var SyncPeakRSSBytes atomic.Int64
SyncPeakRSSBytes holds the most-recent end-of-sync-cycle /proc/self/status VmHWM in bytes (Linux only). Zero means "not Linux" or "no sync yet" — the gauge suppresses observation when zero.
var SyncTypeDeleted metric.Int64Counter
SyncTypeDeleted counts objects deleted per type.
var SyncTypeFallback metric.Int64Counter
SyncTypeFallback counts incremental-to-full fallback events per type.
var SyncTypeFetchErrors metric.Int64Counter
SyncTypeFetchErrors counts PeeringDB API fetch errors per type.
var SyncTypeObjects metric.Int64Counter
SyncTypeObjects counts objects synced per type.
var SyncTypeOrphans metric.Int64Counter
SyncTypeOrphans counts FK-orphan rows observed per sync cycle, broken down by {type, parent_type, field, action} where action is "drop" (row excluded entirely from upsert) or "null" (FK column nulled but row kept). Provides the per-cycle aggregate that replaces the per-row WARN logs which previously blew Tempo's 7.5 MB per-trace budget.
var SyncTypeUpsertErrors metric.Int64Counter
SyncTypeUpsertErrors counts database upsert errors per type.
Functions ¶
func BindInstruments ¶ added in v1.23.0
func BindInstruments()
BindInstruments (re)creates every package-level instrument on the current global MeterProvider. It runs automatically at package init, so the instruments are never nil and callers need no nil-guards or init-ordering discipline: instruments created before the first otel.SetMeterProvider are delegating shims that bind to the real provider when main wires it.
That delegation happens exactly ONCE per process (otel's global delegateMeterOnce). Production sets one provider at startup and never needs this function. Tests that install a fresh MeterProvider after another provider was already set in the same process MUST call BindInstruments afterwards to rebind the instruments to the new provider — otherwise recorded values flow to the previously-bound provider and the test's reader collects nothing.
func InitFreshnessGauge ¶ added in v1.1.0
InitFreshnessGauge registers the sync freshness observable gauge. The lastSyncFn callback returns the time of the last successful sync. Must be called after OTel Setup() and database initialization.
func InitMemoryGauges ¶ added in v1.15.0
func InitMemoryGauges() error
InitMemoryGauges registers observable gauges that report the most-recent end-of-sync-cycle peak heap and RSS (bytes) for the sync-memory dashboard watch. Values are updated by internal/sync.(*Worker).emitMemoryTelemetry via the SyncPeakHeapBytes / SyncPeakRSSBytes atomics. A zero value suppresses the observation (no sync yet, or non-Linux for RSS) so dashboards don't plot misleading zeros.
Bytes is the canonical Prom unit (post-2026-04-26 audit); dashboards format MiB / GiB at render time via Grafana's "bytes" field unit.
func InitObjectCountGauges ¶ added in v1.5.0
InitObjectCountGauges registers an observable Int64Gauge that reports the number of objects stored per PeeringDB type. Reads from a cache function that returns pre-computed counts updated at sync completion time. Must be called after OTel Setup().
func NewDualLogger ¶
NewDualLogger creates a slog.Logger that sends every log record to both a JSON stdout handler and the OTel log pipeline via otelslog. This ensures logs are always visible on stdout even if the OTel backend is unavailable.
The OTel branch is wrapped in a [levelFilterHandler] gated by [otelLevelFromEnv] (env var PDBPLUS_LOG_LEVEL, default INFO) so DEBUG records do not ship to the OTel logging pipeline (and from there to Loki) unless the operator explicitly opts in. The stdout branch keeps its existing INFO gate.
func NewPerRouteSampler ¶ added in v1.18.0
func NewPerRouteSampler(in PerRouteSamplerInput) sdktrace.Sampler
NewPerRouteSampler builds a Sampler that dispatches per HTTP route prefix. The returned Sampler is safe for concurrent use after construction. Internally it pre-computes a sorted-by-length prefix list so longer prefixes win (e.g. "/api/auth/" beats "/api/").
Per-route samplers are sdktrace.TraceIDRatioBased(ratio) so inherent TraceID-deterministic sampling is preserved across services.
func PrewarmCounters ¶ added in v1.18.0
PrewarmCounters emits a single zero-valued .Add(ctx, 0, ...) on each of the 5 zero-rate sync/role counters with the baseline attribute set so every (counter, attribute) tuple registers with the OTel SDK at process startup. Without this, OTel cumulative counters only export a series after their first non-zero .Add(), which means dashboard panels for fallback / fetch-errors / upsert-errors / deletes / role-transitions render "No data" rather than "0" on a healthy fleet that hasn't fired any of those events yet.
The per-type set covers the 4 per-type counters (4 × 13 = 52 baseline series). RoleTransitions is the special case: it labels by direction not type, so it gets 2 baseline series (promoted, demoted) — see internal/sync/worker.go:1634/1651 for the production emission sites that establish the direction attribute.
Total baseline series introduced: 52 + 2 = 54.
MUST be called AFTER otel Setup() has installed the real MeterProvider — instruments are bound at package init and delegate to the provider set by main, so a pre-Setup call would pre-warm a no-op. The call site in cmd/peeringdb-plus/main.go runs after the syncWorker is constructed but before StartScheduler spawns its goroutine.
Types ¶
type PerRouteSamplerInput ¶ added in v1.18.0
PerRouteSamplerInput configures NewPerRouteSampler.
Routes maps URL-path prefix (e.g. "/healthz", "/api/") to a 0–1 sampling ratio. The constructor normalises trailing slashes, so callers may use either "/api" or "/api/" interchangeably. Longest matching prefix wins at sample time, so a future "/api/auth/" entry can override the broader "/api/" entry.
DefaultRatio applies when no prefix matches AND when no url.path / http.target attribute is present (sync-worker spans, internal traces).
type SetupInput ¶
SetupInput holds configuration for initializing the OTel pipeline. SampleRate controls trace sampling (0.0 to 1.0).
type SetupOutput ¶
type SetupOutput struct {
// Shutdown flushes all providers and must be called before program exit.
Shutdown func(context.Context) error
// LogProvider is exposed so the caller can create a dual slog handler
// that bridges slog records into the OTel log pipeline.
LogProvider *sdklog.LoggerProvider
}
SetupOutput holds the OTel shutdown function and LoggerProvider for creating the dual slog handler.
func Setup ¶
func Setup(ctx context.Context, in SetupInput) (*SetupOutput, error)
Setup initializes TracerProvider, MeterProvider, and LoggerProvider using autoexport for environment-driven exporter selection. Individual signals can be disabled via OTEL_*_EXPORTER=none.
Source Files