otel

package
v1.41.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 27, 2026 License: BSD-3-Clause Imports: 26 Imported by: 0

Documentation

Overview

Package otel initializes the OpenTelemetry trace, metric, and log pipelines. It uses autoexport for environment-driven configuration of exporters, supporting all standard OTEL_* env vars.

Index

Constants

View Source
const (
	AttrSyncOrigin  = "pdbplus.origin"
	SyncOriginValue = "sync"
	AttrForceSample = "pdbplus.force_sample"
)

Span-start attributes that the sync worker sets on its root span. The sampler reads them before the route, because a sync root span has no URL path and would otherwise get the 1% default ratio.

AttrSyncOrigin set to SyncOriginValue marks a sync cycle. A cycle without AttrForceSample is a scheduled cycle, and the sampler keeps it at PerRouteSamplerInput.SyncRatio (PDBPLUS_OTEL_SYNC_SAMPLE_RATE). AttrForceSample overrides every ratio. The value true always samples the span (POST /sync). The value false never samples it (POST /sync?trace=0). The constants are untyped strings, so they compare directly with attribute.Key in the ShouldSample scan.

Variables

View Source
var AssetLogoFetches metric.Int64Counter

AssetLogoFetches counts the logo reads of pdbcompat /api/asset by result ∈ {hit, miss, error, skipped}: hit = served from the cache, miss = fetched from the media bucket, error = the fetch failed or was backed off, skipped = the stored URL is not on the media host. Cardinality: 4 series.

CSPReports counts the Content-Security-Policy violation reports that browsers send to /csp-report, by directive. The directive comes from a fixed list, and any other value counts as "other", because the report body is client input. Cardinality: at most 20 series.

View Source
var PeeringDBRateLimitWaitMS metric.Float64Histogram

PeeringDBRateLimitWaitMS is a histogram of per-request rate-limiter wait durations in milliseconds. Replaces the span-event-only signal with a metric so operators can see p50/p95/p99 without enabling sampled tracing. Bucket boundaries cover 0 (always- available bursts) up to 5s (the biggest gap a 1/3s limiter can impose on a single Wait call).

View Source
var PeeringDBRequests metric.Int64Counter

PeeringDBRequests counts outbound HTTP requests to the PeeringDB API by status_class ∈ {2xx, 3xx, 4xx, 5xx, network_error}. The sync-level fk_backfill counter only sees post-decision events; this counter sees every request the transport makes (including retries). Cardinality: 5 values × 1 metric = 5 series (well under any concern).

View Source
var PeeringDBRetries metric.Int64Counter

PeeringDBRetries counts in-transport retries broken down by cause ∈ {429, 5xx, network_error}. The 429 axis catches upstream rate-limit pressure that the limiter under-provisioned for; 5xx catches upstream instability; network_error catches conn/DNS failures. The application-level 5xx ladder in doWithRetry also bumps the 5xx counter — both are intentional (every retry is a retry).

View Source
var ResponseHeapDeltaBytes metric.Int64Histogram

ResponseHeapDeltaBytes records the per-request Go heap HeapInuse delta (exit - entry) for pdbcompat list handlers, in bytes. Populated by internal/pdbcompat.recordResponseHeapDelta via defer at the top of serveList.

Attributes: endpoint (e.g. "/api/net"), entity (e.g. "net"). Low-cardinality by construction — 1 endpoint per type × 13 types = 13 label combinations.

View Source
var RoleTransitions metric.Int64Counter

RoleTransitions counts LiteFS role transition events (promoted/demoted).

SyncDuration records the duration of sync operations in seconds.

View Source
var SyncFKBackfill metric.Int64Counter

SyncFKBackfill counts live FK-backfill attempts during sync, broken down by {type, parent_type, result} where result ∈ {hit, miss, ratelimited, error}. When fkCheckParent finds a missing parent, sync attempts one upstream fetch via ?since=1&id__in=N before declaring the child an orphan. Cap-hit (per-cycle budget exhausted), HTTP failures, and "parent truly absent upstream" each get their own result label so dashboards can split successful recoveries from rate-limit pressure.

View Source
var SyncHistoryRequests metric.Int64Counter

SyncHistoryRequests counts the upstream requests of the history sweep (internal/sync/history_sweep.go), by type and result: ok, rate_limited (429 or WAF block) or error. Cardinality: 12 type values x 3 results.

View Source
var SyncLockRetries metric.Int64Counter

SyncLockRetries counts the retries of short primary writes after a transient SQLite lock error (SQLITE_BUSY or SQLITE_PROTOCOL), by op: the sync_status INSERT and UPDATE and the startup poc scrub and netixlan cascade transactions (internal/sync/lockretry.go). The sync transaction and the sync_status prune are not retried. Cardinality: 5 op values.

View Source
var SyncOperations metric.Int64Counter

SyncOperations counts sync operations by status (success/failed).

View Source
var SyncPeakHeapBytes atomic.Int64

SyncPeakHeapBytes holds the most-recent end-of-sync-cycle Go runtime HeapInuse in bytes. Exposed via pdbplus.sync.peak_heap ObservableGauge (Prom name: pdbplus_sync_peak_heap_bytes) for Grafana / Prometheus dashboards to plot over time. Zero means "no sync has completed yet" — the gauge suppresses observation until the first store. Bytes is the canonical Prom unit; dashboards format MiB / GiB at render time.

View Source
var SyncPeakRSSBytes atomic.Int64

SyncPeakRSSBytes holds the most-recent end-of-sync-cycle /proc/self/status VmHWM in bytes (Linux only). Zero means "not Linux" or "no sync yet" — the gauge suppresses observation when zero.

View Source
var SyncTypeDeleted metric.Int64Counter

SyncTypeDeleted counts the rows that sync itself marks deleted, per type. Only the netixlan cascade of a deleted network emits it (internal/sync/netixlan_cascade.go), after its transaction commits. Tombstones that upstream sends are counted in SyncTypeObjects.

View Source
var SyncTypeFallback metric.Int64Counter

SyncTypeFallback counts incremental-to-full fallback events per type.

View Source
var SyncTypeFetchErrors metric.Int64Counter

SyncTypeFetchErrors counts the failed sync fetch steps per type. The cause can be a cursor read, a PeeringDB request or a scratch write (internal/sync/worker.go failFetchStep).

View Source
var SyncTypeObjects metric.Int64Counter

SyncTypeObjects counts objects synced per type. The sync worker adds the counts of a cycle only after its transaction commits (internal/sync/worker.go recordObjectCounts).

View Source
var SyncTypeOrphans metric.Int64Counter

SyncTypeOrphans counts FK-orphan rows observed per sync cycle, broken down by {type, parent_type, field, action} where action is "drop" (row excluded entirely from upsert) or "null" (FK column nulled but row kept). Provides the per-cycle aggregate that replaces the per-row WARN logs which previously blew Tempo's 7.5 MB per-trace budget.

View Source
var SyncTypeUpsertErrors metric.Int64Counter

SyncTypeUpsertErrors counts database upsert errors per type.

Functions

func BindInstruments added in v1.23.0

func BindInstruments()

BindInstruments (re)creates every package-level instrument on the current global MeterProvider. It runs automatically at package init, so the instruments are never nil and callers need no nil-guards or init-ordering discipline: instruments created before the first otel.SetMeterProvider are delegating shims that bind to the real provider when main wires it.

That delegation happens exactly ONCE per process (otel's global delegateMeterOnce). Production sets one provider at startup and never needs this function. Tests that install a fresh MeterProvider after another provider was already set in the same process MUST call BindInstruments afterwards to rebind the instruments to the new provider — otherwise recorded values flow to the previously-bound provider and the test's reader collects nothing.

func DBSpansOff added in v1.31.0

func DBSpansOff(ctx context.Context) bool

DBSpansOff reports whether ctx carries the mark of WithoutDBSpans.

func InitBuildInfoGauge added in v1.34.0

func InitBuildInfoGauge(version string) error

InitBuildInfoGauge registers the pdbplus.build.info gauge: the value 1 with the attribute service.version set to version. The metric resource has no service.version (buildMetricResource), so this one series per machine carries the version on the metrics path, and a deploy does not start a new copy of every other series. Must be called after OTel Setup().

func InitFreshnessGauge added in v1.1.0

func InitFreshnessGauge(lastSyncFn func(ctx context.Context) (time.Time, bool)) error

InitFreshnessGauge registers the sync freshness observable gauge. The lastSyncFn callback returns the time of the last successful sync. Must be called after OTel Setup() and database initialization.

func InitMemoryGauges added in v1.15.0

func InitMemoryGauges() error

InitMemoryGauges registers observable gauges that report the most-recent end-of-sync-cycle peak heap and RSS (bytes) for the sync-memory dashboard watch. Values are updated by internal/sync.(*Worker).emitMemoryTelemetry via the SyncPeakHeapBytes / SyncPeakRSSBytes atomics. A zero value suppresses the observation (no sync yet, or non-Linux for RSS) so dashboards don't plot misleading zeros.

Bytes is the canonical Prom unit (post-2026-04-26 audit); dashboards format MiB / GiB at render time via Grafana's "bytes" field unit.

func InitObjectCountGauges added in v1.5.0

func InitObjectCountGauges(countsFn func() map[string]int64, isPrimary func() bool) error

InitObjectCountGauges registers an observable Int64Gauge that reports the number of objects stored per PeeringDB type. Reads from a cache function that returns pre-computed counts updated at sync completion time. A collection observes nothing while isPrimary returns false: only the primary runs the sync worker that updates the cache, so the counts of a replica stay at their values from process start. Must be called after OTel Setup().

func InitScratchFreeGauge added in v1.33.0

func InitScratchFreeGauge(dir string, free func(dir string) (uint64, error)) error

InitScratchFreeGauge registers the pdbplus.scratch.free gauge: the free space of the file system of dir (PDBPLUS_SCRATCH_DIR), in bytes, as free reports it. On Fly.io the dir of the primary is on its LiteFS volume. Each collection calls free once. An error, or a value above math.MaxInt64 (free space not known), observes nothing.

func NewDualLogger

func NewDualLogger(stdout io.Writer, logProvider *sdklog.LoggerProvider) *slog.Logger

NewDualLogger creates a slog.Logger that sends every log record to both a JSON stdout handler and the OTel log pipeline via otelslog. This ensures logs are always visible on stdout even if the OTel backend is unavailable.

The OTel branch is wrapped in a [levelFilterHandler] gated by [otelLevelFromEnv] (env var PDBPLUS_LOG_LEVEL, default INFO) so DEBUG records do not ship to the OTel logging pipeline (and from there to Loki) unless the operator explicitly opts in. The stdout branch keeps its existing INFO gate.

func NewPerRouteSampler added in v1.18.0

func NewPerRouteSampler(in PerRouteSamplerInput) sdktrace.Sampler

NewPerRouteSampler builds a Sampler that dispatches per HTTP route prefix. The returned Sampler is safe for concurrent use after construction. Internally it pre-computes a sorted-by-length prefix list so longer prefixes win (e.g. "/api/auth/" beats "/api/").

Per-route samplers are sdktrace.TraceIDRatioBased(ratio) so inherent TraceID-deterministic sampling is preserved across services.

func PrewarmCounters added in v1.18.0

func PrewarmCounters(ctx context.Context)

PrewarmCounters emits a single zero-valued .Add(ctx, 0, ...) on each of the 5 zero-rate sync/role counters with the baseline attribute set so every (counter, attribute) tuple registers with the OTel SDK at process startup. Without this, OTel cumulative counters only export a series after their first non-zero .Add(), which means dashboard panels for fallback / fetch-errors / upsert-errors / deletes / role-transitions render "No data" rather than "0" on a healthy fleet that hasn't fired any of those events yet.

The per-type set covers the 4 per-type counters (4 × 13 = 52 baseline series). RoleTransitions is the special case: it labels by direction not type, so it gets 2 baseline series (promoted, demoted) — see internal/sync/worker.go:1634/1651 for the production emission sites that establish the direction attribute.

Total baseline series introduced: 52 + 2 = 54.

MUST be called AFTER otel Setup() has installed the real MeterProvider — instruments are bound at package init and delegate to the provider set by main, so a pre-Setup call would pre-warm a no-op. The call site in cmd/peeringdb-plus/main.go runs after the syncWorker is constructed but before StartScheduler spawns its goroutine, and only on a node that LiteFS can elect as primary.

func WithoutDBSpans added in v1.31.0

func WithoutDBSpans(ctx context.Context) context.Context

WithoutDBSpans marks ctx so that the traced database handle of internal/database emits no otelsql span for a statement run under it. The sync worker marks each cycle: a full cycle runs thousands of statements, and with their spans its trace is larger than the 5 MB per-trace limit of Grafana Cloud Tempo.

Types

type PerRouteSamplerInput added in v1.18.0

type PerRouteSamplerInput struct {
	DefaultRatio float64
	SyncRatio    float64
	Routes       map[string]float64
}

PerRouteSamplerInput configures NewPerRouteSampler.

Routes maps URL-path prefix (e.g. "/healthz", "/api/") to a 0–1 sampling ratio. The constructor normalises trailing slashes, so callers may use either "/api" or "/api/" interchangeably. Longest matching prefix wins at sample time, so a future "/api/auth/" entry can override the broader "/api/" entry.

DefaultRatio applies when no prefix matches AND when no url.path / http.target attribute is present (internal traces).

SyncRatio applies to scheduled sync cycles: spans with AttrSyncOrigin set to SyncOriginValue and no AttrForceSample.

type SetupInput

type SetupInput struct {
	ServiceName    string
	SampleRate     float64
	SyncSampleRate float64
}

SetupInput holds configuration for initializing the OTel pipeline. SampleRate controls trace sampling of the known app routes (0.0 to 1.0). SyncSampleRate controls trace sampling of scheduled sync cycles (0.0 to 1.0).

type SetupOutput

type SetupOutput struct {
	// Shutdown flushes all providers and must be called before program exit.
	Shutdown func(context.Context) error

	// LogProvider is exposed so the caller can create a dual slog handler
	// that bridges slog records into the OTel log pipeline.
	LogProvider *sdklog.LoggerProvider
}

SetupOutput holds the OTel shutdown function and LoggerProvider for creating the dual slog handler.

func Setup

func Setup(ctx context.Context, in SetupInput) (*SetupOutput, error)

Setup initializes TracerProvider, MeterProvider, and LoggerProvider using autoexport for environment-driven exporter selection. Individual signals can be disabled via OTEL_*_EXPORTER=none.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL