observability

package
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 21, 2026 License: Apache-2.0 Imports: 30 Imported by: 0

Documentation

Index

Constants

View Source
const (
	RequestIDKey = "request_id"
	TenantKey    = "tenant"
	// ClientAddressKey is OpenTelemetry's semantic-convention name, which
	// the HTTP server span already uses for the same value.
	ClientAddressKey = "client.address"
)

The keys a request's correlation goes under, the same on a log record and on a span, so one query finds both.

View Source
const BuildInfoMetric = "wavehouse_build_info"

BuildInfoMetric is the gauge naming the build and roles each process runs.

View Source
const TableAttr attribute.Key = "table"

TableAttr is the metric attribute naming a ClickHouse table. Its values come from the operator's schema, which has no ceiling on table count (Key Design Decision #2), so the provider bounds it: see tableView.

View Source
const TableWarnThreshold = 500

TableWarnThreshold is how many distinct table values the metrics may carry before the provider logs, once, that per-table series can be turned off.

Variables

This section is empty.

Functions

func AnnotateSpan added in v0.2.1

func AnnotateSpan(ctx context.Context)

AnnotateSpan copies the request scope in ctx onto the span in ctx. Call it once the server span exists: the scope is set up before it. Its client.address replaces the one otelhttp derives, which reads X-Forwarded-For whoever sent it.

func ExtractHeaders added in v0.2.0

func ExtractHeaders(ctx context.Context, h map[string][]string) context.Context

ExtractHeaders returns ctx carrying the propagation context found in the message headers h. A nil h returns ctx unchanged. The subscriber calls this when a message is received.

func InitProvider

func InitProvider(ctx context.Context, id Identity, cfg ProviderConfig) (func(context.Context) error, http.Handler, error)

InitProvider sets up the OpenTelemetry pipeline, registering only the signals enabled in cfg. Always installs the W3C TraceContext + Baggage propagator (cheap, harmless when traces are off — in-process span extraction still works against a no-op tracer provider).

Every signal carries id as its resource, and a meter provider exports wavehouse_build_info (see Identity).

Returns a shutdown function (always non-nil) and a Prometheus HTTP handler (non-nil iff PrometheusEnabled). The handler reads from a private prometheus.Registry — global registry pollution is avoided.

func InjectHeaders added in v0.2.0

func InjectHeaders(ctx context.Context, h map[string][]string)

InjectHeaders writes the propagation context (trace + baggage) from ctx into the message headers h, which must be non-nil. The publisher calls this before handing a message to the broker so the consumer can pick the trace back up.

func NewLogger

func NewLogger(component string, level *slog.LevelVar, isJSON bool, otlpSampleRate float64) *slog.Logger

NewLogger creates a production-ready logger with component tags and trace support. level gates both stdout and OTLP. otlpSampleRate (in [0.0, 1.0]) caps OTLP export of the DEBUG/INFO records level admits; WARN/ERROR always export at 100% — dropping them silently during incidents is too dangerous to expose as a knob. Stdout receives 100% of the records level admits.

func PprofHandler added in v0.2.1

func PprofHandler() http.Handler

PprofHandler serves net/http/pprof's endpoints under /debug/pprof/ on a mux of its own. The package's import also registers them on http.DefaultServeMux, which nothing in WaveHouse serves; the handler is mounted only on the profiling listener, never on the API router.

func RegisterSystemMetrics

func RegisterSystemMetrics(mqStats func() (MQStats, error), pebbleStats func() map[string]int64) error

RegisterSystemMetrics creates asynchronous gauges that periodically pull stats from embedded systems (the MQ, Pebble) and push them to OpenTelemetry. Both are read on every scrape; nil skips the MQ gauges, as a nil pebbleStats — or a nil map from it, no store being open — skips the Pebble ones. The Pebble figures are the one instance's that every tenant's seen ids share (dedupe.Embedded.Stats in production).

func RequestID added in v0.2.1

func RequestID(ctx context.Context) string

RequestID returns the id WithRequest stored, or "" outside a request.

func SetRequestClient added in v0.2.1

func SetRequestClient(ctx context.Context, addr string)

SetRequestClient records the address the request came from on its scope and on the span in ctx. Outside a request it still annotates the span. An empty addr marks the address unresolved: log records leave it out, and the span's client.address is blanked rather than left as otelhttp's.

func SetRequestTenant added in v0.2.1

func SetRequestTenant(ctx context.Context, tenant string)

SetRequestTenant records the request's tenant on its scope and on the span in ctx. Outside a request it still annotates the span.

func WithRequest added in v0.2.1

func WithRequest(ctx context.Context, requestID string) context.Context

WithRequest returns ctx carrying a request scope with the given id. Every record TraceHandler handles under it, or under a context derived from it, carries request_id, and tenant once SetRequestTenant has run.

Types

type Identity added in v0.2.1

type Identity struct {
	// ServiceName is service.name unless OTEL_SERVICE_NAME overrides it.
	ServiceName string
	Version     string
	GitCommit   string
	BuildTime   string
	// InstanceID is service.instance.id: the boot config's instance_id, the
	// name a lease's holder carries too.
	InstanceID string
	Roles      []string
}

Identity names the process in its telemetry: the resource every span, metric series and OTLP log record carries, and the wavehouse_build_info series, so replicas of one role and builds in a mixed-version rollout can be told apart.

func (Identity) LogAttrs added in v0.2.1

func (id Identity) LogAttrs() []any

LogAttrs are the fields every log record carries to name the process and its build.

type MQStats added in v0.2.0

type MQStats struct {
	Connections int64 // active client connections
	InMsgs      int64 // messages received, cumulative
}

MQStats is the point-in-time snapshot of the embedded message queue the system gauges observe. Reported by the mq package (mq.Broker.Stats); defined here because mq imports observability, not the other way round.

type ProviderConfig

type ProviderConfig struct {
	TracesEnabled     bool
	TracesSampleRate  float64
	MetricsEnabled    bool
	PrometheusEnabled bool
	LogsEnabled       bool
	// AggregateTables drops the table attribute from every metric, reporting
	// one series summed over all tables (prometheus.per_table_metrics: false).
	AggregateTables bool
}

ProviderConfig wires the metrics/traces/logs pipeline. Each output is independently gated; SampleRate values must be in [0.0, 1.0] and only apply to head-based trace sampling. Log sampling is enforced inside NewLogger (per-level).

MetricsEnabled drives the OTLP-push metric exporter; PrometheusEnabled drives the Prometheus exposition reader. Either, both, or neither may be set — the underlying OTel MeterProvider is the shared substrate. When PrometheusEnabled is true InitProvider returns a non-nil promHandler.

The OTLP exporters take no endpoint/TLS/header options here: the OpenTelemetry SDK reads those from the standard OTEL_EXPORTER_OTLP_* env vars — endpoint (with `https://` selecting TLS via system root CAs), a custom CA via _CERTIFICATE, mutual TLS via _CLIENT_CERTIFICATE/_CLIENT_KEY, and auth _HEADERS. A malformed header is never fatal: traces and metrics skip that entry, while the logs exporter drops the whole variable.

type TraceHandler

type TraceHandler struct {
	slog.Handler
}

TraceHandler intercepts every log call to inject OpenTelemetry IDs and the request's correlation fields.

func (*TraceHandler) Handle

func (h *TraceHandler) Handle(ctx context.Context, r slog.Record) error

Handle pulls the trace_id and span_id from the context if they exist, and request_id, client.address and tenant from the request scope (WithRequest) when there is one.

func (*TraceHandler) WithAttrs

func (h *TraceHandler) WithAttrs(attrs []slog.Attr) slog.Handler

WithAttrs and WithGroup must re-wrap so chained Logger.With / WithGroup calls keep going through TraceHandler.Handle. Without these, Go promotes the calls to the embedded slog.Handler and the returned handler is the unwrapped inner handler — silently dropping stdout trace-ID injection.

func (*TraceHandler) WithGroup

func (h *TraceHandler) WithGroup(name string) slog.Handler

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL