Documentation
¶
Overview ¶
Package telemetry is chok's observability battery: Prometheus metrics, OpenTelemetry traces and, optionally, the App's logs copied to OpenTelemetry Logs, under one Config.
tel := telemetry.MustNew(cfg.Telemetry) d := db.MustOpen(cfg.DB, ...) tel.Instrument(d) app.Use(tel, d, ...)
Registered with app.Use, a Telemetry instruments every request (it is a chok.Instrumenter: its middleware runs between ClientIP and AccessLog) and serves its registry on a listener of its own, not the App's port: GET http://127.0.0.1:9464/metrics by default, reachable from the same host only. Set telemetry.metrics.addr to :9464 for a scraper on another host (a Prometheus in another container or pod), keeping that port away from the public internet. Use it before the components whose shutdown it should still see: Close runs in reverse order, and telemetry's Close flushes the spans and log records the others' Close produced.
Names ¶
The names follow the OpenTelemetry semantic conventions (semconv v1.43.0), written the Prometheus way: dots become underscores and the unit is a suffix.
- http_server_request_duration_seconds{http_request_method, http_route, http_response_status_code}: a histogram, http.server.request.duration.
- http_server_active_requests: requests in flight.
- db_client_operation_duration_seconds{db_system_name, db_operation_name, db_collection_name, error_type}: a histogram, db.client.operation.duration, after Instrument.
- db_client_slow_operations_total{db_system_name, db_operation_name, db_collection_name}: statements at or over db.slow_threshold.
- go_sql_* by db_name: the connection pools, after Instrument.
- go_* and process_*: the Go runtime and the process.
A server span is named "{method} {route}", as in "GET /api/v1/posts/{id}", and carries http.request.method, http.route, http.response.status_code, url.path, url.scheme, server.address, network.protocol.version, user_agent.original and client.address; a 5xx sets error.type and the error status. A request no route matched is "GET unmatched" (route label unmatched). A statement's span is named "{operation} {table}", as in "SELECT posts".
Traces ¶
Tracing is off by default. On, every request continues the trace of an incoming W3C traceparent header, or starts one for sample_ratio of them, and the spans go to stdout or, over OTLP/HTTP, to the collector at tracing.endpoint. Providers remain local by default. Give Globals() to the single process-wide Telemetry when third-party clients use otel's global defaults, or pass TracerProvider() to those clients explicitly. A binary that links telemetry records the trace id in identity's audit events.
Example ¶
Telemetry next to the App: Use puts its middleware in the default stack, and the running App serves the metrics at http://127.0.0.1:9464/metrics, a port of their own that only this host reaches; Instrument adds the database. A Config from chok.Load has the metrics on; this one is built in code. The App here is not run: the registry is read in process.
package main
import (
"context"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/zynthara/chok"
"github.com/zynthara/chok/db"
"github.com/zynthara/chok/telemetry"
)
type greeting struct {
Name string `path:"name"`
}
// Telemetry next to the App: Use puts its middleware in the default
// stack, and the running App serves the metrics at
// http://127.0.0.1:9464/metrics, a port of their own that only this host
// reaches; Instrument adds the database. A Config from chok.Load has the
// metrics on; this one is built in code. The App here is not run: the
// registry is read in process.
func main() {
var cfg telemetry.Config
cfg.Metrics.Enabled = true
tel := telemetry.MustNew(cfg)
defer func() { _ = tel.Close(context.Background()) }()
quiet := slog.New(slog.DiscardHandler)
d := db.MustOpen(db.Config{Driver: "sqlite", SQLite: db.SQLiteConfig{Path: ":memory:"}}, db.Logger(quiet))
defer func() { _ = d.Close(context.Background()) }()
tel.Instrument(d)
app := chok.New(chok.Base{}, chok.Logger(quiet))
app.Use(tel, d)
chok.GET(app.Router(), "/hello/{name}", func(ctx context.Context, req *greeting) (string, error) {
var one int
err := d.Unsafe(ctx).Raw("SELECT 1").Scan(&one).Error
return "hello " + req.Name, err
})
app.Handler().ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodGet, "/hello/ann", nil))
rec := httptest.NewRecorder()
promhttp.HandlerFor(tel.Registry(), promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
for line := range strings.Lines(rec.Body.String()) {
if strings.HasPrefix(line, "http_server_request_duration_seconds_count") ||
strings.HasPrefix(line, "db_client_operation_duration_seconds_count") {
fmt.Print(line)
}
}
}
Output: db_client_operation_duration_seconds_count{db_collection_name="",db_operation_name="SELECT",db_system_name="sqlite",error_type=""} 1 http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="200",http_route="/hello/{name}"} 1
Index ¶
- type Config
- type Option
- type Telemetry
- func (t *Telemetry) BindLogger(l *slog.Logger)
- func (t *Telemetry) Close(ctx context.Context) error
- func (t *Telemetry) Health(context.Context) error
- func (t *Telemetry) Instrument(d *db.DB)
- func (t *Telemetry) Middleware() chok.Middleware
- func (t *Telemetry) Name() string
- func (t *Telemetry) Registry() *prometheus.Registry
- func (t *Telemetry) Start(ctx context.Context) error
- func (t *Telemetry) TracerProvider() trace.TracerProvider
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
Metrics struct {
// Enabled records the HTTP and database metrics, with the Go
// runtime and process collectors, and serves them on Addr and
// Path in the Prometheus text format. Off, the registry stays
// empty.
Enabled bool `yaml:"enabled" default:"true"`
// Addr is the metrics' own listener, apart from http.addr: the
// loopback default keeps them off the network, and an address
// such as :9464 lets a scraper on another host reach them. It
// listens while the App's HTTP server does.
Addr string `yaml:"addr" default:"127.0.0.1:9464"`
// Path is where the metrics listener serves the metrics.
Path string `yaml:"path" default:"/metrics"`
} `yaml:"metrics"`
Tracing struct {
// Enabled records a span for every request and, through
// Instrument, every SQL statement, and exports them.
Enabled bool `yaml:"enabled"`
// Exporter is where spans go: stdout (one JSON object per span,
// for development) or otlp (OTLP over HTTP to Endpoint).
Exporter string `yaml:"exporter" default:"stdout" validate:"oneof=stdout otlp"`
// Endpoint is the OpenTelemetry collector's base URL, as in
// OTEL_EXPORTER_OTLP_ENDPOINT: spans are posted to
// <endpoint>/v1/traces and, with Logs.OTel, log records to
// <endpoint>/v1/logs. https uses TLS with the system roots. The
// other OTEL_EXPORTER_OTLP_* variables (headers, certificates,
// timeout) still apply.
Endpoint string `yaml:"endpoint" default:"http://localhost:4318"`
// ServiceName is the service.name of the spans and log records;
// empty takes OTEL_SERVICE_NAME, else the executable's name.
ServiceName string `yaml:"service_name"`
// SampleRatio is the fraction of new traces recorded, in (0, 1];
// a request whose traceparent says sampled is always recorded.
SampleRatio float64 `yaml:"sample_ratio" default:"1"`
} `yaml:"tracing"`
Logs struct {
// OTel copies every record the App's loggers write (those New
// builds from the log section, the access log included) to
// OpenTelemetry Logs, exported over OTLP to Tracing.Endpoint
// with the trace and span id of the request; the log output
// itself is unchanged.
OTel bool `yaml:"otel"`
} `yaml:"logs"`
}
Config is the telemetry section of an application's configuration:
telemetry:
metrics: # on by default
addr: 127.0.0.1:9464 # the default: this host only
path: /metrics
tracing:
enabled: true
exporter: otlp
endpoint: http://otel-collector:4318
service_name: blog
sample_ratio: 0.25
logs:
otel: true
New gives the zero fields of a Config built in code the defaults below: Metrics.Addr, Metrics.Path, Tracing.Exporter, Tracing.Endpoint and Tracing.SampleRatio. The switches have no zero-value default: a Config built in code sets Metrics.Enabled to record and serve the metrics.
type Option ¶
type Option func(*options)
Option configures New.
type Telemetry ¶
type Telemetry struct {
// contains filtered or unexported fields
}
Telemetry is the observability battery: a Prometheus registry, an OpenTelemetry tracer provider and, with logs.otel, a bridge from the App's logs to OpenTelemetry Logs. Register it with app.Use: its middleware records every request (chok.Instrumenter), its Start serves the metrics on a listener of their own, its Close flushes what is buffered.
func New ¶
New builds the battery from cfg. Providers remain instance-local unless Globals is given. With Globals, OpenTelemetry's own errors are routed through Logger. App.Use binds the Logs bridge only to that App. Spans and log records are exported in the background, in batches; Close flushes what is buffered and stops the exporters.
func (*Telemetry) BindLogger ¶
BindLogger attaches the log bridge to an App-owned logger. App.Use calls it automatically. Injected, unwrapped loggers remain application-owned.
func (*Telemetry) Close ¶
Close stops copying the logs, then flushes the spans and log records still buffered and shuts the exporters down, within ctx. The global tracer provider goes back to a no-op one if it is still this Telemetry's. Close is idempotent.
func (*Telemetry) Health ¶
Health reports nil: telemetry's trouble, such as an unreachable collector, never takes the process out of rotation; failed exports are logged.
func (*Telemetry) Instrument ¶
Instrument observes GORM statement execution and RunInTx transaction phases. Query callbacks include result scanning; Raw().Row()/Rows() callbacks end when the cursor is obtained and exclude later Scan/Next/Close work. Instrument that application-owned streaming work with a span when its duration matters. Statements produce a client span named "{operation} {table}", such as "SELECT posts", carrying the SQL as gorm built it, with placeholders for the bound values (the text of the slow-query log), and its outcome; the statement's duration in db_client_operation_duration_seconds, by operation, table and error type (canceled, timeout or error; a missing record is not an error); and db_client_slow_operations_total, counting the statements that took db.slow_threshold or longer, the threshold of the slow-query log (0 counts none). d's connection pools are exported as the go_sql_* metrics, labelled db_name with the dialect (a SQLite file's read pool with _read; a second handle of the same dialect with _2, and so on).
Instrument may be called at any time, even while d is in use; calling it again for d does nothing. With tracing and metrics off it records nothing.
func (*Telemetry) Middleware ¶
func (t *Telemetry) Middleware() chok.Middleware
Middleware returns the request instrumentation the App puts between ClientIP and AccessLog (t implements chok.Instrumenter): it continues the trace of an incoming W3C traceparent, records a server span named "{method} {route}", such as "GET /api/v1/posts/{id}", and observes the request's duration in http_server_request_duration_seconds by method, route and status code. A request no route matched is named "GET unmatched"; a method outside the standard ones is recorded as _OTHER (the span's name says HTTP), so neither can grow the label sets. A panic counts as a 500 and goes on to the App's Recover.
func (*Telemetry) Registry ¶
func (t *Telemetry) Registry() *prometheus.Registry
Registry returns the Prometheus registry the metrics listener serves: the HTTP, database, Go runtime and process metrics, and whatever the application registers next to them. With metrics off it stays empty and is not served. To serve it on the App's own port as well, behind the application's own guard:
app.Router().Handle(http.MethodGet, "/metrics", promhttp.HandlerFor(tel.Registry(), promhttp.HandlerOpts{}))
func (*Telemetry) Start ¶
Start serves the registry on GET metrics.path at metrics.addr, a listener of its own, apart from the App's port: the loopback default keeps the metrics off the network. It listens only while the App serves HTTP — a worker process (http.enabled false) and an App under choktest open no port — and with metrics on; otherwise it just waits for ctx. An address already in use fails the App's start.
func (*Telemetry) TracerProvider ¶
func (t *Telemetry) TracerProvider() trace.TracerProvider
TracerProvider returns the tracer provider, for spans of the application's own; with tracing off it is a no-op provider, so instrumenting code needs no switch.