telemetry

Quick Start For Local Telemetry

To quickly setup a local telemetry environment where OpenTelemetry data is sent to a local instance of Grafana LGTM:

start the Grafana LGTM docker image:

docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm

Environment Variable

Using the environment variable method will instantiate the OpenTelemetry SDK before global meters and spans. This allows meters and traces to use direct references to the underlying instrument.

Create a basic OpenTelemetry configuration file which will send data to the local instance of Grafana LGTM:

file_format: "1.0-rc.3"
resource:
  attributes:
    - name: service.name
      value: simapp

tracer_provider:
  processors:
    - batch: # NOTE: you should use batch in production!
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317

meter_provider:
  readers:
    - pull:
        exporter:
          prometheus/development: # pushes directly to prometheus backend. 
            host: 0.0.0.0
            port: 9464
            # optional: include resource attributes as constant labels
            with_resource_constant_labels:
              include:
                - service.name
  views:
    - selector:
        instrument_type: histogram
      stream:
        aggregation:
          explicit_bucket_histogram:
            boundaries: [ 0.2, 0.25, 0.3, 0.35, 0.4, 0.5, 0.75, 1, 2, 5 ]

logger_provider:
  processors:
    - batch:
        exporter:
          otlp_grpc:
            endpoint: http://localhost:4317


extensions:
  instruments:
    host: {} # enable optional host instrumentation with go.opentelemetry.io/contrib/instrumentation/host
    runtime: {} # enable optional runtime instrumentation with go.opentelemetry.io/contrib/instrumentation/runtime
    diskio: {} # enable optional disk I/O instrumentation using gopsutil
    # diskio with options:
    # diskio:
    #   disable_virtual_device_filter: true  # include virtual devices (loopback, RAID, partitions) on Linux
  propagators:
    - tracecontext

For a full list of configurable options see: https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml. NOTE: the go implementation may not support all options, so check the go otelconf documentation carefully to see what is actually supported.

1. set the OTEL_EXPERIMENTAL_CONFIG_FILE environment variable to the path of the configuration file: export OTEL_EXPERIMENTAL_CONFIG_FILE=path/to/config.yaml
2. start your application or tests
3. view the data in Grafana LGTM at http://localhost:3000/. The Drilldown views are suggested for getting started.

Node Home OpenTelemetry file

The node's init command will generate an empty otel file in the ~/.<node_home>/config directory. Place your otel configuration here.

When the node's start command is run, the OpenTelemetry SDK will be initialized using this file. If left empty, all meters and tracers will be noop.

OpenTelemetry Initialization

While manual OpenTelemetry initialization is still supported, this package provides a single point of initialization such that end users can just use the official OpenTelemetry declarative configuration spec: https://opentelemetry.io/docs/languages/sdk-configuration/declarative-configuration/ End users only need to set the OTEL_EXPERIMENTAL_CONFIG_FILE environment variable to the path of an OpenTelemetry configuration file, or fill out the otel.yaml file in the node's config directory and that's it. All the documentation necessary is provided in the OpenTelemetry documentation.

Developer Usage

If using the environment variable method, importing the baseapp package will cause the telemetry's initialization to run. Otherwise, ensure the otel.yaml file in the node's config directory is filled out.

IMPORTANT: Make sure Shutdown() is called when the application is shutting down.

Tests can use the TestingMain function at startup to accomplish this.

If these steps are followed, developers can follow the official golang otel conventions of declaring package-level tracer and meter instances using otel.Tracer() and otel.Meter(). NOTE: it is important to thread context.Context properly for spans and metrics to be correlated correctly. When using the SDK's context type, spans must be started with Context.StartSpan to get an SDK context which has the span set correctly.