Documentation
¶
Overview ¶
Package telemetry generates the OpenTelemetry Collector configuration that receives OTLP from Claude Code, Codex, Gemini CLI and Copilot CLI, normalises their harness-specific metrics into the halo.* schema, and exports to ClickHouse and Prometheus.
Normalised CLIs, exactly:
- Claude Code: metrics and log events.
- Gemini CLI: metrics and log events.
- Codex: log events only (its native metrics, including any lines-of-code metric, are not mapped: the vendor metric name is UNVERIFIED in this repo).
- Copilot CLI: metrics only (gen_ai.client.token.usage, gen_ai.client.operation.duration, github.copilot.tool.call.count). It emits spans and span events but no OTel log records, so it produces no halo.api.request / halo.tool.call rows. Its github.copilot.code.lines_added and _removed counters are separate counters; both map to halo.loc with the type attribute (added|removed) stamped from the source name, as Claude Code reports it. Not mapped: the other github.copilot.* / gen_ai.invoke_agent.* metrics. https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-command-reference
Not normalised, on purpose (nothing is invented for them):
- Cursor CLI: no OTel export is documented for the CLI; telemetry is a dashboard/server-side team setting (internal/harness/FACTS.md).
- OpenCode: no OTLP setting; experimental.openTelemetry only switches on AI SDK spans and the exporter comes from OTEL_EXPORTER_OTLP_* environment, which a managed file cannot set (internal/harness/FACTS.md).
For Cursor CLI, OpenCode and any other CLI without native OTel, the honest per-request source is halo-proxy: it exports gen_ai.* GenAI semantic-convention metrics and opt-in spans on otlp/gateway for all traffic that passes through it (see below). That is request-level evidence only (no session, tool or edit signals), and only for traffic routed through the gateway.
Verified end to end by `make obs-e2e` (scripts/obs-e2e.sh): the config loads in otelcol-contrib 0.161.0 (`validate`), the transform renames metrics and stamps halo.harness, and clickhouseexporter writes the rows that deploy/observability/clickhouse/schema.sql projects into halo_metrics. Codex telemetry is event-oriented and its native metrics are not mapped; Codex only matches when it reports gen_ai.client.token.usage with halo.harness=codex set. API/tool events from all three CLIs arrive on the logs pipeline and are turned into halo.api.request / halo.tool.call rows by the schema's log view.
Metrics fan out to two pipelines: ClickHouse gets cumulativetodelta (so per-unit sums never double count cumulative counters); Prometheus keeps the native temporality, takes only the halo.* resource attributes as labels, and drops halo-proxy's halo.gateway.* metrics, whose per-unit halo.unit attribute would explode label cardinality (halo-proxy serves its own /metrics).
Evidence trust: there are two OTLP receivers. `otlp` (CLI, 4317/4318) must be reachable from developer machines, so anything on it is client-controlled: it drops every halo.gateway.* metric and overwrites the data point attribute halo.source with "cli". `otlp/gateway` (OTLP/HTTP, 4319) requires the halo-proxy bearer token (${env:HALO_OTLP_GATEWAY_TOKEN}; the collector refuses to start without it) and stamps halo.source="gateway". internal/promote only counts halo.gateway.* rows carrying halo.source=gateway, and the controller only auto-kills on gateway-sourced evidence. Optionally the CLI receiver requires per-device bearer tokens (Options.CLITokenFile, one per line, hot-reloaded); either way its request size is capped.
halo-proxy also exports OpenTelemetry GenAI semantic-convention metrics (gen_ai.client.*) and, opt-in, per-call spans on otlp/gateway. They ride the same authenticated receiver; metrics land in the exporter's raw tables only (halo_metrics projects halo.* names), spans go through traces/gateway.
Optionally (Options.EvalReceiver) a third receiver, `otlp/eval` (OTLP/HTTP, 4320), takes `halo eval online` quality readings behind its own token (${env:HALO_OTLP_EVAL_TOKEN}, never the gateway token): it keeps only halo.eval.* metrics (so it can never carry halo.gateway.* evidence) and stamps halo.source="eval", which nothing auto-kills on.
Index ¶
Constants ¶
const ( SourceGateway = "gateway" SourceCLI = "cli" SourceEval = "eval" )
Trusted per-receiver values of the halo.source data point attribute.
const EvalTokenEnv = "HALO_OTLP_EVAL_TOKEN" //nolint:gosec // env var name, not a credential
EvalTokenEnv is the collector environment variable holding the bearer token `halo eval online` presents on the eval receiver. It must differ from the gateway token: eval jobs must never be able to write gateway evidence.
const GatewayTokenEnv = "HALO_OTLP_GATEWAY_TOKEN" //nolint:gosec // env var name, not a credential
GatewayTokenEnv is the collector environment variable holding the bearer token halo-proxy presents on the gateway receiver.
Variables ¶
var Mappings = []Mapping{
{"claude-code", "claude_code.cost.usage", "halo.cost.usd"},
{"claude-code", "claude_code.token.usage", "halo.tokens"},
{"claude-code", "claude_code.code_edit_tool.decision", "halo.edit.decision"},
{"claude-code", "claude_code.session.count", "halo.sessions"},
{"claude-code", "claude_code.lines_of_code.count", "halo.loc"},
{"claude-code", "claude_code.pull_request.count", "halo.pull_requests"},
{"claude-code", "claude_code.commit.count", "halo.commits"},
{"claude-code", "claude_code.active_time.total", "halo.active_time.seconds"},
{"gemini-cli", "gen_ai.client.token.usage", "halo.tokens"},
{"gemini-cli", "gen_ai.client.operation.duration", "halo.operation.duration"},
{"gemini-cli", "gemini_cli.session.count", "halo.sessions"},
{"gemini-cli", "gemini_cli.tool.call.count", "halo.tool.calls"},
{"copilot-cli", "github.copilot.tool.call.count", "halo.tool.calls"},
{"copilot-cli", "github.copilot.code.lines_added", "halo.loc"},
{"copilot-cli", "github.copilot.code.lines_removed", "halo.loc"},
}
Mappings is the single source of truth for metric normalisation.
Functions ¶
Types ¶
type Mapping ¶
type Mapping struct {
Harness string // value stamped into resource attribute halo.harness if absent
From string
To string
}
Mapping renames one harness-native metric to its normalised halo.* name.
type Options ¶
type Options struct {
GRPCEndpoint string // CLI OTLP/gRPC listen address, default 0.0.0.0:4317
HTTPEndpoint string // CLI OTLP/HTTP listen address, default 0.0.0.0:4318
GatewayHTTPEndpoint string // halo-proxy OTLP/HTTP listen address (bearer token), default 0.0.0.0:4319
// CLITokenFile, if set, makes the CLI receiver require a bearer token listed
// in this file (one per line; the collector re-reads it on change).
CLITokenFile string
// EvalReceiver adds the authenticated online-eval receiver on
// EvalHTTPEndpoint (default 0.0.0.0:4320), token from EvalTokenEnv.
EvalReceiver bool
EvalHTTPEndpoint string
ClickHouseEndpoint string // required, e.g. tcp://clickhouse:9000
ClickHouseDatabase string // default halo
PrometheusEndpoint string // scrape listen address, default 0.0.0.0:8889
}
Options configures the generated collector.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package gwmetrics is halo-proxy's per-request OTLP metrics emitter: the gateway assigns ring/variant itself, so these are the best-attributed online samples for halo.api.error_rate and halo.latency.* (and the only ones for traffic-axis experiments).
|
Package gwmetrics is halo-proxy's per-request OTLP metrics emitter: the gateway assigns ring/variant itself, so these are the best-attributed online samples for halo.api.error_rate and halo.latency.* (and the only ones for traffic-axis experiments). |
|
Package synth emits synthetic OTLP/HTTP (JSON) telemetry shaped like the real harnesses, for control vs treatment cohorts with a known injected effect.
|
Package synth emits synthetic OTLP/HTTP (JSON) telemetry shaped like the real harnesses, for control vs treatment cohorts with a known injected effect. |