o11y

Overview ¶

O11Y LLM-OBSERVABILITY EVENT INGEST — the native-Go write path for LLM-observability events (traces / observations / scores), folding the RETIRED console-worker (a Node BullMQ→Valkey→Datastore worker) into the one cloud binary.

GROUNDING — why a new SPECIFIC route, not a wire into the embed:

The embedded o11y runtime (embed.go, github.com/hanzoai/o11y) is our infra-observability engine:
it serves INFRA observability (OTLP traces/logs/metrics, dashboards, alerts) and
mounts `app.All("/v1/o11y/*")` as a WILDCARD proxy at subsystem order 70. It has
NO LLM-observability ingestion surface (no traces/observations/scores writer) —
that data is a DIFFERENT product (the console-worker's Datastore tables). So this
file adds the MISSING write path as a cloud-native specific route,
POST /v1/o11y/ingestion.

Fiber matches routes in REGISTRATION ORDER, so a specific /v1/o11y/* route only
binds ahead of the order-70 wildcard when it registers BEFORE it — exactly the
constraint scope.go documents for /v1/o11y/{logs,metrics,status} at order 69.
This subsystem therefore mounts at order 68 (before the wildcard). A route at the
OTLP-ingest order (72) would be SWALLOWED by the proxy and never reached.

PIPELINE — POST /v1/o11y/ingestion (validated tenant) → parse the event batch → group by target Datastore table → batch-insert via the branded github.com/hanzo-ds/go client → oversized event bodies overflow to object storage, only the blob ref is stored inline. The store is the Datastore; datastore-go is verified to bring the ONE ch-go transport line the o11y runtime already uses (MVS-unified), so it coexists in the single binary.

Safety posture (this writes a LIVE, SHARED Datastore):

• Always mounted as a normal o11y subsystem — NO on/off feature flag. The write path is live wherever O11Y_DATASTORE_DSN is set (the SAME knob embed.go reads); the Datastore is a required DEPENDENCY, not a gate. Inert in prod until the console producer repoints to /v1/o11y/ingestion (nothing calls it yet).
• Fail-soft: a missing DSN or a construction error logs and returns nil — never blocks cloud boot (mirrors ingest.go / tracesink.go).
• Durable hand-off: the flush runs INLINE today (always works, mirroring ai's durable-ingest inline fallback). When cloud.EmbeddedTasks() is wired, the accept path enqueues the batch to the ONE in-process durable engine (no BullMQ, no Valkey) and a registered activity flushes it — that worker is the next reviewed step (a Datastore insert must be a durable Activity, not run in a workflow function).

Telemetry INGEST — the in-process OpenTelemetry Collector that folds the standalone otel-collector Deployment into the unified cloud binary.

cloud already embeds the o11y QUERY runtime (embed.go) over the datastore datastore. This file adds the WRITE side: a real OpenTelemetry Collector, constructed IN-PROCESS, that accepts the ZAP span wire (:4317) and writes spans + logs into the SAME datastore the embedded query runtime reads (o11y_traces / o11y_logs on the `insights` cluster). Consumers — cloud itself, console-worker, and third-party OTel SDKs — point at cloud instead of the standalone otel-collector Service, so the standalone Deployment can retire.

Pipeline — trimmed from the standalone collector to the components that both (a) the live consumers exercise and (b) compile against cloud's UPSTREAM datastore driver (datastore-go v2.44.0 / ch-go v0.71.0):

receivers:  zap (the ZAP span wire on 4317)
processors: memory_limiter -> resource(service.namespace=hanzo, deployment.environment) -> batch
exporters:  datastoretraces (traces), datastorelogsexporter (logs)

DEFERRED — the METRICS pipeline (the datastore-metrics exporter + the o11yspanmetrics connector) is intentionally NOT embedded here. That exporter references the upstream dd-sketch fork of ch-go (dsproto.DD/Store/IndexMapping), which does NOT compile against cloud's upstream ch-go — the two driver lines cannot coexist in one binary because cloud's embedded o11y QUERY service pins upstream v2.44.0. Metrics ingest therefore stays on the standalone collector until the metrics exporter is ported onto upstream ch-go. See LLM.md.

Safety posture (this touches a LIVE, SHARED telemetry store):

• Bound by CAPABILITY, not by a flag: ingest runs exactly when a datastore DSN is configured, because the DSN is the thing it writes to. There is no separate on/off env — a boolean that could silently disable the fleet's only ingest path is how every agent ended up in connection-refused with its logs discarded (2026-07-25).
• Fail-soft: any construction error logs and returns nil, so a bad telemetry config can never take cloud down (mirrors embed.go's proxy fallback posture).
• Registered with a ShutdownFunc so the collector flushes on graceful stop.

Package o11y is the ONE owner of the cloud binary's observability plane — registered as a SINGLE `o11y` subsystem (this file's init) that internally mounts, in the load-bearing order, every part of the concept:

READ/SERVE plane (specific /v1/o11y/* routes, registered BEFORE the
hanzoai/o11y wildcard so Fiber's in-order match gives them precedence):
  - tenant-scoped reads  /v1/o11y/{logs,metrics,status}   (scope.go)
  - SuperAdmin VM proxy  /v1/o11y/vm/{query,query_range}   (vmproxy.go)
  - flat builder query   /v1/o11y/{query,query_range}      (query.go)
  - LLM-obs event ingest POST /v1/o11y/ingestion           (event_ingest.go)
RUNTIME handler the hanzoai/o11y wildcard (order 70) delegates to via
  o11y.SetHandler — the in-process runtime (embed.go) or a reverse-proxy
  fallback (this file).
WRITE plane (opt-in, order-independent):
  - OTLP ingest collector (ingest.go)
  - in-process trace sink (tracesink.go)

Decomplection (one and one way): these were five separately-registered subsystems (o11yscope 69, o11y-runtime 71, o11y-event-ingest 68, o11y-otlp-ingest 72, o11y-trace-inproc 73) whose names leaked FIVE public concepts into the registry (five config toggles, five /v1/<name>/health routes). The k8s-style ordering was an internal impl detail. They now collapse to ONE registration of the name `o11y` (order 69): mountO11y performs the ordered sub-mounts in-process, so the PUBLIC concept is a single `o11y`. Behavior is preserved EXACTLY — every route registers at the same point relative to the order-70 wildcard as before (all inside the one order-69 mount, so all before 70).

Co-ownership: the upstream github.com/hanzoai/o11y module ALSO registers the name `o11y` (order 70, the wildcard route surface) from its own init. The two entries are co-owners of ONE public concept; this order-69 entry opts out of the generic HIP-0106 health route (cloud.HealthOwner) so /v1/o11y/health is registered EXACTLY once, by the module's order-70 co-entry.

One way, two backings (mountRuntime):

• PRIMARY: the in-process runtime (buildEmbeddedHandler), enabled by O11Y_TELEMETRYSTORE_DATASTORE_DSN. Serves telemetry from cloud itself.
• FALLBACK: a reverse proxy to a still-running o11y Deployment, used only when the embed is disabled (no DSN) or fails to init. Fail-soft, zero downtime.

Path is preserved verbatim: /v1/o11y/* reaches the o11y runtime unchanged, which rewrites /v1/o11y/* -> /api/* internally (see o11y app.createPublicServer). The gateway terminates auth and propagates identity as X-* headers; the runtime (embedded) or the proxy (fallback) sees the same request.

In-process trace sink — the CONSUMER half of cloud's own span path.

The host owns the tracer provider and the one Send that leaves it (cloud/telemetry.go). This file is what makes that Send free when o11y is linked into the same binary: it registers a handler on the host's trace destination, receives the LIVE span batch by value — zero encode, zero socket, no second collector hop — and writes it to o11y_traces through the REAL dstraces exporter, the one writer that produces the o11y_index_v3 schema the embedded query plane reads.

Registration is process-local, and that is the entire deployment story: linked in, this handler exists and the host's router prefers it; as a plugin, this handler exists in the CHILD, the host's router has no route, and the host's identical Send falls through to the ZAP wire — which lands on the collector this same package mounts (ingest.go). One producer, one call site, both topologies.

The pdata conversion (spanconv.go) lives HERE and not in the host on purpose: it is coupled to the datastore exporter's schema, and hoisting it would drag go.opentelemetry.io/collector into every package that imports cloud for Deps.

Safety posture (this feeds a LIVE, SHARED telemetry store):

• OPT-IN: mounts only when O11Y_TRACES_ZAP_INPROCESS is truthy AND a datastore DSN is set. Inert until the flag flips (verify-then-cutover).
• Fail-soft: any construction error logs and returns nil, leaving cloud's spans on the wire path — activating this can never take cloud down.
• Shutdown deregisters the handler (host falls back to the wire) then flushes the exporter's sending queue to datastore before exit.