eyrie

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 4, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Audit log sink for LLM calls.

AuditEvent captures the privacy-preserving metadata of a single provider call: model/provider, session and key identifiers, sha256 hashes of the prompt and response (never the raw text), token counts, cost, latency, and status.

Sinks are pluggable via the AuditSink interface. The default NoopSink discards events; JSONLFileSink appends one JSON object per line to a configurable file. Everything here is stdlib-only.

OpenTelemetry GenAI semantic convention attribute keys for AI agent spans.

These exported constants are the canonical, ecosystem-wide attribute keys for describing LLM / AI agent operations. They follow the OpenTelemetry GenAI semantic conventions (gen_ai.*) and are shared as the reference set that the other hawk-eco repos (hawk, yaad, tok, trace) should mirror when emitting spans, so dashboards and exporters can correlate cost/usage/identity across the whole ecosystem.

The legacy llm.* attribute keys in observability.go remain for backwards compatibility; new instrumentation should prefer the gen_ai.* keys below. This file is stdlib-only and purely additive (constant declarations only).

Spec: https://opentelemetry.io/docs/specs/semconv/gen-ai/

Package eyrie observability provides OpenTelemetry-compatible structured tracing and metrics collection for all LLM provider calls.

Design follows OpenTelemetry Go SDK patterns (TraceID/SpanID, span lifecycle, metric instruments) while remaining zero-dependency (Go stdlib only).

Usage is opt-in: a nil *Telemetry adds zero overhead.

Index

Constants

View Source
const (
	// AttrGenAISystem identifies the GenAI provider/system handling the request
	// (e.g. "openai", "anthropic", "gemini"). OTel: gen_ai.system.
	AttrGenAISystem = "gen_ai.system"

	// AttrGenAIRequestModel is the model name requested by the caller
	// (e.g. "gpt-4o", "claude-sonnet-4-5"). OTel: gen_ai.request.model.
	AttrGenAIRequestModel = "gen_ai.request.model"

	// AttrGenAIResponseModel is the model that actually served the response, when
	// the provider reports it. OTel: gen_ai.response.model.
	AttrGenAIResponseModel = "gen_ai.response.model"

	// AttrGenAIUsageInputTokens is the number of prompt/input tokens consumed.
	// OTel: gen_ai.usage.input_tokens.
	AttrGenAIUsageInputTokens = "gen_ai.usage.input_tokens"

	// AttrGenAIUsageOutputTokens is the number of completion/output tokens
	// generated. OTel: gen_ai.usage.output_tokens.
	AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens"

	// AttrGenAIOperationName names the GenAI operation (e.g. "chat", "embeddings",
	// "tool_use"). OTel: gen_ai.operation.name.
	AttrGenAIOperationName = "gen_ai.operation.name"

	// AttrCostUSD is the computed monetary cost of the operation in US dollars.
	// Ecosystem extension to the OTel gen_ai.* set; key: cost.usd.
	AttrCostUSD = "cost.usd"

	// AttrToolName is the name of the tool/function invoked during an agent step.
	// OTel: gen_ai.tool.name (exposed here as the stable key "tool.name").
	AttrToolName = "tool.name"

	// AttrSessionID correlates spans belonging to the same logical session or
	// conversation. Ecosystem key: session.id.
	AttrSessionID = "session.id"

	// AttrAgentID identifies the agent instance producing the span. Ecosystem
	// key: agent.id (aligns with OTel gen_ai.agent.id).
	AttrAgentID = "agent.id"
)

GenAI semantic convention attribute keys. These mirror the OpenTelemetry gen_ai.* namespace plus the small set of ecosystem extensions (cost.usd, session.id, agent.id) documented in docs/OTEL-CONVENTIONS.md.

View Source
const (
	SpanLLMChat     = "llm.chat"
	SpanLLMStream   = "llm.stream"
	SpanLLMRetry    = "llm.retry"
	SpanLLMCacheHit = "llm.cache_hit"
)
View Source
const (
	AttrLLMProvider     = "llm.provider"
	AttrLLMModel        = "llm.model"
	AttrLLMInputTokens  = "llm.input_tokens"
	AttrLLMOutputTokens = "llm.output_tokens"
	AttrLLMCostUSD      = "llm.cost_usd"
	AttrLLMLatencyMs    = "llm.latency_ms"
	AttrLLMStatus       = "llm.status"
)

Variables

This section is empty.

Functions

func HashContent

func HashContent(s string) string

HashContent returns the lowercase hex-encoded sha256 of s. Use it to derive PromptHash / ResponseHash without persisting raw prompt or response text.

Types

type AuditEvent

type AuditEvent struct {
	Timestamp    time.Time `json:"timestamp"`
	Model        string    `json:"model"`
	Provider     string    `json:"provider"`
	SessionID    string    `json:"session_id,omitempty"`
	VirtualKeyID string    `json:"virtual_key_id,omitempty"`
	PromptHash   string    `json:"prompt_hash,omitempty"`
	ResponseHash string    `json:"response_hash,omitempty"`
	InputTokens  int       `json:"input_tokens"`
	OutputTokens int       `json:"output_tokens"`
	CostUSD      float64   `json:"cost_usd"`
	LatencyMS    int64     `json:"latency_ms"`
	Status       string    `json:"status"`
}

AuditEvent is a single audit record. It deliberately stores only hashes of prompt/response content so that raw text is never persisted to the audit log.

type AuditSink

type AuditSink interface {
	Record(event AuditEvent) error
}

AuditSink records audit events. Implementations must be safe for concurrent use.

type JSONLFileSink

type JSONLFileSink struct {
	// contains filtered or unexported fields
}

JSONLFileSink appends audit events as newline-delimited JSON to a file. Writes are serialized with a mutex so concurrent callers do not interleave lines.

func NewJSONLFileSink

func NewJSONLFileSink(path string) (*JSONLFileSink, error)

NewJSONLFileSink opens (creating if needed) the file at path for append-only writes and returns a sink that writes to it. The caller owns the sink and should call Close when done.

func (*JSONLFileSink) Close

func (s *JSONLFileSink) Close() error

Close closes the underlying file.

func (*JSONLFileSink) Record

func (s *JSONLFileSink) Record(event AuditEvent) error

Record implements AuditSink, appending the event as one JSON line.

type MetricsCollector

type MetricsCollector struct {
	// contains filtered or unexported fields
}

MetricsCollector aggregates metrics for LLM operations. It uses atomic operations and mutexes for thread safety with minimal contention.

func NewMetricsCollector

func NewMetricsCollector() *MetricsCollector

NewMetricsCollector creates a new MetricsCollector.

func (*MetricsCollector) CacheHitRate

func (mc *MetricsCollector) CacheHitRate() float64

CacheHitRate returns the ratio of cache hits to total cache-eligible requests.

func (*MetricsCollector) CostAccumulator

func (mc *MetricsCollector) CostAccumulator(key string) float64

CostAccumulator returns the total cost in USD for a provider/model key.

func (*MetricsCollector) ErrorRate

func (mc *MetricsCollector) ErrorRate(provider string) float64

ErrorRate returns the error rate for a provider (errors / total requests involving that provider).

func (*MetricsCollector) ExportJSON

func (mc *MetricsCollector) ExportJSON() string

ExportJSON dumps all metrics as a JSON string.

func (*MetricsCollector) ExportPrometheus

func (mc *MetricsCollector) ExportPrometheus() string

ExportPrometheus dumps metrics in Prometheus exposition format.

func (*MetricsCollector) LatencyHistogram

func (mc *MetricsCollector) LatencyHistogram(key string) (p50, p95, p99 float64)

LatencyHistogram returns P50, P95, P99 latencies in milliseconds for a key.

func (*MetricsCollector) RecordCustom

func (mc *MetricsCollector) RecordCustom(name string, value float64, attrs map[string]string)

RecordCustom records a custom named metric.

func (*MetricsCollector) RecordRequest

func (mc *MetricsCollector) RecordRequest(provider, model string, inputTok, outputTok int, latency time.Duration, costUSD float64, isError bool)

RecordRequest records a completed LLM request with all its metrics.

func (*MetricsCollector) RequestCount

func (mc *MetricsCollector) RequestCount(key string) int64

RequestCount returns total requests for a provider/model key.

func (*MetricsCollector) TokensUsed

func (mc *MetricsCollector) TokensUsed(key string) (int64, int64)

TokensUsed returns (inputTokens, outputTokens) for a provider/model key.

func (*MetricsCollector) TotalCost

func (mc *MetricsCollector) TotalCost() float64

TotalCost returns the sum of all accumulated costs in USD.

func (*MetricsCollector) TotalRequests

func (mc *MetricsCollector) TotalRequests() int64

TotalRequests returns the sum of all request counts.

type NoopSink

type NoopSink struct{}

NoopSink is an AuditSink that discards all events. It is the safe default when auditing is not configured.

func (NoopSink) Record

func (NoopSink) Record(AuditEvent) error

Record implements AuditSink and does nothing.

type Span

type Span struct {
	TraceID    string            `json:"trace_id"`
	SpanID     string            `json:"span_id"`
	Name       string            `json:"name"`
	StartTime  time.Time         `json:"start_time"`
	EndTime    time.Time         `json:"end_time,omitempty"`
	Provider   string            `json:"provider,omitempty"`
	Model      string            `json:"model,omitempty"`
	Status     SpanStatus        `json:"status"`
	Attributes map[string]string `json:"attributes,omitempty"`
	Events     []SpanEvent       `json:"events,omitempty"`
}

Span represents a single timed operation in a trace, modeled after OpenTelemetry's Span concept. It carries a TraceID, SpanID, timing information, and arbitrary string attributes.

func (*Span) AddEvent

func (s *Span) AddEvent(name string, attrs map[string]string)

AddEvent adds a timestamped event to the span.

func (*Span) Duration

func (s *Span) Duration() time.Duration

Duration returns the span duration. Returns 0 if not yet ended.

func (*Span) SetAttribute

func (s *Span) SetAttribute(key, value string)

SetAttribute sets a single attribute on the span.

type SpanEvent

type SpanEvent struct {
	Name       string            `json:"name"`
	Timestamp  time.Time         `json:"timestamp"`
	Attributes map[string]string `json:"attributes,omitempty"`
}

SpanEvent records a timestamped event within a span.

type SpanStatus

type SpanStatus string

SpanStatus represents the outcome of a span.

const (
	StatusOK    SpanStatus = "ok"
	StatusError SpanStatus = "error"
	StatusUnset SpanStatus = "unset"
)

type Telemetry

type Telemetry struct {

	// OnSpanEnd is an optional callback invoked when a span ends.
	// Useful for exporting spans to external systems.
	OnSpanEnd func(*Span)
	// contains filtered or unexported fields
}

Telemetry wraps observability for all LLM calls. A nil *Telemetry is safe to use and adds zero overhead (all methods are no-ops on nil receiver).

func NewTelemetry

func NewTelemetry() *Telemetry

NewTelemetry creates a new Telemetry instance with an initialized MetricsCollector.

func (*Telemetry) EndSpan

func (t *Telemetry) EndSpan(span *Span, err error)

EndSpan completes a span, recording its status and duration. If err is non-nil, the span status is set to error and the error message is recorded. No-op if either t or span is nil.

func (*Telemetry) Metrics

func (t *Telemetry) Metrics() *MetricsCollector

Metrics returns the underlying MetricsCollector, or nil if Telemetry is nil.

func (*Telemetry) RecordMetric

func (t *Telemetry) RecordMetric(name string, value float64, attrs map[string]string)

RecordMetric records a named metric value with attributes. This is a general-purpose method for recording custom metrics. No-op if t is nil.

func (*Telemetry) Spans

func (t *Telemetry) Spans() []*Span

Spans returns a copy of all completed spans.

func (*Telemetry) StartSpan

func (t *Telemetry) StartSpan(name string, attrs map[string]string) *Span

StartSpan creates and starts a new Span with the given name and attributes. Returns nil if the Telemetry receiver is nil (opt-in pattern).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL