Documentation
¶
Overview ¶
Package llmobstest provides test infrastructure for LLM Observability integrations.
The Collector captures spans and metrics sent by the LLMObs transport via an in-process RoundTripper — no real TCP listener is started. Wire the collector into the tracer via Collector.TracerOption:
agent, err := tracertest.StartAgent(t)
require.NoError(t, err)
coll := llmobstest.New(t)
_, err = tracertest.Start(t, agent,
tracer.WithLLMObsEnabled(true),
tracer.WithLLMObsMLApp("my-app"),
coll.TracerOption(),
)
require.NoError(t, err)
// ... exercise code ...
tracer.Flush()
span := coll.RequireSpan(t, "my-span")
Index ¶
- type Collector
- func (c *Collector) FindMetric(label string) *LLMObsMetric
- func (c *Collector) FindSpan(name string) *LLMObsSpan
- func (c *Collector) HandleFunc(pattern string, handler http.HandlerFunc)
- func (c *Collector) MetricBatchSizes() []int
- func (c *Collector) MetricCount() int
- func (c *Collector) RequireMetric(tb testing.TB, label string) *LLMObsMetric
- func (c *Collector) RequireSpan(tb testing.TB, name string) *LLMObsSpan
- func (c *Collector) SetSpanResponseDelay(d time.Duration)
- func (c *Collector) SpanBatchSizes() []int
- func (c *Collector) SpanCount() int
- func (c *Collector) Spans() []LLMObsSpan
- func (c *Collector) TracerOption() tracer.StartOption
- type LLMObsMetric
- type LLMObsSpan
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Collector ¶
type Collector struct {
// contains filtered or unexported fields
}
Collector captures LLMObs spans and metrics via an in-process RoundTripper. Use New to create one, then include Collector.TracerOption in the tracer start options so that the LLMObs transport sends data here.
After calling tracer.Flush, all buffered data is guaranteed to be available for querying — no timeouts or polling are needed.
func New ¶
New creates a Collector that routes LLMObs requests via an in-process RoundTripper — no real TCP listener is started.
func (*Collector) FindMetric ¶
func (c *Collector) FindMetric(label string) *LLMObsMetric
FindMetric returns the first collected evaluation metric whose Label equals label, or nil if none is found.
func (*Collector) FindSpan ¶
func (c *Collector) FindSpan(name string) *LLMObsSpan
FindSpan returns the first collected LLMObs span whose Name equals name, or nil if none is found. Call this after tracer.Flush.
func (*Collector) HandleFunc ¶
func (c *Collector) HandleFunc(pattern string, handler http.HandlerFunc)
HandleFunc registers an HTTP handler on the collector's server for the given pattern. Use this to mock dataset, project, or experiment API endpoints in tests. With Collector.TracerOption, all LLMObs transport calls — including dataset and experiment API calls — go to this server, so mock handlers must be registered here rather than on the APM agent.
Patterns use the direct path (no /evp_proxy/v2 prefix), e.g.:
coll.HandleFunc("/api/unstable/llm-obs/v1/", myMockHandler)
func (*Collector) MetricBatchSizes ¶
MetricBatchSizes returns the raw body byte-length of each eval-metric batch HTTP request received so far, in order. Each entry corresponds to one call to the /api/intake/llm-obs/v2/eval-metric endpoint. Use this to verify size-based flushing behaviour: with correct flushing, no single entry should exceed 5 MB.
func (*Collector) MetricCount ¶
MetricCount returns the number of evaluation metrics collected so far.
func (*Collector) RequireMetric ¶
func (c *Collector) RequireMetric(tb testing.TB, label string) *LLMObsMetric
RequireMetric returns the first evaluation metric with the given label. It fails the test immediately if no matching metric is found.
func (*Collector) RequireSpan ¶
func (c *Collector) RequireSpan(tb testing.TB, name string) *LLMObsSpan
RequireSpan returns the first collected LLMObs span with the given name. It fails the test immediately if no matching span is found.
func (*Collector) SetSpanResponseDelay ¶
SetSpanResponseDelay makes the collector sleep for d before responding to each span-batch request, simulating a slow transport. Use it to exercise flush timing behaviour (e.g. that FlushSync blocks until the send completes).
func (*Collector) SpanBatchSizes ¶
SpanBatchSizes returns the raw body byte-length of each span batch HTTP request received so far, in order. Each entry corresponds to one call to the /api/v2/llmobs endpoint. Use this to verify size-based flushing behaviour: with correct flushing, no single entry should exceed 5 MB.
func (*Collector) Spans ¶
func (c *Collector) Spans() []LLMObsSpan
Spans returns a copy of all LLMObs spans collected so far, in arrival order. Use it when a span has no distinguishing name (e.g. spans started with an empty name) and must be addressed positionally. Call after tracer.Flush.
func (*Collector) TracerOption ¶
func (c *Collector) TracerOption() tracer.StartOption
TracerOption returns a tracer.StartOption that routes the LLMObs transport through this collector's in-process RoundTripper. Include it when starting the tracer.
type LLMObsMetric ¶
type LLMObsMetric = llmobstransport.LLMObsMetric
LLMObsMetric is an alias for the LLMObs evaluation metric type.
type LLMObsSpan ¶
type LLMObsSpan = llmobstransport.LLMObsSpanEvent
LLMObsSpan is an alias for the LLMObs span event type.
Source Files