Documentation
¶
Overview ¶
Package testevals provides a testing.T adapter for the evals framework.
Overview ¶
The testevals package provides adapters for *testing.T to implement the evals.Observer interface. This allows evaluation callbacks to report failures and log messages through Go's standard testing framework.
Two constructors are available:
- New(t): Creates a basic adapter
- NewPrefix(t, prefix): Creates an adapter that prefixes all messages
Usage ¶
Basic usage with a test:
func TestLogAnalyzer(t *testing.T) {
// Create a namespaced observer using testevals.NewPrefix
namespacedObs := evals.NewNamespacedObserver(func(name string) evals.Observer {
return testevals.NewPrefix(t, name)
})
// Use evals helpers with the testing adapter
callbacks := []agenttrace.TraceCallback{
evals.Inject(namespacedObs.Child("tool-calls"), evals.ExactToolCalls(1)),
evals.Inject(namespacedObs.Child("errors"), evals.NoErrors()),
}
tracer := agenttrace.ByCode(callbacks...)
// Use tracer with your analyzer
}
Integration with Test Harnesses ¶
The observer adapter is particularly useful when building test harnesses that run multiple test cases with different evaluation criteria:
type TestCase struct {
Name string
Evals map[string]evals.ObservableTraceCallback
}
func runTestCase(t *testing.T, tc TestCase) {
t.Run(tc.Name, func(t *testing.T) {
// Create a namespaced observer using testevals.NewPrefix
namespacedObs := evals.NewNamespacedObserver(func(name string) evals.Observer {
return testevals.NewPrefix(t, name)
})
var callbacks []agenttrace.TraceCallback
for namespace, eval := range tc.Evals {
childObs := namespacedObs.Child(namespace)
callbacks = append(callbacks, evals.Inject(childObs, eval))
}
tracer := agenttrace.ByCode(callbacks...)
// Run analyzer with tracer
})
}
Thread Safety ¶
The observer adapter is thread-safe because it delegates to *testing.T, which is designed to be called from multiple goroutines.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func New ¶
New creates a new Observer from a *testing.T
Example ¶
ExampleNew demonstrates creating a basic testing observer.
package main
import (
"testing"
"chainguard.dev/driftlessaf/agents/agenttrace"
"chainguard.dev/driftlessaf/agents/evals"
"chainguard.dev/driftlessaf/agents/evals/testevals"
)
func main() {
// Create a basic observer from a *testing.T
obs := testevals.New(&testing.T{})
// Use the observer with evaluation callbacks
callback := func(o evals.Observer, trace *agenttrace.Trace[string]) {
o.Log("Processing trace")
if trace.Error != nil {
o.Fail("Trace had an error")
}
}
// Inject the observer into the callback
_ = evals.Inject[string](obs, callback)
}
Output:
func NewPrefix ¶
NewPrefix creates a new Observer from a *testing.T with a message prefix
Example ¶
ExampleNewPrefix demonstrates creating a testing observer with a message prefix.
package main
import (
"testing"
"chainguard.dev/driftlessaf/agents/evals"
"chainguard.dev/driftlessaf/agents/evals/testevals"
)
func main() {
// Create an observer with a prefix for namespaced logging
obs := testevals.NewPrefix(&testing.T{}, "tool-validation")
// Use with a namespaced observer factory
namespacedObs := evals.NewNamespacedObserver(func(name string) evals.Observer {
return testevals.NewPrefix(&testing.T{}, name)
})
// Create child observers for different evaluation aspects
toolObs := namespacedObs.Child("tool-calls")
errorObs := namespacedObs.Child("errors")
// Use the observers
_ = obs
_ = toolObs
_ = errorObs
}
Output:
func WordWrap ¶ added in v0.2.0
WordWrap breaks s into lines of at most width characters, splitting on spaces.
func YAMLScalar ¶ added in v0.2.0
YAMLScalar formats a string as a YAML scalar, using >- for long or multiline strings. The returned string never ends with a newline.
Types ¶
This section is empty.
Source Files