evalmeasure

package
v0.48.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package evalmeasure scores agent run traces for online eval measurements.

Index

Constants

View Source
const (
	MeasurementsFile = "eval-measurements.jsonl"
	LedgerFile       = "eval-measure-ledger.txt"
)
View Source
const (
	EventGenAIEvaluationResult = "gen_ai.evaluation.result"

	AttrGenAIEvaluationName        = "gen_ai.evaluation.name"
	AttrGenAIEvaluationScoreValue  = "gen_ai.evaluation.score.value"
	AttrGenAIEvaluationScoreLabel  = "gen_ai.evaluation.score.label"
	AttrGenAIEvaluationExplanation = "gen_ai.evaluation.explanation"
	AttrFullsendMeasurementVersion = "fullsend.measurement.version"
)

GenAI evaluation event / attribute names.

Attribute names follow OpenTelemetry GenAI semantic conventions (normative pin consulted for this ship: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md#event-gen_aievaluationresult — GenAI events live in semantic-conventions-genai; treat as low-stability. Supporting-library matrix only: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/reference/reports/gen-ai-evaluation-result-event.md). Semconv remains unstable across minor versions (see gen_ai.system → gen_ai.provider.name); bump measurement versions when attribute names change.

Carrier deviation (deliberate): the convention defines gen_ai.evaluation.result as a log-record / Event-API event (SHOULD parent to the GenAI operation span, or set gen_ai.response.id when span id is unavailable). Fullsend emits it as a span event via span.AddEvent on a short fullsend.eval_measure child over the traces OTLP path, because only a traces exporter is configured today — log-side consumers will not auto-discover these scores. A conforming log-record emit is follow-up work when a logs exporter exists; this PR does not add one.

Post-hoc attach: the scored GenAI operation span is already flushed, so the child span is remote-parented to that SpanID. Any OTLP traces backend can correlate by TraceID; vendor score UIs (Assessments panels, etc.) may still need a collector/consumer mapping — fullsend does not call those APIs.

View Source
const (
	ScorerFitness = "trace_fitness"
	LabelPass     = "pass"
	LabelFail     = "fail"
	LabelSkip     = "skip"

	// UnknownSentinel is the CLI fallback for missing work-item / agent
	// identity (resolveWorkItemID). Scorers must treat it as absent.
	UnknownSentinel = "unknown"
)
View Source
const (
	AttrFullsendAgent          = "fullsend.agent"
	AttrFullsendWorkItemID     = "fullsend.work_item_id"
	AttrGenAIAgentName         = "gen_ai.agent.name"
	AttrGenAISystem            = "gen_ai.system" // deprecated; prefer AttrGenAIProviderName
	AttrGenAIProviderName      = "gen_ai.provider.name"
	AttrGenAIRequestModel      = "gen_ai.request.model"
	AttrGenAIUsageInputTokens  = "gen_ai.usage.input_tokens"
	AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens"
	AttrGenAIOperationName     = "gen_ai.operation.name"
)

Attribute names used by EM-001. Most gen_ai.* keys follow OpenTelemetry GenAI semantic conventions. gen_ai.system was renamed to gen_ai.provider.name in semconv v1.37.0; modelOK accepts either so em-001@1 stays green across the emitter migration. Other upstream renames remain an em-001 version bump.

View Source
const PlatformTelemetryFile = "run-telemetry.jsonl"

PlatformTelemetryFile is the host recorder's JSONL at the top of runDir. It matches internal/telemetry.TelemetryFile. Nested copies under iteration-N/output/ are agent-writable and must not be scored.

View Source
const (
	// ScorerRunHealth is EM-002: deterministic run-health scoring over the
	// always-on execute_tool spans (ADR 0108). It is distinct from
	// ScorerFitness (EM-001), which scores whether the telemetry is
	// well-formed, not whether the run behaved correctly.
	ScorerRunHealth = "run_health"
)

Variables

This section is empty.

Functions

func AlreadyScored

func AlreadyScored(ledgerPath, traceID, evalName, version string) (bool, error)

AlreadyScored reports whether the ledger contains this measurement.

func AppendMeasurements

func AppendMeasurements(path string, results []EvaluationResult) (retErr error)

AppendMeasurements writes one NDJSON EvaluationResult per line.

func ExportOTLPScores added in v0.43.0

func ExportOTLPScores(ctx context.Context, results []EvaluationResult, serviceVersion string) (err error)

ExportOTLPScores emits each measurement as a short child span on the same TraceID (remote-parented to the scored SpanID) with a gen_ai.evaluation.result event. Uses the same OTEL_EXPORTER_OTLP_* env as ADR 0050 agent traces. serviceVersion must match telemetry.Setup's version (CLI Version()) so resource identity stays aligned with agent spans. No-op when OTEL is unset or OTEL_SDK_DISABLED=true. Per-score: when inbound TRACEPARENT is valid and unsampled, scores whose TraceID equals that parent TraceID are skipped (same rule as parentSampledProcessor — avoid orphaning score spans on a TraceID that never left the box). Other TraceIDs in the batch are unaffected. Fail-open: returns an error for the caller to warn on; never writes primary telemetry files. Skip rows that lack a parent SpanID are omitted; pass/fail rows with missing or malformed IDs report a warning. Pure ID failures report "N/M scores exported" so operators can tell partial from total failure; a ForceFlush/export transport error uses a distinct "failed for all N" message (row construction is not treated as delivery).

func FindPlatformTelemetry

func FindPlatformTelemetry(outputDir, agent string) ([]string, error)

FindPlatformTelemetry returns run-telemetry.jsonl files that sit at the top of outputDir itself (when outputDir is a runDir) or at the top of a host-created child runDir (when outputDir is the CI output base).

Host runDirs use one of two naming schemes:

  • Legacy: agent-<name>-<pid>-<unix> (full agent name in the directory)
  • Current: fs-<slug>-<hex> (3-char agent slug + hash)

When agent is non-empty, only children whose embedded agent name (legacy) or slug (current) matches are considered. For the current scheme, the slug is a lossy 3-character abbreviation — agents with the same prefix (e.g. "review" and "reverse" both map to "rev") are indistinguishable from the directory name alone. In practice this is acceptable because a single CI job runs one agent at a time.

If several match, only the newest platform file is scored — leftover sibling directories from a previous job are ignored.

Matching child runDirs outrank a root-level run-telemetry.jsonl under outputDir so an agent-planted file at the CI output base cannot displace the real host recorder. If any matching child runDir exists (even without a platform file yet), the root file is ignored. Nested iteration-N/output/ copies are never walked.

func MeasureAndExport

func MeasureAndExport(ctx context.Context, telemetryPath, registryPath, outDir, serviceVersion string) ([]EvaluationResult, ParseStats, error)

MeasureAndExport is MeasureFile with an explicit context used for portable OTLP score export (same OTEL_EXPORTER_OTLP_* path as ADR 0050). Local JSONL/ledger always win; OTLP failures are recorded on ParseStats and never fail the measure. serviceVersion should match telemetry.Setup (CLI Version()) so remote score resources share agent-trace identity.

func ParseTelemetryFile

func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)

ParseTelemetryFile reads OTLP JSON TracesData lines from run-telemetry.jsonl and merges spans by trace id. Truncated or corrupt lines, and spans that fail conversion, are skipped (fail-open). SkippedLines counts whole-line failures; SkippedSpans counts per-span convert failures. An oversized-line scanner error returns traces gathered so far alongside the error so the caller can still score them.

func RecordScored

func RecordScored(ledgerPath, traceID, evalName, version string) (retErr error)

RecordScored appends a ledger entry.

func WithPersistHook

func WithPersistHook(ctx context.Context, fn func()) context.Context

WithPersistHook runs fn after each successful RecordScored. Tests use it to fail a later persist; production callers pass a plain context.

Types

type EvaluationResult

type EvaluationResult struct {
	Name        string `json:"name"`
	Label       string `json:"label"`
	Explanation string `json:"explanation"`
	TraceID     string `json:"trace_id"`
	SpanID      string `json:"span_id"`
	WorkItemID  string `json:"work_item_id,omitempty"`
	Agent       string `json:"agent"`
	Version     string `json:"version"`
	// Value is the numeric score. Skip rows leave this at 0; consumers
	// must key off Label, not Value==0. Do not add omitempty: a real
	// fail can also be 0.0.
	Value float64 `json:"value"`
}

EvaluationResult is a portable measurement score (no vendor types).

func MeasureFile

func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error)

MeasureFile parses telemetry, scores with the manifest, and writes local eval-measurements.jsonl. Idempotent per ledger.

For OTLP, this helper passes an empty serviceVersion into MeasureAndExport, so BuildResource records service.version=unknown when OTEL is configured. Prefer MeasureAndExport with the CLI Version() (same string telemetry.Setup uses) so score resources share agent-trace identity. MeasureFile is mainly for local/tests where OTLP is unset.

func ScoreFitness

func ScoreFitness(tr Trace) EvaluationResult

ScoreFitness implements EM-001 Trace Fitness against a single trace.

func ScoreFitnessNamed

func ScoreFitnessNamed(tr Trace, evalName, version string) EvaluationResult

ScoreFitnessNamed is ScoreFitness with explicit evaluation name/version.

func ScoreRunHealth added in v0.44.0

func ScoreRunHealth(tr Trace) EvaluationResult

ScoreRunHealth implements EM-002 Run Health against a single trace.

func ScoreRunHealthNamed added in v0.44.0

func ScoreRunHealthNamed(tr Trace, evalName, version string) EvaluationResult

ScoreRunHealthNamed scores structural, tool-call-level defects that are decidable from the execute_tool spans (ADR 0108) with no model in the loop. v1 fails only on an unambiguous integrity defect — a tool result with no matching call. Tool errors and unanswered calls are reported as signals, not failures: a tool error is often a legitimate recovery, and an unanswered call is often a legitimate timeout or kill, neither of which is decidable as a defect from the span alone. Runs whose runtime emits no execute_tool spans (pi and codex report no call id) and runs that made no tool calls are skipped, not scored, so the trend measures only runs whose tool behavior was actually observable.

func ScoreTrace

func ScoreTrace(tr Trace, reg Registry) []EvaluationResult

ScoreTrace runs enabled measurements for traces matching the manifest agent. An empty or UnknownSentinel agent name still scores so identity=fail is recorded (EM-001 exists to catch missing identity). A different, known agent name is skipped. Unknown scorer strings write a skip row rather than silent-no-op or a fail that mixes into pass-rate (a newer agents@v0 manifest can name a scorer this binary does not implement yet).

type MeasurementSpec

type MeasurementSpec struct {
	ID      string `yaml:"id"`
	Scorer  string `yaml:"scorer"`
	Name    string `yaml:"name"` // optional display override; default = Scorer
	Version int    `yaml:"version"`
}

MeasurementSpec selects a framework scorer.

type ParseStats

type ParseStats struct {
	NonEmptyLines int
	SkippedLines  int // whole JSONL lines that failed to unmarshal
	SkippedSpans  int // spans inside a valid line that failed convertSpan
	// Incomplete is set when the scanner stopped early (e.g. bufio.ErrTooLong)
	// but some traces were still recovered. Callers should warn; MeasureAndExport
	// still scores those traces and returns a nil error.
	Incomplete string
	// RemoteExportWarning is set when portable OTLP score export failed
	// after local JSONL persistence. Measurements stay fail-open.
	RemoteExportWarning string
}

ParseStats counts telemetry JSONL lines and unusable spans so operators can tell "no traces" from "file present but partially unreadable".

type Registry

type Registry struct {
	Agent        string            `yaml:"agent"`
	Measurements []MeasurementSpec `yaml:"measurements"`
}

Registry is an agent-specific measurement manifest (owned by agents repo).

func LoadRegistry

func LoadRegistry(path string) (Registry, error)

LoadRegistry loads a measurement manifest YAML file.

type Span

type Span struct {
	TraceID       string
	SpanID        string
	ParentSpanID  string
	Name          string
	StartUnixNano uint64
	EndUnixNano   uint64
	StatusCode    int // 0 unset, 1 OK, 2 ERROR (OTLP)
	Attrs         map[string]any
}

Span is a portable view of one OTEL span from run-telemetry.jsonl.

func (Span) AttrBool

func (s Span) AttrBool(key string) (bool, bool)

AttrBool returns a bool attribute.

func (Span) AttrFloat

func (s Span) AttrFloat(key string) (float64, bool)

AttrFloat returns a float64 attribute.

func (Span) AttrInt

func (s Span) AttrInt(key string) (int64, bool)

AttrInt returns an int64 attribute.

func (Span) AttrString

func (s Span) AttrString(key string) (string, bool)

AttrString returns a string attribute.

func (Span) DurationSeconds

func (s Span) DurationSeconds() float64

DurationSeconds returns end-start in seconds.

type Trace

type Trace struct {
	TraceID string
	Spans   []Span
}

Trace groups spans that share a trace id.

func (Trace) AgentName

func (t Trace) AgentName() string

AgentName returns the agent identity from run or agent spans.

func (Trace) SpanByName

func (t Trace) SpanByName(name string) (Span, bool)

SpanByName returns the first span with the given name.

func (Trace) SpansByName

func (t Trace) SpansByName(name string) []Span

SpansByName returns all spans with the given name.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL