Documentation
¶
Overview ¶
Package evalmeasure scores agent run traces for online eval measurements.
Index ¶
- Constants
- func AlreadyScored(ledgerPath, traceID, evalName, version string) (bool, error)
- func AppendMeasurements(path string, results []EvaluationResult) (retErr error)
- func ExportOTLPScores(ctx context.Context, results []EvaluationResult, serviceVersion string) (err error)
- func FindPlatformTelemetry(outputDir, agent string) ([]string, error)
- func MeasureAndExport(ctx context.Context, ...) ([]EvaluationResult, ParseStats, error)
- func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)
- func RecordScored(ledgerPath, traceID, evalName, version string) (retErr error)
- func WithPersistHook(ctx context.Context, fn func()) context.Context
- type EvaluationResult
- func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error)
- func ScoreFitness(tr Trace) EvaluationResult
- func ScoreFitnessNamed(tr Trace, evalName, version string) EvaluationResult
- func ScoreRunHealth(tr Trace) EvaluationResult
- func ScoreRunHealthNamed(tr Trace, evalName, version string) EvaluationResult
- func ScoreTrace(tr Trace, reg Registry) []EvaluationResult
- type MeasurementSpec
- type ParseStats
- type Registry
- type Span
- type Trace
Constants ¶
const ( MeasurementsFile = "eval-measurements.jsonl" LedgerFile = "eval-measure-ledger.txt" )
const ( EventGenAIEvaluationResult = "gen_ai.evaluation.result" AttrGenAIEvaluationName = "gen_ai.evaluation.name" AttrGenAIEvaluationScoreValue = "gen_ai.evaluation.score.value" AttrGenAIEvaluationScoreLabel = "gen_ai.evaluation.score.label" AttrGenAIEvaluationExplanation = "gen_ai.evaluation.explanation" AttrFullsendMeasurementVersion = "fullsend.measurement.version" )
GenAI evaluation event / attribute names.
Attribute names follow OpenTelemetry GenAI semantic conventions (normative pin consulted for this ship: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md#event-gen_aievaluationresult — GenAI events live in semantic-conventions-genai; treat as low-stability. Supporting-library matrix only: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/reference/reports/gen-ai-evaluation-result-event.md). Semconv remains unstable across minor versions (see gen_ai.system → gen_ai.provider.name); bump measurement versions when attribute names change.
Carrier deviation (deliberate): the convention defines gen_ai.evaluation.result as a log-record / Event-API event (SHOULD parent to the GenAI operation span, or set gen_ai.response.id when span id is unavailable). Fullsend emits it as a span event via span.AddEvent on a short fullsend.eval_measure child over the traces OTLP path, because only a traces exporter is configured today — log-side consumers will not auto-discover these scores. A conforming log-record emit is follow-up work when a logs exporter exists; this PR does not add one.
Post-hoc attach: the scored GenAI operation span is already flushed, so the child span is remote-parented to that SpanID. Any OTLP traces backend can correlate by TraceID; vendor score UIs (Assessments panels, etc.) may still need a collector/consumer mapping — fullsend does not call those APIs.
const ( ScorerFitness = "trace_fitness" LabelPass = "pass" LabelFail = "fail" LabelSkip = "skip" // UnknownSentinel is the CLI fallback for missing work-item / agent // identity (resolveWorkItemID). Scorers must treat it as absent. UnknownSentinel = "unknown" )
const ( AttrFullsendAgent = "fullsend.agent" AttrFullsendWorkItemID = "fullsend.work_item_id" AttrGenAIAgentName = "gen_ai.agent.name" AttrGenAISystem = "gen_ai.system" // deprecated; prefer AttrGenAIProviderName AttrGenAIProviderName = "gen_ai.provider.name" AttrGenAIRequestModel = "gen_ai.request.model" AttrGenAIUsageInputTokens = "gen_ai.usage.input_tokens" AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens" AttrGenAIOperationName = "gen_ai.operation.name" )
Attribute names used by EM-001. Most gen_ai.* keys follow OpenTelemetry GenAI semantic conventions. gen_ai.system was renamed to gen_ai.provider.name in semconv v1.37.0; modelOK accepts either so em-001@1 stays green across the emitter migration. Other upstream renames remain an em-001 version bump.
const PlatformTelemetryFile = "run-telemetry.jsonl"
PlatformTelemetryFile is the host recorder's JSONL at the top of runDir. It matches internal/telemetry.TelemetryFile. Nested copies under iteration-N/output/ are agent-writable and must not be scored.
const ( // ScorerRunHealth is EM-002: deterministic run-health scoring over the // always-on execute_tool spans (ADR 0108). It is distinct from // ScorerFitness (EM-001), which scores whether the telemetry is // well-formed, not whether the run behaved correctly. ScorerRunHealth = "run_health" )
Variables ¶
This section is empty.
Functions ¶
func AlreadyScored ¶
AlreadyScored reports whether the ledger contains this measurement.
func AppendMeasurements ¶
func AppendMeasurements(path string, results []EvaluationResult) (retErr error)
AppendMeasurements writes one NDJSON EvaluationResult per line.
func ExportOTLPScores ¶ added in v0.43.0
func ExportOTLPScores(ctx context.Context, results []EvaluationResult, serviceVersion string) (err error)
ExportOTLPScores emits each measurement as a short child span on the same TraceID (remote-parented to the scored SpanID) with a gen_ai.evaluation.result event. Uses the same OTEL_EXPORTER_OTLP_* env as ADR 0050 agent traces. serviceVersion must match telemetry.Setup's version (CLI Version()) so resource identity stays aligned with agent spans. No-op when OTEL is unset or OTEL_SDK_DISABLED=true. Per-score: when inbound TRACEPARENT is valid and unsampled, scores whose TraceID equals that parent TraceID are skipped (same rule as parentSampledProcessor — avoid orphaning score spans on a TraceID that never left the box). Other TraceIDs in the batch are unaffected. Fail-open: returns an error for the caller to warn on; never writes primary telemetry files. Skip rows that lack a parent SpanID are omitted; pass/fail rows with missing or malformed IDs report a warning. Pure ID failures report "N/M scores exported" so operators can tell partial from total failure; a ForceFlush/export transport error uses a distinct "failed for all N" message (row construction is not treated as delivery).
func FindPlatformTelemetry ¶
FindPlatformTelemetry returns run-telemetry.jsonl files that sit at the top of outputDir itself (when outputDir is a runDir) or at the top of a host-created child runDir (when outputDir is the CI output base).
Host runDirs use one of two naming schemes:
- Legacy: agent-<name>-<pid>-<unix> (full agent name in the directory)
- Current: fs-<slug>-<hex> (3-char agent slug + hash)
When agent is non-empty, only children whose embedded agent name (legacy) or slug (current) matches are considered. For the current scheme, the slug is a lossy 3-character abbreviation — agents with the same prefix (e.g. "review" and "reverse" both map to "rev") are indistinguishable from the directory name alone. In practice this is acceptable because a single CI job runs one agent at a time.
If several match, only the newest platform file is scored — leftover sibling directories from a previous job are ignored.
Matching child runDirs outrank a root-level run-telemetry.jsonl under outputDir so an agent-planted file at the CI output base cannot displace the real host recorder. If any matching child runDir exists (even without a platform file yet), the root file is ignored. Nested iteration-N/output/ copies are never walked.
func MeasureAndExport ¶
func MeasureAndExport(ctx context.Context, telemetryPath, registryPath, outDir, serviceVersion string) ([]EvaluationResult, ParseStats, error)
MeasureAndExport is MeasureFile with an explicit context used for portable OTLP score export (same OTEL_EXPORTER_OTLP_* path as ADR 0050). Local JSONL/ledger always win; OTLP failures are recorded on ParseStats and never fail the measure. serviceVersion should match telemetry.Setup (CLI Version()) so remote score resources share agent-trace identity.
func ParseTelemetryFile ¶
func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)
ParseTelemetryFile reads OTLP JSON TracesData lines from run-telemetry.jsonl and merges spans by trace id. Truncated or corrupt lines, and spans that fail conversion, are skipped (fail-open). SkippedLines counts whole-line failures; SkippedSpans counts per-span convert failures. An oversized-line scanner error returns traces gathered so far alongside the error so the caller can still score them.
func RecordScored ¶
RecordScored appends a ledger entry.
Types ¶
type EvaluationResult ¶
type EvaluationResult struct {
Name string `json:"name"`
Label string `json:"label"`
Explanation string `json:"explanation"`
TraceID string `json:"trace_id"`
SpanID string `json:"span_id"`
WorkItemID string `json:"work_item_id,omitempty"`
Agent string `json:"agent"`
Version string `json:"version"`
// Value is the numeric score. Skip rows leave this at 0; consumers
// must key off Label, not Value==0. Do not add omitempty: a real
// fail can also be 0.0.
Value float64 `json:"value"`
}
EvaluationResult is a portable measurement score (no vendor types).
func MeasureFile ¶
func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error)
MeasureFile parses telemetry, scores with the manifest, and writes local eval-measurements.jsonl. Idempotent per ledger.
For OTLP, this helper passes an empty serviceVersion into MeasureAndExport, so BuildResource records service.version=unknown when OTEL is configured. Prefer MeasureAndExport with the CLI Version() (same string telemetry.Setup uses) so score resources share agent-trace identity. MeasureFile is mainly for local/tests where OTLP is unset.
func ScoreFitness ¶
func ScoreFitness(tr Trace) EvaluationResult
ScoreFitness implements EM-001 Trace Fitness against a single trace.
func ScoreFitnessNamed ¶
func ScoreFitnessNamed(tr Trace, evalName, version string) EvaluationResult
ScoreFitnessNamed is ScoreFitness with explicit evaluation name/version.
func ScoreRunHealth ¶ added in v0.44.0
func ScoreRunHealth(tr Trace) EvaluationResult
ScoreRunHealth implements EM-002 Run Health against a single trace.
func ScoreRunHealthNamed ¶ added in v0.44.0
func ScoreRunHealthNamed(tr Trace, evalName, version string) EvaluationResult
ScoreRunHealthNamed scores structural, tool-call-level defects that are decidable from the execute_tool spans (ADR 0108) with no model in the loop. v1 fails only on an unambiguous integrity defect — a tool result with no matching call. Tool errors and unanswered calls are reported as signals, not failures: a tool error is often a legitimate recovery, and an unanswered call is often a legitimate timeout or kill, neither of which is decidable as a defect from the span alone. Runs whose runtime emits no execute_tool spans (pi and codex report no call id) and runs that made no tool calls are skipped, not scored, so the trend measures only runs whose tool behavior was actually observable.
func ScoreTrace ¶
func ScoreTrace(tr Trace, reg Registry) []EvaluationResult
ScoreTrace runs enabled measurements for traces matching the manifest agent. An empty or UnknownSentinel agent name still scores so identity=fail is recorded (EM-001 exists to catch missing identity). A different, known agent name is skipped. Unknown scorer strings write a skip row rather than silent-no-op or a fail that mixes into pass-rate (a newer agents@v0 manifest can name a scorer this binary does not implement yet).
type MeasurementSpec ¶
type MeasurementSpec struct {
ID string `yaml:"id"`
Scorer string `yaml:"scorer"`
Name string `yaml:"name"` // optional display override; default = Scorer
Version int `yaml:"version"`
}
MeasurementSpec selects a framework scorer.
type ParseStats ¶
type ParseStats struct {
NonEmptyLines int
SkippedLines int // whole JSONL lines that failed to unmarshal
SkippedSpans int // spans inside a valid line that failed convertSpan
// Incomplete is set when the scanner stopped early (e.g. bufio.ErrTooLong)
// but some traces were still recovered. Callers should warn; MeasureAndExport
// still scores those traces and returns a nil error.
Incomplete string
// RemoteExportWarning is set when portable OTLP score export failed
// after local JSONL persistence. Measurements stay fail-open.
RemoteExportWarning string
}
ParseStats counts telemetry JSONL lines and unusable spans so operators can tell "no traces" from "file present but partially unreadable".
type Registry ¶
type Registry struct {
Agent string `yaml:"agent"`
Measurements []MeasurementSpec `yaml:"measurements"`
}
Registry is an agent-specific measurement manifest (owned by agents repo).
func LoadRegistry ¶
LoadRegistry loads a measurement manifest YAML file.
type Span ¶
type Span struct {
TraceID string
SpanID string
ParentSpanID string
Name string
StartUnixNano uint64
EndUnixNano uint64
StatusCode int // 0 unset, 1 OK, 2 ERROR (OTLP)
Attrs map[string]any
}
Span is a portable view of one OTEL span from run-telemetry.jsonl.
func (Span) AttrString ¶
AttrString returns a string attribute.
func (Span) DurationSeconds ¶
DurationSeconds returns end-start in seconds.
type Trace ¶
Trace groups spans that share a trace id.
func (Trace) SpanByName ¶
SpanByName returns the first span with the given name.
func (Trace) SpansByName ¶
SpansByName returns all spans with the given name.