Documentation
¶
Overview ¶
Package evalmeasure scores agent run traces for online eval measurements.
Index ¶
- Constants
- func AlreadyScored(ledgerPath, traceID, evalName, version string) (bool, error)
- func AppendMeasurements(path string, results []EvaluationResult) (retErr error)
- func ExportOTLPScores(ctx context.Context, results []EvaluationResult, serviceVersion string) (err error)
- func FindPlatformTelemetry(outputDir, agent string) ([]string, error)
- func MeasureAndExport(ctx context.Context, ...) ([]EvaluationResult, ParseStats, error)
- func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)
- func RecordScored(ledgerPath, traceID, evalName, version string) (retErr error)
- func WithPersistHook(ctx context.Context, fn func()) context.Context
- type EvaluationResult
- type MeasurementSpec
- type ParseStats
- type Registry
- type Span
- type Trace
Constants ¶
const ( MeasurementsFile = "eval-measurements.jsonl" LedgerFile = "eval-measure-ledger.txt" )
const ( EventGenAIEvaluationResult = "gen_ai.evaluation.result" AttrGenAIEvaluationName = "gen_ai.evaluation.name" AttrGenAIEvaluationScoreValue = "gen_ai.evaluation.score.value" AttrGenAIEvaluationScoreLabel = "gen_ai.evaluation.score.label" AttrGenAIEvaluationExplanation = "gen_ai.evaluation.explanation" AttrFullsendMeasurementVersion = "fullsend.measurement.version" )
GenAI evaluation event / attribute names.
Attribute names follow OpenTelemetry GenAI semantic conventions (normative pin consulted for this ship: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md#event-gen_aievaluationresult — GenAI events live in semantic-conventions-genai; treat as low-stability. Supporting-library matrix only: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/reference/reports/gen-ai-evaluation-result-event.md). Semconv remains unstable across minor versions (see gen_ai.system → gen_ai.provider.name); bump measurement versions when attribute names change.
Carrier deviation (deliberate): the convention defines gen_ai.evaluation.result as a log-record / Event-API event (SHOULD parent to the GenAI operation span, or set gen_ai.response.id when span id is unavailable). Fullsend emits it as a span event via span.AddEvent on a short fullsend.eval_measure child over the traces OTLP path, because only a traces exporter is configured today — log-side consumers will not auto-discover these scores. A conforming log-record emit is follow-up work when a logs exporter exists; this PR does not add one.
Post-hoc attach: the scored GenAI operation span is already flushed, so the child span is remote-parented to that SpanID. Any OTLP traces backend can correlate by TraceID; vendor score UIs (Assessments panels, etc.) may still need a collector/consumer mapping — fullsend does not call those APIs.
const ( ScorerFitness = "trace_fitness" LabelPass = "pass" LabelFail = "fail" LabelSkip = "skip" // UnknownSentinel is the CLI fallback for missing work-item / agent // identity (resolveWorkItemID). Scorers must treat it as absent. UnknownSentinel = "unknown" )
const ( AttrFullsendAgent = "fullsend.agent" AttrFullsendWorkItemID = "fullsend.work_item_id" AttrGenAIAgentName = "gen_ai.agent.name" AttrGenAISystem = "gen_ai.system" // deprecated; prefer AttrGenAIProviderName AttrGenAIProviderName = "gen_ai.provider.name" AttrGenAIRequestModel = "gen_ai.request.model" AttrGenAIUsageInputTokens = "gen_ai.usage.input_tokens" AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens" AttrGenAIOperationName = "gen_ai.operation.name" )
Attribute names used by EM-001. Most gen_ai.* keys follow OpenTelemetry GenAI semantic conventions. gen_ai.system was renamed to gen_ai.provider.name in semconv v1.37.0; modelOK accepts either so em-001@1 stays green across the emitter migration. Other upstream renames remain an em-001 version bump.
const PlatformTelemetryFile = "run-telemetry.jsonl"
PlatformTelemetryFile is the host recorder's JSONL at the top of runDir. It matches internal/telemetry.TelemetryFile. Nested copies under iteration-N/output/ are agent-writable and must not be scored.
Variables ¶
This section is empty.
Functions ¶
func AlreadyScored ¶
AlreadyScored reports whether the ledger contains this measurement.
func AppendMeasurements ¶
func AppendMeasurements(path string, results []EvaluationResult) (retErr error)
AppendMeasurements writes one NDJSON EvaluationResult per line.
func ExportOTLPScores ¶ added in v0.43.0
func ExportOTLPScores(ctx context.Context, results []EvaluationResult, serviceVersion string) (err error)
ExportOTLPScores emits each measurement as a short child span on the same TraceID (remote-parented to the scored SpanID) with a gen_ai.evaluation.result event. Uses the same OTEL_EXPORTER_OTLP_* env as ADR 0050 agent traces. serviceVersion must match telemetry.Setup's version (CLI Version()) so resource identity stays aligned with agent spans. No-op when OTEL is unset or OTEL_SDK_DISABLED=true. Per-score: when inbound TRACEPARENT is valid and unsampled, scores whose TraceID equals that parent TraceID are skipped (same rule as parentSampledProcessor — avoid orphaning score spans on a TraceID that never left the box). Other TraceIDs in the batch are unaffected. Fail-open: returns an error for the caller to warn on; never writes primary telemetry files. Skip rows that lack a parent SpanID are omitted; pass/fail rows with missing or malformed IDs report a warning. Pure ID failures report "N/M scores exported" so operators can tell partial from total failure; a ForceFlush/export transport error uses a distinct "failed for all N" message (row construction is not treated as delivery).
func FindPlatformTelemetry ¶
FindPlatformTelemetry returns run-telemetry.jsonl files that sit at the top of outputDir itself (when outputDir is a runDir) or at the top of a host-created child runDir (when outputDir is the CI output base).
Host runDirs use one of two naming schemes:
- Legacy: agent-<name>-<pid>-<unix> (full agent name in the directory)
- Current: fs-<slug>-<hex> (3-char agent slug + hash)
When agent is non-empty, only children whose embedded agent name (legacy) or slug (current) matches are considered. For the current scheme, the slug is a lossy 3-character abbreviation — agents with the same prefix (e.g. "review" and "reverse" both map to "rev") are indistinguishable from the directory name alone. In practice this is acceptable because a single CI job runs one agent at a time.
If several match, only the newest platform file is scored — leftover sibling directories from a previous job are ignored.
Matching child runDirs outrank a root-level run-telemetry.jsonl under outputDir so an agent-planted file at the CI output base cannot displace the real host recorder. If any matching child runDir exists (even without a platform file yet), the root file is ignored. Nested iteration-N/output/ copies are never walked.
func MeasureAndExport ¶
func MeasureAndExport(ctx context.Context, telemetryPath, registryPath, outDir, serviceVersion string) ([]EvaluationResult, ParseStats, error)
MeasureAndExport is MeasureFile with an explicit context used for portable OTLP score export (same OTEL_EXPORTER_OTLP_* path as ADR 0050). Local JSONL/ledger always win; OTLP failures are recorded on ParseStats and never fail the measure. serviceVersion should match telemetry.Setup (CLI Version()) so remote score resources share agent-trace identity.
func ParseTelemetryFile ¶
func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)
ParseTelemetryFile reads OTLP JSON TracesData lines from run-telemetry.jsonl and merges spans by trace id. Truncated or corrupt lines, and spans that fail conversion, are skipped (fail-open). SkippedLines counts whole-line failures; SkippedSpans counts per-span convert failures. An oversized-line scanner error returns traces gathered so far alongside the error so the caller can still score them.
func RecordScored ¶
RecordScored appends a ledger entry.
Types ¶
type EvaluationResult ¶
type EvaluationResult struct {
Name string `json:"name"`
Label string `json:"label"`
Explanation string `json:"explanation"`
TraceID string `json:"trace_id"`
SpanID string `json:"span_id"`
WorkItemID string `json:"work_item_id,omitempty"`
Agent string `json:"agent"`
Version string `json:"version"`
// Value is the numeric score. Skip rows leave this at 0; consumers
// must key off Label, not Value==0. Do not add omitempty: a real
// fail can also be 0.0.
Value float64 `json:"value"`
}
EvaluationResult is a portable measurement score (no vendor types).
func MeasureFile ¶
func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error)
MeasureFile parses telemetry, scores with the manifest, and writes local eval-measurements.jsonl. Idempotent per ledger.
For OTLP, this helper passes an empty serviceVersion into MeasureAndExport, so BuildResource records service.version=unknown when OTEL is configured. Prefer MeasureAndExport with the CLI Version() (same string telemetry.Setup uses) so score resources share agent-trace identity. MeasureFile is mainly for local/tests where OTLP is unset.
func ScoreFitness ¶
func ScoreFitness(tr Trace) EvaluationResult
ScoreFitness implements EM-001 Trace Fitness against a single trace.
func ScoreFitnessNamed ¶
func ScoreFitnessNamed(tr Trace, evalName, version string) EvaluationResult
ScoreFitnessNamed is ScoreFitness with explicit evaluation name/version.
func ScoreTrace ¶
func ScoreTrace(tr Trace, reg Registry) []EvaluationResult
ScoreTrace runs enabled measurements for traces matching the manifest agent. An empty or UnknownSentinel agent name still scores so identity=fail is recorded (EM-001 exists to catch missing identity). A different, known agent name is skipped. Unknown scorer strings write a skip row rather than silent-no-op or a fail that mixes into pass-rate (a newer agents@v0 manifest can name a scorer this binary does not implement yet).
type MeasurementSpec ¶
type MeasurementSpec struct {
ID string `yaml:"id"`
Scorer string `yaml:"scorer"`
Name string `yaml:"name"` // optional display override; default = Scorer
Version int `yaml:"version"`
}
MeasurementSpec selects a framework scorer.
type ParseStats ¶
type ParseStats struct {
NonEmptyLines int
SkippedLines int // whole JSONL lines that failed to unmarshal
SkippedSpans int // spans inside a valid line that failed convertSpan
// Incomplete is set when the scanner stopped early (e.g. bufio.ErrTooLong)
// but some traces were still recovered. Callers should warn; MeasureAndExport
// still scores those traces and returns a nil error.
Incomplete string
// RemoteExportWarning is set when portable OTLP score export failed
// after local JSONL persistence. Measurements stay fail-open.
RemoteExportWarning string
}
ParseStats counts telemetry JSONL lines and unusable spans so operators can tell "no traces" from "file present but partially unreadable".
type Registry ¶
type Registry struct {
Agent string `yaml:"agent"`
Measurements []MeasurementSpec `yaml:"measurements"`
}
Registry is an agent-specific measurement manifest (owned by agents repo).
func LoadRegistry ¶
LoadRegistry loads a measurement manifest YAML file.
type Span ¶
type Span struct {
TraceID string
SpanID string
ParentSpanID string
Name string
StartUnixNano uint64
EndUnixNano uint64
StatusCode int // 0 unset, 1 OK, 2 ERROR (OTLP)
Attrs map[string]any
}
Span is a portable view of one OTEL span from run-telemetry.jsonl.
func (Span) AttrString ¶
AttrString returns a string attribute.
func (Span) DurationSeconds ¶
DurationSeconds returns end-start in seconds.
type Trace ¶
Trace groups spans that share a trace id.
func (Trace) SpanByName ¶
SpanByName returns the first span with the given name.
func (Trace) SpansByName ¶
SpansByName returns all spans with the given name.