Documentation
¶
Overview ¶
Package evalmeasure scores agent run traces for online eval measurements.
Index ¶
- Constants
- func AlreadyScored(ledgerPath, traceID, evalName, version string) (bool, error)
- func AppendMeasurements(path string, results []EvaluationResult) (retErr error)
- func FindPlatformTelemetry(outputDir, agent string) ([]string, error)
- func MeasureAndExport(ctx context.Context, telemetryPath, registryPath, outDir string) ([]EvaluationResult, ParseStats, error)
- func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)
- func RecordScored(ledgerPath, traceID, evalName, version string) (retErr error)
- func WithPersistHook(ctx context.Context, fn func()) context.Context
- type EvaluationResult
- type MeasurementSpec
- type ParseStats
- type Registry
- type Span
- type Trace
Constants ¶
const ( MeasurementsFile = "eval-measurements.jsonl" LedgerFile = "eval-measure-ledger.txt" )
const ( ScorerFitness = "trace_fitness" LabelPass = "pass" LabelFail = "fail" LabelSkip = "skip" // UnknownSentinel is the CLI fallback for missing work-item / agent // identity (resolveWorkItemID). Scorers must treat it as absent. UnknownSentinel = "unknown" )
const ( AttrFullsendAgent = "fullsend.agent" AttrFullsendWorkItemID = "fullsend.work_item_id" AttrGenAIAgentName = "gen_ai.agent.name" AttrGenAISystem = "gen_ai.system" // deprecated; prefer AttrGenAIProviderName AttrGenAIProviderName = "gen_ai.provider.name" AttrGenAIRequestModel = "gen_ai.request.model" AttrGenAIUsageInputTokens = "gen_ai.usage.input_tokens" AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens" AttrGenAIOperationName = "gen_ai.operation.name" )
Attribute names used by EM-001. Most gen_ai.* keys follow OpenTelemetry GenAI semantic conventions. gen_ai.system was renamed to gen_ai.provider.name in semconv v1.37.0; modelOK accepts either so em-001@1 stays green across the emitter migration. Other upstream renames remain an em-001 version bump.
const PlatformTelemetryFile = "run-telemetry.jsonl"
PlatformTelemetryFile is the host recorder's JSONL at the top of runDir. It matches internal/telemetry.TelemetryFile. Nested copies under iteration-N/output/ are agent-writable and must not be scored.
Variables ¶
This section is empty.
Functions ¶
func AlreadyScored ¶
AlreadyScored reports whether the ledger contains this measurement.
func AppendMeasurements ¶
func AppendMeasurements(path string, results []EvaluationResult) (retErr error)
AppendMeasurements writes one NDJSON EvaluationResult per line.
func FindPlatformTelemetry ¶
FindPlatformTelemetry returns run-telemetry.jsonl files that sit at the top of outputDir itself (when outputDir is a runDir) or at the top of a host-created child runDir (when outputDir is the CI output base).
Host runDirs use one of two naming schemes:
- Legacy: agent-<name>-<pid>-<unix> (full agent name in the directory)
- Current: fs-<slug>-<hex> (3-char agent slug + hash)
When agent is non-empty, only children whose embedded agent name (legacy) or slug (current) matches are considered. For the current scheme, the slug is a lossy 3-character abbreviation — agents with the same prefix (e.g. "review" and "reverse" both map to "rev") are indistinguishable from the directory name alone. In practice this is acceptable because a single CI job runs one agent at a time.
If several match, only the newest platform file is scored — leftover sibling directories from a previous job are ignored.
Matching child runDirs outrank a root-level run-telemetry.jsonl under outputDir so an agent-planted file at the CI output base cannot displace the real host recorder. If any matching child runDir exists (even without a platform file yet), the root file is ignored. Nested iteration-N/output/ copies are never walked.
func MeasureAndExport ¶
func MeasureAndExport(ctx context.Context, telemetryPath, registryPath, outDir string) ([]EvaluationResult, ParseStats, error)
MeasureAndExport is MeasureFile with an explicit context (reserved for future portable OTLP score export on the same OTEL_* path as ADR 0050).
func ParseTelemetryFile ¶
func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)
ParseTelemetryFile reads OTLP JSON TracesData lines from run-telemetry.jsonl and merges spans by trace id. Truncated or corrupt lines, and spans that fail conversion, are skipped (fail-open). SkippedLines counts whole-line failures; SkippedSpans counts per-span convert failures. An oversized-line scanner error returns traces gathered so far alongside the error so the caller can still score them.
func RecordScored ¶
RecordScored appends a ledger entry.
Types ¶
type EvaluationResult ¶
type EvaluationResult struct {
Name string `json:"name"`
Label string `json:"label"`
Explanation string `json:"explanation"`
TraceID string `json:"trace_id"`
SpanID string `json:"span_id"`
WorkItemID string `json:"work_item_id,omitempty"`
Agent string `json:"agent"`
Version string `json:"version"`
// Value is the numeric score. Skip rows leave this at 0; consumers
// must key off Label, not Value==0. Do not add omitempty: a real
// fail can also be 0.0.
Value float64 `json:"value"`
}
EvaluationResult is a portable measurement score (no vendor types).
func MeasureFile ¶
func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error)
MeasureFile parses telemetry, scores with the manifest, and writes local eval-measurements.jsonl. Idempotent per ledger.
func ScoreFitness ¶
func ScoreFitness(tr Trace) EvaluationResult
ScoreFitness implements EM-001 Trace Fitness against a single trace.
func ScoreFitnessNamed ¶
func ScoreFitnessNamed(tr Trace, evalName, version string) EvaluationResult
ScoreFitnessNamed is ScoreFitness with explicit evaluation name/version.
func ScoreTrace ¶
func ScoreTrace(tr Trace, reg Registry) []EvaluationResult
ScoreTrace runs enabled measurements for traces matching the manifest agent. An empty or UnknownSentinel agent name still scores so identity=fail is recorded (EM-001 exists to catch missing identity). A different, known agent name is skipped. Unknown scorer strings write a skip row rather than silent-no-op or a fail that mixes into pass-rate (a newer agents@v0 manifest can name a scorer this binary does not implement yet).
type MeasurementSpec ¶
type MeasurementSpec struct {
ID string `yaml:"id"`
Scorer string `yaml:"scorer"`
Name string `yaml:"name"` // optional display override; default = Scorer
Version int `yaml:"version"`
}
MeasurementSpec selects a framework scorer.
type ParseStats ¶
type ParseStats struct {
NonEmptyLines int
SkippedLines int // whole JSONL lines that failed to unmarshal
SkippedSpans int // spans inside a valid line that failed convertSpan
// Incomplete is set when the scanner stopped early (e.g. bufio.ErrTooLong)
// but some traces were still recovered. Callers should warn; MeasureAndExport
// still scores those traces and returns a nil error.
Incomplete string
}
ParseStats counts telemetry JSONL lines and unusable spans so operators can tell "no traces" from "file present but partially unreadable".
type Registry ¶
type Registry struct {
Agent string `yaml:"agent"`
Measurements []MeasurementSpec `yaml:"measurements"`
}
Registry is an agent-specific measurement manifest (owned by agents repo).
func LoadRegistry ¶
LoadRegistry loads a measurement manifest YAML file.
type Span ¶
type Span struct {
TraceID string
SpanID string
ParentSpanID string
Name string
StartUnixNano uint64
EndUnixNano uint64
StatusCode int // 0 unset, 1 OK, 2 ERROR (OTLP)
Attrs map[string]any
}
Span is a portable view of one OTEL span from run-telemetry.jsonl.
func (Span) AttrString ¶
AttrString returns a string attribute.
func (Span) DurationSeconds ¶
DurationSeconds returns end-start in seconds.
type Trace ¶
Trace groups spans that share a trace id.
func (Trace) SpanByName ¶
SpanByName returns the first span with the given name.
func (Trace) SpansByName ¶
SpansByName returns all spans with the given name.