evalmeasure

package
v0.37.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 23, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package evalmeasure scores agent run traces for online eval measurements.

Index

Constants

View Source
const (
	MeasurementsFile = "eval-measurements.jsonl"
	LedgerFile       = "eval-measure-ledger.txt"
)
View Source
const (
	ScorerFitness = "trace_fitness"
	LabelPass     = "pass"
	LabelFail     = "fail"
	LabelSkip     = "skip"

	// UnknownSentinel is the CLI fallback for missing work-item / agent
	// identity (resolveWorkItemID). Scorers must treat it as absent.
	UnknownSentinel = "unknown"
)
View Source
const (
	AttrFullsendAgent          = "fullsend.agent"
	AttrFullsendWorkItemID     = "fullsend.work_item_id"
	AttrGenAIAgentName         = "gen_ai.agent.name"
	AttrGenAISystem            = "gen_ai.system" // deprecated; prefer AttrGenAIProviderName
	AttrGenAIProviderName      = "gen_ai.provider.name"
	AttrGenAIRequestModel      = "gen_ai.request.model"
	AttrGenAIUsageInputTokens  = "gen_ai.usage.input_tokens"
	AttrGenAIUsageOutputTokens = "gen_ai.usage.output_tokens"
	AttrGenAIOperationName     = "gen_ai.operation.name"
)

Attribute names used by EM-001. Most gen_ai.* keys follow OpenTelemetry GenAI semantic conventions. gen_ai.system was renamed to gen_ai.provider.name in semconv v1.37.0; modelOK accepts either so em-001@1 stays green across the emitter migration. Other upstream renames remain an em-001 version bump.

View Source
const PlatformTelemetryFile = "run-telemetry.jsonl"

PlatformTelemetryFile is the host recorder's JSONL at the top of runDir. It matches internal/telemetry.TelemetryFile. Nested copies under iteration-N/output/ are agent-writable and must not be scored.

Variables

This section is empty.

Functions

func AlreadyScored

func AlreadyScored(ledgerPath, traceID, evalName, version string) (bool, error)

AlreadyScored reports whether the ledger contains this measurement.

func AppendMeasurements

func AppendMeasurements(path string, results []EvaluationResult) (retErr error)

AppendMeasurements writes one NDJSON EvaluationResult per line.

func FindPlatformTelemetry

func FindPlatformTelemetry(outputDir, agent string) ([]string, error)

FindPlatformTelemetry returns run-telemetry.jsonl files that sit at the top of outputDir itself (when outputDir is a runDir) or at the top of a host-created child runDir (when outputDir is the CI output base).

Host runDirs use one of two naming schemes:

  • Legacy: agent-<name>-<pid>-<unix> (full agent name in the directory)
  • Current: fs-<slug>-<hex> (3-char agent slug + hash)

When agent is non-empty, only children whose embedded agent name (legacy) or slug (current) matches are considered. For the current scheme, the slug is a lossy 3-character abbreviation — agents with the same prefix (e.g. "review" and "reverse" both map to "rev") are indistinguishable from the directory name alone. In practice this is acceptable because a single CI job runs one agent at a time.

If several match, only the newest platform file is scored — leftover sibling directories from a previous job are ignored.

Matching child runDirs outrank a root-level run-telemetry.jsonl under outputDir so an agent-planted file at the CI output base cannot displace the real host recorder. If any matching child runDir exists (even without a platform file yet), the root file is ignored. Nested iteration-N/output/ copies are never walked.

func MeasureAndExport

func MeasureAndExport(ctx context.Context, telemetryPath, registryPath, outDir string) ([]EvaluationResult, ParseStats, error)

MeasureAndExport is MeasureFile with an explicit context (reserved for future portable OTLP score export on the same OTEL_* path as ADR 0050).

func ParseTelemetryFile

func ParseTelemetryFile(path string) ([]Trace, ParseStats, error)

ParseTelemetryFile reads OTLP JSON TracesData lines from run-telemetry.jsonl and merges spans by trace id. Truncated or corrupt lines, and spans that fail conversion, are skipped (fail-open). SkippedLines counts whole-line failures; SkippedSpans counts per-span convert failures. An oversized-line scanner error returns traces gathered so far alongside the error so the caller can still score them.

func RecordScored

func RecordScored(ledgerPath, traceID, evalName, version string) (retErr error)

RecordScored appends a ledger entry.

func WithPersistHook

func WithPersistHook(ctx context.Context, fn func()) context.Context

WithPersistHook runs fn after each successful RecordScored. Tests use it to fail a later persist; production callers pass a plain context.

Types

type EvaluationResult

type EvaluationResult struct {
	Name        string `json:"name"`
	Label       string `json:"label"`
	Explanation string `json:"explanation"`
	TraceID     string `json:"trace_id"`
	SpanID      string `json:"span_id"`
	WorkItemID  string `json:"work_item_id,omitempty"`
	Agent       string `json:"agent"`
	Version     string `json:"version"`
	// Value is the numeric score. Skip rows leave this at 0; consumers
	// must key off Label, not Value==0. Do not add omitempty: a real
	// fail can also be 0.0.
	Value float64 `json:"value"`
}

EvaluationResult is a portable measurement score (no vendor types).

func MeasureFile

func MeasureFile(telemetryPath, registryPath, outDir string) ([]EvaluationResult, error)

MeasureFile parses telemetry, scores with the manifest, and writes local eval-measurements.jsonl. Idempotent per ledger.

func ScoreFitness

func ScoreFitness(tr Trace) EvaluationResult

ScoreFitness implements EM-001 Trace Fitness against a single trace.

func ScoreFitnessNamed

func ScoreFitnessNamed(tr Trace, evalName, version string) EvaluationResult

ScoreFitnessNamed is ScoreFitness with explicit evaluation name/version.

func ScoreTrace

func ScoreTrace(tr Trace, reg Registry) []EvaluationResult

ScoreTrace runs enabled measurements for traces matching the manifest agent. An empty or UnknownSentinel agent name still scores so identity=fail is recorded (EM-001 exists to catch missing identity). A different, known agent name is skipped. Unknown scorer strings write a skip row rather than silent-no-op or a fail that mixes into pass-rate (a newer agents@v0 manifest can name a scorer this binary does not implement yet).

type MeasurementSpec

type MeasurementSpec struct {
	ID      string `yaml:"id"`
	Scorer  string `yaml:"scorer"`
	Name    string `yaml:"name"` // optional display override; default = Scorer
	Version int    `yaml:"version"`
}

MeasurementSpec selects a framework scorer.

type ParseStats

type ParseStats struct {
	NonEmptyLines int
	SkippedLines  int // whole JSONL lines that failed to unmarshal
	SkippedSpans  int // spans inside a valid line that failed convertSpan
	// Incomplete is set when the scanner stopped early (e.g. bufio.ErrTooLong)
	// but some traces were still recovered. Callers should warn; MeasureAndExport
	// still scores those traces and returns a nil error.
	Incomplete string
}

ParseStats counts telemetry JSONL lines and unusable spans so operators can tell "no traces" from "file present but partially unreadable".

type Registry

type Registry struct {
	Agent        string            `yaml:"agent"`
	Measurements []MeasurementSpec `yaml:"measurements"`
}

Registry is an agent-specific measurement manifest (owned by agents repo).

func LoadRegistry

func LoadRegistry(path string) (Registry, error)

LoadRegistry loads a measurement manifest YAML file.

type Span

type Span struct {
	TraceID       string
	SpanID        string
	ParentSpanID  string
	Name          string
	StartUnixNano uint64
	EndUnixNano   uint64
	StatusCode    int // 0 unset, 1 OK, 2 ERROR (OTLP)
	Attrs         map[string]any
}

Span is a portable view of one OTEL span from run-telemetry.jsonl.

func (Span) AttrBool

func (s Span) AttrBool(key string) (bool, bool)

AttrBool returns a bool attribute.

func (Span) AttrFloat

func (s Span) AttrFloat(key string) (float64, bool)

AttrFloat returns a float64 attribute.

func (Span) AttrInt

func (s Span) AttrInt(key string) (int64, bool)

AttrInt returns an int64 attribute.

func (Span) AttrString

func (s Span) AttrString(key string) (string, bool)

AttrString returns a string attribute.

func (Span) DurationSeconds

func (s Span) DurationSeconds() float64

DurationSeconds returns end-start in seconds.

type Trace

type Trace struct {
	TraceID string
	Spans   []Span
}

Trace groups spans that share a trace id.

func (Trace) AgentName

func (t Trace) AgentName() string

AgentName returns the agent identity from run or agent spans.

func (Trace) SpanByName

func (t Trace) SpanByName(name string) (Span, bool)

SpanByName returns the first span with the given name.

func (Trace) SpansByName

func (t Trace) SpansByName(name string) []Span

SpansByName returns all spans with the given name.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL