Documentation
¶
Overview ¶
Package sevaluation provides integration between structured-evaluation reports and omniobserve llmops providers (Opik, Phoenix, Langfuse).
This package enables:
- Exporting EvaluationReport scores and findings to observability platforms
- Converting platform evaluation results into EvaluationReport format
- Correlating evaluations with LLM traces for debugging
Exporting to Providers ¶
Use Export to send an EvaluationReport to any llmops provider:
provider, _ := llmops.Open("opik", llmops.WithAPIKey("..."))
report := evaluation.NewEvaluationReport("prd", "document.md")
// ... populate report ...
err := sevaluation.Export(ctx, provider, traceID, report)
This will:
- Add feedback scores for each category (normalized 0-1)
- Create annotations for each finding with severity labels
- Add an overall evaluation score with the decision status
Score Normalization ¶
structured-evaluation uses 0-10 scores while llmops uses 0-1. This package automatically normalizes scores during export.
Index ¶
- func AnnotationToFinding(ann llmops.Annotation) evaluation.Finding
- func DenormalizeScore(score float64) float64
- func Export(ctx context.Context, provider llmops.Provider, traceID string, ...) error
- func ExportToSpan(ctx context.Context, provider llmops.Provider, spanID string, ...) error
- func ImportEvalResult(result *llmops.EvalResult, opts ...ImportOptions) *evaluation.EvaluationReport
- func ImportMetricScores(scores []llmops.MetricScore, opts ...ImportOptions) *evaluation.EvaluationReport
- func MetricScoreToCategory(score llmops.MetricScore, weight float64) evaluation.CategoryScore
- func NormalizeScore(score float64) float64
- type ExportOptions
- type ImportOptions
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AnnotationToFinding ¶
func AnnotationToFinding(ann llmops.Annotation) evaluation.Finding
AnnotationToFinding converts an llmops Annotation to an evaluation Finding.
func DenormalizeScore ¶
DenormalizeScore converts a 0-1 score to 0-10 range.
func Export ¶
func Export(ctx context.Context, provider llmops.Provider, traceID string, report *evaluation.EvaluationReport, opts ...ExportOptions) error
Export sends an EvaluationReport to an llmops provider, attaching to a trace. It exports category scores as feedback scores and findings as annotations.
func ExportToSpan ¶
func ExportToSpan(ctx context.Context, provider llmops.Provider, spanID string, report *evaluation.EvaluationReport, opts ...ExportOptions) error
ExportToSpan exports an EvaluationReport to a specific span instead of a trace.
func ImportEvalResult ¶
func ImportEvalResult(result *llmops.EvalResult, opts ...ImportOptions) *evaluation.EvaluationReport
ImportEvalResult converts an llmops EvalResult into an EvaluationReport. Each MetricScore becomes a CategoryScore in the report.
func ImportMetricScores ¶
func ImportMetricScores(scores []llmops.MetricScore, opts ...ImportOptions) *evaluation.EvaluationReport
ImportMetricScores converts a slice of MetricScores into an EvaluationReport.
func MetricScoreToCategory ¶
func MetricScoreToCategory(score llmops.MetricScore, weight float64) evaluation.CategoryScore
MetricScoreToCategory converts a single MetricScore to a CategoryScore.
func NormalizeScore ¶
NormalizeScore converts a 0-10 score to 0-1 range.
Types ¶
type ExportOptions ¶
type ExportOptions struct {
// IncludeFindings controls whether findings are exported as annotations.
// Default: true
IncludeFindings bool
// IncludeCategories controls whether category scores are exported as feedback scores.
// Default: true
IncludeCategories bool
// IncludeOverall controls whether the overall score and decision are exported.
// Default: true
IncludeOverall bool
// ScorePrefix is prepended to score names (e.g., "eval_" -> "eval_problem_definition").
// Default: ""
ScorePrefix string
// Source identifies the evaluation source in annotations.
// Default: "structured-evaluation"
Source string
}
ExportOptions configures the export behavior.
func DefaultExportOptions ¶
func DefaultExportOptions() ExportOptions
DefaultExportOptions returns the default export configuration.
type ImportOptions ¶
type ImportOptions struct {
// ReviewType sets the review type for the generated report.
// Default: "llm_evaluation"
ReviewType string
// Document sets the document name for the report metadata.
// Default: ""
Document string
// DefaultWeight sets the weight for imported categories.
// Default: 1.0
DefaultWeight float64
// PassCriteria sets the criteria for the report.
// Default: evaluation.DefaultPassCriteria()
PassCriteria evaluation.PassCriteria
}
ImportOptions configures the import behavior.
func DefaultImportOptions ¶
func DefaultImportOptions() ImportOptions
DefaultImportOptions returns the default import configuration.
Source Files