structured-evaluation

Structured Evaluation

A reusable evaluation framework for LLM-as-Judge and multi-agent workflows.

Overview

structured-evaluation provides standardized types for evaluation reports, enabling:

• ⚖️ LLM-as-Judge assessments with categorical and 1-5 integer scoring
• 🔧 Automated repair via reason codes with repair prompts
• 📊 Coverage tracking for spec completeness metrics
• 📈 Confidence & routing for human review of low-confidence evaluations
• ✅ GO/NO-GO summary reports for deterministic checks (CI, tests, validation)
• 🔗 Multi-agent coordination with DAG-based report aggregation
• 📋 Claims validation for factual claim extraction and source verification
• 🟦 TypeScript / Zod bindings generated from the same Go structs, never hand-maintained

Architecture

┌───────────────────────────────────────────────────────────┐
│                    SummaryReport (GO/NO-GO)               │
│  ┌──────────────────────┐  ┌──────────────────────┐       │
│  │  Embedded Reports    │  │   Team Sections      │       │
│  │  (Full-Fidelity)     │  │   (Task Results)     │       │
│  └──────────────────────┘  └──────────────────────┘       │
└───────────────────────────────────────────────────────────┘
                              ▲
              ┌───────────────┴───────────────┐
              │                               │
┌─────────────┴─────────────┐   ┌─────────────┴─────────────┐
│     Rubric (rubric/)      │   │   ClaimsReport (claims/)  │
│  ┌─────────────────────┐  │   │  ┌─────────────────────┐  │
│  │ Category Results    │  │   │  │ Claims + Validation │  │
│  │ (pass/partial/fail) │  │   │  │ (verified/rejected) │  │
│  ├─────────────────────┤  │   │  ├─────────────────────┤  │
│  │ Findings            │  │   │  │ Sources             │  │
│  │ (severity-based)    │  │   │  │ (external/internal) │  │
│  └─────────────────────┘  │   │  └─────────────────────┘  │
│  LLM-as-Judge scoring     │   │  Fact verification        │
└───────────────────────────┘   └───────────────────────────┘

Three complementary report types:

Installation

go get github.com/plexusone/structured-evaluation

Packages

Report Types

Rubric (LLM-as-Judge)

For subjective quality assessments with detailed findings:

import "github.com/plexusone/structured-evaluation/rubric"

report := rubric.NewRubric("prd", "document.md")

// Add category with 1-5 integer score and confidence
result := rubric.NewCategoryResultWithIntScore(
    "problem_definition",
    rubric.ScoreGood,  // 4/5
    0.9,               // High confidence
    "Clear problem statement with measurable goals",
)
report.AddCategoryResult(*result)

// Add finding with reason code for automated repair
finding := rubric.NewFindingWithCode(
    "f1", "metrics",
    rubric.CodeMETRICNoBaseline,
    "Missing baseline metrics",
    "No baseline measurements defined for success metrics",
)
finding.SetRecommendation("Add current baseline measurements")
report.AddFinding(*finding)

// Set minimum score threshold
report.PassCriteria.MinIntScore = rubric.ScoreGood  // Require 4+

report.Finalize(nil, "sevaluation check document.md")
// report.IntScore, report.Confidence, report.Blocking are computed

Summary Report (GO/NO-GO)

For deterministic checks with pass/fail status:

import "github.com/plexusone/structured-evaluation/summary"

report := summary.NewSummaryReport("my-service", "v1.0.0", "Release Validation")
report.AddTeam(summary.TeamSection{
    ID:   "qa",
    Name: "Quality Assurance",
    Tasks: []summary.TaskResult{
        {ID: "unit-tests", Status: summary.StatusGo, Detail: "Coverage: 92%"},
        {ID: "e2e-tests", Status: summary.StatusWarn, Detail: "2 flaky tests"},
    },
})

Claims Report (v0.6.0)

For factual claim extraction and source validation:

import "github.com/plexusone/structured-evaluation/claims"

report := claims.NewClaimsReport("security-advisory.md")

// External source: CVE from NVD
claim := claims.NewClaim("cvss", "CVSS 8.8 High", claims.ClaimRiskAssessment,
    claims.Location{Section: "severity"})
claim.SetValidation(claims.NewExternalValidation(
    "https://nvd.nist.gov/vuln/detail/CVE-2026-25253",
    claims.ExternalNVD,
))
report.AddClaim(*claim)

// Internal validation: exploit confirmed via code
exploit := claims.NewClaim("exploit", "RCE confirmed", claims.ClaimTechnicalFinding,
    claims.Location{Section: "impact"})
exploit.SetValidation(claims.NewInternalValidation(
    claims.MethodCodeExecution, "poc.py", true,
))
report.AddClaim(*exploit)

report.Finalize()
// report.Decision.Passed, report.Summary.Counts

Severity Levels

Following InfoSec conventions:

CategoryResult.Severity (v0.11.0) is the worst severity among a category's findings, computed automatically — never set independently by the judge, so it can't drift from Findings. Useful for sorting or highlighting which categories to fix first, distinct from Score/IntScore (which measure quality, not urgency).

Pass Criteria

Default criteria (zero blocking findings, all categories passing):

criteria := rubric.DefaultPassCriteria()
// MaxCritical: 0, MaxHigh: 0, MaxMedium: -1 (unlimited), RequireAllPass: false

criteria := rubric.StrictPassCriteria()
// MaxCritical: 0, MaxHigh: 0, MaxMedium: 3, RequireAllPass: true

Report Validation (v0.7.0)

Validate evaluation reports for correctness:

result := rubric.ValidateReport(&report)

if !result.Valid {
    fmt.Printf("Invalid: %d errors, %d warnings\n", result.ErrorCount, result.WarningCount)
    for _, issue := range result.Issues {
        fmt.Printf("[%s] %s: %s\n", issue.Severity, issue.Path, issue.Message)
    }
}

// Get valid enum values for tooling
scores := rubric.ValidScoreValues()         // ["pass", "partial", "fail"]
severities := rubric.ValidSeverityValues()  // ["critical", "high", "medium", "low", "info"]

CLI Tool

# Install
go install github.com/plexusone/structured-evaluation/cmd/sevaluation@latest

# Render reports
sevaluation render report.json --format=detailed
sevaluation render report.json --format=terminal   # ANSI colors + UTF8 icons
sevaluation render report.json --format=markdown   # Markdown output
sevaluation render report.json --format=box
sevaluation render report.json --format=json

# Lint reports for correctness (v0.7.0)
sevaluation lint report.json              # Basic validation
sevaluation lint report.json --strict     # Warnings are errors
sevaluation lint report.json --format=json

# Check pass/fail (exit code 0/1)
sevaluation check report.json

# Validate structure
sevaluation validate report.json

# Generate JSON Schema
sevaluation schema generate -o ./schema/

DAG-Based Aggregation

For multi-agent workflows with dependencies:

import "github.com/plexusone/structured-evaluation/combine"

results := []combine.AgentResult{
    {TeamID: "qa", Tasks: qaTasks},
    {TeamID: "security", Tasks: secTasks, DependsOn: []string{"qa"}},
    {TeamID: "release", Tasks: relTasks, DependsOn: []string{"qa", "security"}},
}

report := combine.AggregateResults(results, "my-project", "v1.0.0", "Release")
// Teams are topologically sorted: qa → security → release

JSON Schema

Schemas are embedded for runtime validation:

import "github.com/plexusone/structured-evaluation/schema"

rubricSchema := schema.RubricSchemaJSON
claimsSchema := schema.ClaimsSchemaJSON
summarySchema := schema.SummarySchemaJSON

TypeScript / Zod (v0.11.0)

For consumers that read reports in TypeScript, @plexusone/structured-evaluation provides Zod schemas and TS types (Rubric, RubricSet, ClaimsReport, SummaryReport) generated from the same JSON Schema above — downstream of the Go structs, never hand-maintained:

npm install @plexusone/structured-evaluation

import { RubricSchema, type Rubric } from '@plexusone/structured-evaluation'

const report: Rubric = RubricSchema.parse(JSON.parse(rawJson))
console.log(report.intScore) // the 1-5 score, correctly typed
console.log(report.categories[0].severity) // "critical" | "high" | "medium" | "low" | "info" | undefined

Every schema is .strict() — an unrecognized key (a stale consumer still expecting a field that was renamed upstream) fails parsing loudly instead of silently reading undefined. See ts/README.md for regeneration instructions and known limitations.

RubricSet (v0.4.0)

Define explicit criteria for consistent categorical evaluations:

cat := rubric.NewCategory("quality", "Output Quality", "Overall quality assessment").
    WithPassPartialFail(
        []string{"Meets all requirements, no significant issues"},
        []string{"Meets most requirements, minor issues"},
        []string{"Missing key requirements or major issues"},
    )

rubricSet := rubric.NewRubricSet("output-review", "Output Review", "1.0").
    AddCategory(*cat)

Rich Weighted Criteria (v0.10.0)

Categories can decompose into weighted sub-criteria, each with pass/partial/fail bands carrying a description and concrete indicators, scored by numeric scoreThresholds. Rubric definitions also carry yaml tags, so they can be authored as YAML and parsed directly into a RubricSet. See docs/features/rubrics.md.

Judge Metadata (v0.2.0)

Track LLM judge configuration for reproducibility:

judge := rubric.NewJudgeMetadata("claude-3-opus").
    WithProvider("anthropic").
    WithPrompt("prd-eval-v1", "1.0").
    WithTemperature(0.0).
    WithTokenUsage(1500, 800)

report.SetJudge(judge)

Pairwise Comparison (v0.2.0)

Compare two outputs instead of absolute scoring:

comparison := rubric.NewPairwiseComparison(input, outputA, outputB)
comparison.SetWinner(rubric.WinnerA, "A is more accurate", 0.9)

// Aggregate multiple comparisons
result := rubric.ComputePairwiseResult(comparisons)
// result.WinRateA, result.OverallWinner

Multi-Judge Aggregation (v0.4.0)

Combine evaluations from multiple judges:

result := rubric.AggregateEvaluations(evaluations, rubric.AggregationMajority)

// Methods: AggregationMajority, AggregationConservative, AggregationOptimistic
// result.Agreement - inter-judge agreement (0-1)
// result.Disagreements - categories with significant disagreement
// result.ConsolidatedDecision - final aggregated decision

Likert Scales (v0.5.0)

Use 1-5 numeric scales for human comparison studies:

// Create a Likert-scale category
cat := rubric.NewCategory("quality", "Content Quality", "Overall quality").
    WithLikert5(rubric.StandardLikert5Anchors())

// Record a Likert score (automatically maps to categorical)
result := rubric.NewCategoryResultFromLikert("quality", 4, config, "Good quality")
// result.Score = ScorePass, result.NumericScore = 4.0

// Or record both categorical and numeric
result := rubric.NewCategoryResultWithNumeric("quality", rubric.ScorePass, 4.5, "reasoning")

Inter-Rater Reliability (v0.5.0)

Compare LLM evaluations with human ground truth:

// Compute IRR metrics
metrics := rubric.ComputeIRRFromResults(humanResults, llmResults)

fmt.Printf("Exact Agreement: %.1f%%\n", metrics.ExactAgreement*100)
fmt.Printf("Adjacent Agreement: %.1f%%\n", metrics.AdjacentAgreement*100)
fmt.Printf("Pearson r: %.3f\n", metrics.PearsonCorrelation)

// Categorical agreement with confusion matrix
agreement := rubric.ComputeCategoricalAgreement(humanResults, llmResults)

Claims Validation (v0.6.0)

Validate factual claims have proper source backing:

import "github.com/plexusone/structured-evaluation/claims"

report := claims.NewClaimsReport("article.md")

// Source types: external (URL), internal (code/lab), derived, subjective
// Reliability tiers: authoritative, high, medium, low
// Verdicts: verified, unverified, needs-review, rejected

// Configure pass criteria
report.SetCriteria(claims.ClaimsCriteria{
    RequireAllVerified:           true,
    AllowSubjectiveWithDisclaimer: false,
    MinReliabilityTier:           claims.ReliabilityHigh,
})

report.Finalize()
if report.IsPassing() {
    fmt.Println("Ready for publication")
}

Embedded Reports (v0.6.0)

Archive full-fidelity reports within SummaryReport:

report := summary.NewSummaryReport("project", "v1.0.0", "RELEASE")

// Embed detailed reports
report.EmbedRubricReport("quality-review", rubricReport)
report.EmbedClaimsReport("source-validation", claimsReport)

// Retrieve later
var r rubric.Rubric
report.GetEmbeddedRubricReport("quality-review", &r)

OmniObserve Integration

Export evaluations to Opik, Phoenix, or Langfuse:

import "github.com/plexusone/omniobserve/integrations/sevaluation"

// Export to observability platform
err := sevaluation.Export(ctx, provider, traceID, report)

Integration

Designed to work with:

• github.com/plexusone/omniobserve - LLM observability (Opik, Phoenix, Langfuse)
• github.com/grokify/structured-requirements - PRD evaluation templates
• github.com/plexusone/multi-agent-spec - Agent coordination
• github.com/grokify/structured-changelog - Release validation

License

MIT License - see LICENSE for details.